Towards better understanding of meta-features contributions

This paper will be divided into three parts (see Figure 1). Section 2 appeal to collecting meta-features to meta-learning problem. Next part is concentrated on selection of meta-model. In section 4 we present reservoir of eXplainable AI (XAI) tools to examine built surrogate model structure. We draw conclusions about importance of three types meta-features: data properties, landmarkers and hyperparameters.

Methods proposed in third section are universal and can be applied to diverse definition of meta-learning problem with different definition of meta-fetures space. That work may be perceived as sort of case study of application of these methods. For the case study we used the OpenML 100 database.

In this study we have focused on OpenML100 database Bischl et al. (2017) so that considered datasets are real-world tasks. On this platform every uploaded tabular data has provided automated computing of 61 characteristics. Summary of available dataset properties is included in Vanschoren (2018). As follow in (Bilalli et al., 2017), some of statistics may be generated only for datasets which contain continuous variables, some for datasets with at least one categorical column and third group are calculated for any type of data. We have decided to analyse binary classification problem and selected 20 data sets whose all predictive variables are continuous. For these datasets are 38 available statistical and information-theoretic properties. See the Table 1 for the list of all selected datasets.

We have decided to explore gradient boosting classifiers (gbm) sensitivity to tunability so very algorithm was fitted on each dataset with various parameters. 100 random configurations of hyperparameters have been drawn to examine their influence model predictive power. The region, which hyperparameters are sampled, is according to Probst et al. (2018). Additionally, one of considered configurations is set of default values in R software R Core Team (2019). In this study we consider following hyperparameters for gbm model:

• n.trees,

• interaction.depth,

• n.minobsinnode,

• shrinkage,

• bag.fraction.

For every pair of algorithm with hyperparameters configuration and dataset we have specified 20 train and test data splits. Model has been fitted on each train subset and afterwards AUC was computed on the test frame.

On the base of AUC results, for individual dataset we have created ranking of models - the higher AUC, the higher position in ranking. Ratings are scaled to [0,1] interval. Every configurations for each algorithms appear in list 20 times beacuse of train-test splits. To aggregate this to one value for every model we have computed average rating for model.

We have used landmarkers technique to distinct approach to characterise learning problem of given dataset. As landmarkers models we have considered five machine learning algorithms with default hyperparameter configurations:

• generalized linear regression with regularization,

• gradient boosting,

• k nearest neighbours,

• two random forest implementations: randomForest and ranger.

To evaluate their predictive power we have also applied 20 train/test split method and computed AUC score. These five algorithms have been ranked according to methodology in section 2.3. Because we are mainly concentrated on analysing gradient boosting models we have computed landmarkers as ratio of models rankings (knn, glmnet, ranger and randomForest) to default gbm configuration. In a result, we have four landmarkers meta-features.

Meta-dataset with all meta-features can be found in github repository anonymous URL.

First aspect is examination of influence of individual meta-variables and ordering them according to this. This investigation helps to identify presumptive noisy aspects and may be significant in deliberation to exclude these meta-features from new generations meta-models. We have used permutation based measures of variable importance Fisher et al. (2018). In Figure 3 are presented the most influential meta-features. On the x-axis there is dropout values of root mean square error score caused by permutation of consecutive variable.

The most important meta-data are shrinkage, interaction depth and number of trees. These hyperparameters are followed by landmarkers for k nearest neighbours and ranger. Datasets properties play minor role in this ranking, significant permutational feature importance is obtained by MajorityClassSize, NumberOfInstances and MinorityClassSize. These meta-features are related to dimension and balancing in dataset.

Considered meta-features form three groups because of different approach to creating them. Figure 3 presents the assessment of groups influence. As we see, the most important class of meta-features is hyperparameters and this conclusion is consistent with importance measure for individual variables. Landmarkers and dataset characteristics has similar dropout values.

In previous section we analyse importance of three groups related to definitions of meta-features. We also apply automatically group meta-features based on meta-features correlation. Resulted dendrogram is presented in Figure 4.

Some of dataset statistical characteristcs are closely related and pairwise have high correlation. For example quartile of means and standard deviation create correlated cluster. Hyperparameters are uncorrelated with landmarkers and dataset properties. Moreover they have close to zero correlation within their group. This observation agrees with definition since hyperparameters were sampled independently. Landmarkers for ranger and randomForest are strongly correlated because this algorithms different implementation of the same algorithm.

So far, we investigate overall impact of explanatory variables which can be estimated as single value. Now we focus on the type of effect of selected meta-predictor and how meta-model response is sensitive to changes in this feature. In this analysis we apply Individual Conditional Expectations (ICE) Goldstein et al. (2015) and its R implementation called Ceteris Paribus (CP) (Biecek, 2018).

ICE/CP analysis for selected hyperparameters is presented in Figure 5. In view of distribution of hyperparameters we apply loarithmic scale in x-axis. On each plot we present ceteris paribus profiles for all datasets, grey lines correspond to datasets which are in train subset of meta-data in this particular fold of one-dataset-out crossvalidation and black dotted line is related to eeg-eye-state test dataset.

For every dataset in meta-data we have 100 observations because of varying gbm hyperparameters but profiles for instances of selected dataset have similar shape and are nearly parallel. We are especially interested in curve outline since the translation may be caused by other meta-features values. So we present profiles for averaged meta-features for each dataset. In fact, we averaged only hyperparameters because dataset characteristics and landmarkers has constant values for all dataset observations.

To recall, the higher ranking, the better predictive power of given for gradient boosting model. So observing increasing lines in Ceteris Paribus profiles indicate the better rating of considered gbm models.

We start with the most important variables correspondingly to Figure 3. For shrinkage, interaction depth and number of trees hyperparameters we detect three patterns of profiles in train datasets subset. We apply hierarchical clustering for profiles and for each of three groups aggregated profiles are shown in plots 5. These groups are indicated with distinctive colors and termed as A, B and C. It is worth emphasizing that groups indicate as A for two hyperparameters may consist of different datasets because clustering was executed independently.

For each of these three hyperparameters group A has increasing CP profile with very strong trend for interior values of variables. Profiles stabilise on high prediction for larger meta-features values. Similar behaviour we can observe in group B, with this difference, that maximum predicted rating is achieved for smaller values. Group C are strictly different and the highest prediction of rankings are obtained by gbm models with rather lower number of trees, shallow trees. Because of different types of profiles for above meta-features we infer that for different dataset meta-model predict various hyperparameters as optimal.

Next meta-variable from importance ranking is relative landmark for k nearest neighbours. Corresponding CP profile is presented in Figure 6. In this case all profiles has similar shape and this is difficult to distinguish any clusters.

On every plot we observe that Ceteris Paribus profile for test dataset has similar shape to training datasets curves. So according to low MSE score on test dataset we can conclude that meta-gbm is accurate and generalise meta-problem.