Dynamic classifier chains for multi-label learning

Dynamic classifier chains for multi-label learning

Pawel Trajdos Department of Systems and Computer Networks, Wroclaw University of Technology,
Wybrzeze Wyspianskiego 27, 50-370 Wroclaw, Poland
pawel.trajdos@pwr.wroc.pl
   Marek Kurzynski Department of Systems and Computer Networks, Wroclaw University of Technology,
Wybrzeze Wyspianskiego 27, 50-370 Wroclaw, Poland
pawel.trajdos@pwr.wroc.pl
Abstract

In this paper, we deal with the task of building a dynamic ensemble of chain classifiers for multi-label classification. To do so, we proposed two concepts of classifier chains algorithms that are able to change label order of the chain without rebuilding the entire model. Such modes allows anticipating the instance-specific chain order without a significant increase in computational burden. The proposed chain models are built using the Naive Bayes classifier and nearest neighbour approach as a base single-label classifiers. To take the benefits of the proposed algorithms, we developed a simple heuristic that allows the system to find relatively good label order. The heuristic sort labels according to the label-specific classification quality gained during the validation phase. The heuristic tries to minimise the phenomenon of error propagation in the chain. The experimental results showed that the proposed model based on Naive Bayes classifier the above-mentioned heuristic is an efficient tool for building dynamic chain classifiers.

Keywords:
multi-label, classifier chains, naive bayes, dynamic chains, nearest neighbour

1 Introduction

Under well-known single-label classification framework, an object is assigned to only one class which provides a full description of the object. However, many real-world datasets contain objects that are assigned to different categories at the same time. All of these categories constitute a full description of the object. Omitting of one of these concepts induces a loss of information. Classification process in which such kind of data is involved is called multi-label classification Gibaja2014 (). A great example of a multi-label dataset is a gallery of tagged photos. Each photo may be described using such tags as mountains, sea, forest, beach, sunset, etc. Multi-label classification is a relatively new idea that is explored extensively for last two decades. As a consequence, it was employed in a wide range of practical applications including text classification Jiang2012 (), multimedia classification Sanden2011 () and bioinformatics Wu2014 () to name a few.

Multi-label classification algorithms can be broadly partitioned into two main groups i.e. dataset transformation algorithms and algorithm adaptation approaches Gibaja2014 ().

Methods belong to the group of algorithm adaptation approaches provides a generalisation of an existing multi-class algorithm. The generalised algorithm is able to solve multi-label classification problem in a direct way. Among the others, the most known approaches from this group are: multi label KNN algorithm Jiang2012 (), the Structured SVM approach Diez2014 () or deep-learning-based algorithms Wei2015 ().

In this paper, we investigate only dataset transformation algorithms that decompose a multi-label problem into a set of single-label classification tasks. To reconstruct a multi-label response, during the inference phase, outputs of the underlying single-label classifiers are combined in order to create a multi-label prediction.

Let’s focus on one of the simplest decomposition methods. That is the binary relevance (BR) approach that decomposes a multi-label classification task into a set of one-vs-rest binary classification problems AlvaresCherman2010 (). This approach assumes that labels are conditionally independent. However the assumption does not hold in most of real-life recognition problems, the BR framework is one of the most widespread multi-label classification methods Tsoumakas_Katakis_Vlahavas_2008 (). This is due to its excellent scalability and acceptable classification quality Luaces2012 ().

To preserve scalability of BR systems, and provide a model of inter-label relations, Read et al. Read2009 (); Read2011 () provided us with the Classifier Chain model (CC) which establish a linked chain of modified one-vs-rest binary classifiers. The modification consists of an extension of the input space of single-label classifiers along the chain sequence. To be more strict, for a given label sequence, the feature space of each classifier along the chain is extended with a set of binary variables corresponding to the labels that precede the given one. The model implies that, during the training phase, input space of given classifier is extended using the ground-truth labels extracted from the training set. During the inference step, due to lack of the ground-truth labels, we employ binary labels predicted by preceding classifiers. The inference is done in a greedy way that makes the best decision for each of considered labels. That is, the described approach passes along the chain, information allowing CC to take into account inter-label relations at the cost of allowing the label-prediction-errors to propagate along the chain Read2011 (). This way of performing classification induces a major drawback of the CC system. That is, the CC classifier uses a kind of greedy strategy during the inference phase. This design allows classification errors to propagate along the chain. As a consequence, the performance of a chain classifier strongly depends on chain configuration Senge2013 (). To overcome these effects, the authors suggested to generate an ensemble of chain classifiers (ECC). The ensemble consists of classifiers trained using different label sequences Read2009 ().

The originally proposed ECC ensemble uses randomly generated label orders. Additionally, each chain classifier is built using a resampled dataset. This approach provides an additional diversity into the ensemble classifier. This simple, yet effective approach allows improving the classification quality significantly in comparison to single chain classifier. However, the intuition says that there is still room for improvement when we employ a more data-driven approach.

Indeed, later research shows that the members of the ensemble may be chosen in such a way that provides further improvement of classification quality. That is, Read et al. proposed a strategy which uses Monte Carlo sampling to explore the label sequence space in order to find a classifier chain that offers the highest classification quality Read2014 (). Another approach was proposed by Liu et al. Liu2010 (). They introduced a method that builds a model of inter-label relations as a directed acyclic graph (DAG). The weights of the graph are calculated using confusion and support for each pair of labels. Then, the ensemble is generated using topological sorting of the graph. Chen et al. Chen2016 () proposed a method that makes clusters of labels. Then, for each cluster of labels, an undirected graph describing inter-label relations is built. Then, a minimum spanning tree is created for the graph. After that, breadth-first search algorithm determines sequences for a cluster-specific ensemble of CC classifiers. A similar approach was proposed by Huang et al Huang2015 (). They proposed to build the clusters using a meta-space that mixes input space and label space. Then inter-label relations are modeled using correlation. The model is expressed using DAG structure. Finally, the CC classifier is built for each cluster. The chain structure may also be induced using Bayesian Network approach Zhang2014 ().

Chain sequence can be also found using meta-heuristic approach. That is, Goncalves et al. developed a strategy that utilises a genetic algorithm (GA) to find a good chain structure for the entire dataset Goncalves2013 (); Gonalves2015 (). The proposed approach using wrapper-based approach. That is each chromosome codes different chain order. To evaluate those label orders each corresponding classifier must be built and evaluated using a validation set. A similar approach was also used by Trajdos and Kurzynski who proposed to use a multi-objective genetic algorithm to optimize classification quality and chain diversity simultaneously Trajdos2017 (). Although those methods are rather time-consuming, they provide a significant improvement in terms of classification quality.

Another way of dealing with the error propagation is to build a classifier that combines CC algorithm BR-based approach Montaes2014 (). The authors proposed a stacking based architecture to combine the above-mentioned classifiers. That is, the first layer is a simple BR classifier that predicts each label separately. The attribute set of the classifiers from the second layer is extended using all labels except the predicted one. During the training phase, both layers are trained separately. During the prediction phase, on the other hand, classifiers from the second layer mix outcomes of the BR classifiers with the outcomes provides by preceding classifiers. That is, the first classifier of the chain structure has its attribute space extended by the outcomes of the BR classifier. The second one uses the prediction of the first one and the remaining attributes are taken from the prediction of the BR classifier. Finally, the last classifier along the chain has the attribute space extended using only labels predicted by the preceding classifiers. Another way of combining the CC classifier with the BR classifier is described in Madjarov2012 ().

The previously cited methods build ensemble structure during the training procedure. Consequently, throughout this paper, this kind of methods will be called static methods. The dynamic chain classifiers, on the other hand, determines the best label order at the prediction phase daSilva2014 (). The above-mentioned classifier produces a set of randomly generated label sequences and then validates the chain classifiers. During the validation phase, each point from the validation set is assigned with a label order that produces the most accurate output vector for this point. As the experimental research shows, the dynamic methods of building a label order may achieve better classification quality daSilva2014 ().

We observed that during the building of a dynamic chain classifier, multiple chain classifiers must be learned. These classifiers are built using the same training set and differ only in chain order. As a consequence, the computational burden of the algorithm may be reduced if there exists a classifier that is trained once and changing the label sequence is done without rebuilding the model. To address this issue, we built two models based on the Naive Bayes Hand2001 () approach and the nearest neighbour approach Cover1967 () that meet the above-mentioned properties.

Additionally, we proposed a dynamic method of determining the chain order based on classification quality for each label separately.

A part of this paper was previously published in Trajdos2017b (). This paper is an extended version of the previously-published work. The main elements that has been changed/extended:

  • The literature review has been extended.

  • We have added the results of the experimental comparison of the BR and CC versions of different base classifiers.

  • We have proposed a new model of the dynamic chain classifier. That is, we introduce the CC model based on the nearest neighbour approach.

  • New experimental results have been provided.

The rest of the paper is organised as follows. Next Section 2 provides a formal description of the multi-label classification problem and describes the developed algorithms. Section 3 contains a description of the conducted experiments. The results are presented and discussed in Section 4. Finally, Section 5 concludes the paper.

2 Proposed Methods

In this section, we introduce a formal notation of multi-label classification problem and provide a description of the proposed method.

2.1 Preliminaries

Under the multi-label (ML) formalism a object is assigned to a set of labels indicated by a binary vector of length : , where denotes the number of labels.

In this paper, we follow a statistical classification framework. As a consequence, it is assumed that object and its set of labels are realizations of corresponding random vectors , and the joint probability distribution on is known.

Because the above-mentioned assumption is never meet in real world, in this study, we assume that multi-label classifier , which maps feature space to the set , is built in a supervised learning procedure using the training set containing pairs of feature vectors and corresponding class labels :

(1)

2.2 Naive Bayes Classifier for Dynamic Classifier Chains

In this paper, we consider ML classifiers build according to the chain rule. That is, the classifier is an ensemble of single-label classifiers that constitutes a linked chain which is built according to a permutation of label sequence . As it was mentioned earlier, in this paper we follow the statistical classification framework. Consequently, each single-label classifier along with the chain makes its decision according to the following rule:

(2)

where is a random event defined below:

(3)

and for

(4)

Conditioning on the random event instead of allows the chain to take inter-label dependencies into account. The above-mentioned classification rule is a greedy rule that calculates the probability (2) using predictions of preceding classifiers. The optimal prediction under the chaining rule may be found using the PCC approach cheng2010bayes (). However, the approach requires the number of calculations that grows exponentially with the number of labels.

The probability defined in (2) is then computed using the Bayes rule:

(5)

The term does not depend on event . Consequently, the decision rule (2) is rewritten:

(6)

Now, to improve the readability we simplify the notation:

(7)

Then, following the Naive Bayes rule, we assume that all random variables that constitute are conditionally independent given . Consequently, is defined using the following formula:

(8)

Now, it is easy to see that the term , contrary to , depends on the chain structure. Furthermore, all probability distributions used in the above-mentioned terms can be estimated during the training phase when the chain structure is unknown.

The training and inference phases are described in detail using pseudocode shown in Algorithms 1 and 2.

Input data:
- training set;

BEGIN
Split into and so that:
and ,
;
Using build estimators of
the following distributions:
END
Algorithm 1 Pseudocode of the learning procedure of the Naive Bayes classifier.
Input data:
-- input instance;
-- validation set;

BEGIN
#Query the BR models
FOR :
;
;
END FOR;
Determine label permutaion using and ;
SET ;
DO:
FOR :
END FOR;
WHILE();
RETURN ;
END
Algorithm 2 Pseudocode of the inference procedure of the Naive Bayes classifier.

2.2.1 Computational complexity

In this section, we assess the increase in computational complexity that the proposed algorithm causes.

First of all, it is easy to see that for both the original and the proposed algorithm the number of estimators that must be built to assess is: .

The number of estimators of that must be built is also the same for both classifiers: L.

The key difference is in the number of estimators of that must be built. For the original CC classifier the number of estimators that is built is . On the other hand our method builds estimators.

At the inference phase, the only additional calculations are performed to determine the permutation of labels. Since the validation set is involved in this process, a number of calculations is proportional to .

2.3 KNN Classifier for Dynamic Classifier Chains

In this section, we define a dynamic classifier chain algorithm based on the nearest neighbours approach.The nearest neighbour algorithm is an instance-based classifier that does not build an explicit model of mapping between the feature space and the label space. Instead, the classifier performs classification in a lazy manner. That is, the R nearest instances and then the class is predicted using labels of the neighbour instances.

Let’s begin with the definition of a distance function that depends on label permutation and the position along the chain. The distance function is defined in the extended feature space that combines the input space and the label space. For the first position, the distance is a simple Euclidean distance in the input space:

(9)

For the other positions, the distance function uses both the input space and the label space:

(10)

Such defined distance function allows us to make the prediction using chaining rule. Since the distance is modified in order to fit the chain structure. During the inference phase, the distance calculates the extended distance using labels predicted at the preceding steps of the procedure. The above-defined distance function is used to build the neighbourhood of a given point in the extended feature space: . The neighbourhood contains the closest instances selected from the training set according to the distance function .

Given the neighbourhood, the probability is estimated as follows:

(11)

The label is also predicted using rule (2).

Input data:
- training set;

BEGIN
Split into and so that:
and ,
;
Save the training set
END
Algorithm 3 Pseudocode of the learning procedure of the nearest neighbour classifier.
Input data:
-- input instance;
-- validation set;
-- Training set;

BEGIN
SET: ;
Determine label permutaion using and ;
For
BEGIN
END
RETURN ;
END
Algorithm 4 Pseudocode of the inference procedure of the nearest neighbour classifier.

The training procedure is described in Algorithm 3. The procedure is very simple. That is, it splits the original training set into actual training set and the validation set .

The inference procedure begins with assigning undefined values into the prediction vector . Then the predictions are updated sequentially according to the permutation . The precodure is shown in Algorithm 4

2.4 Dynamic Chain order

In this subsection, we define a local measure of classification quality. To do so, we employed a modified version of the well-known measure.

First of all, we defined a fuzzy neighbourhood in the input space. The neighbourhood of an instance is defined using the following fuzzy set Zadeh1965 ():

(12)

where each tripplet defines fuzzy set with the membership coefficient . The membership function is defined using gaussian potential function:

(13)

The distance function is simple euclidean distance and the coefficient is tuned during the experiments.

Then, we define set of points that belongs to given label and that are classified as given label :

(14)
(15)

The above-mentioned classifier responses are related to the binary relevance classifier that can be built without knowing the order of the chain. The classifier is defined using the following classification rule:

(16)

Since the neighbourhood of a given instance is defined as a fuzzy set, consistently the above-mentioned sets are also defined as fuzzy. However, the sets are fuzzy singletons. The visualisation of aforementioned sets is provided in Figure 1.

Figure 1: Visualisation of ground truth labels and the decision set of the algorithm.

Using the above-mentioned sets we define local True Positive rate, False Positive rate, False Negative rate respectively:

(17)
(18)
(19)

where is the cardinality of a fuzzy set Dhar2013 (). Then, we define the local measure of classification quality:

(20)

Finally, the label order is chosen so that the following inequalities are met:

(21)

That is labels for whom the classification quality is higher precedes other labels in the chain structure. In other words, this simple heuristic is aimed at dealing with error propagation in the chain structure by employing the most accurate models at the beginning of the chain.

2.5 The Ensemble Classifier

Now, let us define a ML classifier ensemble: . The ensemble is built using classifier chain algorithms defined in previous sections. Each ensemble classifier is built using a subset of the original dataset. The size of subset is of the original training set.

The BR transformation may produce imbalanced single-label dataset. To prevent the classifier from learning from a highly imbalanced dataset, we applied the random undersampling technique Garca2012 (). The majority class is undersampled when imbalance ratio is higher than 20. The goal of undersampling is to keep the imbalance ratio at the level of 20.

The research on the application of Naive Bayes algorithm under the CC framework shows that when the number of features in the input space is significantly higher in comparison to the number of labels the Naive Bayes classifier may not perform well daSilva2014 (). To prevent the proposed system from being affected by this phenomenon, we applied the feature selection procedure for each single-label separately. That is, the attributes are selected in order to improve the classification quality for given label. The feature selection removes only attributes related to the original input space. Features related to labels are passed through the chain without selection. We employed the selection procedure based on correlation. In other words, we select attributes that are highly correlated to the predicted label and their inter-correlations are low Hall1999 (). Additionally, if the number of selected features is higher than 300, we select 300 random features from the set of previously selected features.

The final prediction vector of the ensemble is obtained via is a simple averaging of response vectors corresponding to base classifiers of the ensemble followed by the thresholding procedure:

(22)

where is the Iverson bracket.

3 Experimental Setup

The experimental study is divided into three main sections. The first one assesses the impact of employing chaining approach. In the section, we compare binary relevance and classifier chains algorithms built using the following base classifiers:

In this section, we compare BR and CC ensembles built using a genetic algorithm tailored to optimise the macro-averaged loss. For each ensemble, the size of the committee was set to . For the algorithm based on the genetic algorithm, the initial size of the committee was set to . Each numeric attribute in the training and validation datasets was also standardised. After the standardisation, the mean value of the attribute is 0 and its standard deviation is 1.

During the experimental study, the parameters of the SVM classifier (, ) were tuned using grid search and 3-fold cross validation. The number of nearest neighbours was also tuned using 3-fold cross validation. The number of neighbours was chosen among the following values .

In two remaining sections, the conducted experimental study provides an empirical evaluation of the classification quality of the proposed methods and compares it to reference methods. Namely, we conducted our experiments using the following algorithms of building a CC ensemble:

  1. The proposed approach (Section 2.4).

  2. Static ensemble generated using a genetic algorithm Trajdos2017 (). The enesmble is tuned to optimise the macro-averaged measure

  3. ECC ensemble with randomly generated chain orders Read2009 ().

  4. OOCC dynamic method proposed by da Silva et al. daSilva2014 (). The ensemble is tuned to optimise the example based measure. Additionally, the reference method uses single split into training and validation sets.

The above-mentioned methods of building CC systems were evaluated using Naive Bayes and the nearest neighbour algorithms as base classifiers. Systems built using different base classifiers are investigated in two separate sections. In the sections, we will refer to the investigated algorithms using the above-said numbers.

The reference algorithm also uses Naive Bayes/nearest neighbour algorithm with data preprocessing procedures described in Section 2.5.

The extraction of training and test datasets was performed using fold cross-validation. For each ensemble, the proportion of the training set was fixed at of the original training set (see Algorithm 1). For each ensemble, the size of the committee was set to . For the algorithm based on the genetic algorithm, the initial size of the committee was set to . Each numeric attribute in the training and validation datasets was also standardised. After the standardisation, the mean value of the attribute is 0 and its standard deviation is 1.

The coefficient was tuned during the training procedure using 3 CV approach. The best value among is chosen.

Single label classifiers were implemented using WEKA software Hall2009 (). Multi-label classifier were implemented using Mulan software mulan_software ().

The experiments were conducted using 30 multi-label benchmark sets. The main characteristics of the datasets are summarized in Table 1. We used datasets from the sources abbreviated as follows:A Charte2015 (), B meka () M–Tsoumakas2011_mulan (); W–Wu2014 (); X–Xu2013 (); Z–Zhou2012 (); T–Tomas2014 (); O – thorsten1998 (). Some of the employed sets needed some preprocessing. That is, we used multi-label multi-instance Zhou2012 () sets (sources Z and W) which were transformed to single-instance multi-label datasets according to the suggestion made by Zhou et al. Zhou2012 (). Multi-target regression sets (No 9, 30) were binarised using simple thresholding strategy. That is if the response is greater than 0 the resulting label is set relevant. Two of the used datasets are synthetic ones (source T) and they were generated using algorithm described in Tomas2014 (). To reduce the computational burden, we use only a subset of original Tmc2007 and IMDB sets. Additionally, the number of labels in Stackex datasets is reduced to 15.

The algorithms were compared in terms of 11 different quality criteria coming from three groups Luaces2012 (): Instance-based (Hamming, Zero-One, , False Discovery Rate, False Negative Rate); Label-based. The last group contains the following measures: Macro Averaged (False Discovery Rate (FDR, 1- Precision), False Negative Rate (FNR, 1-Recall), ) and Micro Averaged versions of the above-mentioned criteria.

Statistical evaluation of the results was performed using the Wilcoxon signed-rank test demsar2006 (); wilcoxon1945 () and the family-wise error rates were controlled using the Holm procedure demsar2006 (); holm1979 (). For all statistical tests, the significance level was set to . Additionally, we also applied the Friedman Friedman1940 () test followed by the Nemenyi post-hoc procedure demsar2006 ().

No Name Sr LC LD avIR
1 Arts1 M 7484 1733 26 1.654 .064 94.74
2 Azotobacter W 407 20 13 1.469 .113 2.225
3 Birds M 645 260 19 1.014 .053 5.407
4 Caenorhabditis W 2512 20 21 2.419 .115 2.347
5 Drosophila W 2605 20 22 2.656 .121 1.744
6 Emotions M 593 72 6 1.868 .311 1.478
7 Enron M 1702 1001 53 3.378 .064 73.95
8 Flags X 194 43 7 3.392 .485 2.255
9 Flare2 M 1066 27 3 0.209 .070 14.15
10 Genbase M 662 1186 27 1.252 .046 37.32
11 Geobacter W 379 20 11 1.264 .115 2.750
12 Haloarcula W 304 20 13 1.602 .123 2.419
13 Human X 3106 440 14 1.185 .085 15.29
14 Image M 2000 294 5 1.236 .247 1.193
15 IMDB M 3042 1001 28 1.987 .071 24.61
16 LLOG B 1460 1004 75 1.180 .016 39.27
17 Medical M 978 1449 45 1.245 .028 89.50
18 MimlImg Z 2000 135 5 1.236 .247 1.193
19 Ohsumed O 13929 1002 23 1.663 .072 7.869
20 Plant X 978 440 12 1.079 .090 6.690
21 Pyrococcus W 425 20 18 2.136 .119 2.421
22 Saccharomyces W 3509 20 27 2.275 .084 2.077
23 Scene X 2407 294 6 1.074 .179 1.254
24 SimpleHC T 3000 30 10 1.900 .190 1.138
25 SimpleHS T 3000 30 10 2.307 .231 2.622
26 SLASHDOT B 3782 1079 22 1.181 .054 17.69
27 Stackex_chess A 1675 585 15 1.137 .076 4.744
28 Tmc2007-500 M 2857 500 22 2.222 .101 17.15
29 water-quality M 1060 16 14 5.073 .362 1.767
30 yeast M 2417 103 14 4.237 .303 7.197
Table 1: Summarised properties of the datasets employed in the experimental study. Sr denotes the source of dataset, No. is the ordinal number of a set, is the number of instances, is the dimensionality of input space, denotes the number of labels. , , are label cardinality, label density and average imbalance ratio respectively Luaces2012 (); Charte2014 ().

4 Results and Discussion

4.1 Assessing the impact of chaining approach

In this section, we assess the consequences of employing chaining approach. That is, we compare binary relevance ensembles with classifier chain ensembles built using the same base classifier. The results are shown in Figure 2 and Table 2. Full results are presented in the appendix in Tables 5, 6 and 7. The compared algorithms are numbered as follows:

  1. BR ensemble built using J48 algorithm.

  2. CC ensemble built using J48 algorithm.

  3. BR ensemble built using SVM algorithm.

  4. CC ensemble built using SVM algorithm.

  5. BR ensemble built using NB algorithm.

  6. CC ensemble built using NB algorithm.

  7. BR ensemble built using KNN algorithm.

  8. CC ensemble built using KNN algorithm.

The analysis of the results clearly shows that there is a noticeable difference between two groups of measures. That is, the differences between BR-based and CC-based algorithms are greater in terms of example based criteria. On the other hand, the differences in mean ranks are lower for example based measures.

For the example based measures, the average ranks achieved by CC-based algorithms are lower than for BR-based algorithms. However, only for algorithms based on J48 classifier, the differences are significant for example-based FDR, FNR and measures. A similar trend is observed for the zero-one loss. In this case, only differences for the nearest neighbour classifier are insignificant. It means that CC-based classifiers obtain the higher number of ’perfect match’ results.

On the other hand, for the label-based measures and Hamming loss, almost no significant differences are observed. However, the average ranks suggest that for this group of measures, the classification quality may deteriorate.

The results clearly show that although label-specific quality measures do not change in a significant way, the prediction of the entire label-vector improves. This is an expected result since the CC-based approach incorporates the inter-label relations. This is a well-known fact that has been reported by authors that have previously compared both approaches Madjarov2012 ().

The results also show that there are almost no significant differences between J48, NB and KNN based algorithms. Contrary, SVM algorithm tends to outperform the remaining ones in terms of example-based criteria, hamming loss and zero-one loss. It means that although J48 algorithm takes the biggest advantage of employing the chain rule, NB and KNN based classifiers are comparable to J48-based ensembles.

Figure 2: Base classifier comparison. Visualisation of average ranks achieved by algorithms and corresponding critical distances for the Nemenyi post-hoc test. Each axis of the radar plot corresponds to given quality criterion. The closer a point is to the centre of the radar plot, the lower average rank is (lower is better). Black bars parallel to criterion-specific axes denote critical difference for the Nemenyi tests.
Alg. 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
Hamming Zero-One EX FDR EX FNR
Rnk 4.94 5.03 2.81 3.03 4.72 4.63 5.38 5.47 5.50 5.00 3.30 2.25 5.17 4.16 5.88 4.75 5.64 4.44 3.13 2.66 5.00 4.66 5.61 4.88 5.47 4.66 4.22 3.53 3.97 3.91 5.19 5.06
Frd
Wp-1 1.00 .002 .148 1.00 1.00 1.00 .432 .032 .001 .000 1.00 .478 1.00 1.00 .070 .001 .001 1.00 .650 1.00 1.00 .011 1.00 .009 .205 .054 1.00 1.00
Wp-2 .000 .025 1.00 1.00 1.00 1.00 1.00 .001 1.00 1.00 .044 1.00 .845 .001 1.00 1.00 .056 1.00 1.00 .223 1.00 1.00 1.00 1.00
Wp-3 1.00 .014 .022 .000 .000 .032 .005 .627 .001 .597 1.00 .002 .097 .001 .573 .937 1.00 1.00 1.00 1.00
Wp-4 .552 .552 .017 .002 .000 .004 .000 .002 .002 .006 .000 .002 1.00 1.00 .169 .113
Wp-5 1.00 1.00 .591 .041 1.00 1.00 1.00 .964 1.00 1.00 .221 1.00
Wp-6 1.00 1.00 .478 1.00 .666 1.00 .090 .266
Wp-7 1.00 .220 1.00 1.00
EX Macro FDR Macro FNR Macro
Rnk 5.72 4.47 4.06 2.53 4.53 4.25 5.63 4.81 4.25 5.50 3.00 4.91 3.97 5.25 4.00 5.13 4.63 4.45 5.20 4.97 4.03 4.78 3.41 4.53 4.81 5.09 4.25 4.84 3.94 5.16 3.41 4.50
Frd
Wp-1 .005 .041 .000 .200 .072 1.00 1.00 .019 .552 1.00 1.00 .390 1.00 .479 1.00 1.00 1.00 1.00 1.00 .022 1.00 1.00 1.00 1.00 1.00 1.00 .000 1.00
Wp-2 1.00 .001 1.00 1.00 .189 1.00 .003 1.00 .697 1.00 .152 1.00 1.00 1.00 1.00 1.00 .441 1.00 1.00 1.00 1.00 1.00 .031 1.00
Wp-3 .272 1.00 1.00 .089 1.00 .066 .552 .023 .349 .003 1.00 .077 1.00 .058 1.00 1.00 1.00 1.00 1.00 1.00
Wp-4 .014 .007 .000 .005 1.00 1.00 1.00 1.00 .419 1.00 .501 1.00 1.00 1.00 .824 1.00
Wp-5 1.00 .200 1.00 .208 1.00 1.00 .998 1.00 1.00 .355 1.00 1.00
Wp-6 .090 1.00 .101 1.00 1.00 1.00 .278 1.00
Wp-7 1.00 .552 .143 .203
Micro FDR Micro FNR Micro
Rnk 4.81 4.84 2.72 3.13 4.81 4.78 5.19 5.72 4.66 4.44 5.31 4.66 4.31 4.66 3.78 4.19 4.56 4.69 4.41 3.75 4.69 5.03 4.25 4.63
Frd
Wp-1 1.00 .001 .141 1.00 1.00 1.00 .045 1.00 .706 1.00 1.00 1.00 .136 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
Wp-2 .000 .023 1.00 1.00 1.00 .564 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
Wp-3 1.00 .001 .001 .000 .001 1.00 .233 1.00 .215 .132 1.00 1.00 1.00 1.00 1.00
Wp-4 .091 .079 .015 .007 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
Wp-5 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00 1.00
Wp-6 1.00 1.00 1.00 1.00 1.00 1.00
Wp-7 1.00 1.00 1.00
Table 2: Base classifier comparison. Result of the statistical evaluation. Rnk stands for average rank over all datasets, Frd is the p-value obtained using the Friedman test and Wp-i denotes the p-value associated with the Wilcoxon test that compares the i-th algorithm against the others.

4.2 Naive Bayes Classifier

The results of the experimental study are presented in Table 3 and Figure (a)a. Tables 86 and 10 show full results of the experiment. Table 3 provides results of the statistical evaluation of the experiments. Figure (a)a visualises the average ranks and provide a view of the Nemenyi post-hoc procedure.

First, let’s analyse differences between the proposed heuristic and the simple ECC ensemble. The proposed method is tailored to optimise the macro-averaged loss so we begin with investigating macro-averaged measures. It is easy to see that both methods are comparable in terms of recall but the proposed one is significantly better in terms of precision. It means that the proposed method makes significantly less false positive predictions. Consequently, under the macro-averaged loss the proposed method outperforms the ECC ensemble. The same pattern is also present in results related to micro-averaged measures. However, the difference for the micro-averaged measure is not significant. In contrast, under example based measures, except the Hamming loss, there are no significant differences between investigated methods.

The results show that the proposed heuristic provides an effective way of improving classification quality for classifier chains ensemble. Moving the best performing label-specific models at the beginning of the chain reduces the error that propagates along the chain. What is more, the experimental study also showed that the Naive Bayes classifier combined with proper data preprocessing may be effectively employed in classifier chain ensembles.

Now, let’s compare the proposed method to the other algorithm based on the dynamic chain approach. When we investigate the example-based criteria it is easy to see that the OOCC algorithm outperforms the proposed one in terms of FDR and Hamming loss. Those results combined with results achieved in terms of macro and micro averaged measures shows that the OOCC mthod seems to be too much conservative. That is, it tends to makes many false negative predictions in comparison to the other methods. The outstanding results for the Hamming loss are a consequence of the imbalanced nature of the multi-label data. That is, the presence of labels is relatively rare and the prediction that contains many false negatives may achieve inadequately hight performance under the Hamming loss Luaces2012 ().

On the other hand, the average ranks clearly show that the method based on genetic algorithm achieves the best results in comparison to the other investigated methods. The main reason is that the GA-based approach optimises the entire ensemble structure, whereas the investigated dynamic chain methods, choose the best label order for single classifier chain. Then the locally chosen chains are combined into an ensemble. It gives us an important clue. That is when we consider an algorithm for dynamic chain order selection, we should think about a single chain and the global structure of the entire ensemble as well.

(a) Naive Bayes classifier
(b) Nearest Neighbour classifier.
Figure 3: Visualisation of average ranks achieved by algorithms.
Alg. 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
Hamming Zero-One EX FDR EX FNR
Rnk 2.45 2.67 3.03

1.84

2.52

2.14

2.97 2.38 2.84

1.97

2.91 2.28 2.56

2.03

2.53 2.88
Frd
Wp-1 .702 .025 .063 .358 .199 .761 .019 .295 .134 .053 .700 .700
Wp-2 .319 .319 .006 .368 .005 .239 .136 .015
Wp-3 .001 .097 .040 .136
EX Macro FDR Macro FNR Macro
Rnk 2.69

2.03

2.84 2.44 2.31

2.22

3.16 2.31 2.56

1.94

2.36 3.14 2.47

1.81

2.84 2.88
Frd
Wp-1 .066 .821 .821 1.00 .012 1.00 .035 .919 .022 .156 .096 .248
Wp-2 .017 .112 .031 1.00 .174 .002 .003 .022
Wp-3 .590 .012 .105 .733
Micro FDR Micro FNR Micro
Rnk 2.47 2.66 3.22

1.66

2.69

1.72

2.34 3.25 2.75

1.78

2.81 2.66
Frd
Wp-1 1.00 .028 .000 .005 .254 .239 .044 .610 1.00
Wp-2 1.00 .008 .052 .000 .002 .044
Wp-3 .000 .014 1.00
Table 3: Naive Bayes Based CC. Results of the statistical evaluation.

4.3 Nearest Neighbour Classifier

The results of the experimental study are presented in Table 4 and Figure (b)b. Tables 1112 and 13 show full results of the experiment. Table 3 provides results of the statistical evaluation of the experiments. Figure (b)b visualises the average ranks and provide a view of the Nemenyi post-hoc procedure.

The results show that for the group of example based measures and the zero-one loss, there are no significant differences in classification quality between all investigated algorithms.

For macro and micro averaged measures, the best performing algorithm is an ensemble optimised using the genetic algorithm. The proposed nearest-neighbour-based classifier does not differ significantly from ECC OOCC algorithms. However, it tends to be more conservative because it achieves lower FDR and higher FDR. In other words, the classifier tends to decrease the false positive rate at the cost of decreasing the true positive rate. This phenomenon causes the highest classification quality in terms of the Hamming loss. The reason is that for the multi-label set with low label density, it is easy to obtain high classification by setting all possible labels as irrelevant.

The results confirm the findings described in Section 4.1. The nearest-neighbour-based CC algorithm is unable to take all benefits of the chaining approach. On the other hand, the method is still comparable to chains built using different base classifiers. The main goal of this paper was to propose the model that can change the chain structure without retraining. This goal was achieved.

Alg. 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4
Hamming Zero-One EX FDR EX FNR
Rnk 1.72 2.83 2.80 2.65 2.22 2.73 2.42 2.63 2.17 2.57 2.57 2.70 2.60 2.47 2.40 2.53
Frd
Wp-1 .004 .000 .006 1.00 1.00 1.00 1.00 1.00 .948 1.00 .985 1.00
Wp-2 .441 .173 1.00 1.00 1.00 1.00 1.00 1.00
Wp-3 .441 1.00 1.00 1.00
EX Macro FDR Macro FNR Macro
Rnk 2.43 2.57 2.47 2.53 2.27 1.67 3.10 2.97 2.97 1.70 2.78 2.55 2.600 1.533 3.033 2.833
Frd
Wp-1 1.00 1.00 1.00 .017 .010 .012 .000 .617 .617 .000 .276 .579
Wp-2 1.00 1.00 .000 .003 .000 .004 .000 .000
Wp-3 1.00 .271 .561 .579
Micro FDR Micro FNR Micro
Rnk 1.67 2.97 2.77 2.60 2.97 1.70 2.77 2.57 2.63 1.63 3.00 2.73
Frd
Wp-1 .009 .000 .028 .000 .926 .926 .002 .532 .657
Wp-2 .382 .070 .002 .019 .001 .006
Wp-3 .715 .926 .657
Table 4: Nearest Neighbour based CC. Result of statistical evaluation.

5 Conclusions

The main goal of this research was to provide an effective chain classifier that allows changing label order at relatively low computational cost. We achieved it using a classifier based on the Naive Bayes approach. To prove that the proposed method allows handling inter-label relations in an efficient way, we proposed a simple heuristic method that determines label order that should minimise label propagation error. Indeed, the experimental results showed that the proposed method is able to produce a good chain structure at a low computational cost. However, the proposed method of building a dynamic ensemble does not allow to outperform the static system that optimizes the entire ensemble structure. The obtained results are very promising. We believe that there is still a room for improvement. In our opinion, the performance of the system may be improved if we provide better, a better heuristic that optimises the entire ensemble in a dynamic way. The proposed dynamic classifier is a first step in the process of investigating dynamic classifier chain ensembles.

Another way of improving this idea is to build different classifiers that would be able to change the chain structure without retraining the entire model.

Acknowledgment

The work was supported by the statutory funds of the Department of Systems and Computer Networks, Wroclaw University of Science and Technology, under agreement 0401/0159/16.

References

  • (1) Alvares Cherman, E., Metz, J., Monard, M.C.: A simple approach to incorporate label dependency in multi-label classification. In: Advances in Soft Computing, pp. 33–43. Springer Berlin Heidelberg (2010). DOI 10.1007/978-3-642-16773-7˙3
  • (2) Chang, C.C., Lin, C.J.: LIBSVM. ACM Transactions on Intelligent Systems and Technology 2(3), 1–27 (2011). DOI 10.1145/1961189.1961199
  • (3) Charte, F., Rivera, A., del Jesus, M.J., Herrera, F.: Concurrence among imbalanced labels and its influence on multilabel resampling algorithms. In: Lecture Notes in Computer Science, pp. 110–121. Springer International Publishing (2014). DOI 10.1007/978-3-319-07617-1˙10
  • (4) Charte, F., Rivera, A.J., del Jesus, M.J., Herrera, F.: Quinta: A question tagging assistant to improve the answering ratio in electronic forums. In: IEEE EUROCON 2015 - International Conference on Computer as a Tool (EUROCON). IEEE (2015). DOI 10.1109/eurocon.2015.7313677
  • (5) Chen, B., Li, W., Zhang, Y., Hu, J.: Enhancing multi-label classification based on local label constraints and classifier chains. In: 2016 International Joint Conference on Neural Networks (IJCNN). IEEE (2016). DOI 10.1109/ijcnn.2016.7727370
  • (6) Cheng, W., Hüllermeier, E., Dembczynski, K.J.: Bayes optimal multilabel classification via probabilistic classifier chains. In: Proceedings of the 27th international conference on machine learning (ICML-10), pp. 279–286 (2010)
  • (7) Cortes, C., Vapnik, V.: Support-vector networks. Machine Learning 20(3), 273–297 (1995). DOI 10.1007/bf00994018
  • (8) Cover, T., Hart, P.: Nearest neighbor pattern classification. IEEE Transactions on Information Theory 13(1), 21–27 (1967). DOI 10.1109/tit.1967.1053964
  • (9) Díez, J., Luaces, O., del Coz, J.J., Bahamonde, A.: Optimizing different loss functions in multilabel classifications. Progress in Artificial Intelligence 3(2), 107–118 (2014). DOI 10.1007/s13748-014-0060-7
  • (10) Demšar, J.: Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research 7, 1–30 (2006)
  • (11) Dhar, M.: On cardinality of fuzzy sets. International Journal of Intelligent Systems and Applications 5(6), 47–52 (2013). DOI 10.5815/ijisa.2013.06.06
  • (12) Friedman, M.: A comparison of alternative tests of significance for the problem of rankings. The Annals of Mathematical Statistics 11(1), 86–92 (1940). DOI 10.1214/aoms/1177731944
  • (13) García, V., Sánchez, J., Mollineda, R.: On the effectiveness of preprocessing methods when dealing with different levels of class imbalance. Knowledge-Based Systems 25(1), 13–21 (2012). DOI 10.1016/j.knosys.2011.06.013
  • (14) Gibaja, E., Ventura, S.: Multi-label learning: a review of the state of the art and ongoing research. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 4(6), 411–444 (2014). DOI 10.1002/widm.1139
  • (15) Gonçalves, E.C., Plastino, A., Freitas, A.A.: Simpler is better. In: Proceedings of the 2015 on Genetic and Evolutionary Computation Conference - GECCO ’15. ACM Press (2015). DOI 10.1145/2739480.2754650
  • (16) Goncalves, E.C., Plastino, A., Freitas, A.A.: A genetic algorithm for optimizing the label ordering in multi-label classifier chains. In: 2013 IEEE 25th International Conference on Tools with Artificial Intelligence. IEEE (2013). DOI 10.1109/ictai.2013.76
  • (17) Hall, M., Frank, E., Holmes, G., Pfahringer, B., Reutemann, P., Witten, I.H.: The weka data mining software. ACM SIGKDD Explorations Newsletter 11(1), 10 (2009). DOI 10.1145/1656274.1656278
  • (18) Hall, M.A.: Correlation-based feature selection for machine learning. Ph.D. thesis, The University of Waikato (1999)
  • (19) Hand, D.J., Yu, K.: Idiot’s bayes: Not so stupid after all? International Statistical Review / Revue Internationale de Statistique 69(3), 385 (2001). DOI 10.2307/1403452
  • (20) Holm, S.: A Simple Sequentially Rejective Multiple Test Procedure. Scandinavian Journal of Statistics 6(2), 65–70 (1979). DOI 10.2307/4615733
  • (21) Huang, J., Li, G., Wang, S., Zhang, W., Huang, Q.: Group sensitive classifier chains for multi-label classification. In: 2015 IEEE International Conference on Multimedia and Expo (ICME). IEEE (2015). DOI 10.1109/icme.2015.7177400
  • (22) Jiang, J.Y., Tsai, S.C., Lee, S.J.: Fsknn: Multi-label text categorization based on fuzzy similarity and k nearest neighbors. Expert Systems with Applications 39(3), 2813–2821 (2012). DOI 10.1016/j.eswa.2011.08.141
  • (23) Joachims, T.: Text categorization with suport vector machines: Learning with many relevant features. In: Proc. 10th European Conference on Machine Learning, pp. 137–142 (1998)
  • (24) Liu, X., Shi, Z., Li, Z., Wang, X., Shi, Z.: Sorted label classifier chains for learning images with multi-label. In: Proceedings of the international conference on Multimedia - MM ’10. ACM Press (2010). DOI 10.1145/1873951.1874121
  • (25) Luaces, O., Díez, J., Barranquero, J., del Coz, J.J., Bahamonde, A.: Binary relevance efficacy for multilabel classification. Progress in Artificial Intelligence 1(4), 303–313 (2012). DOI 10.1007/s13748-012-0030-x
  • (26) Madjarov, G., Kocev, D., Gjorgjevikj, D., Džeroski, S.: An extensive experimental comparison of methods for multi-label learning. Pattern Recognition 45(9), 3084–3104 (2012). DOI 10.1016/j.patcog.2012.03.004
  • (27) Montañes, E., Senge, R., Barranquero, J., Ramón Quevedo, J., José del Coz, J., Hüllermeier, E.: Dependent binary relevance models for multi-label classification. Pattern Recognition 47(3), 1494–1508 (2014). DOI 10.1016/j.patcog.2013.09.029
  • (28) Quinlan, J.R.: C4.5 : Programs for machine learning. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA (1993)
  • (29) Read, J., Martino, L., Luengo, D.: Efficient monte carlo methods for multi-dimensional learning with classifier chains. Pattern Recognition 47(3), 1535–1546 (2014). DOI 10.1016/j.patcog.2013.10.006
  • (30) Read, J., Peter, R.: Meka:http://meka.sourceforge.net/ (2017). URL http://meka.sourceforge.net/
  • (31) Read, J., Pfahringer, B., Holmes, G., Frank, E.: Classifier chains for multi-label classification. In: Machine Learning and Knowledge Discovery in Databases, pp. 254–269. Springer Berlin Heidelberg (2009). DOI 10.1007/978-3-642-04174-7˙17
  • (32) Read, J., Pfahringer, B., Holmes, G., Frank, E.: Classifier chains for multi-label classification. Machine Learning 85(3), 333–359 (2011). DOI 10.1007/s10994-011-5256-5
  • (33) Sanden, C., Zhang, J.Z.: Enhancing multi-label music genre classification through ensemble techniques. In: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information - SIGIR ’11. ACM Press (2011). DOI 10.1145/2009916.2010011
  • (34) Senge, R., del Coz, J.J., Hüllermeier, E.: On the problem of error propagation in classifier chains for multi-label classification. In: Studies in Classification, Data Analysis, and Knowledge Organization, pp. 163–170. Springer International Publishing (2013). DOI 10.1007/978-3-319-01595-8˙18
  • (35) da Silva, P.N., Gonçalves, E.C., Plastino, A., Freitas, A.A.: Distinct chains for different instances: An effective strategy for multi-label classifier chains. In: Machine Learning and Knowledge Discovery in Databases, pp. 453–468. Springer Berlin Heidelberg (2014). DOI 10.1007/978-3-662-44851-9˙29
  • (36) Spyromitros-Xioufis, E., Tsoumakas, G., Groves, W., Vlahavas, I.: Multi-target regression via input space expansion: treating targets as inputs. Machine Learning 104(1), 55–98 (2016). DOI 10.1007/s10994-016-5546-z
  • (37) Spyromitros-Xioufis, E., Tsoumakas, G., Groves, W., Vlahavas, I.: Multi-target regression via input space expansion: treating targets as inputs. Machine Learning 104(1), 55–98 (2016). DOI 10.1007/s10994-016-5546-z
  • (38) Tomás, J.T., Spolaôr, N., Cherman, E.A., Monard, M.C.: A framework to generate synthetic multi-label datasets. Electronic Notes in Theoretical Computer Science 302, 155–176 (2014). DOI 10.1016/j.entcs.2014.01.025
  • (39) Trajdos, P., Kurzynski, M.: Naive bayes classifier for dynamic chaining approach in multi-label learning. International Journal of Education and Learning Systems 2, 133–142 (2017). URL http://www.iaras.org/iaras/filedownloads/ijels/2017/002-0019(2017).pdf
  • (40) Trajdos, P., Kurzynski, M.: Permutation-based diversity measure for classifier-chain approach. In: Advances in Intelligent Systems and Computing, pp. 412–422. Springer International Publishing (2017). DOI 10.1007/978-3-319-59162-9˙43
  • (41) Tsoumakas, G., Katakis, I., Vlahavas, I.: Effective and efficient multilabel classification in domains with large number of labels, p. 30–44 (2008)
  • (42) Wei, Y., Xia, W., Lin, M., Huang, J., Ni, B., Dong, J., Zhao, Y., Yan, S.: Hcp: A flexible cnn framework for multi-label image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 38(9), 1901–1907 (2016). DOI 10.1109/tpami.2015.2491929
  • (43) Wilcoxon, F.: Individual comparisons by ranking methods. Biometrics Bulletin 1(6), 80 (1945). DOI 10.2307/3001968
  • (44) Wu, J.S., Huang, S.J., Zhou, Z.H.: Genome-wide protein function prediction through multi-instance multi-label learning. IEEE/ACM Transactions on Computational Biology and Bioinformatics 11(5), 891–902 (2014). DOI 10.1109/tcbb.2014.2323058
  • (45) Xu, J.: Fast multi-label core vector machine. Pattern Recognition 46(3), 885–898 (2013). DOI 10.1016/j.patcog.2012.09.003
  • (46) Zadeh, L.: Fuzzy sets. Information and Control 8(3), 338–353 (1965). DOI 10.1016/s0019-9958(65)90241-x
  • (47) Zhang, P., Yang, Y., Zhu, X.: Approaching multi-dimensional classification by using bayesian network chain classifiers. In: 2014 Sixth International Conference on Intelligent Human-Machine Systems and Cybernetics. IEEE (2014). DOI 10.1109/ihmsc.2014.129
  • (48) Zhou, Z.H., Zhang, M.L., Huang, S.J., Li, Y.F.: Multi-instance multi-label learning. Artificial Intelligence 176(1), 2291–2320 (2012). DOI 10.1016/j.artint.2011.10.002

Appendix A Full results

Macro FDR Macro FNR Macro
No. 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
1 .668 .622 .758 .645 .789 .780 .666 .590 .682 .642 .775 .604 .802 .782 .692 .622 .694 .651 .781 .649 .810 .796 .697 .625
2 .639 .608 .507 .527 .568 .565 .722 .664 .638 .593 .529 .530 .564 .561 .678 .623 .663 .620 .528 .534 .575 .570 .729 .678
3 .386 .402 .384 .418 .367 .389 .394 .402 .408 .429 .411 .423 .400 .413 .422 .438 .410 .432 .411 .433 .398 .414 .424 .433
4 .526 .449 .409 .447 .449 .440 .534 .678 .556 .486 .428 .471 .464 .474 .567 .655 .557 .481 .425 .468 .467 .470 .572 .693
5 .582 .476 .411 .453 .453 .482 .547 .623 .605 .487 .434 .469 .459 .496 .565 .608 .621 .486 .438 .467 .470 .494 .586 .639
6 .351 .365 .330 .333 .368 .373 .342 .343 .335 .337 .322 .310 .264 .265 .324 .302 .376 .382 .359 .350 .353 .352 .365 .354
7 .466 .510 .442 .498 .493 .517 .481 .515 .408 .398 .397 .414 .418 .419 .404 .399 .472 .485 .450 .483 .495 .506 .476 .492
8 .293 .280 .280 .283 .276 .271 .284 .294 .208 .216 .231 .215 .248 .213 .194 .221 .262 .258 .271 .263 .276 .255 .251 .267
9 .210 .216 .196 .198 .194 .204 .195 .190 .210 .216 .198 .200 .196 .207 .198 .192 .211 .217 .198 .200 .196 .207 .197 .192
10 .007 .006 .008 .007 .009 .008 .007 .005 .051 .052 .052 .049 .052 .053 .051 .052 .037 .037 .038 .035 .038 .038 .037 .037
11 .617 .543 .532 .528 .552 .556 .693 .598 .628 .550 .549 .548 .565 .581 .677 .601 .640 .561 .553 .551 .568 .577 .714 .609
12 .507 .538 .527 .512 .574 .524 .668 .626 .554 .546 .575 .527 .597 .571 .662 .628 .551 .562 .565 .532 .597 .565 .691 .646
13 .730 .703 .700 .640 .703 .699 .724 .739 .667 .633 .615 .577 .590 .575 .628 .594 .719 .691 .682 .628 .676 .668 .703 .699
14 .416 .405 .380 .336 .530 .524 .411 .393 .427 .431 .400 .378 .239 .264 .411 .425 .439 .434 .410 .374 .453 .454 .430 .424
15 .838 .785 .836 .746 .812 .716 .834 .751 .794 .690 .794 .683 .793 .731 .785 .680 .841 .769 .839 .739 .827 .756 .833 .743
16 .713 .728 .716 .699 .692 .704 .715 .699 .689 .680 .720 .711 .647 .672 .682 .652 .715 .723 .727 .712 .689 .706 .714 .692
17 .258 .244 .259 .228 .266 .262 .256 .232 .213 .227 .211 .235 .220 .239 .211 .233 .255 .249 .254 .243 .262 .266 .252 .245
18 .522 .498 .446 .411 .469 .488 .472 .470 .516 .511 .447 .440 .387 .388 .472 .489 .537 .521 .465 .443 .459 .471 .491 .494
19 .525 .471 .516 .452 .542 .524 .526 .459 .516 .488 .488 .474 .540 .522 .511 .493 .554 .513 .538 .499 .573 .556 .553 .511
20 .800 .761 .783 .727 .774 .751 .806 .757 .693 .604 .706 .605 .714 .652 .707 .638 .772 .717 .762 .688 .760 .724 .779 .721
21 .703 .664 .606 .610 .648 .627 .742 .749 .701 .686 .630 .644 .652 .671 .642 .712 .725 .687 .630 .639 .666 .663 .748 .762
22 .541 .486 .455 .457 .489 .476 .826 .720 .560 .502 .467 .470 .497 .486 .785 .689 .564 .501 .468 .469 .501 .487 .841 .728
23 .366 .335 .346 .289 .380 .366 .323 .277 .354 .342 .332 .303 .191 .210 .273 .281 .369 .345 .348 .302 .324 .320 .313 .287
24 .261 .232 .165 .117 .200 .198 .190 .166 .485 .471 .433 .414 .404 .398 .438 .440 .428 .408 .361 .331 .361 .358 .373 .365
25 .568 .636 .543 .580 .551 .561 .698 .706 .673 .596 .765 .679 .713 .699 .486 .500 .664 .652 .710 .669 .682 .676 .651 .657
26 .694 .572 .677 .546 .682 .561 .701 .559 .687 .551 .667 .551 .656 .524 .681 .576 .701 .577 .684 .560 .684 .559 .704 .577
27 .578 .553 .562 .555 .591 .577 .562 .570 .575 .559 .575 .581 .590 .570 .575 .582 .596 .576 .586 .585 .609 .594 .586 .595
28 .345 .348 .334 .333 .350 .343 .349 .355 .357 .342 .344 .342 .295 .291 .357 .348 .396 .391 .384 .379 .369 .364 .397 .396
29 .509 .506 .419 .510 .531 .529 .502 .484 .335 .319 .458 .334 .240 .256 .351 .360 .460 .457 .470 .468 .445 .447 .464 .460
30 .456 .389 .392 .398 .422 .405 .436 .515 .281 .340 .299 .310 .288 .303 .309 .313 .406 .390 .375 .383 .389 .384 .409 .460
Table 5: Base classifier comparison – full results – example based criteria.
Macro FDR Macro FNR Macro Hamming
No. 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
1 . 569 .613 .563 .648 .583 .635 .564 .647 .760 .767 .811 .754 .798 .798 .769 .771 .715 .732 .768 .755 .769 .778 .718 .744 .060 .064 .061 .086 .062 .065 .058 .068
2 . 797 .879 .804 .909 .835 .901 .807 .856 .804 .857 .894 .919 .896 .926 .729 .817 .822 .883 .875 .918 .880 .924 .784 .853 .168 .168 .124 .113 .135 .124 .214 .199
3 . 717 .739 .708 .738 .713 .729 .697 .738 .777 .783 .777 .809 .789 .790 .767 .825 .768 .779 .761 .793 .768 .779 .752 .803 .054 .055 .053 .058 .051 .054 .053 .053
4 . 606 .917 .282 .941 .502 .866 .565 .870 .774 .970 .807 .982 .792 .974 .726 .775 .726 .962 .720 .976 .723 .959 .675 .838 .127 .118 .099 .115 .111 .117 .126 .258
5 . 724 .981 .520 .975 .584 .980 .682 .856 .764 .985 .812 .981 .796 .983 .701 .829 .758 .984 .757 .979 .753 .983 .697 .852 .156 .122 .117 .121 .130 .123 .155 .228
6 . 335 .345 .310 .327 .391 .394 .329 .341 .360 .353 .341 .320 .275 .279 .340 .319 .358 .356 .338 .332 .344 .349 .345 .338 .207 .212 .195 .200 .224 .223 .202 .207
7 . 763 .770 .751 .767 .759 .773 .759 .766 .729 .731 .739 .750 .716 .733 .728 .735 .758 .762 .757 .768 .752 .766 .755 .761 .066 .070 .060 .066 .070 .076 .066 .071
8 . 320 .342 .342 .357 .338 .308 .326 .347 .280 .295 .312 .327 .339 .300 .264 .307 .324 .332 .345 .362 .358 .325 .312 .344 .248 .244 .255 .250 .248 .239 .240 .253
9 . 618 .625 .547 .618 .554 .573 .566 .563 .693 .690 .692 .721 .647 .645 .692 .682 .672 .673 .654 .691 .621 .627 .656 .644 .081 .084 .075 .078 .075 .079 .075 .073
10 . 250 .245 .255 .238 .242 .245 .246 .240 .246 .242 .250 .235 .239 .245 .242 .241 .248 .244 .252 .237 .241 .246 .244 .241 .005 .005 .005 .005 .005 .005 .005 .005
11 . 847 .873 .831 .859 .844 .864 .829 .821 .818 .844 .853 .857 .870 .894 .745 .885 .847 .872 .852 .865 .864 .892 .805 .868 .152 .137 .126 .120 .126 .122 .206 .145
12 . 820 .843 .859 .844 .806 .883 .819 .834 .839 .807 .881 .831 .852 .880 .811 .837 .844 .841 .877 .849 .841 .889 .829 .852 .123 .146 .123 .128 .127 .127 .176 .171
13 . 837 .832 .800 .789 .795 .802 .817 .862 .846 .808 .806 .790 .789 .796 .803 .782 .847 .831 .812 .803 .812 .822 .832 .843 .111 .123 .112 .110 .108 .114 .124 .156
14 . 291 .304 .275 .268 .502 .488 .323 .287 .447 .455 .425 .404 .269 .290 .439 .449 .383 .393 .361 .346 .445 .443 .389 .382 .169 .175 .162 .158 .305 .293 .178 .170
15 . 911 .903 .906 .907 .907 .899 .908 .914 .899 .866 .890 .895 .902 .896 .893 .891 .914 .902 .912 .912 .907 .903 .909 .911 .124 .136 .129 .114 .117 .110 .127 .118
16 . 653 .662 .668 .669 .651 .650 .658 .657 .658 .651 .672 .671 .646 .653 .658 .650 .662 .662 .674 .673 .653 .656 .662 .659 .020 .025 .018 .018 .022 .022 .022 .026
17 . 382 .380 .381 .380 .383 .388 .383 .384 .360 .366 .358 .375 .361 .372 .360 .377 .377 .378 .375 .382 .378 .384 .377 .385 .012 .012 .012 .011 .013 .012 .012 .011
18 . 407 .417 .342 .357 .477 .504 .398 .381 .536 .529 .469 .463 .408 .404 .490 .503 .483 .484 .417 .420 .453 .468 .454 .455 .213 .219 .190 .196 .240 .256 .208 .205
19 . 513 .505 .536 .513 .505 .507 .524 .504 .591 .581 .567 .580 .610 .595 .581 .594 .578 .564 .574 .566 .585 .574 .577 .569 .064 .062 .065 .063 .064 .063 .065 .062
20 . 849 .848 .814 .805 .835 .819 .816 .849 .784 .749 .811 .759 .834 .773 .806 .782 .837 .819 .821 .802 .843 .812 .832 .834 .151 .172 .128 .151 .122 .138 .145 .161
21 . 809 .891 .857 .945 .853 .904 .830 .887 .810 .933 .919 .947 .898 .940 .692 .851 .826 .929 .911 .948 .888 .934 .789 .878 .168 .135 .127 .121 .141 .123 .240 .209
22 . 825 .952 .826 .950 .828 .934 .882 .894 .927 .970 .964 .973 .945 .969 .756 .809 .909 .970 .945 .968 .923 .965 .843 .866 .098 .087 .084 .085 .091 .087 .215 .206
23 . 185 .182 .172 .173 .346 .332 .249 .207 .356 .343 .335 .311 .202 .220 .280 .289 .286 .275 .268 .253 .290 .287 .272 .256 .092 .090 .088 .085 .129 .122 .100 .090
24 . 151 .131 .106 .091 .202 .211 .141 .111 .583 .572 .543 .527 .516 .513 .542 .546 .446 .432 .400 .382 .405 .405 .407 .402 .126 .122 .114 .110 .124 .125 .118 .115
25 . 680 .721 .733 .762 .679 .657 .744 .750 .787 .695 .882 .788 .828 .809 .563 .576 .774 .726 .881 .803 .809 .787 .678 .687 .249 .308 .221 .274 .234 .240 .392 .396
26 . 516 .505 .522 .469 .535 .517 .523 .502 .633 .588 .620 .594 .609 .566 .621 .619 .598 .570 .591 .558 .589 .557 .596 .585 .048 .051 .049 .046 .052 .051 .052 .048
27 . 598 .597 .586 .608 .574 .605 .571 .608 .644 .647 .681 .674 .701 .697 .673 .682 .658 .661 .676 .678 .671 .676 .669 .679 .083 .080 .077 .079 .080 .081 .077 .084
28 . 449 .466 .434 .455 .438 .454 .423 .459 .545 .521 .506 .549 .464 .485 .550 .558 .528 .521 .498 .528 .474 .489 .520 .538 .074 .075 .073 .073 .073 .073 .075 .076
29 . 532 .533 .491 .533 .534 .526 .534 .522 .37