Unsupervised Feature Construction for Improving Data Representation and Semantics

Unsupervised Feature Construction for Improving Data Representation and Semantics

Marian-Andrei Rizoiu Marian-Andrei Rizoiu Julien Velcin Stéphane Lallich ERIC Laboratory, University Lumière Lyon 2
5, avenue Pierre Mendès France, 69676 Bron Cedex, France
Tel. +33 (0)4 78 77 31 54
Fax. +33 (0)4 78 77 23 75
Marian-Andrei Rizoiu 66email: Marian-Andrei.Rizoiu@univ-lyon2.frJulien Velcin 88email: Julien.Velcin@univ-lyon2.frStéphane Lallich 1010email: Stephane.Lallich@univ-lyon2.fr
   Julien Velcin Marian-Andrei Rizoiu Julien Velcin Stéphane Lallich ERIC Laboratory, University Lumière Lyon 2
5, avenue Pierre Mendès France, 69676 Bron Cedex, France
Tel. +33 (0)4 78 77 31 54
Fax. +33 (0)4 78 77 23 75
Marian-Andrei Rizoiu 66email: Marian-Andrei.Rizoiu@univ-lyon2.frJulien Velcin 88email: Julien.Velcin@univ-lyon2.frStéphane Lallich 1010email: Stephane.Lallich@univ-lyon2.fr
  
Stéphane Lallich
Marian-Andrei Rizoiu Julien Velcin Stéphane Lallich ERIC Laboratory, University Lumière Lyon 2
5, avenue Pierre Mendès France, 69676 Bron Cedex, France
Tel. +33 (0)4 78 77 31 54
Fax. +33 (0)4 78 77 23 75
Marian-Andrei Rizoiu 66email: Marian-Andrei.Rizoiu@univ-lyon2.frJulien Velcin 88email: Julien.Velcin@univ-lyon2.frStéphane Lallich 1010email: Stephane.Lallich@univ-lyon2.fr
Received: 27/01/2012 / Accepted: 29/01/2013
Abstract

Feature-based format is the main data representation format used by machine learning algorithms. When the features do not properly describe the initial data, performance starts to degrade. Some algorithms address this problem by internally changing the representation space, but the newly-constructed features are rarely comprehensible. We seek to construct, in an unsupervised way, new features that are more appropriate for describing a given dataset and, at the same time, comprehensible for a human user. We propose two algorithms that construct the new features as conjunctions of the initial primitive features or their negations. The generated feature sets have reduced correlations between features and succeed in catching some of the hidden relations between individuals in a dataset. For example, a feature like would be true for non-urban images and is more informative than simple features expressing the presence or the absence of an object. The notion of Pareto optimality is used to evaluate feature sets and to obtain a balance between total correlation and the complexity of the resulted feature set. Statistical hypothesis testing is used in order to automatically determine the values of the parameters used for constructing a data-dependent feature set. We experimentally show that our approaches achieve the construction of informative feature sets for multiple datasets.

Keywords:
Unsupervised feature construction Feature evaluation Nonparametric statistics Data mining Clustering Representations Algorithms for data and knowledge management Heuristic methods Pattern analysis

1 Introduction

Most machine learning algorithms use a representation space based on a feature-based format. This format is a simple way to describe an instance as a measurement vector on a set of predefined features. In the case of supervised learning, a class label is also available. One limitation of the feature-based format is that supplied features sometimes do not adequately describe, in terms of classification, the semantics of the dataset. This happens, for example, when general-purpose features are used to describe a collection that contains certain relations between individuals.

In order to obtain good results in classification tasks, many algorithms and preprocessing techniques (e.g., SVM (Cortes and Vapnik, 1995), PCA (Dunteman, 1989) etc.) deal with non-adequate variables by internally changing the description space. The main drawback of these approaches is that they function as a black box, where the new representation space is either hidden (for SVM) or completely synthetic and incomprehensible to human readers (PCA).

The purpose of our work is to construct a new feature set that is more descriptive for both supervised and unsupervised classification tasks. In the same way that frequent itemsets (Piatetsky-Shapiro, 1991) help users to understand the patterns in transactions, our goal with the new features is to help understand relations between individuals of datasets. Therefore, the new features should be easily comprehensible by a human reader. Literature proposes algorithms that construct features based on the original user-supplied features (called primitives). However, to our knowledge, all of these algorithms construct the feature set in a supervised way, based on the class information, supplied a priori with the data.

In order to construct new features, we propose two algorithms that create new feature sets in the absence of classified examples, in an unsupervised manner. The first algorithm is an adaptation of an established supervised algorithm, making it unsupervised. For the second algorithm, we have developed a completely new heuristic that selects, at each iteration, pairs of highly correlated features and replaces them with conjunctions of literals that do not co-occur. Therefore, the overall redundancy of the feature set is reduced. Later iterations create more complex Boolean formulas, which can contain negations (meaning absence of features). We use statistical considerations (hypothesis testing) to automatically determine the value of parameters depending on the dataset, and a Pareto front (Sawaragi et al, 1985)-inspired method for the evaluation. The main advantage of the proposed methods over PCA or the kernel of the SVM is that the newly-created features are comprehensible to human readers (features like and are easily interpretable).

In Sections 2 and 3, we present our proposed algorithms and in Section 4 we describe the evaluation metrics and the complexity measures. In Section 5, we perform a set of initial experiments and outline some of the inconveniences of the algorithms. In Section 6, by use of statistical hypothesis testing, we address these weak points, notably the choice of the threshold parameter. In Section 7, a second set of experiments validates the proposed improvements. Finally, Section 8 draws the conclusion and outlines future works.

1.1 Motivation: why construct a new feature set

(a)
(b)
(c)
(d)
(e)
(f)
Figure 1: Example of images tagged with , , ,

In the context of classification (supervised or unsupervised), a useful feature needs to portray new information. A feature , that is highly correlated with another feature , does not bring any new information, since the value of can be deduced from that of . Subsequently, one could filter out “irrelevant” features before applying the classification algorithm. But by simply removing certain features, one runs the risk of losing important information of the hidden structure of the feature set, and this is the reason why we perform feature construction. Feature construction attempts to increase the expressive power of the original features by discovering missing information about relationships between features.

We deal primarily with datasets described with Boolean features. Any dataset described by using the feature-value format can be converted to a binary format using discretization and binarization. In real-life datasets, most binary features have specific meanings. Let us consider the example of a set of images that are tagged using Boolean features. Each feature marks the presence (true) or the absence (false) of a certain object in the image. These objects could include: , , , , or . In this case, part of the semantic structure of the feature set can be guessed quite easily. Relations like “is-a” and “part-of” are fairly intuitive: is a sort of , is part of etc. But other relations might be induced by the semantics of the dataset (images in our example). will co-occur with , for they usually take place in the city. Fig. 1 depicts a simple image dataset described using the feature set , , , . The feature set is quite redundant and some of the features are non-informative (e.g., feature is present for all individuals). Considering co-occurrences between features, we could create the more eloquent features (describing the top row) and (describing the bottom row).

The idea is to create a data-dependent feature set, so that the new features are as independent as possible, limiting co-occurrences between the new features. At the same time they should be comprehensible to the human reader.

1.2 Related work

The literature proposes methods for augmenting the descriptive power of features. Liu and Motoda (1998) collects some of them and divides them into three categories: feature selection, feature extraction and feature construction.

Feature selection (Lallich and Rakotomalala, 2000; Mo and Huang, 2011) seeks to filter the original feature set in order to remove redundant features. This results in a representation space of lower dimensionality. Feature extraction is a process that extracts a set of new features from the original features through functional mapping (Motoda and Liu, 2002). For example, the SVM algorithm (Cortes and Vapnik, 1995) constructs a kernel function that changes the description space into a new separable one. Supervised and non-supervised algorithms can be boosted by pre-processing with principal component analysis (PCA) (Dunteman, 1989). PCA is a mathematical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of uncorrelated variables, called principal components. Manifold learning (Huo et al, 2005) can be seen as a classification approach where the representation space is changed internally in order to boost the performances. Feature extraction mainly seeks to reduce the description space and redundancy between features. Newly-created features are rarely comprehensible and very difficult to interpret. Both feature selection and feature extraction are inadequate for detecting relations between the original features.

Feature Construction is a process that discovers missing information about the relationships between features and augments the space of features by inferring or creating additional features (Motoda and Liu, 2002). This usually results in a representation space with a larger dimension than the original space. Constructive induction (Michalski, 1983) is a process of constructing new features using two intertwined searches (Bloedorn and Michalski, 1998): one in the representation space (modifying the feature set) and another in the hypothesis space (using classical learning methods). The actual feature construction is done using a set of constructing operators and the resulted features are often conjunctions of primitives, therefore easily comprehensible to a human reader. Feature construction has mainly been used with decision tree learning. New features served as hypotheses and were used as discriminators in decision trees. Supervised feature construction can also be applied in other domains, like decision rule learning (Zheng, 1995).

0:   – set of primitive user-given features
0:   – the data expressed using which will be used to construct features Inner parameters: – set of operators for constructing features, – machine learning algorithm to be employed
0:   – set of new (constructed and/or primitives) features.
  
  
  repeat
     
      convert()
      Run M()
      new feat. constructed with Op
     prune useless features in
  until  stopping criteria are met.
Algorithm 1 General feature construction schema.

Algorithm 1, presented in Gomez and Morales (2002); Yang et al (1991), represents the general schema followed by most constructive induction algorithms. The general idea is to start from , the dataset described with the set of primitive features. Using a set of constructors and the results of a machine learning algorithm, the algorithm constructs new features that are added to the feature set. In the end, useless features are pruned. These steps are iterated until some stopping criterion is met (e.g., a maximum number of iterations performed or a maximum number of created features).

Most constructive induction systems construct features as conjunctions or disjunctions of literals. Literals are the features or their negations. E.g., for the feature set the literal set is . Operator sets and are both complete sets for the Boolean space. Any Boolean function can be created using only operators from one set. FRINGE (Pagallo and Haussler, 1990) creates new features using a decision tree that it builds at each iteration. New features are conjunctions of the last two nodes in each positive path (a positive path connects the root with a leaf having the class label true). The newly-created features are added to the feature set and then used in the next iteration to construct the decision tree. This first algorithm of feature construction was initially designed to solve replication problems in decision trees.

Other algorithms have further improved this approach. CITRE (Matheus, 1990) adds other search strategies like root (selects first two nodes in a positive path) or root-fringe (selects the first and last node in the path). It also introduces domain-knowledge by applying filters to prune the constructed features. CAT (Zheng, 1998) is another example of a hypothesis-driven constructive algorithm similar to FRINGE. It also constructs conjunctive features based on the output of decision trees. It uses a dynamic-path based approach (the conditions used to generate new features are chosen dynamically) and it includes a pruning technique.

There are alternative representations, other than conjunctive and disjunctive. The M-of-N and X-of-N representations use feature-value pairs. An feature-value pair is true for an instance if and only if the feature has the value for that instance. The difference between M-of-N and X-of-N is that, while the second one counts the number of true feature-value pairs, the first one uses a threshold parameter to assign a value of truth for the entire representation. The algorithm  (Murphy and Pazzani, 1991) uses M-of-N representations for the newly-created features. It has a specialization and a generalization construction operator and it does not need to construct a new decision tree at each step, but instead integrates the feature construction in the decision tree construction. The  (Zheng, 1995) algorithm functions similarly, except that it uses the X-of-N representation. It also takes into account the complexity of the features generated.

Comparative studies like Zheng (1996) show that conjunctive and disjunctive representations have very similar performances in terms of prediction accuracy and theoretical complexity. M-of-N, while more complex, has a stronger representation power than the two before. The X-of-N representation has the strongest representation power, but the same studies show that it suffers from data fragmenting more than the other three.

The problem with all of these algorithms is that they all work in a supervised environment and they cannot function without a class label. In the following sections, we will propose two approaches towards unsupervised feature construction.

2 uFRINGE - adapting FRINGE for unsupervised learning

We propose uFRINGE, an unsupervised version of FRINGE, one of the first feature construction algorithms. FRINGE (Pagallo and Haussler, 1990) is a framework algorithm (see Section 1.2), following the same general schema shown in Algorithm 1. It creates new features using a logical decision tree, created using a traditional algorithm like ID3 (Quinlan, 1986) or C4.5 (Quinlan, 1993). Taking a closer look at FRINGE, one would observe that its only component that is supervised is the decision tree construction. The actual construction of features is independent of the existence of a class attribute. Hence, using an unsupervised decision tree construction algorithm renders FRINGE unsupervised.

Clustering trees (Blockeel et al, 1998) were introduced as generalized logical decision trees. They are constructed using a top-down strategy. At each step, the cluster under a node is split into two, seeking to maximize the intra-cluster variance. The authors argue that supervised indicators, used in traditional decision trees algorithms, are special cases of intra-cluster variance, as they measure intra-cluster class diversity. Following this interpretation, clustering trees can be considered generalizations of decision trees and are suitable candidates for replacing ID3 in uFRINGE.

Adapting FRINGE to use clustering trees is straightforward: it is enough to replace M in Algorithm 1 with the clustering trees algorithm. At each step, uFRINGE constructs a clustering tree using the dataset and the current feature set. Just like in FRINGE, new features are created using the conditions under the last two nodes in each path connecting the root to a leaf. FRINGE constructs new features starting only from positive leaves (leaves labelled true). But unlike decision trees, in classification trees the leaves are not labelled using class features. Therefore, uFRINGE constructs new features based on all paths from root to a leaf.

Newly-constructed features are added to the feature set and used in the next classification tree construction. The algorithm stops when either no more features can be constructed from the clustering tree or when a maximum allowed number of features have already been constructed.

Limitations. uFRINGE is capable of constructing new features in an unsupervised context. It is also relatively simple to understand and implement, as it is based on the same framework as FRINGE. However, it suffers from a couple of drawbacks. Constructed features tend to be redundant and contain doubles. Newly-constructed features are added to the feature set and are used, alongside old features, in later iterations. Older features are never removed from the feature set and they can be combined multiple times, thus resulting in doubles in the constructed feature set. What is more, old features can be combined with new features in which they already participated, therefore constructing redundant features (e.g., and resulting in ). Another limitation is controlling the number of constructed features. The algorithm stops when a maximum number of features are constructed. This is very inconvenient, as the dimension of the new feature set cannot be known in advance and is highly dependent on the dataset. Furthermore, constructing too many features leads to overfitting and an overly complex feature set. These shortcomings could be corrected by refining the constructing operator and by introducing a filter operator.

3 uFC - a greedy heuristic

We address the limitations of uFRINGE by proposing a second, innovative approach. We propose an iterative algorithm that reduces the overall correlation of features of a dataset by iteratively replacing pairs of highly correlated features with conjunctions of literals. We use a greedy search strategy to identify the features that are highly correlated, then use a construction operator to create new features. From two correlated features and we create three new features: , and . In the end, both and are removed from the feature set. The algorithm stops when no more new features are created or when it has performed a maximum number of iterations. The formalization and the different key parts of the algorithm (e.g., the search strategy,construction operators or feature pruning) will be presented in the next sections.

Fig. 2 illustrates visually, using Venn diagrams, how the algorithm replaces the old features with new ones. Features are represented as rectangles, where the rectangle for each feature contains the individuals having that feature set to true. Naturally, the individuals in the intersection of two rectangles have both features set to true. and have a big intersection, showing that they co-occur frequently. On the contrary, and have a small intersection, suggesting that their co-occurrence is less than that of the hazard (negatively correlated). is included in the intersection of and , while has no common elements with any other. is incompatible with all of the others.

In the first iteration, and are combined and 3 features are created: , and . These new features will replace and , the original ones. At the second iteration, is combined with . As is contained in , the feature will have a support equal to zero and will be removed. Note that and are never combined, as they are considered uncorrelated. The final feature set will be

(a)
(b)
(c)
Figure 2: Graphical representation of how new features are constructed - Venn diagrams. (a) Iter. 0: Initial features (Primitives), (b) Iter. 1: Combining and and (c) Iter. 2: Combining and

3.1 uFC - the proposed algorithm

We define the set of user-supplied initial features and the dataset described using . We start from the hypothesis that even if the primitive set cannot adequately describe the dataset , there is a data-specific feature set that can be created in order to represent the data better. New features are iteratively created using conjunctions of primitive features or their negations (as seen in Fig. 2). Our algorithm does not use the output of a learning algorithm in order to create the new features. Instead we use a greedy search strategy and a feature set evaluation function that can determine if a newly-obtained feature set is more appropriate than the former one.

0:   – set of primitive user-given features
0:   – the data expressed using which will be used to construct features
  Inner parameters: – correlation threshold for searching, – max no of iterations.
   – set of newly-constructed features.
  
  
  repeat
     
      search_correlated_pairs()
     
     while   do
         highest_scoring_pair()
         construct_new_feat()
        remove_candidate()
     end while
     prune_obsolete_features()
      convert()
  until  OR
  
Algorithm 2 uFC - Unsupervised feature construction

The schema of our proposal is presented in Algorithm 2. The feature construction is performed starting from the dataset and the primitives . The algorithm follows the general inductive schema presented in Algorithm 1. At each iteration, uFC searches for frequently co-occurring pairs in the feature set created at the previous iteration (). It determines the candidate set and then creates new features as conjunctions of the highest scoring pairs. The new features are added to the current set (), after which the set is filtered in order to remove obsolete features. At the end of each iteration, the dataset is translated to reflect the feature set . A new iteration is performed as long as new features were generated in the current iteration and a maximum number of iterations have not yet been reached ( is a parameter for the algorithm).

3.2 Searching co-occurring pairs

The search_correlated_pairs function searches for frequently co-occurring pairs of features in a feature set . We start with an empty set and we investigate all possible pairs of features . We use a function () to measure the co-occurrence of a pair of features and compare it to a threshold . If the value of the function is above the threshold, then their co-occurrence is considered as significant and the pair is added to . Therefore, will be

The function is the empirical Pearson correlation coefficient, which is a measure of the strength of the linear dependency between two variables. and it is defined as the covariance of the two variables divided by the product of their standard deviations. The sign of the function gives the direction of the correlation (inverse correlation for and positive correlation for ), while the absolute value or the square gives the strength of the correlation. A value of 0 implies that there is no linear correlation between the variables. When applied to Boolean variables, having the contingency table as shown in Table 1, the function has the following formulation:

a b
c d
Table 1: Contingency table for two Boolean features

The threshold parameter will serve to fine-tune the number of selected pairs. Its impact on the behaviour of the algorithm will be studied in Section 5.3. A method of automatic choice of using statistical hypothesis testing is presented in Section 6.1.

3.3 Constructing and pruning features

Once is constructed, uFC performs a greedy search. The function highest_scoring_pair is iteratively used to extract from the pair that has the highest co-occurrence score.

The function construct_new_feat constructs three new features: , and . They represent, respectively, the intersection of the initial two features and the relative complements of one feature in the other. The new features are guaranteed by construction to be negatively correlated. If one of them is set to true for an individual, the other two will surely be false. At each iteration, very simple features are constructed: conjunctions of two literals. The creation of more complex and semantically rich features appears through the iterative process. and can be either primitives or features constructed in previous iterations.

After the construction of features, the remove_candidate function removes from the pair , as well as any other pair that contains of . When there are no more pairs in , prune_obsolete_features is used to remove from the feature set two types of features:

  • features that are false for all individuals. These usually appear in the case of hierarchical relations. We consider that and have a hierarchical relation if all individuals that have feature true, automatically have feature true (e.g., “is a type of” or “is a part of” ). One of the generated features (in the example ) is false for all individuals and, therefore, eliminated. In the example of and , we create only and , since there cannot exist a cascade without water (considering that a value of false means the absence of a feature and not missing data).

  • features that participated in the creation of a new feature. Effectively, all are replaced by the newly-constructed features.

4 Evaluation of a feature set

To our knowledge, there are no widely accepted measures to evaluate the overall correlation between the features of a feature set. We propose a measure inspired from the “inclusion-exclusion” principle (Feller, 1950). In set theory, this principle permits to express the cardinality of the finite reunion of finite ensembles by considering the cardinality of those ensembles and their intersections. In the Boolean form, it is used to estimate the probability of a clause (disjunction of literals) as a function of its composing terms (conjunctions of literals).

Given the feature set , we have:

which, by putting apart the first term, is equivalent to:

Without loss of generality, we can consider that each individual has at least one feature set to true. Otherwise, we can create an artificial feature “null” that is set to true when all the others are false. Consequently, the left side of the equation is equal to 1. On the right side, the second term is the probability of intersections of the features. Knowing that , this probability of intersection has a value of zero when all features are incompatible (no overlapping). It has a “worst case scenario” value of , when all individuals have all the features set to true.

Based on these observations, we propose the Overlapping Index evaluation measure:

where and “better” towards zero. Hence, a feature set describes a dataset better than another feature set when .

4.1 Complexity of the feature set.

Number of features Considering the case of the majority of machine learning datasets, where the number of primitives is inferior to the number of individuals in the dataset, reducing correlations between features comes at the expense of increasing the number of features. Consider the pair of features judged correlated. Unless or , the algorithm will replace by , thus increasing the total number of features. A feature set that contains too many features is no longer informative, nor comprehensible. The maximum number of features that can be constructed is mechanically limited by the number of unique combinations of primitives in the dataset (the number of unique individuals).

where is the constructed feature set and is the dataset.

To measure the complexity in terms of number of features, we use:

where is the primitive feature set. measures the ratio between how many extra features are constructed and the maximum number of features that can be constructed. and a value closer to means a feature set less complex.

The average length of features At each iteration, simple conjunctions of two literals are constructed. Complex Boolean formulas are created by combining features constructed in previous iterations. Long and complicated expressions generate incomprehensible features, which are more likely a random side-effect rather than a product of underlying semantics.

We define as the average number of literals (a primitive or its negation) that appear in a Boolean formula representing a new feature.

where is the primitive set and .

As more iterations are performed, the feature set contains more features ( grows) which are increasingly more complex ( grows). This suggests a correlation between the two. What is more, since can potentially double at each iteration and can have at most a linear increase, the correlation is exponential. For this reason, in the following sections we shall use only as the complexity measure.

Overfitting All algorithms that learn from data risk overfitting the solution to the learning set. There are two ways in which uFC can overfit the resulted feature set, corresponding to the two complexity measures above: a) constructing too many features (measure ) and b) constructing features that are too long (measure ). The worst overfitting of type a) is when the algorithm constructs as many features as the maximum theoretical number (one for each individual in the dataset). The worst overfitting of type b) appears in the same conditions, where each constructed feature is a conjunction of all the primitives appearing for the corresponding individual. The two complexity measures can be used to quantify the two types of ovefitting. Since and are correlated, both types of overfitting appear simultaneously and can be considered as two sides of a single phenomenon.

4.2 The trade-off between two opposing criteria

is a measure of how overfitted a feature set is. In order to avoid overfitting, feature set complexity should be kept at low values, while the algorithm optimizes the co-occurrence score of the feature set. Optimizing both the correlation score and the complexity at the same time is not possible, as they are opposing criteria. A compromise between the two must be achieved. This is equivalent to the optimization of two contrary criteria, which is a very well-known problem in multi-objective optimization. To acquire a trade-off between the two mutually contradicting objectives, we use the concept of Pareto optimality (Sawaragi et al, 1985), originally developed in economics. Given multiple feature sets, a set is considered to be Pareto optimal if there is no other set that has both a better correlation score and a better complexity for a given dataset. Pareto optimal feature sets will form the Pareto front. This means that no single optimum can be constructed, but rather a class of optima, depending on the ratio between the two criteria.

We plot the solutions in the plane defined by the complexity, as one axis, and the co-occurrence score, as the other. Constructing the Pareto front in this plane makes a visual evaluation of several characteristics of the uFC algorithm possible, based on the deviation of solutions compared to the front. The distance between the different solutions and the constructed Pareto front visually shows how stable the algorithm is. The convergence of the algorithm can be visually evaluated by how fast (in number of performed iterations) the algorithm transits the plane from the region of solutions with low complexity and high co-occurrence score to solutions with high complexity and low co-occurrence. We can visually evaluate overfitting, which corresponds to the region of the plane with high complexity and low co-occurence score. Solutions found in this region are overfitted.

In order to avoid overfitting, we propose the “closest-point” heuristic for finding a compromise between and . We consider the two criteria to have equal importance. We consider as a good compromise, the solution in which the gain in co-occurrence score and the loss in complexity are fairly equal. If one of the indicators has a value considerably larger than the other, the solution is considered to be unsuitable. Such solutions would have either a high correlation between features or a high complexity. Therefore, we perform a battery of tests and we search a posteriori the Pareto front for solutions for which the two indicators have essentially equal values. In the space of solutions, this translates into a minimal Euclidian distance between the solution and the ideal point (the point ).

5 Initial Experiments

(a)
(b)
(c)
Figure 3: Images related to the newly-constructed feature on hungarian: (a) Hungarian puszta, (b)(c) Hungarian Mátra mountains

Throughout the experiments, uFC was executed by varying only the two parameters: and . We denote an execution with specific values for parameters as uFC, whereas the execution where the parameters were determined a posteriori using the “closest-point” strategy will be noted uFC*. For uFRINGE, the maximum number of features was set at 300. We perform a comparative evaluation of the two algorithms seen from a qualitative and quantitative point of view, together with examples of typical executions. Finally, we study the impact of the two parameters of uFC.

Experiments were performed on three Boolean datasets. The hungarian dataset111http://eric.univ-lyon2.fr/~arizoiu/files/hungarian.txt is a real-life collection of images, depicting Hungarian urban and countryside settings. Images were manually tagged using one or more of the 13 tags. Each tag represents an object that appears in the image (eg. tree, cascade etc.). The tags serve as features and a feature takes the value true if the corresponding object is present in the image or false otherwise. The resulted dataset contains 264 individuals, described by 13 Boolean features. Once the dataset was constructed, the images were not used any more. The street dataset222http://eric.univ-lyon2.fr/~arizoiu/files/street.txt was constructed in a similar way, starting from images taken from the LabelMe dataset (Russell et al, 2008). 608 urban images from Barcelona, Madrid and Boston were selected. Image labels were transformed into tags depicting objects by using the uniformization list provided with the toolbox. The dataset contains 608 individuals, described by 66 Boolean features.

The third dataset is “Spect Heart”333http://archive.ics.uci.edu/ml/datasets/SPECT+Heart from the UCI. The corpus is provided with a “class” attribute and divided into a learning corpus and a testing one. We eliminated the class attribute and concatenated the learning and testing corpus into a single dataset. It contains 267 instances described by 22 Boolean features. Unlike the first two datasets, the features of spect have no specific meaning, being called “F1”, “F2”, … , “F22”.

5.1 uFC and uFRINGE: Qualitative evaluation

(a)
(b)
(c)
Figure 4: Images related to the newly-constructed feature on hungarian

For the human reader, it is quite obvious why and have the tendency to appear together or why and have the tendency to appear separately. One would expect, that based on a given dataset, the algorithms would succeed in making these associations and catching the underlying semantics. Table 2 shows the features constructed with uFRINGE and uFC* on hungarian. A quick overview shows that constructed features manage to make associations that seem “logical” to a human reader. For example, one would expect the feature to denote images where there is a panoramic view and the sky, but no buildings, therefore suggesting images outside the city. Fig. 3 supports this expectation. Similarly, the feature covers urban images, where groups of people are present and denotes a cascade in the forest (Fig. 4).

(a)
(b)
(c)
Figure 5: Images related to the newly-constructed feature on street

Comprehension quickly deteriorates when the constructed feature set is overfitted, when the constructed features are too complex. The execution of uFC reveals features like:

Even if the formula is not in the Disjunctive Normal Form (DNF), it is obvious that it is too complex to make any sense. If uFC tends to construct overly complex features, uFRINGE suffers from another type of dimensionality curse. Even if the complexity of features does not impede comprehension, the fact that there are over 300 hundred features constructed from 13 primitives makes the newly-constructed feature set unusable. The number of features is actually greater than the number of individuals in the dataset, which proves that some of the features are redundant. The actual correlation score of the newly-created feature set is even greater than the initial primitive set. What is more, new features present redundancy, just as predicted in section 2. For example, the feature which contains two times the primitive .

The same conclusions are drawn from execution on the street dataset. uFC* creates comprehensible features. For example (Fig. 5) suggests images in which the front part of cars appear. It is especially interesting how the algorithm specifies in conjunction with in order to differenciate between people () and objects that have heads (but no arms).

primitives uFRINGE uFC(0.194, 2)
... and 284 others
Table 2: Feature sets constructed by uFC and uFRINGE

5.2 uFC and uFRINGE: Quantitative evaluation

Strategy

hungar.

Primitives 13 1.00 0.24 0.00
uFC*(0.194, 2) 21 2.95 0.08 0.07
uFC(0.184, 5) 36 11.19 0.03 0.20
uFRINGE 306 3.10 0.24 2.53

street

Primitives 66 1.00 0.12 0.00
uFC*(0.446, 3) 81 2.14 0.06 0.04
uFC(0.180, 5) 205 18.05 0.02 0.35
uFRINGE 233 2.08 0.20 0.42

spect

Primitives 22 1.00 0.28 0.00
uFC*(0.432, 3) 36 2.83 0.09 0.07
uFC(0.218, 4) 62 8.81 0.03 0.20
uFRINGE 307 2.90 0.25 1.45
Table 3: Values of indicators for multiple runs on each dataset

Table 3 shows, for the three datasets, the values of certain indicators, like the size of the feature set, the average length of a feature (), the and indicators. For each dataset, we compare four feature sets: the initial feature set (primitives), the execution of uFC* (parameters determined by the “closest-point” heuristic), uFC with another random set of parameters and uFRINGE. For the hungarian and street datasets, the same parameter combinations are used as in the qualitative evaluation.

On all three datasets, uFC* creates feature sets that are less correlated than the primitive sets, while the increase in complexity is only marginal. Very few (2-3) iterations are needed, as uFC converges very fast. Increasing the number of iterations has very little impact on , but results in very complex vocabularies (large and feature lengths). In the feature set created by uFC on street, on average, each feature contains more than 18 literals. This is obviously too much for human comprehension.

For uFRINGE, the indicator shows very marginal or no improvement on spect and hungarian datasets, and even a degradation on street (compared to the primitive set). Features constructed using this approach have an average length between 2.08 and 3.1 literals, just as much as the selected uFC* configuration. But, it constructs between 2.6 and 13.9 times more features than uFC*. We consider this to be due to the lack of filtering in uFRINGE, which would also explain the low score. Old features remain in the feature set and amplify the total correlation by adding the correlation between old and new features.

5.3 Impact of parameters and

In order to understand the impact of parameters, we executed uFC with a wide range of values for and and studied the evolution of the indicators and . For each dataset, we varied between and with a step of . For each value of , we executed uFC by varying between and for the hungarian dataset, and between and for street and spect. We study the evolution of the indicators as a function of , respectively , we plot the solution in the () space and construct the Pareto front.

(a)
(b)
Figure 6: Variation of indicator with on hungarian (a) and with on street (b)

For the study of , we hold fixed at various values and we vary only . The evolution of the correlation indicator is given in Fig. (a)a. As expected, the measure ameliorates with the number of iterations. has a very rapid descent and needs less than 10 iterations to converge on all datasets towards a value dependent on . The higher the value of , the higher the value of convergence. The complexity has a very similar evolution, but in the inverse direction: it increases with the number of iterations performed. It also converges towards a value that is dependent on : the higher the value of , the lower the complexity of the resulting feature set.

Similarly, we study by fixing . Fig. (b)b shows how evolves when varying . As foreseen, for all values of , the indicator increases with , while decreases with . shows an abrupt increase between 0.2 and 0.3, for all datasets. For lower values of , many pairs get combined as their correlation score is bigger than the threshold. As increases, only highly correlated pairs get selected and this usually happens in the first iterations. Performing more iterations does not bring any change and indicators are less dependent on . For hungarian, no pair has a correlation score higher than 0.4. Setting higher than this value causes uFC to output the primitive set (no features are created).

(a)
(b)
Figure 7: The distribution of solutions, the Pareto front and the closest-point on spect dataset

To study Pareto optimality, we plot the generated solutions in the (, ) space. Fig. (a)a presents the distribution of solutions, the Pareto front and the solution chosen by the “closest-point” heuristic. The solutions generated by uFC with a wide range of parameter values are not dispersed in the solution space, but their distribution is rather close together. This shows good algorithm stability. Even if not all the solutions are Pareto optimal, none of them are too distant from the front and there are no outliers.

Most of the solutions densely populate the part of the curve corresponding to low and high . As pointed out in the Section 4.2, the area of the front corresponding to high feature set complexity (high ) represents the overfitting area. This confirms that the algorithm converges fast, then enters overfitting. Most of the improvement in quality is done in the first 2-3 iterations, while further iterating improves quality only marginally with the cost of an explosion of complexity. The “closest-point” heuristic keeps the constructing out of overfitting, by stopping the algorithm at the point where the gain of co-occurence score and the loss in complexity are fairly equal. Fig. (b)b magnifies the region of the solution space corresponding for low numbers of iterations.

5.4 Relation between number of features and feature length

Both the average length of a feature () and the number of features () increase with the number of iterations. In Section 4.1 we have speculated that the two are correlated: . For each in the batch of tests, we create the and series depending on the and we perform a statistical hypothesis test, using the Kendall rank coefficient as the test statistic. The Kendall rank coefficient is particularly useful as it makes no assumptions about the distributions of and . For all values of , for all datasets, the statistical test revealed a p-value of the order of . This is consistently lower than habitually used significance levels and makes us reject the null independence hypothesis and conclude that and are statistically dependent.

6 Improving the uFC algorithm

The major difficulty of uFC, shown by the initial experiments, is setting the values of parameters. An unfortunate choice would result in either an overly complex feature set or a feature set where features are still correlated. But both parameters and are dependent on the dataset and finding the suitable values would prove to be a process of trial and error for each new corpus. The “closest-point” heuristic achieves acceptable equilibrium between complexity and performance, but requires multiple executions with large choices of values for parameters and the construction of the Pareto front, which might not always be desirable or even possible.

We propose a new method for choosing based on statistical hypothesis testing and a new stopping criterion inspired from the “closest-point” heuristic. These will be integrated into a new “risk-based” heuristic that approximates the best solution while avoiding the time consuming construction of multiple solutions and the Pareto front. The only parameter is the significance level , which is independent of the dataset, and makes the task of running uFC on new, unseen datasets easy. A pruning technique is also proposed.

6.1 Automatic choice of

We propose replacing the user-supplied co-occurrence threshold with a technique that selects only pairs of features for whom the positive linear correlation is statistically significant. These pairs are added to the set of co-occurring pairs (defined in Section 3.2) and, starting from , new features are constructed. We use a statistical method: the hypothesis testing. For each pair of candidate features, we test the independence hypothesis against the positive correlation hypothesis .

We use as a test statistic the Pearson correlation coefficient (calculated as defined in Section 3.2) and test the following formally defined hypothesis: and , where is the theoretical correlation coefficient between two candidate features. We can show that in the case of Boolean variables, having the contingency table shown in Table 1, the observed value of the of independence is ( is the size of the dataset). Consequently, considering true the hypothesis , is approximately following a distribution with one degree of freedom (), resulting in following a standard normal distribution (), given that is large enough.

We reject the hypothesis in favour of if and only if , where is the right critical value for the standard normal distribution. Two features will be considered significantly correlated when . The significance level represents the risk of rejecting the independence hypothesis when it was in fact true. It can be interpreted as the false discovery risk in data mining. In the context of feature construction it is the false construction risk, since this is the risk of constructing new features based on a pair of features that are not really correlated. Statistical literature usually sets at or , but levels of or even are often used.

The proposed method repeats the independence test a great number of times, which inflates the number of type I errors. Ge et al (2003) presents several methods for controlling the false discoveries. Setting aside the Bonferroni correction, often considered too simplistic and too drastic, one has the option of using sequential rejection methods (Benjamini and Liu, 1999; Holm, 1979), the q-value method of Storey (Storey, 2002) or making use of bootstrap (Lallich et al, 2006). In our case, applying these methods is not clear-cut, as tests performed at each iteration depend on the results of the tests performed at previous iterations. It is noteworthy that a trade-off must be acquired between the inflation of false discoveries and the inflation of missed discoveries. This makes us choose a risk between and , where is the theoretical number of tests to be performed.

6.2 Candidate pruning technique. Stopping criterion.

Pruning In order to apply the independence test, it is necessary that the expected frequencies considering true the hypothesis be greater or equal than 5. We add this constraint to the new feature search strategy (subsection 3.2). Pairs for whom the values of , , and are not greater than 5, will be filtered from the set of candidate pairs . This will impede the algorithm from constructing features that are present for very few individuals in the dataset.

Risk-based heuristic We introduced in Section 4.2 the “closest-point” for choosing the values for parameters and . It searches the solution on the Pareto front for which the indicators are sensibly equal. We transform the heuristic into a stopping criterion: and are combined into a single formula, the root mean square (RMS). The algorithm will stop iterating when RMS has reached a minimum. Using the generalized mean inequality, we can prove that has only one global minimum, as with each iteration the complexity increases and descends.

The parameter, which is data-dependent, is replaced by the automatic RMS stopping criterion. This stopping criterion together with the automatic choice strategy, presented in Section 6.1, form a data-independent heuristic for choosing parameters. We will call the new heuristic risk-based heuristic. This new heuristic will make it possible to approximate the best parameter compromise and avoid the time consuming task of computing a batch of solutions and constructing the Pareto front.

7 Further Experiments

We test the proposed ameliorations, similarly to what was shown in Section 5, on the same three datasets: hungarian, spect and street. We execute uFC in two ways: the classical uFC (Section 3) and the improved uFC (Section 6). The classical uFC needs to have parameters and set (noted uFC). uFC* denotes the execution with parameters which were determined a posteriori using the “closest-point” heuristic. The improved uFC will be denoted as uFC(). The “risk-based” heuristic will be used to determine the parameters and control the execution.

7.1 Risk-based heuristic for choosing parameters

Root Mean Square In the first batch of experiments, we study the variation of the Root Means Square aggregation function for a series of selected values of . We vary between 0 and 30, for hungarian, and between 0 and 20 for spect and street. The evolution of RMS is presented in Fig. 8.

Figure 8: RMS vs. on spect

For all the RMS starts by decreasing, as descends more rapidly than the increases. In just 1-3 iterations, RMS reaches its minimum and afterwards its value starts to increase. This is due to the fact that complexity increases rapidly, with only marginal improvement of quality. This behaviour is consistent with the results presented in Section 5. As already discussed in Section 5.3, has a bounding effect over complexity, thus explaining why RMS reaches a maximum for higher values of .

Figure 9: “Closest-point” and “risk-based” for multiple on hungarian

The “risk-based” heuristic The second batch of experiments deals with comparing the “risk-based” heuristic to the “closest-point” heuristic. The “closest-point” was determined as described in Section 5. The “risk-based” heuristic was executed multiple times, with values for parameter , , , , , , , , ,

Strategy

hung.

Primitives - - 13 - 1.00 0.235 0.000
uFC*(0.194, 2) 0.194 2 21 19 2.95 0.076 0.069
uFC(0.001) 0.190 2 22 3.18 0.071 0.078

street

Primitives - - 66 - 1.00 0.121 0.000
uFC*(0.446, 3) 0.446 3 87 33 2.14 0.062 0.038
uFC(0.0001) 0.150 1 90 1.84 0.060 0.060

spect

Primitives - - 22 - 1.00 0.279 0.000
uFC*(0.432, 3) 0.432 3 36 19 2.83 0.086 0.071
uFC(0.0001) 0.228 2 39 2.97 0.078 0.086
Table 4: “closest-point” and “risk-based” heuristics

Table 4 gives a quantitative comparison between the two heuristics. A risk of is used for hungarian and for spect and street. The feature sets created by the two approaches are very similar, considering all indicators. Not only the differences between values for , , average feature length and feature set dimension are negligible, but most of the created features are identical. On hungarian, 19 of the 21 features created by the two heuristics are identical. Table 5 shows the two features sets, with non-identical features in bold.

primitives uFC*(0.194, 2) uFC(0.001)