IMMIGRATE: A Margin-based Feature Selection Method with Interaction Terms

Given a distance d(\vec{x}_{i},\vec{x}_{j}) between two instances, \vec{x}_{i} and \vec{x}_{j}, a hypothesis-margin (Gilad-Bachrach et al., 2004) is defined as \rho_{n,h,m}=d(\vec{x}_{n},\vec{x}_{m})-d(\vec{x}_{n},\vec{x}_{h}), where \vec{x}_{h} and \vec{x}_{m} represent the nearest hit and nearest miss for instance \vec{x}_{n}, respectively. We adopt the probabilistic hypothesis-margin defined by Sun and Li (2006) as

where \alpha_{n,h}\geq 0, \beta_{n,m}\geq 0, \sum_{h\in\mathcal{H}_{n}}\alpha_{n,h}=1, \sum_{m\in\mathcal{M}_{n}}\beta_{n,m}=1, for \forall\,n\,\in\,\{1,\cdots,N\}. As in the above design, the hidden random variable \alpha_{n,h} represents the probability that \vec{x}_{h} is the nearest hit of instance \vec{x}_{n}, while \beta_{n,m} indicates the probability that \vec{x}_{m} is the nearest miss of instance \vec{x}_{n}. In the rest of the paper, for conciseness, we will use margin to indicate hypothesis-margin.

The distributions of hits and misses can be used to evaluate the stability of margins (i.e., margin quality). A more stable margin can be obtained by considering the distributions of instances with the same or different labels with respect to the target instance. A margin is deemed stable if it will not be greatly reduced by changes to only a few neighbors of the target instance. Considering an instance \vec{x}_{n}, its probabilities \{\alpha_{n,h}\} and \{\beta_{n,m}\} represent the distributions of its hits and misses, respectively. We can use the hit entropy E_{hit}(\vec{x}_{n})=-\sum_{h\in\mathcal{H}_{n}}\alpha_{n,h}\log\alpha_{n,h} and miss entropy E_{miss}(\vec{x}_{n})=-\sum_{m\in\mathcal{M}_{n}}\beta_{n,m}\log\beta_{n,m} to evaluate the stability of \vec{x}_{n}’s margin. The following two scenarios help explain the intuition of using these entropy. Scenario A: all neighbors are distributed evenly around the target instance; scenario B: the neighbor distribution is highly uneven. An extreme example for scenario B is that one instance is quite close to the target and the rest are quite far away from the target. An easy experiment to test the stability is to discard one instance from the system and to check how it influences the margin. In scenario A, if the closest neighbor (no matter if it is hit or miss) is discarded, the margin changes only slightly because there are many other hits/misses evenly distributed around the target. In scenario B, if the closest neighbor is a miss, its removal can increase the margin significantly. On the contrary, if the closest neighbor is a hit, removing it can decrease the margin significantly. Intuitively speaking, hits prefer scenario A and misses favor scenario B.

Since scenarios A and B correspond to high and low entropy, respectively, the margin can benefit from a large hit entropy E_{hit} (e.g., scenario A) and a low miss entropy E_{miss} (e.g., scenario B). We can set up a framework to maximize the hit entropy and minimize the miss entropy, which is equivalent to make the margin in Equation 3.2 the most stable. Bei and Hong (2015) use the term max-min entropy principle to describe the process that maximizes the hit entropy and minimize the loss entropy to maximize the margin quality. The process of stabilizing margin is an extension of the large margin principle.

We extend the margin in Equation 3.2 by using a new quadratic-Manhattan measurement defined as:

where W is a non-negative symmetric matrix (element-wise non-negative) with its Frobenius norm \|\textbf{W}\|_{F}=1. The quadratic-Manhattan measurement is a natural extension of the weight vector, and the distance defined in Equation 3.3 is a natural extension of the weighted Manhattan distance in Equation 2.1. Off-diagonal elements in W capture feature interactions and diagonal elements in W capture main effects. To understand why quadratic-Manhattan measurement can capture the influence of interactions, we observe that the effect of element w_{a,b} (a\neq b) in W enters into (3.3) as the coefficient for the combination of the a-th and b-th elements in vector \big{|}\vec{x}_{i}-\vec{x}_{j}\big{|}. In Relief-based algorithms, the weighted Manhattan distance Equation 2.1 can be equivalently captured by the feature weight update equation in Algorithm 1. Similarly, w_{a,b} can be updated using the combination of the a-th and b-th features based on a randomly given instance. We thus define our new margin using the quadratic-Manhattan measurement as

We design the following cost function to maximize our new margin, and simultaneously, the hit entropy and miss entropy are optimized.

where E_{miss}(z_{n})=-\sum_{m\in\mathcal{M}_{n}}\beta_{n,m}\log\beta_{n,m}, E_{hit}(z_{n})=-\sum_{h\in\mathcal{H}_{n}}\alpha_{n,h}\log\alpha_{n,h}, and \sigma is a hyperparameter that can be tuned via internal cross-validation.

We also design the following optimization procedure containing two iterative steps to find W that minimizes the cost function. The framework starts from a randomly initialized W and stops when the change of cost function is less than a preset limit or the iteration number reaches a preset threshold. In practice, we find that it typically takes less than 10 iterations to stop and obtain good results. Based on our experiments, different initialization of W does not influence the results of the iterative optimization. The computation time of IMMIGRATE is comparable to other interaction related methods such as SODA (Li and Liu, 2018), hierNet (Bien et al., 2013).

As depicted by the flow-chart in Figure 1, the IMMIGRATE algorithm iteratively optimizes the cost function Equation 3.5. It starts with a random initiation satisfying certain boundary conditions, and proceeds to iterate the two steps as detailed below in Algorithm 2.

Boosting (Schapire, 1990; Freund and Schapire, 1996; Freund and Mason, 1999) has been widely used to create ensemble learners that produce the state-of-the-art results in many tasks. Boosting combines a set of relatively weak base learners to create a much stronger learner. To use IMMIGRATE as the base classifier in the AdaBoost algorithm (Freund and Schapire, 1996), we modify the cost function Equation 3.5 to include sample weights and use the modified version in the boosting iterations. We name the algorithm BIM, standing for Boosted IMMIGRATE (Refer to Equation 3.19 and Algorithm 3 for more details about BIM. BIM schedules the adjustment of the hyperparameter \sigma in its boosting iterations. It starts with \sigma being a predefined \sigma_{max} and gradually reduces \sigma by multiplying it with (\sigma_{min}/\sigma_{max})^{1/T} at each interaction until reaching \sigma_{min}, where T is a predefined maximum number of boosting iterations.

where E_{miss}(z_{n})=-\sum_{m\in\mathcal{M}_{n}}\beta_{n,m}\log\beta_{n,m}, E_{hit}(z_{n})=-\sum_{h\in\mathcal{H}_{n}}\alpha_{n,h}\log\alpha_{n,h}, \sum_{n=1}^{N}D(\vec{x}_{n})=1, and D(\vec{x}_{n})\geq 0,\,\,\,\forall\,\,n.

When applied to high-dimensional data, IMMIGRATE can incur a high computational cost because it considers the interactions between every feature pair. To reduce the computational cost, we first use IM4E (Bei and Hong, 2015) to learn a feature weight vector, which is used to initialize the diagonal elements of W in the proposed quadratic-Manhattan measurement. We also use the learned feature weight vector to help pre-screen the features, and keep only those with weights above a preset limit. In the remaining computation, we only model interactions between those chosen features. The discarded features after pre-screening can be added back empirically based on the need of a specific application. We term this procedure IM4E-IMMIGRATE, which is effective and computationally efficient. It can also be boosted (Boosted IM4E-IMMIGRATE) to be stronger.

We first test the robustness of the IMMIGRATE algorithm using a synthesized dataset where we have two interacting features following Gaussian distributions in a binary classification setting. The simulated dataset contains 100 samples from one class governed by a Gaussian distribution with mean (4,2)^{T} and the covariance matrix \left(\begin{array}[]{cc}1&0.5\\ 0.5&1\\ \end{array}\right) and another 100 samples from the other class governed by a Gaussian distribution with mean (6,0)^{T} and the same covariance matrix. In addition, we add noises following a Gaussian distribution with mean (8,-2)^{T} and the covariance matrix \left(\begin{array}[]{cc}8&4\\ 4&8\\ \end{array}\right) to the fist class, and add noises following a Gaussian distribution with mean (2,4)^{T} and the same covariance matrix to the second class. Figure 2 shows a scatter plot of the synthesized dataset containing 10% samples from the noise distributions. The slope of the orange dotted line in Figure 2 is 1, which separates data with different labels.

The noises are included to disturb the detection of the interaction term. The noise level starts from 5%, and gradually increases by 5% to 50%. As the baseline, we apply logistic regression and observe that the t-test p-value of the interaction coefficient increases from 3\times 10^{-11} to 7\times 10^{-5} and 0.7 when the noise level increases from 0% to 10% and 50%. Local Feature Extraction (LFE, Sun and Wu (2008)) is a Relief-based algorithm which considers interaction terms indirectly, though the interaction information is only used for feature extraction. We run IMMIGRATE and LFE on the synthesized datasets and compare the weights of the interaction term between features 1 and 2 in Figure 3, which shows IMMIGRATE is more robust than LFE.

We compare IMMIGRATE with several existing popular methods using real datasets from the UCI database. The following algorithms are considered in the comparison: Support Vector Machine (Soentpiet, 1999) with Sigmoid Kernel (SV1), Support Vector Machine with Radial basis function Kernel (SV2), LASSO (LAS) (Tibshirani, 1996), Decision Tree (DT) (Freund and Mason, 1999), Naive Bayes Classifier (NBC) (John and Langley, 1995), Radial basis function Network (RBF) (Haykin, 1994), 1-Nearest Neighbor (1NN) (Aha et al., 1991), 3-Nearest Neighbor (3NN), Large Margin Nearest Neighbor (LMN) (Weinberger and Saul, 2009), Relief (REL) (Kira and Rendell, 1992), ReliefF (RFF) (Kononenko, 1994; Robnik-Šikonja and Kononenko, 2003), Simba (SIM) (Gilad-Bachrach et al., 2004), and Linear Discriminant Analysis (LDA) (Fisher, 1936). In addition, several methods designed for detecting interaction terms are included: LFE (Sun and Wu, 2008), Stepwise conditional likelihood variable selection for Discriminant Analysis (SOD) (Li and Liu, 2018), and hierNet (HIN) (Bien et al., 2013). We also include three most widely used and competitive ensemble learners: Adaptive Boosting (ADB) (Freund and Schapire, 1996; Freund and Mason, 1999), Random Forest (RF) (Breiman, 2001), and XgBoost (XGB) (Chen and Guestrin, 2016). We use the following abbreviations when presenting the results: IM4 for IM4E, IGT for IMMIGRATE, and B4G for the boosted IM4E-IMMIGRATE.

Whenever possible, we use the settings of the aforementioned methods reported in their original papers: LMNN uses 3-NN classifier; Relief and Simba use Euclidean distance and 1-NN classifier; ReliefF uses Manhattan distance and k-NN classifier (k=1,3,5 is decided by internal cross-validation); in SODA, gam (=0, 0.5, 1) is determined by internal cross-validation and logistic regression is used for prediction. The IM4E algorithm has two hyperparameters \lambda and \sigma. We fix \lambda=1 as it has no actual contribution and tune \sigma as suggested by Bei and Hong (2015). Hence, the IMMIGRATE algorithm only has one hyperparameter \sigma. When tuning \sigma, we gradually decrease \sigma from \sigma_{0}=4 by half each time until it is not larger than 0.2. The preset limit for weight pruning is 1/A, where A is the number of features. Furthermore, the preset iteration number is chosen to be 10. For each dataset, \sigma and whether weight pruning is applied are determined by the best internal cross-validation results. For BIM, we use \sigma_{max}=4, \sigma_{min}=0.2, and the maximal number of boosting iterations T is 100. The preset threshold in IM4E-IMMIGRATE is 2/A.

We repeat ten-fold cross-validation ten times for each algorithm on each dataset, i.e., 100 trials are carried out. When comparing two algorithms (i.e., A vs. B), we calculate the paired Student’s t-test using the results of 100 trials. First, the null hypothesis is there is no difference between the performances of A and those of B. When the p-value is larger than the significant level cutoff 0.05, we say A "Tie" B, which means there is no significant difference between their performances. When the p-value is smaller than the significant level cutoff 0.05, the second null hypothesis is the performances of B are no worse than those of A. When the new p-value is smaller than the significant level cutoff 0.05, we say A "wins", which means A on average performs significantly better than B on this dataset, and vice versa.