Regularising Non-linear Models Using Feature Side-information

# Regularising Non-linear Models Using Feature Side-information

Amina Mollaysa    Pablo Strasser    Alexandros Kalousis
###### Abstract

Very often features come with their own vectorial descriptions which provide detailed information about their properties. We refer to these vectorial descriptions as feature side-information. In the standard learning scenario, input is represented as a vector of features and the feature side-information is most often ignored or used only for feature selection prior to model fitting. We believe that feature side-information which carries information about features intrinsic property will help improve model prediction if used in a proper way during learning process. In this paper, we propose a framework that allows for the incorporation of the feature side-information during the learning of very general model families to improve the prediction performance. We control the structures of the learned models so that they reflect features’ similarities as these are defined on the basis of the side-information. We perform experiments on a number of benchmark datasets which show significant predictive performance gains, over a number of baselines, as a result of the exploitation of the side-information.

feature side-information, non-linear model

## 1 Introduction

Side-information in machine learning is a very general term used in very different learning scenarios with quite different connotations. Nevertheless, generally it is understood as any type of information, other than the learning instances, which can be used to support the learning process; typically such information will live in a different space than the learning instances. Examples include learning with privileged information (Vapnik & Izmailov, 2015) in which during training a teacher provides additional information for the learning instances; this information is not available in testing. In metric learning and clustering, it has been used to denote the availability of additional similarity information on instances, i.e. pairs of similar and dissimilar instances, (Xing et al., 2002). In this paper we focus on side-information describing the features. We will consider learning problems in which we have additional information describing the properties and/or the relations of the features. The features will have their own vectorial descriptions in some space in which we will describe their properties.

Real world problems with such properties are very common. For example in drug efficiency prediction problems, and more general in chemical formulae property prediction problems, drugs/formulae are collections of molecules. Each molecule comes with its own description, for example in terms of its physio-chemical properties, and/or its molecular structure. In language modeling, words are features and the words’ semantic and syntactic properties are their side-information. In image recognition, pixels are features and their position is the side-information, and so on. Similar ideas also appear in tasks such as matrix completion, robust PCA and collaborative filtering (Rao et al., 2015; Chiang et al., 2016, 2015). There one seeks low rank matrix decompositions in which the component matrices are constrained to follow relationships given by side-information matrices, typically matrices which contain user and item descriptors.

Despite the prevalence of such problems, there has been surprisingly limited work on learning with feature side-information. Krupka et al., 2008, used the feature side-information to perform feature selection as a preprocessing step prior to any modelling or learning. More interestingly (Krupka & Tishby, 2007) exploit the feature side-information directly within the learning process, by forcing features that have similar side-information to have a similar weight within a SVM model. One can think of this as a model regularisation technique in which we force the model structure, i.e. the feature parameters, to reflect the feature manifold as this is given by the feature side-information. In the same work the authors also provide an ad-hoc way to apply the same idea for non-linear models, more precisely polynomials of low degree. However the solution that they propose requires an explicit construction of the different non-linear terms, as well as appropriate definitions of the feature side-information that is associated with them. These definitions are hand-crafted and depend on the specific application problem. Beyond this ad-hoc approach it is far from clear how one could regularise general non-linear models so that they follow the feature manifold.

In this paper we present a method for the exploitation of feature side-information in non-linear models. The main idea is that the learned model will treat in a similar manner features that are similar. Intuitively, exchanging the values of two very similar features should only have a marginal effect on the model output. This is straightforward for linear models since we have direct access to how the model treats the features, i.e. the feature weights. In such a case one can design regularisers as Krupka & Tishby, 2007, did which force the feature weights to reflect the feature manifold. An obvious choice would be to apply a Laplacian regulariser to the linear model, where the Laplacian is based on the feature similarity. Such regularisers have been previously used for parameter shrinkage but only in the setting of linear models where one has direct access to the model parameters (Huang et al., 2011). However, in general non-linear models we no longer have access to the feature weights; the model parameters are shared between the features and we cannot disentangle them.

We present a regulariser which forces the learned model to be invariant/symmetric to relative changes in the values of similar features. It directly reflects the intuition that small changes in the values of similar features should have a small effect on the model output. The regulariser relies on a measure of the model output sensitivity to changes in all possible pairs of features. The model sensitivity measure quantifies the norm of the change of the model output under all possible relative changes of the values of two features. We compute this norm by integrating over the relative changes and the data distribution. Integrating over the relative changes is problematic we thus give two ways to approximate the sensitivity measure. In the first approach we rely on a first order Taylor expansion of the learned model under which the sensitivity measure boils down to the squared norm of the difference of the partial derivatives of the model with respect to the input features. Under this approach the regulariser finally boils down to the application of a Laplacian regulariser on the Jacobian of the model. In the second approach we rely on sampling and data augmentation to generate instances with appropriate relative changes over different feature pairs. We approximate the value of the regulariser only on the augmented data.

We implement the above ideas in the context of neural networks, nevertheless it is relatively straightforward to use them in other non-linear models such as SVMs and kernels.

We experiment on a number of text classification datasets in which the side-information is the word2vec representation of the words. We compare against a number of baselines and we show significant performance improvements.

## 2 Learning Symmetric Models with Respect to Feature Similarity

We consider supervised learning settings in which, in addition to the classical data matrix containing instances and features, and the target matrix , we are also given a matrix , the th row of which, denoted by , contains a description of the th feature. We call the feature side-information matrix. Note how the matrix is fixed and independent of the training instances. As in the standard supervised setting, instances, , are drawn i.i.d from some non-observed probability distributions and targets, , are assigned according to some non-observed conditional distribution . In the standard setting we learn a mapping from the input to the output using the matrices by optimizing some loss function . In this paper we learn the input-output mapping using in addition to the , matrices the feature side-information matrix.

We bring the feature side-information in the learning process through the feature similarity matrix which we construct from as follows. Given two features , with , side-information vectors the element of contains their similarity given by some similarity function. We will denote by the Laplacian of the similarity matrix ; is the diagonal degree matrix with .

We use the similarity and the Laplacian matrices to constraint the learned model to treat in a similar manner features that have similar side-information. This is relatively straightforward with linear models such as , and can be achieved through the introduction of the Laplacian regulariser in the objective function where is the th column vector of containing the model parameters associated with the th feature, (Huang et al., 2011). The Laplacian regulariser forces the parameter vectors of the features to cluster according to the feature similarity.

However in non-linear models such neat separation of the model parameters is not possible since these are shared between the different input features. In order to achieve the same effect we will now operate directly on the model output. We will do so by requiring that the change in the model’s output is marginal if we change the relative proportion of two very similar features. Concretely, let and be such features, and , , be the -dimensional unit vectors with the th and th dimensions respectively equal to one. We want that:

 (1)
 ∀λi,λj,λ′i,λ′j∈R such that λi+λj=λ′i+λ′j

Equation (1) states that as long as the total contribution of the , , features is kept fixed, the model’s output should be left almost unchanged. The exact equality will hold when the , , are on the limit identical, i.e. . More general the level of the model’s change should reflect the similarity of the features, thus a more accurate reformulation of equation 1 is:

 ||ϕ(x+λiei+λjej)−ϕ(x+λ′iei+λ′jej)||2∝1Sij (2)
 ∀λi,λj,λ′i,λ′j∈R such that λi+λj=λ′i+λ′j

Thus the norm of the change in the model output, that we get when we alter the relative proportion of two features and , while keeping their total contribution fixed, should be inversely proportional to the features similarity, i.e. large similarity, small output change. The result is that the model is symmetric to similar features and its output does not depend on the individual contributions/values of two similar features but only on their total contribution. In figure 1 we visualise the effect of the model constraint given in eq. 2. Given some instance and two features , , that are on the limit identical the constraint forces the model output to be constant on the line defined by , , for some given . We can think of the whole process as the model clustering together, to some latent factor, features that have very high similarity. The latent factor captures the original features total contribution leaving the model’s output unaffected to relative changes in their values.

To unclutter notation we will define the vector . We want the constraint of eq. 2 to be valid over all instances drawn from as well as for all vectors that satisfy the equality constraint eq 2. A natural measure of the degree to which the constraint holds for the feature pair is given by:

 Rij(ϕ) = ∫||ϕ(x+λiei+λjej) −ϕ(x+λ′iei+λ′jej)||2

where . Since we want to define a regulariser that accounts for all feature pairs and their similarities we simply have:

 R(ϕ)=∑ijRdij(ϕ) (4)

Calculating the regularizer is problematic due to the presence of the function that selects the subspace over which the integration is performed. In the next two sections we will give two ways to approximate it. The first one will be analytical relying on the first order Taylor expansion of and its Jacobian. The second one stochastic, essentially performing data augmentation and defining a regularisation term along the lines of eq. 2.

### 2.1 An analytical approximation

We will use the first order Taylor expansion of to simplify the squared term in eq. 2 by removing the variable. We will start by using the first order Taylor expansion to approximate the value of at

 ≈ ϕ(x)+J(x)(λiei+λjej)

is the Jacobian of evaluated at . Then plugging the Taylor expansion in eq 2 we get:

 (5)

and since we have and eq 5 becomes:

 ||J(x)ei−J(x)ej||2=||∇iϕ(x)−∇jϕ(x)||2∝1Sij (6)

where is the -dimensional partial derivative of with respect to the th input feature. Using eq 6 we can approximate as follows:

 Rij(ϕ) ≈

from which we get the following approximation of the regulariser:

 R(ϕ) ≈ ∑ij∫||∇iϕ(x)−∇jϕ(x)||2SijP(x)dx (7) ≈ ∫∑ij||∇iϕ(x)−∇jϕ(x)||2SijP(x)dx ≈ ∫Tr[J(x)LJT(x)P(x)dx

which is the local linear approximation of the original regulariser eq 4 on the input instances. Since we only have access to the training sample and not to we will get the sample estimate of eq. 7 given by

 ^R(ϕ) = ∑ij∑k||∇iϕ(xk)−∇jϕ(xk)||2Sij (8) = ∑k∑ij||∇iϕ(xk)−∇jϕ(xk)||2Sij = ∑kTr[J(xk)LJT(xk)]

So the sample based estimate of the regulariser is a sum of Laplacian regularisers applied on the Jacobian of each one of the training samples. It forces the partial derivatives of the model with respect to the input, or equivalently the model’s sensitivity to the input features, to reflect the features similarity in the local neighborhood around each training point. Or in other words it will constrain the learned model in a small neighborhood around each training point to have similar slop in the dimensions that are associated with similar features. Note that if then and reduces to the standard Laplacian regulariser on the columns of associated with the input features. Adding the sample based estimate of the regulariser to the loss function we get the final objective function which we minimize with giving the following minimization problem under the analytical approximation:

 minϕ∑kL(yk,ϕ(xk))+λ∑kTr[J(xk)LJT(xk)] (9)

The approximation of the requlariser is only effective locally around each training point since it relies on first order Taylor expansion. When the learned function is highly nonlinear, it can force model invariance only to small relative changes in the values of two similar features. However, as the size of the relative changes increases and we move away from the local region the approximation is no longer effective. The regulariser will not be powerful enough to make the invariance hold away from the training points. If we want a less local approximation we can either use higher order Taylor approximation which is computationally prohibitive or rely on a more global approximation through data augmentation as we will see in the next section. Note also that the presence of the Jacobian in the objective function means that if we optimise it using gradient descent we will need to compute second order partial derivatives which come with an increasing computational cost.

### 2.2 A stochastic approximation

Instead of using the first order Taylor expansion to simplify the squared term required by the regulariser we can use sampling to approximate it. Concretely for a given feature pair, , and a given instance we randomly sample quadruples such that , , which we use to generate new instance pairs as follows:

 x→⎧⎨⎩x+λ(l)iei+λ(l)jejx+λ(l)′iei+λ(l)′jej

We can now use the training sample and the sampling process to get an estimate of by:

 ∑k∑l||(ϕ(xk+λ(l)iei+λ(l)jej) −ϕ(xk+λ(l)′iei+λ(l)′jej))||2Sij

and of the final regulariser by:

 ~R(ϕ) = ∑ij∑k∑l||(ϕ(xk+λ(l)iei+λ(l)jej) (10) −ϕ(xk+λ(l)′iei+λ(l)′jej))||2Sij

So the final optimization problem will now become:

 minϕ∑kL(yk,ϕ(xk))+λ~R(ϕ) (11)

Note that the new instances appear only in the regulariser and not in the loss. The regulariser will penalise models which do not have the invariance property with respect to pairs of similar features. In practice when computing we do not want to go through all the pairs of features but only through the most similar. We do not want to spend sampling time on data augmentation for dissimilar pairs since for these there is no effective constraint on the values of the model’s output. So we simplify the sum run only over the pairs of similar features. One motivation for the stochastic approach was the fact that the analytical one relies in an approximation which is only effective locally in the neighborhood of each learning instance. In the stochastic approach we have control on the size of the neighborhood over which the constraint is enforced through the Euclidean norm of the change vector ; the larger its value the larger the neighborhood. The smaller the neighborhood the closer we are to the local behavior of the analytical approximation. We should note here that the sampling of stochastic approximation will naturally blend with the stochastic gradient optimization that we will use to optimize our objective functions.

### 2.3 Optimization

We learn with a standard feed forward neural network with sigmoid activation functions applied on the hidden layers using stochastic gradient descent.

The objective function of the analytical approach contains the Jacobian of the model with respect to its input. Calculating the gradient over this results in the introduction of second order partial derivatives of the model with respect to the inputs and the model parameters. Bishop, 1992, gave a backpropagation algorithm for the exact calculation of the Hessian of the loss of a multi-layer perceptron. We have adapted this algorithm so that we can compute the gradient of objective functions that contain the Jacobian with respect to the input features, we give the complete gradient calculation procedure in the appendix.

We will give now the computational complexity of each the two methods. We will denote by the number of layers, the output dimension of the network, the number of hidden units of the th layer and we will define . The computational complexity of computing the gradient for a single instance of the objective function of the analytical approach is for networks with a single hidden layer and for networks with more than one hidden layers. To reduce this computational complexity in our experiments we sparsify by keeping only the entries correponding to top biggest elements and zero out the rest. The complexity now becomes for one layer networks and for networks with more than one layer. The computational complexity of the stochastic approach is while the computational complexity for standard feed forward network is .

## 3 Related Work

Krupka & Tishby, 2007, use feature side-information, they call it meta-features, within a linear SVM model. They force the SVM’s weights to be similar for features that have similar side-information. They achieve that through the introduction of a Gaussian prior on the feature weight vector. The covariance matrix of the Gaussian is a function of the features similarity. The authors show how to extend their approach from linear to polynomial models. However, their approach requires explicit calculation of all the higher order terms limiting its applicability to low order polynomials. Very similar in spirit is all the body of work on Laplacian regularisation for feature regularisation; (Krupka & Tishby, 2007) contains an extensive review. Such regularisers constrain the feature weights to reflect relations that are given by the Laplacian. The Laplacian matrix is constructed from available domain knowledge, what here we call feature side information. However, it can also be constructed from the data; for example as a function of the feature correlation matrix.

The Taylor expansion we use in the analytical approximation of the regulariser results in the use of the Jacobian of the model. Regularisers that use the Jacobian have previously been successfully used to control the stability/robustness of models to noisy inputs. Relevant work includes contractive auto encoders, (Rifai et al., 2011b), and (Zhai & Zhang, 2015). (Rifai et al., 2011b) use the Frobenius of the Jacobian at the input instances to force the model to be relatively constant in small neighbors around the input instances. Such a regulariser introduces invariance to small input variations. In a different setting Rosasco et al., 2013, used the Jacobian to learn sparse non-linear models in the context of kernels.

Optimizing the Jacobian in networks with more than one layer is cumbersome, thus very often the stochastic approach is preferred over the analytic e.g. (Zhai & Zhang, 2015). Denoising autoencoders, (Vincent et al., 2010) follow the stochastic paradigm and require that small random variations in the inputs have only a limited effect on the model output. Zheng et al., 2016, used Gaussian perturbations to stabilise the network’s output with respect to variations in the input, essentially augmenting the training data. Regularisers on higher order derivatives, Hessian, are also used, (Rifai et al., 2011a), in such cases the stochastic approach is the only choice due to the prohibitive cost of optimizing the Hessian term.

Data augmentation is a well-established approach for learning models with built-in invariance to noise and/or robustness to data perturbations. In addition it is also used when we have additional prior knowledge on the instance structures to which the models should be invariant. In imaging problems such structures include translations, rotations, scalings,etc, (Simard et al., 1991; Decoste & Schölkopf, 2002).

The works that are closer to our work are (Krupka & Tishby, 2007) as well as the works that use Laplacian based regularisers for model regularisation, e.g.  (Huang et al., 2011). However to the best of our knowledge all previous work was strictly limited to linear models. We are the first ones who show how such regularisers and constraints can be applied to general classes of non-linear models.

## 4 Experiments

We will experiment and evaluate our regularisers in two settings, a synthetic and a real world one. We will compare the analytical and the stochastic regulariser, which we will denote by AN and ST respectively, against popular regularisers used with neural networks, namely and Dropout (Srivastava et al., 2014), over different network architectures. In the real world datasets we also give the results of the Word Mover’s Distance, WMD, (Kusner et al., 2015) which makes direct use of the side-information to compute document distances. Obviously our regularisers and WMD have an advantage over and dropout since it exploit side-information which and dropout do not.

We trained both the analytical and the stochastic models, as well as all baselines against which we compare, using Adam (Kingma & Ba, 2014). We used for one hidden layer networks, and for the networks with more hidden layers. We initialize all networks parameters using (Glorot & Bengio, 2010). Due to the large computational complexity of the analytical approach we set the mini-batch size to five. For the stochastic model, as well as for all the baseline models, we set the mini-batch size to 20. For the analytical model we set the maximum number of iterations to 5000. For the stochastic model we set the maximum number of iterations to 10000 for the one layer networks and to 20000 for networks with more layers. We used early stopping where we keep 20% of the training data as the validation set. Every five parameter updates we compute the validation error. Training terminates either if we reach the maximum iteration number or the validation error keeps increasing more than ten times in a row.

In the stochastic approach we do the sampling for the generation of the instance pairs within the stochastic gradient descent process. Concretely for each instance in a mini batch we randomly chose a feature pair , from the set of similar feature pairs. We sample a quadruple from respecting the constraint: from which we generate the respective instance pairs. We repeat the process times each time sampling a new feature pair , and a new quadruple. We fix the set of similar feature pairs to be the top of most similar feature pairs. Thus within each mini-batch of size we generate instance pairs and we accumulate the norm of the respective model output differences in the objective. In the experiments we set

### 4.1 Artificial datasets

We design a simple data generation process in order to test the performance of our regularisers when the data generation mechanism is compatible with the assumptions of our models. We randomly generate an instance matrix by uniformly sampling instances from . We create feature clusters as follows. To each one of these clusters we initially assign one of the original input features without replacement. We assign randomly and uniformly the remaining features to the clusters. We use the feature clusters to define a latent space where every feature cluster gives rise to a latent feature. The value of each latent feature is the sum of the values of the features that belong to its cluster. On the latent space representation of the training data we apply a linear transformation that projects the latent space to a new space with lower dimensionality . On this lower dimensionality space we apply an element-wise sigmoid and the final class assignment is given by the index of the maximum sigmoid value. The similarity of the , features of the original space is 1 if they ended up in the same cluster and 0 otherwise. We set , , . We will call this dataset A1. The generating procedure gave a very sparse matrix with only of its entries being non-zero. Each feature had an average of 1.3 similar features. We used 4000 instances for training and the rest for testing. During training 20% of the instances are used for the validation set. We measure performance with the classification error, i.e. percentage of wrong predictions. We train all algorithms on the original input space. For all regularisers we used a single layer with 100 hidden units. We tune the hyperparameters based on the performance on the validation set. We select the hyperparameters of of AN, ST, and from ; we select the of dropout from . We set the in the augmentation process, that controls the size of the neigborhood within which the output constraints should hold, to one.

Both the analytical and the stochastic regulariser bring performance improvements of roughly 10% when compared to the regulariser and to Dropout, results in table 1. In figure 2 we plot the learning curves, i.e. error on the validation set for each epoch, of the four regularisers. We can see that both the analytical and the stochastic regulariser converge much faster and to significantly lower error values than either or dropout.

The regularizer we propose constrains the model structure by forcing the model to reflect the feature similarity structures as these are given in the similarity matrix. Thus we expect the structure of the similarity matrix to have an impact on the performance of the regulariser. To see that let us consider the trivial case in which is diagonal, i.e. there are no similar features. In this case the input and the latent spaces are equivalent. Under such setting the regulariser will have no effect since there are no similarity constraints to impose on the model. If on the other hand, all features are identical, i.e. , then the latent space will have a dimensionality of one, in such a case the regulariser has the strongest effect.

To explore this dependency we generate two additional synthetic datasets where we use the same generating mechanism as in A1 but vary the proportion of features we cluster together to generate latent factors. Concretely in the synthetic dataset we will call A2 we randomly select a set A of features over which we will perform clustering to define latent factors. We use the remaining set B of features directly as they are in the latent space. We cluster the features of the A set to clusters—latent factors, making sure that as in A1 each cluster has at least one feature in it. As a result the final latent space has a dimensionality of . To generate the class assignments we proceed as in A1. To generate the third dataset, A3, we select features to generate A and the remaining for B. We now cluster the features in A to clusters, again making sure that there is at least one feature pre cluster. The dimensionality of the latent space is now ; class assignments are generated as above. We used the same values for , , as in A1. As we move from A1 to A3 we reduce the number of features that are similar to other features, thus we increase the sparsity of . For A2 and A3 the percentage of non-zero elements is and respectively, compared to we had in A1. So A1 is the datasets that has most constraints while A3 is the one with the least constraints. We apply the different regularisers in these two datasets using exactly the same protocol as in A1. The results are also given in table 1. As we see the classification error of both ST and AN increases as the dataset sparsity increases and it approaches that of the standard regularisers.

### 4.2 Real world datasets

We evaluated both approaches on the eight classification datasets used in (Kusner et al., 2015). The datasets are: BBC sports articles (BBCSPORT) labeled as one of athletics, cricket, footbal, rugby, tennis; tweets labeled with sentiments âpositiveâ, ânegativeâ, or âneutralâ (TWITTR); recipes labeled by their region of origin (RECIPE); of medical abstracts labeled by different cardiovascular disease groups (OHSUMED); sentences from academic papers labeled by publisher name (CLASSIC); amazon reviews labeled by product category (AMAZON); news dataset labeled by the news topics (REUTER); news articles classified into 20 different categories (20NEWS). We removed all the words in the SMART stop word list (Salton & Buckley, 1988). Documents are represented as bag of words. To speed up training, we removed words that appear very few times over all the documents of a dataset. Concretely, in 20NEWS we reduce the dictionary size by removing words with a frequency less or equal to three. In the OHSUMED and CLASSIC datasets we remove words with frequency one and the in REUTER dataset words with frequency equal or less than two. As feature side-information we use the word2vec representation of the words which have a dimensionality of 300 (Mikolov et al., 2013); other possibilities include knowledge-based side-information, e.g. based on WordNet (Miller, 1995). In table 2 we give a description of the final datasets on which we experiment including the number of classes () and average number of unique words per document.

We compute the similarity matrix from the word2vect word representations using the heat kernel with bandwidth parameter , i.e. the similarity of features is given by: . We select so that roughly 20% of the entries of the similarity matrix are in interval.

For those datasets that do not come with a predefined train/test split (BBCSPORT, TWITTER, CLASSIC, AMAZON, RECIPE), we use five-fold cross validation and report the average error. We compare the statistical significance of the results using the MacNemar’s test with a significance level of 0.05. For hyperparameter tuning we use three-fold inner cross validation. We select the hyperparameters of of AN, ST, and from ; we select the of dropout from . We do a series of experiments in which we vary the number of hidden layers. Due to the computational complexity of the backprogation for the AN regulariser we only give results for the single layer architecture.

In the first set of experiments we use a neural network with one hidden layer and 100 hidden units, we give the results in table 3. ST is significantly better than the AN in five out of the eight datasets, significantly worse once, and equivalent in one dataset. ST is significantly better than the in six out of the eight daasets, while it is equivalent in one. Compared to dropout it is four times significantly better and three times significantly worse.

When we increase the number of hidden layers to two with 500 and 100 units on the first and second layer ST method is significantly better compared to three times, significantly worse three times, while there is no significant difference in two datasets. A similar picture emerges with respect to Dropout with ST being significantly better three times, significantly worse twise, while in three cases there is no significant difference. We give the detailed results in table 4.

## 5 Conclusion and Future Work

Many real world applications come with additional information describing the properties of the features. Despite that, quite limited attention has been given to such setting. In this paper we develop a regulariser that exploits exactly such information for general non-linear models. It relies on the simple intuition that features which have similar properties should be treated by the learned model in a similar manner. The regulariser imposes a stability constraint over the model output. The constraint forces the model to produce similar outputs for instances the feature values of which differ only on similar features. We give two ways to approximate the value of the regulariser. An analytical one which boils down to the imposition of a Laplacian regulariser on the Jacobian of the learned model with respect to the input features and a stochastic one which relies on sampling.

We experiment with neural networks with the two approximations of the regulariser and compare their performance to well established model regularisers, namely and dropout, on artificial and real world datasets. In the artificial datasets, for which we know that they match the assumptions of our regulariser we demonstrate significant performance improvements. In the real world datasets the performance improvements are less striking. One of the main underlying assumptions of our model is that the feature side-information is indeed relevant for the learning problem, when this is indeed the case we will have performance improvements. If it is not the case then the regulariser will not be selected, as a result of the tuning of the parameter. Along the same lines we want to perform a more detailed study on how the structure of the similarity matrix, namely its sparsity and the underlying feature cluster structure, determines the regularisation strength of our regulariser. It is clear that a sparse similarity matrix will lead to a rather limited regularisation effect since only few features will be affected. This points to the fact that the regulariser should be used together with more traditional sparsity inducing regularisers, especially in the case of a sparse feature feature similarity matrix. Finally since we use the feature information through a similarity function it might be the case that the similarity function that we are using is not appropriate and better results can be obtained if we also learn the feature similarity. We leave this for future work.

## References

• Bishop (1992) Bishop, Chris. Exact calculation of the hessian matrix for the multilayer perceptron, 1992.
• Bishop (1995) Bishop, Christopher M. Neural networks for pattern recognition. Oxford university press, 1995.
• Chiang et al. (2015) Chiang, Kai-Yang, Hsieh, Cho-Jui, and Dhillon, Inderjit S. Matrix completion with noisy side information. In Advances in Neural Information Processing Systems, pp. 3447–3455, 2015.
• Chiang et al. (2016) Chiang, Kai-Yang, Hsieh, Cho-Jui, and Dhillon, EDU Inderjit S. Robust principal component analysis with side information. In Proceedings of The 33rd International Conference on Machine Learning, pp. 2291–2299, 2016.
• Decoste & Schölkopf (2002) Decoste, Dennis and Schölkopf, Bernhard. Training invariant support vector machines. Machine Learning, 46(1):161–190, 2002. ISSN 1573-0565.
• Glorot & Bengio (2010) Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. In Aistats, volume 9, pp. 249–256, 2010.
• Huang et al. (2011) Huang, Jian, Ma, Shuangge, Li, Hongzhe, and Zhang, Cun-Hui. The sparse laplacian shrinkage estimator for high-dimensional regression. The Annals of Statistics, 39(4):2021–2046, 2011. ISSN 00905364.
• Kingma & Ba (2014) Kingma, Diederik and Ba, Jimmy. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
• Krupka & Tishby (2007) Krupka, Eyal and Tishby, Naftali. Incorporating prior knowledge on features into learning. In AISTATS, volume 2, 2007.
• Krupka et al. (2008) Krupka, Eyal, Navot, Amir, and Tishby, Naftali. Learning to select features using their properties. Journal of Machine Learning Research, 9(Oct):2349–2376, 2008.
• Kusner et al. (2015) Kusner, Matt J, Sun, Yu, Kolkin, Nicholas I, and Weinberger, Kilian Q. From word embeddings to document distances. In Proceedings of the 32nd International Conference on Machine Learning (ICML 2015), pp. 957–966, 2015.
• Mikolov et al. (2013) Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg S, and Dean, Jeff. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013.
• Miller (1995) Miller, George A. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41, 1995.
• Rao et al. (2015) Rao, Nikhil, Yu, Hsiang-Fu, Ravikumar, Pradeep K, and Dhillon, Inderjit S. Collaborative filtering with graph information: Consistency and scalable methods. In Advances in neural information processing systems, pp. 2107–2115, 2015.
• Rifai et al. (2011a) Rifai, Salah, Mesnil, Grégoire, Vincent, Pascal, Muller, Xavier, Bengio, Yoshua, Dauphin, Yann, and Glorot, Xavier. Higher order contractive auto-encoder. In Machine Learning and Knowledge Discovery in Databases - European Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part II, pp. 645–660, 2011a.
• Rifai et al. (2011b) Rifai, Salah, Vincent, Pascal, Muller, Xavier, Glorot, Xavier, and Bengio, Yoshua. Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th international conference on machine learning (ICML-11), pp. 833–840, 2011b.
• Rosasco et al. (2013) Rosasco, L, Villa, S, Mosci, S, Santoro, M, and others. Nonparametric sparsity and regularization. J. Mach. Learn. Res., 2013.
• Salton & Buckley (1988) Salton, Gerard and Buckley, Christopher. Term-weighting approaches in automatic text retrieval. Information processing & management, 24(5):513–523, 1988.
• Simard et al. (1991) Simard, Patrice Y., Victorri, Bernard, LeCun, Yann, and Denker, John S. Tangent prop - A formalism for specifying selected invariances in an adaptive network. In Moody, John E., Hanson, Stephen Jose, and Lippmann, Richard (eds.), Advances in Neural Information Processing Systems 4, [NIPS Conference, Denver, Colorado, USA, December 2-5, 1991], pp. 895–903. Morgan Kaufmann, 1991.
• Srivastava et al. (2014) Srivastava, Nitish, Hinton, Geoffrey E, Krizhevsky, Alex, Sutskever, Ilya, and Salakhutdinov, Ruslan. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958, 2014.
• Vapnik & Izmailov (2015) Vapnik, Vladimir and Izmailov, Rauf. Learning using privileged information: Similarity control and knowledge transfer. Journal of Machine Learning Research, 16:2023–2049, 2015.
• Vincent et al. (2010) Vincent, Pascal, Larochelle, Hugo, Lajoie, Isabelle, Bengio, Yoshua, and Manzagol, Pierre-Antoine. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 11:3371–3408, 2010.
• Xing et al. (2002) Xing, Eric P., Ng, Andrew Y., Jordan, Michael I., and Russell, Stuart J. Distance metric learning with application to clustering with side-information. In Advances in Neural Information Processing Systems 15 [Neural Information Processing Systems, NIPS 2002, December 9-14, 2002, Vancouver, British Columbia, Canada], pp. 505–512, 2002.
• Zhai & Zhang (2015) Zhai, Shuangfei and Zhang, Zhongfei. Manifold regularized discriminative neural networks. arXiv preprint arXiv:1511.06328, 2015.
• Zheng et al. (2016) Zheng, Stephan, Song, Yang, Leung, Thomas, and Goodfellow, Ian J. Improving the robustness of deep neural networks via stability training. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pp. 4480–4488, 2016.

## Appendix A Appendix

### a.1 Modified Backpropogation

For notion simplicity, we consider stochastic gradient descent. The objective function we want to minimize is as following:

 E=L(y,ϕ(x))+λ1∑ij||∂ϕ(x)∂xi−∂ϕ(x)∂xj||2Sij (12)

Notice that the objective function includes derivative of the learned function with respect to the input features, if we use neural network to learn the model, the conventional backpropagation algorithm can’t be applied directly. Therefore, we developed a modified version of the backpropagation algorithm to find the gradient of the objective.
We keep the notation consistent with the notation used in the book of (Bishop, 1995). n is the total layers (including input and out put layer) number of the network, is the pre-activation units in layer , is the number of hidden units in hidden layer , is the number of output units, and stands for the non-linear activation function.

 z0=xak=wkzk−1+bkzk=h(ak)ϕ(x)=zn (13)

To find the gradient of (12), we define as the Jacobian of the learned function with respect to pre-activations at the layer :

 δk=⎡⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢⎣∂ϕ1∂ak1∂ϕ2∂ah1⋯.∂ϕm∂ak1∂ϕ1∂ak2∂ϕ2∂ak2⋯∂ϕm∂ak2⋮⋮⋮∂ϕ1∂akk1∂ϕ2∂akk1⋯∂ϕm∂akk1⎤⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥⎦ (14)

for all can be achieved by the following backpropagation equation.

 δk=((Wk+1)Tδk+1)⊙h′(ak)∀k=1,2,...,n−1 (15)

Where stands for the element wise multiplication of a column vector to every column of the matrix.

 δn=⎡⎢ ⎢ ⎢ ⎢⎣h′(an1)0...00h′(an2)...0.........00...h′(anm)⎤⎥ ⎥ ⎥ ⎥⎦ (16)

Defining the term in such a away, we can rewrite the regularizer term in equation (12) as following:

 ∑ij||(W1(:,i))−W1(:,j))Tδ1||2Sij (17)

If the network only has one hidden layer, we can derive derivative of the regularizer with respect to weights using and (15). When hidden layer’s number is more than one, we need to introduce two more term, one to the backward path and one to the forward path: Define as the jacobian of pre-activation unit at layer with respect to pre-activation at first hidden layer, note layer corresponding to first hidden layer.

 Gkmg=∂akm∂a1g∀k=1,2,3,...,n (18)

We know that:

 G1mg=∂a1m∂a1g={1if m=g0others (19)

And for all can be achieved during forward path by the following forward propagation equation and

 Gkmg=∑lWkmlGk−1lgh′(ak−1l)∀k=2,3,...,n (20)

Define which gives the derivative of the with respect to the pre-activation units in the first hidden layers:

 Bkljg=∂δklj∂a1g∀k=1,2,...,n (21)

We know that:

 Bnljg=∂δnlj∂a1g=h′′(anl)1ljGnlg (22)

for all can be obtained by the following propagating equation during backward path using as following:

 \scalebox0.80[1]$Bkljg=h′′(akl)Gklg∑pδk+1pjWk+1pl+h′(akl)∑pWk+1plBk+1pjg$ (23)
 ∀k=1,2,...,n−1

Finally, the gradient of the regularizer, i.e. second term of the equation (12), can be calculated as follwoing:
For , i.e. first hidden layer:

 \scalebox0.60[1]$∂R∂W1lm=4λ1∑sSms∑j(W1(:,m)−W1(:,s))Tδ1(:,j)δ1(lj)$\scalebox0.70[1]$+2λ1∑ksSks∑j(W1(:,k)−W1(:,s))Tδ1(:,j)∑g(W1(g,k)−W1(g,s))B1ljgz0m)$ (24)

For :

 \scalebox0.80[0.9]$∂R∂Wklm=2λ1∑ksSks∑j(W1(:,k)−W1(:,s))Tδ1(:,j)$\scalebox0.70[0.9]$∑g(W1(g,k)−W1(g,s))(zk−1mBkljg+δkljh′(ak−1m)Gk−1mg)$ (25)

Gradient with respect to bias term, for all :

 ∂R∂bklm=2λ1∑ksSks∑j(W1(:,k)−W1(:,s))Tδ1(:,j)∑g(W1(g,k)−W1(g,s))Bkljg (26)

The gradient of the first part of the objective which is some loss function we chose, is same as in the standard Backpropagation algorithm, here we just need to rewrite it in terms of the newly defined . For example, if we use sigmoid on all layers as activation function and cross entropy loss, we have the following:

 E=−m∑i=1(yilogϕ(x)i+(1−yi)log(1−ϕ(x)i)) (27)
 ∂E∂Wk=δkϕ−yϕ(1−ϕ)(zk−1)T (28)
 ∂E∂bk=δkϕ−yϕ(1−ϕ) (29)

Now we can find the gradient of the loss with respect to weights in all layers. Compared to the conventional back propagation algorithm, except we have term which is defined differently than the conventional backprop algorithm, we have one more extra term to add to the backward path and one more term to the forward path.

You are adding the first comment!
How to quickly get a good reply:
• Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
• Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
• Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters