Multi-Stage Multi-Task Feature Learning
Multi-task sparse feature learning aims to improve the generalization performance by exploiting the shared features among tasks. It has been successfully applied to many applications including computer vision and biomedical informatics. Most of the existing multi-task sparse feature learning algorithms are formulated as a convex sparse regularization problem, which is usually suboptimal, due to its looseness for approximating an -type regularizer. In this paper, we propose a non-convex formulation for multi-task sparse feature learning based on a novel non-convex regularizer. To solve the non-convex optimization problem, we propose a Multi-Stage Multi-Task Feature Learning (MSMTFL) algorithm; we also provide intuitive interpretations, detailed convergence and reproducibility analysis for the proposed algorithm. Moreover, we present a detailed theoretical analysis showing that MSMTFL achieves a better parameter estimation error bound than the convex formulation. Empirical studies on both synthetic and real-world data sets demonstrate the effectiveness of MSMTFL in comparison with the state of the art multi-task sparse feature learning algorithms.
Keywords: Multi-Task Learning, Multi-Stage, Non-convex, Sparse Learning
Multi-task learning (MTL) (Caruana, 1997) exploits the relationships among multiple related tasks to improve the generalization performance. It has been successfully applied to many applications such as speech classification (Parameswaran and Weinberger, 2010), handwritten character recognition (Obozinski et al., 2006; Quadrianto et al., 2010) and medical diagnosis (Bi et al., 2008). One common assumption in multi-task learning is that all tasks should share some common structures including the prior or parameters of Bayesian models (Schwaighofer et al., 2005; Yu et al., 2005; Zhang et al., 2006), a similarity metric matrix (Parameswaran and Weinberger, 2010), a classification weight vector (Evgeniou and Pontil, 2004), a low rank subspace (Chen et al., 2010; Negahban and Wainwright, 2011) and a common set of shared features (Argyriou et al., 2008; Gong et al., 2012; Kim and Xing, 2009; Kolar et al., 2011; Lounici et al., 2009; Liu et al., 2009; Negahban and Wainwright, 2008; Obozinski et al., 2006; Yang et al., 2009; Zhang et al., 2010).
Multi-task feature learning, which aims to learn a common set of shared features, has received a lot of interests in machine learning recently, due to the popularity of various sparse learning formulations and their successful applications in many problems. In this paper, we focus on a specific multi-task feature learning setting, in which we learn the features specific to each task as well as the common features shared among tasks. Although many multi-task feature learning algorithms have been proposed in the past, many of them require the relevant features to be shared by all tasks. This is too restrictive in real-world applications (Jalali et al., 2010). To overcome this limitation, Jalali et al. (2010) proposed an regularized formulation, called dirty model, to leverage the common features shared among tasks. The dirty model allows a certain feature to be shared by some tasks but not all tasks. Jalali et al. (2010) also presented a theoretical analysis under the incoherence condition (Donoho et al., 2006; Obozinski et al., 2011) which is more restrictive than RIP (Candes and Tao, 2005; Zhang, 2012). The regularizer is a convex relaxation for the -type one, in which a globally optimal solution can be obtained. However, a convex regularizer is known to too loose to approximate the -type one and often achieves suboptimal performance (either require restrictive conditions or obtain a suboptimal error bound) (Zhang and Zhang, 2012; Zhang, 2010, 2012). To remedy the limitation, a non-convex regularizer can be used instead. However, the non-convex formulation is usually difficult to solve and a globally optimal solution can not be obtained in most practical problems. Moreover, the solution of the non-convex formulation heavily depends on the specific optimization algorithms employed. Even with the same optimization algorithm adopted, different initializations usually lead to different solutions. Thus, it is often challenging to analyze the theoretical behavior of a non-convex formulation.
Contributions: We propose a non-convex formulation, called capped-, regularized model for multi-task feature learning. The proposed model aims to simultaneously learn the features specific to each task as well as the common features shared among tasks. We propose a Multi-Stage Multi-Task Feature Learning (MSMTFL) algorithm to solve the non-convex optimization problem. We also provide intuitive interpretations of the proposed algorithm from several aspects. In addition, we present a detailed convergence analysis for the proposed algorithm. To address the reproducibility issue of the non-convex formulation, we show that the solution generated by the MSMTFL algorithm is unique (i.e., the solution is reproducible) under a mild condition, which facilitates the theoretical analysis of the MSMTFL algorithm. Although the MSMTFL algorithm may not obtain a globally optimal solution, we show that this solution achieves good performance. Specifically, we present a detailed theoretical analysis on the parameter estimation error bound for the MSMTFL algorithm. Our analysis shows that, under the sparse eigenvalue condition which is weaker than the incoherence condition used in Jalali et al. (2010), MSMTFL improves the error bound during the multi-stage iteration, i.e., the error bound at the current iteration improves the one at the last iteration. Empirical studies on both synthetic and real-world data sets demonstrate the effectiveness of the MSMTFL algorithm in comparison with the state of the art algorithms.
Notations: Scalars and vectors are denoted by lower case letters and bold face lower case letters, respectively. Matrices and sets are denoted by capital letters and calligraphic capital letters, respectively. The norm, Euclidean norm, norm and Frobenius norm are denoted by , and , respectively. denotes the absolute value of a scalar or the number of elements in a set, depending on the context. We define the norm of a matrix as . We define as and as the normal distribution with mean and variance . For a matrix and sets , we let be the vector with the -th entry being , if , and , otherwise. We also let be a matrix with the -th entry being , if , and , otherwise.
Organization: In Section 2, we introduce a non-convex formulation and present the corresponding optimization algorithm. In Section 3, we discuss the convergence and reproducibility issues of the MSMTFL algorithm. In Section 4, we present a detailed theoretical analysis on the MSMTFL algorithm, in terms of the parameter estimation error bound. In Section 5, we provide a sketch of the proof of the presented theoretical results and the detailed proof is provided in the Appendix. In Section 6, we report the experimental results and we conclude the paper in Section 7.
2 The Proposed Formulation and the Optimization Algorithm
In this section, we first present a non-convex formulation for multi-task feature learning. Then, we show how to solve the corresponding optimization problem. Finally, we provide intuitive interpretations and discussions for the proposed algorithm.
2.1 A Non-convex Formulation
Assume we are given learning tasks associated with training data , where is the data matrix of the -th task with each row as a sample; is the response of the -th task; is the data dimensionality; is the number of samples for the -th task. We consider learning a weight matrix consisting of the weight vectors for linear predictive models: . In this paper, we propose a non-convex multi-task feature learning formulation to learn these models simultaneously, based on the capped-, regularization. Specifically, we first impose the penalty on each row of , obtaining a column vector. Then, we impose the capped- penalty (Zhang, 2010, 2012) on that vector. Formally, we formulate our proposed model as follows:
where is an empirical loss function of ; is a parameter balancing the empirical loss and the regularization; is a thresholding parameter; is the -th row of the matrix . In this paper, we focus on the following quadratic loss function:
Intuitively, due to the capped- penalty, the optimal solution of Eq. (1) denoted as has many zero rows. For a nonzero row , some entries may be zero, due to the -norm imposed on each row of . Thus, under the formulation in Eq. (1), some features can be shared by some tasks but not all the tasks. Therefore, the proposed formulation can leverage the common features shared among tasks.
2.2 Optimization Algorithm
The formulation in Eq. (1) is non-convex and is difficult to solve. In this paper, we propose an algorithm called Multi-Stage Multi-Task Feature Learning (MSMTFL) to solve the optimization problem (see details in Algorithm 1). In this algorithm, a key step is how to efficiently solve Eq. (3). Observing that the objective function in Eq. (3) can be decomposed into the sum of a differential loss function and a non-differential regularization term, we employ FISTA (Beck and Teboulle, 2009) to solve the sub-problem. In the following, we present some intuitive interpretations of the proposed algorithm from several aspects.
2.2.1 Locally Linear Approximation
First, we define two auxiliary functions:
We note that is a concave function and we say that a vector is a sub-gradient of at , if for all vector , the following inequality holds:
where denotes the inner product. Using the functions defined above, Eq. (1) can be equivalently rewritten as follows:
Based on the definition of the sub-gradient for a concave function given above, we can obtain an upper bound of using a locally linear approximation at :
where is a sub-gradient of at . Furthermore, we can obtain an upper bound of the objective function in Eq. (4), if the solution at the -th iteration is available:
It can be shown that a sub-gradient of at is
which is used in Step 4 of Algorithm 1. Since both and are constant with respect to , we have
which, as shown in Step 3 of Algorithm 1, obtains the next iterative solution by minimizing the upper bound of the objective function in Eq. (4). Thus, in the viewpoint of the locally linear approximation, we can understand Algorithm 1 as follows: The original formulation in Eq. (4) is non-convex and is difficult to solve; the proposed algorithm minimizes an upper bound in each step, which is convex and can be solved efficiently. It is closely related to the Concave Convex Procedure (CCCP) (Yuille and Rangarajan, 2003). In addition, we can easily verify that the objective function value decreases monotonically as follows:
An important issue we should mention is that a monotonic decrease of the objective function value does not guarantee the convergence of the algorithm, even if the objective function is strictly convex and continuously differentiable (see an example in the book (Bertsekas, 1999, Fig 1.2.6)). In Section 3.1, we will formally discuss the convergence issue.
2.2.2 Block Coordinate Descent
Recall that is a concave function. We can define its conjugate function as (Rockafellar, 1970):
Since is also a closed function (i.e., the epigraph of is convex), the conjugate function of is the original function (Bertsekas, 1999, Chap. 5.4), that is:
which corresponds to Step 3 of Algorithm 1.
The block coordinate descent procedure is intuitive, however, it is non-trivial to analyze its convergence behavior. We will present the convergence analysis in Section 3.1.
If we terminate the algorithm with , the MSMTFL algorithm is equivalent to the regularized multi-task feature learning algorithm (Lasso). Thus, the solution obtained by MSMTFL can be considered as a multi-stage refinement of that of Lasso. Basically, the MSMTFL algorithm solves a sequence of weighted Lasso problems, where the weights ’s are set as the product of the parameter in Eq. (1) and a -valued indicator function. Specifically, a penalty is imposed in the current stage if the -norm of some row of in the last stage is smaller than the threshold ; otherwise, no penalty is imposed. In other words, MSMTFL in the current stage tends to shrink the small rows of and keep the large rows of in the last stage. However, Lasso (corresponds to ) penalizes all rows of in the same way. It may incorrectly keep the irrelevant rows (which should have been zero rows) or shrink the relevant rows (which should have been large rows) to be zero vectors. MSMTFL overcomes this limitation by adaptively penalizing the rows of according to the solution generated in the last stage. One important question is whether the MSMTFL algorithm can improve the performance during the multi-stage iteration. In Section 4, we will theoretically show that the MSMTFL algorithm indeed achieves the stagewise improvement in terms of the parameter estimation error bound. That is, the error bound in the current stage improves the one in the last stage. Empirical studies in Section 6 also validate the presented theoretical analysis.
3 Convergence and Reproducibility Analysis
In this section, we first present the convergence analysis. Then, we discuss the reproducibility issue for the MSMTFL algorithm.
3.1 Convergence Analysis
The main convergence result is summarized in the following theorem, which is based on the block coordinate descent interpretation.
Let be a limit point of the sequence generated by the block coordinate descent algorithm. Then is a critical point of Eq. (1).
It follows that
which indicates that the sequence is monotonically decreasing. Since is a limit point of , there exists a subsequence such that
We observe that
where the first inequality above is due to Eq. (7). Thus, is bounded below. Together with the fact that is decreasing, exists. Since is continuous, we have
Taking limits on both sides of Eq. (11) with , we have
Therefore, the zero matrix must be a sub-gradient of the objective function in Eq. (12) at :
where denotes the sub-differential (which is a set composed of all sub-gradients) of at . We observe that
which implies that :
Taking limits on both sides of the above inequality with , we have:
which implies that is a sub-gradient of at , that is:
Note that the above theorem holds by assuming that there exists a limit point. Next, we need to prove that the sequence has a limit point. For any bounded initial point , based on Eq. (7), Eq. (8) and the monotonicity of , we have:
Assume that the sequence is unbounded, that is, there exist some such that . It implies that (We exclude the case that some columns of are zero vectors. Otherwise, we can simply remove the corresponding zero columns.) and hence . This leads to a contradiction with Eq. (15). Thus, the sequence is bounded and there exists at least one limit point , since any bounded sequence has limit points.
Due to the equivalence between Algorithm 1 and the block coordinate descent algorithm above, Theorem 1 and its remark indicate that the sequence generated by Algorithm 1 has at least one limit point that is also a critical point of Eq. (1). The remaining issue is to analyze the performance of the critical point. In the sequel, we will conduct analysis in two aspects: reproducibility and the parameter estimation performance.
3.2 Reproducibility of The Algorithm
In general, it is difficult to analyze the performance of a non-convex formulation, as different solutions can be obtained due to different initializations. One natural question is whether the solution generated by Algorithm 1 (based on the initialization of in Step 1) is reproducible. In other words, is the solution of Algorithm 1 unique? If we can guarantee that, for any , the solution of Eq. (3) is unique, then the solution generated by Algorithm 1 is unique. That is, the solution is reproducible. The main result is summarized in the following theorem:
If has entries drawn from a continuous probability distribution on , then, for any , the optimization problem in Eq. (3) has a unique solution with probability one.
Proof Eq. (3) can be decomposed into independent smaller minimization problems:
Next, we only need to prove the solution of the above optimization problem is unique. To simplify the notations, we unclutter the above equation (by ignoring some superscripts and subscripts) as follows:
The first order optimal condition is :
where , if ; , if ; and , otherwise. We define
where denotes the matrix composed of the columns of indexed by . Then, the optimal solution of Eq. (16) satisfies
where denotes the vector composed of entries of indexed by .
Since is drawn from the continuous probability distribution,
has columns in general positions with probability one and hence
(or equivalently ), due to
Lemma 3, Lemma 4 and their discussions in Tibshirani (2012). Therefore, the objective function in Eq. (18) is strictly convex, which implies that is unique. Thus, the optimal solution of Eq. (16) is also unique and so is the optimization problem in Eq. (3) for any . This completes the proof of Theorem 3.
Theorem 3 is important in the sense that it makes the theoretical analysis for the parameter estimation performance of Algorithm 1 possible. Although the solution may not be globally optimal, we show in the next section that the solution has good performance in terms of the parameter estimation error bound.
4 Parameter Estimation Error Bound
In this section, we theoretically analyze the parameter estimation performance of the solution obtained by the MSMTFL algorithm. To simplify the notations in the theoretical analysis, we assume that the number of samples for all the tasks are the same. However, our theoretical analysis can be easily extended to the case where the tasks have different sample sizes.
Let be the underlying sparse weight matrix and , where is a random vector with all entries being independent sub-Gaussians: there exists such that :
We call the random variable satisfying the condition in Assumption 1 sub-Gaussian, since its moment generating function is bounded by that of a zero mean Gaussian random variable. That is, if a normal random variable , then we have:
Based on the Hoeffding’s Lemma, for any random variable and , we have . Therefore, both zero mean Gaussian and zero mean bounded random variables are sub-Gaussians. Thus, the sub-Gaussian noise assumption is more general than the Gaussian noise assumption which is commonly used in the multi-task learning literature (Jalali et al., 2010; Lounici et al., 2009).
Given , we define
is in fact the maximum (minimum) eigenvalue of , where is a set satisfying and is a submatrix composed of the columns of indexed by . In the MTL setting, we need to exploit the relations of among multiple tasks.
We present our parameter estimation error bound on MSMTFL in the following theorem:
Let Assumption 1 hold. Define and . Denote as the number of nonzero rows of . We assume that
where is some integer satisfying . If we choose and such that for some :
then the following parameter estimation error bound holds with probability larger than :
where is a solution of Eq. (3).
Eq. (19) assumes that the -norm of each nonzero row of is away from zero. This requires the true nonzero coefficients should be large enough, in order to distinguish them from the noise. Eq. (20) is called the sparse eigenvalue condition (Zhang, 2012), which requires the eigenvalue ratio to grow sub-linearly with respect to . Such a condition is very common in the analysis of sparse regularization (Zhang and Huang, 2008; Zhang, 2009) and it is slightly weaker than the RIP condition (Candes and Tao, 2005; Huang and Zhang, 2010; Zhang, 2012).
When (corresponds to Lasso), the first term of the right-hand side of Eq. (23) dominates the error bound in the order of
since satisfies the condition in Eq. (21). Note that the first term of the right-hand side of Eq. (23) shrinks exponentially as increases. When is sufficiently large in the order of , this term tends to zero and we obtain the following parameter estimation error bound:
Jalali et al. (2010) gave an -norm error bound as well as a sign consistency result between and . A direct comparison between these two bounds is difficult due to the use of different norms. On the other hand, the worst-case estimate of the -norm error bound of the algorithm in Jalali et al. (2010) is in the same order with Eq. (24), that is: . When is large and the ground truth has a large number of sparse rows (i.e., is a small constant), the bound in Eq. (25) is significantly better than the ones for the Lasso and Dirty model.
Jalali et al. (2010) presented an -norm parameter estimation error bound and hence a sign consistency result can be obtained. The results are derived under the incoherence condition which is more restrictive than the RIP condition and hence more restrictive than the sparse eigenvalue condition in Eq. (20). From the viewpoint of the parameter estimation error, our proposed algorithm can achieve a better bound under weaker conditions. Please refer to (Van De Geer and Bühlmann, 2009; Zhang, 2009, 2012) for more details about the incoherence condition, the RIP condition, the sparse eigenvalue condition and their relationships.
The capped- regularized formulation in Zhang (2010) is a special case of our formulation when . However, extending the analysis from the single task to the multi-task setting is nontrivial. Different from previous work on multi-stage sparse learning which focuses on a single task (Zhang, 2010, 2012), we study a more general multi-stage framework in the multi-task setting. We need to exploit the relationship among tasks, by using the relations of sparse eigenvalues and treating the -norm on each row of the weight matrix as a whole for consideration. Moreover, we simultaneously exploit the relations of each column and each row of the matrix.
5 Proof Sketch of Theorem 8
In this section, we present a proof sketch of Theorem 8. We first provide several important lemmas (detailed proofs are available in the Appendix) and then complete the proof of Theorem 8 based on these lemmas.
Lemma 13 gives bounds on the residual correlation () with respect to . We note that Eq. (26) and Eq. (27) are closely related to the assumption on in Eq. (21) and the second term of the right-hand side of Eq. (23) (error bound), respectively. This lemma provides a fundamental basis for the proof of Theorem 8.
Denote and notice that . Lemma 14 says that is upper bounded in terms of , which indicates that the error of the estimated coefficients locating outside of should be small enough. This provides an intuitive explanation why the parameter estimation error of our algorithm can be small.