Optimal Stochastic Strongly Convex Optimization
with a Logarithmic Number of Projections
We consider stochastic strongly convex optimization with a complex inequality constraint. This complex inequality constraint may lead to computationally expensive projections in algorithmic iterations of the stochastic gradient descent (SGD) methods. To reduce the computation costs pertaining to the projections, we propose an Epoch-Projection Stochastic Gradient Descent (Epro-SGD) method. The proposed Epro-SGD method consists of a sequence of epochs; it applies SGD to an augmented objective function at each iteration within the epoch, and then performs a projection at the end of each epoch. Given a strongly convex optimization and for a total number of iterations, Epro-SGD requires only projections, and meanwhile attains an optimal convergence rate of , both in expectation and with a high probability. To exploit the structure of the optimization problem, we propose a proximal variant of Epro-SGD, namely Epro-ORDA, based on the optimal regularized dual averaging method. We apply the proposed methods on real-world applications; the empirical results demonstrate the effectiveness of our methods.
Optimal Stochastic Strongly Convex Optimization
with a Logarithmic Number of Projections
Recent years have witnessed an increased interest in adopting the stochastic (sub)gradient (SGD) methods [1, 3, 21] for solving large-scale machine learning problems. In each of the algorithmic iterations, SGD reduces the computation cost by sampling one (or a small number of) example for computing a stochastic (sub)gradient. Thus the computation cost in SGD is independent of the size of the data available for training; this property makes SGD appealing for large-scale optimization. However, when the optimization problems involve a complex domain (for example a positive definite constraint or a polyhedron one), the projection operation in each iteration of SGD, which is used to ensure the feasibility of the intermediate solutions, may become the computational bottleneck.
In this paper we consider to solve the following constrained optimization problem
where is -strongly convex  and is convex. We assume a stochastic access model for , in which the only access to is via a stochastic gradient oracle; in other words, given arbitrary , this stochastic gradient oracle produces a random vector , whose expectation is a subgradient of at the point , i.e., , where denotes the subdifferential set of at . On the other hand we have the full access to the (sub)gradient of .
and then returning as the final solution for a total number of iterations . Note that is a projection operator defined as
If the involved constraint function is complex (e.g., a polyhedral or a positive definite constraint), computing the associated projection may be computationally expensive; for example, a projection onto a positive definite cone over requires a full singular value decomposition (SVD) operation with time complexity of .
In this paper, we propose an epoch-based SGD method, called Epro-SGD, which requires only a logarithmic number of projections (onto the feasible set), and meanwhile achieves an optimal convergence rate for stochastic strongly convex optimization. Specifically, the proposed Epro-SGD method consists of a sequence of epochs; within each of the epochs, the standard SGD is applied to optimize a composite objective function augmented by the complex constraint function, hence avoiding the expensive projections steps; at the end of every epoch, a projection operation is performed to ensure the feasibility of the intermediate solution. Our analysis shows that given a strongly convex optimization and for a total number of iterations, Epro-SGD requires only projections, and meanwhile achieves an optimal rate of convergence at , both in expectation and with a high probability.
To exploit the structure (for example the sparisty) of the optimization problem, we propose a proximal variant of the Epro-SGD method, namely Epro-ORDA, which utilizes an existing optimal dual averaging method to solve the involved proximal mapping. Our analysis shows that Epro-ORDA similarly requires only a logarithmic number of projections while enjoys an optimal rate of convergence.
For illustration we apply the proposed Epro-SGD methods on two real-world applications, i.e., the constrained Lasso formulation and the large margin nearest neighbor (LMNN) classification. Our experimental results demonstrate the efficiency of the proposed methods, in comparison to the existing methods.
2 Related Work
The present work is inspired from the break-through work in , which proposed two novel one-projection-based stochastic gradient descent (OneProj) methods for stochastic convex optimizations. Specifically the first OneProj method was developed for general convex optimization; it introduces a regularized Lagrangian function as
then applies SGD to the convex-concave problem , and finally performs only one projection at the end of all iterations, where is a bounded ball subsuming as a subset.
The second OneProj method was developed for strongly convex optimization. The proposed method introduced an augmented objective function
where is a parameter dependent on the total number of iterations , and is a problem specific parameter . OneProj applies SGD to the augmented objective function, specifically using a stochastic subgradient of and a subgradient of , and then performs a projection step after all iterations. For a total number iterations, the OneProj method achieves a rate of convergence at , which is suboptimal for stochastic strongly convex optimization.
Several recent works [15, 26] propose optimal methods with optimal rates of convergence at for stochastic strongly convex optimization. In particular, the Epoch-SGD method  consists of a sequence of epochs, each of which has a geometrically decreasing step size and a geometrically increasing iteration number. This method however needs to project the intermediate solutions onto a feasible set at every algorithmic iteration; when the involved constraint is complex, the involved projection is usually computationally expensive. This limitation restricts the practical applications on large scale data analysis. Therefore we are motivated to develop an optimal stochastic algorithm for strongly convex optimization but with a constant number of projections.
Another closely related work is the logT-SGD  for stochastic strongly convex and smooth optimization. LogT-SGD achieves an optimal rate of convergence, while requires to perform projections, where is the ratio of the smoothness parameter to the strong convexity parameter. There are several key differences between our proposed Epro-SGD method and logT-SGD: (i) logT-SGD and its analysis rely on both the smoothness and the strong convexity of the objective function; in contrast, Epro-SGD only assumes that the objective function is strongly convex; (ii) the number of the required projections in logT-SGD is , where the conditional number can be very large in real applications; in contrast, Epro-SGD requires at most projections.
Besides reducing the number of projections in SGD, another line of research is based on the conditional gradient algorithms [7, 14, 17, 18, 32]; this type of algorithms mostly build upon the Frank-Wolfe technique , which eschews the projection in favor of a linear optimization step; however in general, they require the smoothness assumption in the objective function. On the other hand, [12, 16] extended Frank-Wolfe techniques to stochastic or online setting for general and strongly convex optimizations. Specifically  presents an online/stochastic Frank-Wolfe (OFW) algorithm with a convergence rate for general convex optimization problems, which is slower than the optimal rate .  presents an algorithm for online strongly convex optimization with an regret bound, implying an convergence rate for stochastic stronlgy convex optimization. This algorithm requires the problem domain to be a polytope, instead of a convex inequality constraint used in this paper; it also hinges on an efficient local linear optimization oracle that amounts to approximately solving a linear optimization problem over an intersection of a ball and and the feasible domain; furthermore the convergence result only holds in expectation and is sub-optimal.
3 Epoch-Projection Sgd Algorithm
In this section, we present an epoch-projection SGD method, called Epro-SGD, for solving Eq. (1) and discuss its convergence result. Based on a stochastic dual averaging algorithm, we then present a proximal variant of the proposed Epro-SGD method.
3.1 Setup and Background
Denote the optimal solution to Eq. (1) by and its domain set by . Since is -strongly convex  and is convex, the optimization problem in Eq. (1) is strongly convex. Note that the strong convexity in implies that for any . Our analysis is based on the following assumptions:
The stochastic subgradient is uniformly bounded by , i.e., .
The subgradient is uniformly bounded by , i.e., for any .
There exists a positive value such that
Remarks Assumptions A1 and A2 respectively impose an upper bound on the stochastic subgradient of the objective function and the constraint function . Assumption A3 ensures that the projection of a point onto a feasible domain does not deviate too much from this intermediate point. Note that Assumption A1 is previously used in ; a condition similar to Assumption A3 is used in , which however simply assumes that , without considering possible non-differentiability in .
A key consequence of Assumption A3 is presented in the following lemma.
For any , let . If Assumption A3 holds, then
where is a hinge operator defined as if , and otherwise.
If , we have ; the inequality in Eq. (6) trivially holds. If , we can verify that , and there exists and such that (using duality theory). It follows that (), and thus is the same direction as . It follows that
where the last inequality uses Assumption A3. This completes the proof of this lemma. ∎
The result in Lemma 1 is closely related to the polyhedral error bound condition [13, 31]; this condition shows that the distance of a point to the optimal set of a convex optimization problem whose epigraph is a polyhedron is bounded by the distance of the objective value at this point to the optimal objective value scaled by a constant. For illustration, we consider the optimization problem
with an optimal set as . If , is the closest point in the optimal set to . Therefore, by the polyhedral error bound condition of a polyhedral convex optimization, if is a polyhedral function, there exists a such that
Below we present three examples in which Assumption A3 or Lemma 1 is satisfied. Example : an affine function with . Example : the norm constraint where . Example : the maximum of a finite number of affine functions satisfying Lemma 1 as well as the polyhedral error bound condition .
3.2 Main Algorithm
To solve Eq. (1) (using Epro-SGD), we introduce an augmented objective function by incorporating the constraint function as
It is worth noting that the augmented function in Eq. (7) does not have any iteration-dependent parameter, for example the parameter in Eq. (4). is a prescribed parameter satisfying , as illustrated in Lemma 2.
The details of our proposed Epro-SGD algorithm is presented in Algorithm 1. Similar to Epoch-SGD , Epro-SGD consists of a sequence of epochs, each of which has a geometrically decreasing step size and a geometrically increasing iteration number (Line in Algorithm 1). The updates in every intra-epoch (Line - ) are standard SGD steps applied to the augmented objective function with . Epro-SGD is different from Epoch-SGD in that the former computes a projection only at the end of each epoch, while the latter computes a projection at each iteration. Consequently, when the projection step is computationally expensive (e.g., projecting onto a positive definite constraint), Epro-SGD may require much less computation time than Epoch-SGD.
Under Assumptions A1A3, if we apply the update for a number of iterations, the following equality holds
Let and denote by the expectation conditioned on the randomness until round . It is easy to verify that , and . For any , we have
where . Furthermore by the convexity of , we have
Noting that , taking expectation over randomness and summation over , we have
Let . Since , we have
It follows that
If , we have . Following from and , we can verify that and also holds.
Next we show that holds when . From Lemma 1, we have
Moreover it follows from and that the following inequality holds
By some rearrangement, we have . Furthermore we have
where the second inequality follows from , and for any . This completes the proof of the lemma. ∎
We present a main convergence result of the Epro-SGD algorithm in the following theorem.
Under Assumptions A1A3 and given that is -strongly convex, if we let , , and set , the total number of epochs in Algorithm 1 is given by
the solution enjoys a convergence rate of
From the updating rule , we can easily verify Eq. (12). Since , the inequality trivially holds.
Let . It follows that and . Next we show the inequality
holds by induction. Note that Eq. (14) implies , due to . Let . It follows from Lemma 5, , and , the inequality in Eq. (14) holds when . Assuming that Eq. (14) holds for , we show that Eq. (14) holds for .
For a random variable measurable with respect to the randomness up to epoch . Let denote the expectation conditioned on all the randomness up to epoch . Following Lemma 2, we have
Since by the strong convexity in , we have
which completes the proof of this theorem. ∎
Remark We compare the obtained main results in Theorem 1 with several existing works. Firstly Eq. (13) implies that Epro-SGD achieves an optimal bound , matching the lower bound for a strongly convex problem . Secondly in contrast to the OneProj method  with a convergence rate , Epro-SGD uses no more than projections to obtain an convergence rate. Epro-SGD thus has better control over the solution for not deviating (too much) from the feasibility domain in the intermediate iterations. Thirdly compared to Epoch-SGD with its convergence rate bounded by , the convergence rate bound of Epro-SGD is only worse by a factor of constant . Particularly consider a positive definite constraint with , , and , we have and the bound of Epro-SGD is only worse by a factor of than Epoch-SGD. Finally compared to the logT-SGD algorithm  which requires projections ( is the conditional number), the number of projections in Epro-SGD is independent of the conditional number.
The main results in Lemma 2 and Theorem 1 are expected convergence bounds. In Theorem 2 (proof provided in Appendix) we show that Epro-SGD also enjoys a high probability bound under a boundedness assumption, i.e., for all . Note that the existing Epoch-SGD method  uses two different methods to derive its high probability bounds. Specifically the first method relies on an efficient function evaluator to select the best solutions among multiple trials of run; while the second one modifies the updating rule by projecting the solution onto the intersection of the domain and a center-shifted bounded ball with decaying radius. These two methods however may lead to additional computation steps, if being adopted for deriving high probability bounds for Epro-SGD.
Under Assumptions A1A3 and given for all . If we let , , and set , , the total number of epochs in Algorithm 1 is given by
and the final solution enjoys a convergence rate of
with a probability at least , where .
Remark The assumption can be satisfied practically, if we estimate the value of such that , and then project the intermediate solutions onto at every iteration. Note that Epoch-SGD  requires a total number of projections, and its high probability bound of Epoch-SGD is denoted by with a probability at lest , where .
3.3 A Proximal Variant
We propose a proximal extension of Epro-SGD, by exploiting the structure of the objective function. Let the objective function in Eq. (1) be a sum of two components
where is a relatively simple function, for example a squared -norm or -norm, such that the involved proximal mapping
is easy to compute. The optimization problem in Eq. (1) can be rewritten as
Denote by the optimal solution to Eq. (15). We similarly introduce an augmented objective function as
If is a sparse regularizer, the proximal SGD can guarantee the sparsity in the intermediate solutions and usually yields better convergence than the standard SGD. However, given a complex constraint, solving the proximal mapping may be computational expensive. Therefore, we consider a proximal variant of Epro-SGD which involves only the proximal mapping of without the constraint . An instinctive solution is to use the following update in place of step 6 in Algorithm 1:
Under Assumptions A1A3 and setting , by applying the update in Eq. (18) a number of iterations, we have
where , and denotes the projected solution of the averaged solution .
Different from the main result in Lemma 2, Eq. (19) has an additional term ; it makes the convergence analysis in Epro-SGD difficult. To overcome this difficulty, we adopt the optimal regularized dual averaging (ORDA) algorithm  for solving Eq. (16). The details of ORDA are presented in Algorithm 2. The main convergence results of ORDA are summarized in the following lemma (proof provided in Appendix).
Under Assumptions A1A3 and setting , by running ORDA a number of iterations for solving the augmented objective (16), we have
where denotes the projected solution of the final solution .
We present a proximal variant of Epro-SGD, namely Epro-ORDA, in Algorithm 3, and summarize its convergence results in Theorem 3. Note that Algorithm 2 and the convergence analysis in Lemma 4 are independent of the strong convexity in ; the strong convexity is however used for analyzing the convergence of Epro-ORDA in Theorem 3 (proof provided in Appendix).
Under Assumptions A1A3 and given that is -strongly convex, if we let and , and set , then the total number of epochs in Algorithm 3 is given by
and the final solution enjoys a convergence rate of
4 An Example of Solving Lmnn via Epro-Sgd
In this section, we discuss an application of applying the proposed Epro-SGD to solve a high dimensional distance metric learning (DML) with a large margin formulation, i.e., the large margin nearest neighbor (LMNN) classification method . LMNN classification is one of the state-of-the-art methods for k-nearest neighbor classification. It learns a positive semi-definite distance metric, based on which the examples from the k-nearest neighbors always belong to the same class, while the examples from different classes are separated by a large margin.
To describe the LMNN method, we first present some notations. Let , be a set of data points, where and denote the feature representation and the class label, respectively. Let be a positive definite matrix that defines a distance metric as . To learn a distance metric that separates the examples from different classes by a large margin, one needs to extract a set of similar examples (from the same class) and dissimilar examples (from a different class), denoted by , where shares the same class label to and a different class from . To this end, for each example one can form by extracting the k nearest neighbors (defined by an Euclidean distance metric) that share the same class label to , and form by extracting a set of examples that have a different class label. Then an appropriate distance metric could be obtained from the following constrained optimization problem
where is a hinge loss and is a trade-off parameter. In Eq. (4), is used as the constraint to ensure that Assumption A3 holds. Minimizing the first term is equivalent to maximizing the margin between and . The matrix encodes certain prior knowledge about the distance metric; for example, the original LMNN work  defines as , where are all k-nearest neighbor pairs from the same class. Other works  have used a weighted summation of distances between all data pairs or intra-class covariance matrix . The last term is used as a regularization term and also makes the objective function strongly convex.
For data sets of very high dimensionality, i.e., , LMNN in Eq. (4) usually produces a sub-optimal solution , as this formulation does not capture the sparsity structure of the features. Therefore we add a sparse regularizer and express the formulation below
where is an elmenent-wise -norm excluding the diagonal entries. Note that this sparse regularizer have been previously used in  for a different purpose.
Many standard optimization solvers or algorithms may not be efficient for solving Eq. (4). Firstly, the optimization problem in Eq. (4) can be formulated as a semi-definite program (SDP); however, general SDP solvers usually scale poorly with the number of triplets and is not suitable for large scale data analysis. Secondly, the gradient decent method presented in  requires to project intermediate solutions onto a positive definite cone; this operation invokes expensive singular value decomposition (SVD) for a large matrix and this limitation restricts the real-world applications of the gradient descent method. Thirdly,  employs a block coordinate descent (BCD) method to solve an L-penalized log-det optimization problem; the BCD method is not suitable for solving Eq. (4), as the loss function is not linear in the variable .
We employ the proposed Epro-SGD algorithm to solve the LMNN formulation in Eq. (4). Let and . The positive definite constraint can be rewritten into an inequality constraint as , where denotes the minimum eigen-value of the matrix . We also make the correspondences , , , and provide necessary details below in a question-answer form
How to compute the stochastic gradient of ? First sample one triplet (or a small number of triplets) and then compute if