Generalization Error Bounds for Optimization Algorithms via Stability

# Generalization Error Bounds for Optimization Algorithms via Stability

Qi Meng1, Yue Wang2, Wei Chen3, Taifeng Wang3, Zhi-Ming Ma4, Tie-Yan Liu3
1 School of Mathematical Sciences, Peking University, qimeng13@pku.edu.cn
2Beijing Jiaotong University, 11271012@bjtu.edu.cn
3Microsoft Research, wche, taifengw, tie-yan.liu@microsoft.com
###### Abstract

Many machine learning tasks can be formulated as Regularized Empirical Risk Minimization (R-ERM), and solved by optimization algorithms such as gradient descent (GD), stochastic gradient descent (SGD), and stochastic variance reduction (SVRG). Conventional analysis on these optimization algorithms focuses on their convergence rates during the training process, however, people in the machine learning community may care more about the generalization performance of the learned model on unseen test data. In this paper, we investigate on this issue, by using stability as a tool. In particular, we decompose the generalization error for R-ERM, and derive its upper bound for both convex and non-convex cases. In convex cases, we prove that the generalization error can be bounded by the convergence rate of the optimization algorithm and the stability of the R-ERM process, both in expectation (in the order of , where is the convergence error and is the number of iterations) and in high probability (in the order of with probability ). For non-convex cases, we can also obtain a similar expected generalization error bound. Our theorems indicate that 1) along with the training process, the generalization error will decrease for all the optimization algorithms under our investigation; 2) Comparatively speaking, SVRG has better generalization ability than GD and SGD. We have conducted experiments on both convex and non-convex problems, and the experimental results verify our theoretical findings.

Generalization Error Bounds for Optimization Algorithms via Stability

Qi Meng1, Yue Wang2, Wei Chen3, Taifeng Wang3, Zhi-Ming Ma4, Tie-Yan Liu3 1 School of Mathematical Sciences, Peking University, qimeng13@pku.edu.cn 2Beijing Jiaotong University, 11271012@bjtu.edu.cn 3Microsoft Research, wche, taifengw, tie-yan.liu@microsoft.com 4Academy of Mathematics and Systems Science, Chinese Academy of Sciences, mazm@amt.ac.cn

## 1 Introduction

Many machine learning tasks can be formulated as Regularized Empirical Risk Minimization (R-ERM). Specifically, given a training dataset, the goal of R-ERM is to learn a model from a hypothesis space by minimizing the regularized empirical risk defined as the average loss on the training data plus a regularization term.

While the aforementioned convergence analysis can characterize the behaviors of the optimization algorithms in the training process, what the machine learning community cares more is the generalization performance of the learned model on unseen test data. 111Under a related but different setting, i.e., the data instances are successively generated from the underlying distribution, people have proven regret bounds for algorithms like SGD (??) and SVRG (?). As we know, the generalization error of a machine learning algorithm can be decomposed into three parts, the approximation error, the estimation error, and the optimization error. The approximation error is caused by the limited representation power of the hypothesis space ; the estimation error (which measures the difference between the empirical risk and the expected risk) is caused by the limited amount of training data (?)(?); and the optimization error (which measures the difference between expected risks of the model obtained by the optimization algorithm after iterations and the true optimum of the regularized empirical risk) is caused by the limited computational power. In (?), Bottou and Bousquet proved generalization error bounds for GD and SGD based on VC-dimension (?), which unavoidably are very loose in their nature.222In (?), Hardt studied convex risk minimization via stability, but they did not consider the influence of hypothesis space and the tradeoff between approximation error and estimation error. The goal of our paper is to develop more general and tighter generalization error bounds for the widely used optimization algorithms in R-ERM.

To this end, we leverage stability (?) as a tool and obtain the following results:

(1) For convex objective functions, we prove that, the generalization error of an optimization algorithm can be upper bounded by a quantity related to its stability plus its convergence rate in expectation. Specifically, the generalization error bound is in the order of , where is the optimization convergence error and is the number of iterations. This indicates that along with the optimization process on the training data, the generalization error will decrease, which is consistent with our intuition.

(2) For convex objective functions, we can also obtain a high probability bound for the generalization error. In particular, the bound is in the order of with probability at least . That is, if an algorithm has a high-probability convergence bound, we can get a high-probability generalization error bound too, and our bound is sharper than those derived in the previous literature.

(3) Based on our theorems, we analyze the time for different optimization algorithms to achieve the same generalization error, given the same amount of training data. We find that SVRG outperforms GD and SGD in most cases, and although SGD can quickly reduce the test error at the beginning of the training process, it slows down due to the decreasing step size and can hardly obtain the same test error as GD and SVRG when is large.

(4) Some of our theoretical results can be extended to the nonconvex objective functions, with some additional assumptions on the distance between the global minimizer and the stationary local minimizers.

We have conducted experiments on linear regression, logistic regression, and fully-connected neural networks to verify our theoretical findings. The experimental results are consistent with our theory: (1) when the training process goes on, the test error decreases; (2) in most cases, SVRG has better generalization performance than GD and SGD.

## 2 Preliminaries

In this section, we briefly introduce the R-ERM problem, and popular optimization algorithms to solve it.

### 2.1 R-ERM and its Stability

Suppose that we have a training set with instances that are i.i.d. sampled from according to an unknown distribution . The goal is to learn a good prediction model , whose prediction accuracy at instance is measured by a loss function . Different learning tasks may use different loss functions, such as the least square loss for regression, and the logistic loss for classification. We learn the prediction model from the training set , and will use this model to give predictions for unseen test data.

R-ERM is a very common way to achieve the above goal. Given loss function , we aim to learn a model that minimizes the expected risk

 R(f)=Ez∼Pl(f,z).

Because the underlying distribution is unknown, in practice, we learn the prediction model by minimizing the regularized empirical risk over the training instances, which is defined as below,

 RrS(f)=1nn∑i=1l(f,zi)+λN(f). (1)

Here, the regularization term helps to restrict the capacity of the hypothesis space to avoid overfitting. In this paper, we consider as a norm in a reproducing kernel Hilbert space (RKHS): where refers to the kernel (?).

As aforementioned, our goal is expected risk minimization but what we can do in practice is empirical risk minimization instead. The gap between these two goals is measured by the so-called estimation error, which is usually expressed in the following way: the expected risk is upper bounded by the empirical risk plus a quantity related to the capacity of the hypothesis space (?)(?). One can choose different ways to measure the capacity of the hypothesis space, and stability is one of them, which is proved to be able to produce tighter estimation error bound than VC dimension (?). There has been a venerable line of research on estimation error analysis based on stability, dated back more than thirty years ago (?????). The landmark work by Bousquet and Elisseeff (?) introduced the following definitions of uniform loss stability and output stability.

###### Definition 2.1

(Uniform Loss Stability) An algorithm has uniform stability with respect to loss function if the following holds

 ∣∣EA[l(AS,⋅)]−EA[l(AS∖j,⋅)]∣∣≤β0, (2)

where are the outputs of algorithm based on and , respectively.

###### Definition 2.2

(Output Stability) An algorithm has output stability if the following holds

 ∥AS−AS∖j∥Fc≤β1, (3)

where denotes the norm in hypothesis space .

From the above definitions, we can see that stability measures the change of the loss function or the produced model of a given learning algorithm if one instance in the training set is changed. For example, if the loss function is convex and -Lipschitz w.r.t. , the corresponding R-ERM algorithm with regularization term has stability and , where is the upper bound of the kernel norm (?).

### 2.2 Optimization Algorithms

Many optimization methods can be used to solve the R-ERM problem, including the first-order methods such as Gradient Descent (GD) (?), Stochastic Gradient Descent (SGD) (?), and Stochastic Variance Reduction (SVRG) (?), as well as the second-order methods such as Newton’s methods (?) and quasi-Newton’s methods (?). We will take the first-order methods as examples in this paper, although many of our analysis can be easily extended to other optimization algorithms.

Let us consider model parameterized by . The update rules of GD, SGD, and SVRG are summarized as follows. Gradient Descent (GD)

 wt+1=wt−η∇RrS(wt). (4)

 wt+1=wt−ηtg(wt). (5)

 vts=g(wts)−∇RrS(wts)+∇RrS(~wt−1) (6) wts+1=wts−ηvts. (7)

where is the gradient of at randomly sampled training instances, is the output parameter at the -th iteration in the -th stage, and is the final output in stage .

When the loss function is strongly convex and smooth with respect to the model parameters, GD can achieve linear convergence rate; SGD can only achieve sublinear convergence rate due to the variance introduced by stochastic sampling (but in each iteration, it only needs to compute the gradient over one instance and thus can be much faster in speed); SVRG can achieve linear convergence rate by reducing the variance and in most iterations it only needs to compute the gradient over one instance. 333The second-order methods can get quadratic convergence rate (?). However, as compared with the first-order methods, the computation complexity of the second-order methods could be much higher due to the calculation of the second-order information. When the loss functions are nonconvex w.r.t. the model parameters (e.g., neural networks), GD (?), SGD (?), and SVRG (?) still have convergence properties (although regarding a different measure of convergence). For ease of reference, we summarize the convergence rates of the aforementioned optimization algorithms in both convex and nonconvex cases in Table 1.

## 3 Generalization Analysis

In this section, we will analyze the generalization error for optimization algorithms by using stability as a tool. Firstly, we introduce the definition of generalization error and its decomposition. Then, we prove the generalization error bounds of optimization algorithms in both convex and nonconvex cases. The proof details of all the lemmas and theorems are placed in the supplementary materials due to space limitation.

### 3.1 Generalization Error and its Decomposition

As we mentioned in Section 2, R-ERM minimizes the regularized empirical risk, i.e.,

 f∗S,r:=argminf∈FRrS(f) (8)

as an approximation of the expected risk minimization:

 f∗:=argminfR(f). (9)

Denote the empirical risk . It is clear that, the minimization of in is equivalent to the minimization of in for some constant . That is,

 f∗S,r=f∗S,Fc:=argminf∈FcRS(f). (10)

Denote the minimizer of the expected risk in the hypothesis space as , i.e.,

 f∗Fc:=argminf∈FcR(f). (11)

In many practical cases, neither nor has a closed form. What people do is to implement an iterative optimization algorithm to produce the prediction model. We denote the output model of algorithm at iteration over training instances as . We use generalization error to denote the difference between the expected risk of this learnt model and the optimal expected risk, as follows,

 E(A,n,Fc,T)=R(fT(A,n,Fc))−R(f∗). (12)

As known, the generalization error can be decomposed into the three components,

 E(A,n,Fc,T) (13) = R(fT)−R(f∗S,Fc)+R(f∗S,Fc)−R(f∗Fc) +R(f∗Fc)−R(f∗) := Eopt(A,n,Fc,T)+Eest(n,Fc)+Eapp(Fc). (15)

The item , is called approximation error, which is caused by the limited representation power of the hypothesis space . With the hypothesis space increasing, (i.e., is increasing), the approximation error will decrease. The item , is called estimation error, which is caused by the limited amount of the training data (which leads to the gap between the empirical risk and the expected risk). It will decrease with the increasing training data size , and the decreasing capacity of the hypothesis space . The item , is called optimization error, which measures the sub-optimality of the optimization algorithms in terms of the expected risk. It is caused by the limited computational resources. 444For simplicity, we sometimes denote , , , , as , , , , , respectively.

Please note that, the optimization error under our study differs from the target in the conventional convergence analysis of optimization algorithms. In the optimization community, the following two objectives

 ρ0(T)=RS(fT)−RS(f∗S,Fc);ρ1(T)=∥fT−f∗S,Fc∥2Fc (16)

are commonly used in convex cases, and

 ρ2(T)=∥∇RrS(fT)∥2 (17)

is commonly used in nonconvex cases. To avoid confusion, we call them convergence error and their corresponding upper bounds convergence error bounds. Please note although convergence error is different from optimization error, having a convergence error bound plays an important role in guaranteeing a generalization error bound. In the following subsections, we will prove the generalization error bound for typical optimization algorithms, by using the stability techniques, based on their convergence error bounds.

### 3.2 Expected Generalization Bounds for Convex Case

The following theorem gives an expected generalization error bounds in the convex case.

###### Theorem 3.1

Consider an R-ERM problem, if the loss function is -Lipschitz continuous, -smooth, and convex with respect to the prediction output vector, we have

 ES,AE ≤ Eapp+2β0+ES,Aρ0(T)+γES,Aρ1(T)2 (18) +√ES,Aρ1(T)(L22n+6Lγβ1),

where are the uniform stability and output stability of the R-ERM process as defined in 2.1 and 2.2, and are the convergence errors defined in Eqn 16.

From Theorem 3.1, we can see that the generalization error can be upper bounded by the stability and , the convergence errors of the optimization algorithms and , and the well-studied approximation error (?). As the training process goes on, both and will decrease. Therefore, the expected generalization error will decrease too. This is consistent with our intuition. Better optimizations will lead to better expected generalization performance.

In order to prove Theorem 3.1, we need the following two lemmas, whose proofs are placed in the supplementary materials due to space restrictions.

###### Lemma 3.2

For R-ERM problems, we have :

 ES[R(f∗S,Fc)−RS(f∗S,Fc)]=ES[l(f∗S,Fc,z′j)−l(f∗Sj,Fc,z′j)] (19)

and

 ES[∇R(f∗S,Fc)−∇RS(f∗S,Fc)]=ES[∇fl(f∗S,Fc,z′j)−∇fl(f∗Sj,Fc,z′j)], (20)

where , and is the minimizer of in .

###### Lemma 3.3

Assume that the loss function is -Lipschitz and -smooth w.r.t. the prediction output vector, we have

 ES[∇R(f∗S,Fc)−∇RS(f∗S,Fc)]2≤L22n+6Lγβ1. (21)

Proof Sketch of Theorem 3.1:

Step 1: Since the loss function is convex and -smooth w.r.t. , we can get that is -smooth and is convex w.r.t . We decompose as below:

 Eopt ≤ (∇R(f∗S,Fc)−∇RS(f∗S,Fc))T(fT−f∗S,Fc) +RS(fT)−RS(f∗S,Fc)+γ2∥fT−f∗S,Fc∥2Fc,

We can use , and Lemma 3.3, to get an upper bound of .

Step 2: Since , we have

 Eest≤[R(f∗S,Fc)−RS(f∗S,Fc)]+[RS(f∗Fc)−R(f∗Fc)].

We have . By using Lemma 3.2, we can bound . By combining the upper bounds of and , we can get the results.

After proving the general theorem, we consider a special case - an R-ERM problem with kernel regularization term . In this case, we can derive the concrete expressions of the stability and convergence error. In particular, , and is equivalent to . If the loss function is convex and smooth w.r.t. parameter , with is strongly convex and smooth w.r.t . In this case, dominates , i.e., is larger than w.r.t the order of . Therefore, we can obtain the following corollary.

###### Corollary 3.4

For an R-ERM problem with a regularization term , under the same assumptions in Theorem 3.1, and further assuming that the loss function is convex and smooth w.r.t parameter , we have

 (22)

### 3.3 High-Probability Generalization Bounds for Convex Case

The following theorem gives a high-probability bound of in the convex case. Due to space limitation, we put the proof in the supplementary materials.

###### Theorem 3.5

For an R-ERM problem, if the loss function is L-Lipschitz continuous, -smooth and convex with respect to the prediction output vector, and for arbitrary and , then with probability at least , we have

 E ≤ Eapp+2β0+ρ0(T)+γ2ρ1(T)+2γβ1√ρ1(T) +(4nβ0+2M+(4nγβ1+L)√ρ1(T))√ln4/δ2n.

The high-probability bound is consistent with the expected bound given in the previous subsection. That is, the high-probability generalization bound will also decrease along with the training process. In addition, we can also get a corollary for the special case of R-ERM with kernel regularization.

###### Corollary 3.6

For an R-ERM problem with kernel regularization term , under the same assumptions in Theorem 3.5, and further assuming that the loss function is convex and smooth w.r.t parameter , we have, with probability at least ,

 E ≤ Eapp+O⎛⎝√log1/δn+ρ0(T)⎞⎠.

Rakhlin et.al. (?) proved a high-probability convergence rate for SGD. For GD, the training process is deterministic. By plugging the order of and in SGD and GD, we have the following corollary.

###### Corollary 3.7

For an R-ERM problem with kernel regularization, under the assumptions in Corollary 3.6, with probability at least , the generalization error of SGD and GD can be upper bounded as follows,

 ESGD ≤ Eapp+O⎛⎝√ln1/δn⎞⎠+O(κ2log(log(T)/δ)T); EGD ≤ Eapp+O⎛⎝√ln1/δn⎞⎠+O(e−κT),

where is the condition number.

### 3.4 Expected Generalization Bounds for Nonconvex Case

In this subsection, we consider the case in which the loss function is convex w.r.t. the prediction output vector, but non-convex w.r.t. the model parameter. This case can cover deep neural networks, which are state-of-the-art AI techniques nowadays.

For the non-convex case, the definition of convergence error is a little different, as shown by Eq. (17). It measures whether the solution is close to a critical point, which is defined and further categorized as follows.

###### Definition 3.8

Consider the objective and parameter . If , we say is a critical point of ; if has at least one strictly negative eigenvalue, we say is a strict saddle point. If each critical point is either a local minimum or a strict saddle point, we say that satisfies the strict saddle property.

The following theorem gives the expected generalization error bound for non-convex cases under the widely used assumptions (???).

###### Theorem 3.9

If is -strongly convex in the - neighborhood of arbitrary local minimum , satisfies strict saddle point property, - Lipschitz continuous, -smooth and continuously twice differential w.r.t the model parameter , and the loss function is convex w.r.t , then we have

 ES,AE≤Eapp+2β0+R(wloc)−R(w∗S,Fc)+Lμ√mint=1,⋯,TES,Aρ2(t),

where and is the number of iterations to achieve .

Similarly to the convex case, from the above theorem we can see that with the training process going on, the generalization error in the nonconvex case will also decrease. In addition, we can also derive specific bound for the R-ERM with kernel regularization.

## 4 Sufficient Training and Optimal Generalization Error

In this section, we make further discussions on the generalization bound. In particular, we will explore the sufficient training iterations, and the optimal generalization error given the training data size.

As shown in Section 3, the generalization error bounds consist of an estimation error related to the training data size and an optimization error related to the training iteration . Given a machine learning task with fixed training size , at the early stage of the training process (i.e., is relatively small), the optimization error will dominate the generalization error; when becomes larger than a threshold, the optimization error will decrease to be smaller than the estimation error (i.e. ), and then the estimation error will dominate the generalization error. We call this threshold sufficient training iteration and the corresponding training time sufficient training time. The generalization error with the optimization algorithm sufficiently trained is called optimal generalization error. Given the generalization error bound, we can derive the sufficient training iteration/time. For ease of analysis, we list the sufficient training iteration/time of GD, SGD, and SVRG for both convex and nonconvex cases in Table 2.

From Table 2, we have the following observations. For the convex case, when the condition number is much smaller than , GD, SGD and SVRG have no big differences from each other in their sufficient training iterations; when is comparable with , e.g., , 555In some cases, is related to the regularization coefficient and is determined by the data size (?)(?). the sufficient training time for GD, SGD and SVRG is , , , respectively. That is, SVRG corresponds to a shorter sufficient training time than GD and SVRG. For the non-convex case, if , which is more likely to happen for small data size , the first term in the sufficient training time dominates, and it is fine to terminate the training process at . SVRG requires shorter training time than GD and SGD by at least an order of and , respectively. If is larger than , which is more likely to happen for large data size , the sufficient training time for GD, SGD, and SVRG is , , and , respectively. In this case, SVRG requires shorter training time than GD and SGD by an order of and , respectively.

## 5 Experiments

In this section, we report experimental results to validate our theoretical findings. We conducted experiments on three tasks: linear regression, logistic regression, and fully connected neural networks, whose objective functions are least square loss, logistic loss, and cross-entropy loss respectively, plus an regularization term with . The first two tasks are used to verify our results for convex problems, and the third task is used to verify our theory on nonconvex problems. For each task, we report three figures. The horizontal axis of each figure corresponds to the number of data passes and the vertical axis corresponds to the training loss, test loss, and log-scaled test loss, respectively. For linear regression, we independently sample data instances with size from a dimension Gaussian distribution. We use half of them as the training data and the other as the test data. We set the step size for GD, SGD, SVRG as , and , respectively, according to the smoothness and strong-convexity coefficients. For our simulated data, the condition number . The results are shown in Fig.111. For logistic regression, we conduct binary classification on benchmark dataset rcv1. We set the step sizes for GD, SGD, SVRG as , and , respectively. The results are shown in Fig. 111. For neural networks, we work on a model with one fully connected hidden layer of 100 nodes, ten softmax output nodes, and sigmoid activation (?). We tune the step size for GD, SGD, SVRG and eventually choose , and , respectively, which correspond to the best performances in our experiments. The inner loop size for SVRG for convex problems is set as and that for nonconvex problem is set as . The results are shown in Fig.111.

From the results for all the three tasks, we have the following observations. (1) As training error decreases, the test error also decreases. (2) According to Fig.1, SVRG is faster than GD by a factor of and faster than SGD by a factor of more than . (3) According to Fig. 111, SGD is the slowest although it is fast in the beginning, which is consistent with our discussions in Section 4.

By comparing the results of logistic regression and linear regression, we have the following observations. (1) The test error for logistic regression converges after fewer rounds of data passes than linear regression. This is because the condition number for logistic regression is smaller than linear regression. (2) SVRG is faster than GD and SGD but the differences between them are less significant for logistic regression, due to a smaller . As compared to the results for logistic regression and linear regression, we have the following observations on the results of neural networks. (1) The convergence rate is slower and the accuracy is lower. This is because of the nonconvexity and the gap between global optimum and local optimum. (2) SVRG is faster than GD and SGD but the differences between them are not as significant as in the convex cases, which is consistent with our discussions in Section 4 by considering the data size of CIFAR 10.

## 6 Conclusion

In this paper, we have studied the generalization error bounds for optimization algorithms to solve R-ERM problems, by using stability as a tool. For convex problems, we have obtained both expected bounds and high-probability bounds. Some of our results can be extended to the nonconvex case. Roughly speaking, our theoretical analysis has shown: (1) Along with the training process, the generalization error will decrease; (2) SVRG outperforms GD and SGD in most cases. We have verified the theoretical findings by using experiments on linear regression, logistic regression and fully connected neural networks. In the future, we plan to study the stability of R-ERM with other regularization terms, e.g., the regularizer, which is usually associated with non-smooth optimization methods.

## References

• [Bousquet and Bottou 2008] Bousquet, O., and Bottou, L. 2008. The tradeoffs of large scale learning. In Advances in neural information processing systems, 161–168.
• [Bousquet and Elisseeff 2002] Bousquet, O., and Elisseeff, A. 2002. Stability and generalization. Journal of Machine Learning Research 2(Mar):499–526.
• [Byrd et al. 2016] Byrd, R. H.; Hansen, S.; Nocedal, J.; and Singer, Y. 2016. A stochastic quasi-newton method for large-scale optimization. SIAM Journal on Optimization 26(2):1008–1031.
• [Cesa-Bianchi, Conconi, and Gentile 2004] Cesa-Bianchi, N.; Conconi, A.; and Gentile, C. 2004. On the generalization ability of on-line learning algorithms. IEEE Transactions on Information Theory 50(9):2050–2057.
• [Devroye and Wagner 1979] Devroye, L., and Wagner, T. 1979. Distribution-free performance bounds for potential function rules. IEEE Transactions on Information Theory 25(5):601–604.
• [Frostig et al. 2015] Frostig, R.; Ge, R.; Kakade, S. M.; and Sidford, A. 2015. Competing with the empirical risk minimizer in a single pass. In Conference on Learning Theory.
• [Ghadimi and Lan 2013] Ghadimi, S., and Lan, G. 2013. Stochastic first-and zeroth-order methods for nonconvex stochastic programming. SIAM Journal on Optimization 23(4):2341–2368.
• [Hardt, Recht, and Singer 2015] Hardt, M.; Recht, B.; and Singer, Y. 2015. Train faster, generalize better: Stability of stochastic gradient descent. arXiv preprint arXiv:1509.01240.
• [Johnson and Zhang 2013] Johnson, R., and Zhang, T. 2013. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in Neural Information Processing Systems, 315–323.
• [Kakade and Tewari 2009] Kakade, S. M., and Tewari, A. 2009. On the generalization ability of online strongly convex programming algorithms. In Advances in Neural Information Processing Systems, 801–808.
• [Kearns and Ron 1999] Kearns, M., and Ron, D. 1999. Algorithmic stability and sanity-check bounds for leave-one-out cross-validation. Neural Computation 11(6):1427–1453.
• [Lee et al. 2016] Lee, J. D.; Simchowitz, M.; Jordan, M. I.; and Recht, B. 2016. Gradient descent converges to minimizers. University of California, Berkeley 1050:16.
• [Lian et al. 2015] Lian, X.; Huang, Y.; Li, Y.; and Liu, J. 2015. Asynchronous parallel stochastic gradient for nonconvex optimization. In Advances in Neural Information Processing Systems, 2737–2745.
• [Mukherjee et al. 2006] Mukherjee, S.; Niyogi, P.; Poggio, T.; and Rifkin, R. 2006. Learning theory: stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization. Advances in Computational Mathematics 25(1-3):161–193.
• [Nesterov 2013] Nesterov, Y. 2013. Introductory lectures on convex optimization: A basic course, volume 87. Springer Science & Business Media.
• [Nocedal and Wright 2006] Nocedal, J., and Wright, S. 2006. Numerical optimization. Springer Science & Business Media.
• [Rakhlin, Shamir, and Sridharan 2011] Rakhlin, A.; Shamir, O.; and Sridharan, K. 2011. Making gradient descent optimal for strongly convex stochastic optimization. arXiv preprint arXiv:1109.5647.
• [Reddi et al. 2016] Reddi, S. J.; Hefny, A.; Sra, S.; Póczós, B.; and Smola, A. 2016. Stochastic variance reduction for nonconvex optimization. arXiv preprint arXiv:1603.06160.
• [Shalev-Shwartz et al. 2010] Shalev-Shwartz, S.; Shamir, O.; Srebro, N.; and Sridharan, K. 2010. Learnability, stability and uniform convergence. Journal of Machine Learning Research 11(Oct):2635–2670.
• [Shamir, Srebro, and Zhang 2014] Shamir, O.; Srebro, N.; and Zhang, T. 2014. Communication-efficient distributed optimization using an approximate newton-type method. In ICML, volume 32, 1000–1008.
• [Vapnik and Kotz 1982] Vapnik, V. N., and Kotz, S. 1982. Estimation of dependences based on empirical data, volume 40. Springer-Verlag New York.
• [Vapnik and Vapnik 1998] Vapnik, V. N., and Vapnik, V. 1998. Statistical learning theory, volume 1. Wiley New York.
• [Wahba 2000] Wahba, G. 2000. An introduction to model building with reproducing kernel hilbert spaces. Statistics Department TR 1020.

## 7 Appendices

### 7.1 Proofs of Lemma 3.2, Lemma 3.3 and Theorem 3.1

Lemma 3.2: For R-ERM problems, we have :

 ES[R(f∗S,Fc)−RS(f∗S,Fc)]=ES[l(f∗S,Fc,z′j)−l(f∗Sj,Fc,z′j)] (23)

and

 ES[∇R(f∗S,Fc)−∇RS(f∗S,Fc)] (24) = ES[∇fl(f∗S,Fc,z′j)−∇fl(f∗Sj,Fc,z′j)],

where , and is the minimizer of .

Proof:

The proofs of Eq.(23) and Eq.(24) are very similar, we only prove Eq.(24).

 ES[∇RS(f∗S,Fc)] (25) = (26) = 1nn∑j=1ES,z′j[∇fl(f∗S,Fc,zj)] (27) = 1nn∑j=1ES,z′j[∇fl(f∗Sj,Fc,z′j)] (28)

Using the definition of , we can get

 ES∇R(f∗S,Fc)=ES,z∇fl(f∗S,Fc,z)=ES,z′j∇fl(f∗S,Fc,z′j). (29)

By combining Eq.(28) and (29), we can get the results.

Lemma 3.3 : Assume that the loss function is -Lipschitz and -smooth w.r.t. the prediction output vector, we have

 ES[∇R(f∗S,Fc)−∇RS(f∗S,Fc)]2≤L22n+6Lγβ1. (30)

Proof:

The proof is following Lemma 9 and Lemma 25 in (?). We just need to relpalce which is the upper bound of the loss function by the upper bound of the derivative of loss function , and replace the lip By the assumption that the loss function is Lipschitz continous and smooth w.r.t. the prediction output vector, we can get that and is Lipschitz continous. Following Lemma 9 in (?), we have

 ES[∇R(f∗S,Fc)−∇RS(f∗S,Fc)]2 (31) ≤ L22n+3LES,z′j[|∇l(f∗S,Fc,zj)−∇l(f∗Sj,Fc,zj)|] (32) ≤ L22n+3LγES,z′j[∥f∗S,Fc−f∗Sj,Fc∥Fc] (33) ≤ L22n+3LγES,z′j[∥f∗S,Fc−f∗S∖j,Fc∥Fc] (35) +3LγESj,z′j[∥f∗S,Fc−f∗S∖j,Fc∥Fc] ≤ L22n+6Lγβ1.□ (36)

Theorem 3.1 : Consider an R-ERM problem, if the loss function is -Lipschitz continuous, -smooth, and convex with respect to the prediction output vector, we have

 ES,AE ≤ Eapp+2β0+ES,Aρ0(T)+γES,Aρ1(T)2 (37) +√ES,Aρ1(T)(L22n+6Lγβ1),

where are the uniform stability and output stability of the R-ERM process as defined in Def.2.1 and Def.2.2, and are the convergence errors defined in Eqn.(16).

Proof:
If is convex and -smooth w.r.t. , we can get that is -smooth and is convex. First, we decompose as follows:

 Eopt (38) = R(fT)−R(f∗S,Fc) (39) ≤ ∇R(f∗S,Fc)T(fT−f∗S,Fc)+γ2∥fT−f∗S,Fc∥2Fc (40) = (∇R(f∗S,Fc)−∇RS(f∗S,Fc))T(fT−f∗S,Fc) (42) +∇RS(f∗S,Fc)T(fT−f∗S,Fc)+γ2∥fT−f∗S,Fc∥2Fc ≤ (∇R(f∗S,Fc)−∇RS(f∗S,Fc))T(fT−f∗S,Fc) +RS(fT)−RS(f∗S,Fc)+γ2∥fT−f∗S,Fc∥2Fc,

where the first inequality is established by using the -smoothness condition and the third inequality is established by using the convexity condition. Taking expectation w.r.t. and the optimization algorithm , we can get

 ES,Aγ2∥fT−f∗S,Fc∥2 = γ2ES,Aρ1(T) (44) ES,A[RS(fT)−RS(f∗S,Fc)] = ES,Aρ0(T) (45)

For the term , by using Cauthy-Schwarz inequality, we can get:

 ES,A(∇R(f∗S,Fc)−∇RS(f∗S,Fc))T(fT−f∗S,Fc) ≤√ES,A∥fT−f∗S,Fc∥2ES(∇R(f∗S,Fc)−∇RS(f∗S,Fc))2 ≤√ES,Aρ1(T)(L22n+6Lγβ1) (46)

where the second inequality holds according to Lemma 3.3.

Next we decompose as follows:

 Eest (48) = [R(f∗S,Fc)−RS(f∗S,Fc)]+[RS(f∗S,Fc)−RS(f∗Fc)] +[RS(f∗Fc)−R(f∗Fc)] ≤ [R(f∗S,Fc)−RS(f∗S,Fc)]+[RS(f∗Fc)−R(f∗Fc)], (49)

where the second inequality is established because is the minimizer of restricted to the hypothesis space .

Since is independent of , we have . Then by using Eq.(19) and the definition of uniform stability, we can get

 ESEest ≤ ES[R(f∗S,Fc)−RS(f∗S,Fc)] (50) ≤ ES,z′j[|l(f∗S,Fc,z′j)−l(f∗Sj,Fc,z′j)|] ≤ ES,z′j[|l(f∗S,Fc,z′j)−l(f∗S∖j,Fc,z′j)|] +ES,z′j[|l(f∗S∖j,Fc,z′j)−l(f∗Sj,Fc,z′j)|] ≤ 2β0.

By combining Ineq.(