Understanding the Acceleration Phenomenon via HighResolution Differential Equations
Abstract
Gradientbased optimization algorithms can be studied from the perspective of limiting ordinary differential equations (ODEs). Motivated by the fact that existing ODEs do not distinguish between two fundamentally different algorithms—Nesterov’s accelerated gradient method for strongly convex functions (NAGSC) and Polyak’s heavyball method—we study an alternative limiting process that yields highresolution ODEs. We show that these ODEs permit a general Lyapunov function framework for the analysis of convergence in both continuous and discrete time. We also show that these ODEs are more accurate surrogates for the underlying algorithms; in particular, they not only distinguish between NAGSC and Polyak’s heavyball method, but they allow the identification of a term that we refer to as “gradient correction” that is present in NAGSC but not in the heavyball method and is responsible for the qualitative difference in convergence of the two methods. We also use the highresolution ODE framework to study Nesterov’s accelerated gradient method for (nonstrongly) convex functions, uncovering a hitherto unknown result—that NAGC minimizes the squared gradient norm at an inverse cubic rate. Finally, by modifying the highresolution ODE of NAGC, we obtain a family of new optimization methods that are shown to maintain the accelerated convergence rates of NAGC for smooth convex functions.
Keywords. Convex optimization, firstorder method, Polyak’s heavy ball method, Nesterov’s accelerated gradient methods, ordinary differential equation, Lyapunov function, gradient minimization, dimensional analysis, phase space representation, numerical stability
1 Introduction
Machine learning has become one of the major application areas for optimization algorithms during the past decade. While there have been many kinds of applications, to a wide variety of problems, the most prominent applications have involved largescale problems in which the objective function is the sum over terms associated with individual data, such that stochastic gradients can be computed cheaply, while gradients are much more expensive and the computation (and/or storage) of Hessians is often infeasible. In this setting, simple firstorder gradient descent algorithms have become dominant, and the effort to make these algorithms applicable to a broad range of machine learning problems has triggered a flurry of new research in optimization, both methodological and theoretical.
We will be considering unconstrained minimization problems,
(1.1) 
where is a smooth convex function. Perhaps the simplest firstorder method for solving this problem is gradient descent. Taking a fixed step size , gradient descent is implemented as the recursive rule
given an initial point .
As has been known at least since the advent of conjugate gradient algorithms, improvements to gradient descent can be obtained within a firstorder framework by using the history of past gradients. Modern research on such extended firstorder methods arguably dates to Polyak [Pol64, Pol87], whose heavyball method incorporates a momentum term into the gradient step. This approach allows past gradients to influence the current step, while avoiding the complexities of conjugate gradients and permitting a stronger theoretical analysis. Explicitly, starting from an initial point , the heavyball method updates the iterates according to
(1.2) 
where is the momentum coefficient. While the heavyball method
provably attains a faster rate of local convergence than gradient
descent near a minimum of , it does not come with global guarantees.
Indeed, [LRP16] demonstrate that even for strongly convex
functions the method can fail to converge for some choices of the step size.
The next major development in firstorder methodology was due to Nesterov, who discovered a class of accelerated gradient methods that have a faster global convergence rate than gradient descent [Nes83, Nes13]. For a strongly convex objective with Lipschitz gradients, Nesterov’s accelerated gradient method (NAGSC) involves the following pair of update equations:
(1.3)  
given an initial point . Equivalently, NAGSC can be written in a singlevariable form that is similar to the heavyball method:
(1.4) 
starting from and . Like the heavyball method, NAGSC blends gradient and momentum contributions into its update direction, but defines a specific momentum coefficient . Nesterov also developed the estimate sequence technique to prove that NAGSC achieves an accelerated linear convergence rate:
if the step size satisfies . Moreover, for a (weakly) convex objective with Lipschitz gradients, Nesterov defined a related accelerated gradient method (NAGC), that takes the following form:
(1.5)  
with . The choice of momentum coefficient , which tends to one, is fundamental to the estimatesequencebased argument used by Nesterov to establish the following inverse quadratic convergence rate:
(1.6) 
for any step size . Under an oracle model of optimization complexity, the convergence rates achieved by NAGSC and NAGC are optimal for smooth strongly convex functions and smooth convex functions, respectively [NY83].
1.1 Gradient Correction: Small but Essential
Throughout the present paper, we let and to define a specific implementation of the heavyball method in (1.2). This choice of the momentum coefficient and the second initial point renders the heavyball method and NAGSC identical except for the last (small) term in (1.4). Despite their close resemblance, however, the two methods are in fact fundamentally different, with contrasting convergence results (see, for example, [Bub15]). Notably, the former algorithm in general only achieves local acceleration, while the latter achieves acceleration method for all initial values of the iterate [LRP16]. As a numerical illustration, Figure 1 presents the trajectories that arise from the two methods when minimizing an illconditioned convex quadratic function. We see that the heavyball method exhibits pronounced oscillations throughout the iterations, whereas NAGSC is monotone in the function value once the iteration counter exceeds .
This striking difference between the two methods can only be attributed to the last term in (1.4):
(1.7) 
which we refer to henceforth as the gradient correction
A recent line of research has taken a different point of view on the theoretical
analysis of acceleration, formulating the problem in continuous time and obtaining
algorithms via discretization [SBC14, KBB15, WWJ16]). This can be done by taking continuoustime limits
of existing algorithms to obtain ordinary differential equations (ODEs) that
can be analyzed using the rich toolbox associated with ODEs, including Lyapunov
functions
(1.8) 
with initial conditions and , is the exact limit of NAGC (1.5) by taking the step size . Alternatively, the starting point may be a Lagrangian or Hamiltonian framework [WWJ16]. In either case, the continuoustime perspective not only provides analytical power and intuition, but it also provides design tools for new accelerated algorithms.
Unfortunately, existing continuoustime formulations of acceleration stop short of differentiating between the heavyball method and NAGSC. In particular, these two methods have the same limiting ODE (see, for example, [WRJ16]):
(1.9) 
and, as a consequence, this ODE does not provide any insight into the stronger convergence results for NAGSC as compared to the heavyball method. As will be shown in Section 2, this is because the gradient correction is an orderofmagnitude smaller than the other terms in (1.4) if . Consequently, the gradient correction is not reflected in the lowresolution ODE (1.9) associated with NAGSC, which is derived by simply taking in both (1.2) and (1.4).
1.2 Overview of Contributions
Just as there is not a singled preferred way to discretize a differential equation, there is not a single preferred way to take a continuoustime limit of a difference equation. Inspired by dimensionalanalysis strategies widely used in fluid mechanics in which physical phenomena are investigated at multiple scales via the inclusion of various orders of perturbations [Ped13], we propose to incorporate terms into the limiting process for obtaining an ODE, including the (Hessiandriven) gradient correction in (1.7). This will yield highresolution ODEs that differentiate between the NAG methods and the heavyball method.
We list the highresolution ODEs that we derive in the paper here
Highresolution ODEs are more accurate continuoustime counterparts for the
corresponding discrete algorithms than lowresolution ODEs, thus allowing
for a better characterization of the accelerated methods. This is illustrated
in Figure 2, which presents trajectories and convergence of the discrete
methods, and the low and highresolution ODEs. For both NAGs, the highresolution
ODEs are in much better agreement with the discrete methods than the lowresolution
ODEs
The three new ODEs include terms that are not present in the corresponding lowresolution ODEs (compare, for example, (1.12) and (1.8)). Note also that if we let , each highresolution ODE reduces to its lowresolution counterpart. Thus, the difference between the heavyball method and NAGSC is reflected only in their highresolution ODEs: the gradient correction (1.7) of NAGSC is preserved only in its highresolution ODE in the form . This term, which we refer to as the (Hessiandriven) gradient correction, is connected with the discrete gradient correction by the approximate identity:
for small , with the identification . The gradient correction
in NAGC arises in the same
fashion
Despite being small, the gradient correction has a fundamental effect on the behavior of both NAGs, and this effect is revealed by inspection of the highresolution ODEs. We provide two illustrations of this.

Effect of the gradient correction in acceleration. Viewing the coefficient of as a damping ratio, the ratio of in the highresolution ODE (1.11) of NAGSC is adaptive to the position , in contrast to the fixed damping ratio in the ODE (1.10) for the heavyball method. To appreciate the effect of this adaptivity, imagine that the velocity is highly correlated with an eigenvector of with a large eigenvalue, such that the large friction effectively “decelerates” along the trajectory of the ODE (1.11) of NAGSC. This feature of NAGSC is appealing as taking a cautious step in the presence of high curvature generally helps avoid oscillations. Figure 1 and the left plot of Figure 2 confirm the superiority of NAGSC over the heavyball method in this respect.
If we can translate this argument to the discrete case we can understand why NAGSC achieves acceleration globally for strongly convex functions but the heavyball method does not. We will be able to make this translation by leveraging the highresolution ODEs to construct discretetime Lyapunov functions that allow maximal step sizes to be characterized for the NAGSC and the heavyball method. The detailed analyses is given in Section 3.

Effect of gradient correction in gradient norm minimization. We will also show how to exploit the highresolution ODE of NAGC to construct a continuoustime Lyapunov function to analyze convergence in the setting of a smooth convex objective with Lipschitz gradients. Interestingly, the time derivative of the Lyapunov function is not only negative, but it is smaller than . This bound arises from the gradient correction and, indeed, it cannot be obtained from the Lyapunov function studied in the lowresolution case by [SBC16]. This finer characterization in the highresolution case allows us to establish a new phenomenon:
That is, we discover that NAGC achieves an inverse cubic rate for minimizing the squared gradient norm. By comparison, from (1.6) and the Lipschitz continuity of we can only show that . See Section 4 for further elaboration on this cubic rate for NAGC.
As we will see, the highresolution ODEs are based on a phasespace representation that provides a systematic framework for translating from continuoustime Lyapunov functions to discretetime Lyapunov functions. In sharp contrast, the process for obtaining a discretetime Lyapunov function for lowresolution ODEs presented by [SBC16] relies on “algebraic tricks” (see, for example, Theorem 6 of [SBC16]).
1.3 Related Work
There is a long history of using ODEs to analyze optimization methods [HM12, Sch00, Fio05]. Recently, the work of [SBC14, SBC16] has sparked a renewed interest in leveraging continuous dynamical systems to understand and design firstorder methods and to provide more intuitive proofs for the discrete methods. Below is a rather incomplete review of recent work that uses continuoustime dynamical systems to study accelerated methods.
In the work of [WWJ16, WRJ16, BJW18], Lagrangian and Hamiltonian frameworks are used to generate a large class of continuoustime ODEs for a unified treatment of accelerated gradientbased methods. Indeed, [WWJ16] extend NAGC to nonEuclidean settings, mirror descent and accelerated higherorder gradient methods, all from a single “Bregman Lagrangian.” In [WRJ16], the connection between ODEs and discrete algorithms is further strengthened by establishing an equivalence between the estimate sequence technique and Lyapunov function techniques, allowing for a principled analysis of the discretization of continuoustime ODEs. Recent papers have considered symplectic [BJW18] and Runge–Kutta [ZMSJ18] schemes for discretization of the lowresolution ODEs.
An ODEbased analysis of mirror descent has been pursued in another line of work by [KBB15, KBB16, KB17], delivering new connections between acceleration and constrained optimization, averaging and stochastic mirror descent.
In addition to the perspective of continuoustime dynamical systems, there has also been work on the acceleration from a controltheoretic point of view [LRP16, HL17, FRMP18] and from a geometric point of view [BLS15, CML17]. See also [OC15, FB15, GL16, LMH18, DFR18] for a number of other recent contributions to the study of the acceleration phenomenon.
1.4 Organization and Notation
The remainder of the paper is organized as follows. In Section 2, we briefly introduce our highresolution ODEbased analysis framework. This framework is used in Section 3 to study the heavyball method and NAGSC for smooth strongly convex functions. In Section 4, we turn our focus to NAGC for a general smooth convex objective. In Section 5 we derive some extensions of NAGC. We conclude the paper in Section 6 with a list of future research directions. Most technical proofs are deferred to the Appendix.
We mostly follow the notation of [Nes13], with slight modifications tailored to the present paper. Let be the class of smooth convex functions defined on ; that is, if for all and its gradient is Lipschitz continuous in the sense that
where denotes the standard Euclidean norm and is the Lipschitz constant. (Note that this implies that is also Lipschitz for any .) The function class is the subclass of such that each has a Lipschitzcontinuous Hessian. For , let denote the subclass of such that each member is strongly convex for some . That is, if and
for all . Note that this is equivalent to the convexity of , where denotes a minimizer of the objective .
2 The HighResolution ODE Framework
This section introduces a highresolution ODE framework for analyzing gradientbased methods, with NAGSC being a guiding example. Given a (discrete) optimization algorithm, the first step in this framework is to derive a highresolution ODE using dimensional analysis, the next step is to construct a continuoustime Lyapunov function to analyze properties of the ODE, the third step is to derive a discretetime Lyapunov function from its continuous counterpart and the last step is to translate properties of the ODE into that of the original algorithm. The overall framework is illustrated in Figure 3.
Step 1: Deriving HighResolution ODEs
Our focus is on the singlevariable form (1.4) of NAGSC. For any nonnegative integer , let and assume for some sufficiently smooth curve . Performing a Taylor expansion in powers of , we get
(2.1)  
We now use a Taylor expansion for the gradient correction, which gives
(2.2) 
Multiplying both sides of (1.4) by and rearranging the equality, we can rewrite NAGSC as
(2.3) 
Next, plugging (2.1) and (2.2)
into (2.3), we have
which can be rewritten as
Multiplying both sides of the last display by , we obtain the following highresolution ODE of NAGSC:
where we ignore any terms but retain the terms (note that ).
Our analysis is inspired by dimensional analysis [Ped13], a strategy widely used in physics to construct a series of differential equations that involve increasingly highorder terms corresponding to small perturbations. In more detail, taking a small , one first derives a differential equation that consists only of terms, then derives a differential equation consisting of both and , and next, one proceeds to obtain a differential equation consisting of and terms. Highorder terms in powers of are introduced sequentially until the main characteristics of the original algorithms have been extracted from the resulting approximating differential equation. Thus, we aim to understand Nesterov acceleration by incorporating terms into the ODE, including the (Hessiandriven) gradient correction which results from the (discrete) gradient correction (1.7) in the singlevariable form (1.4) of NAGSC. We also show (see Appendix A.1 for the detailed derivation) that this term appears in the highresolution ODE of NAGC, but is not found in the highresolution ODE of the heavyball method.
As shown below, each ODE admits a unique global solution under mild conditions on the objective, and this holds for an arbitrary step size . The solution is accurate in approximating its associated optimization method if is small. To state the result, we use to denote the class of twicecontinuouslydifferentiable maps from to for (the heavyball method and NAGSC) and (NAGC).
Proposition 2.1.
In fact, Proposititon 2.1 holds for because both the discrete iterates and the ODE trajectories converge to the unique minimizer when the objective is stongly convex.
Proposition 2.2.
For any , the ODE (1.12) with the specified initial conditions has a unique global solution . Moreover, NAGC converges to its highresolution ODE in the sense that
for any fixed .
Step 2: Analyzing ODEs Using Lyapunov Functions
With these highresolution ODEs in place, the next step is to construct Lyapunov functions for analyzing the dynamics of the corresponding ODEs, as is done in previous work [SBC16, WRJ16, LRP16]. For NAGSC, we consider the Lyapunov function
(2.4) 
The first and second terms and can be regarded, respectively, as the potential energy and kinetic energy, and the last term is a mix. For the mixed term, it is interesting to note that the time derivative of equals .
The differentiability of will allow us to investigate properties of the ODE (1.11) in a principled manner. For example, we will show that decreases exponentially along the trajectories of (1.11), recovering the accelerated linear convergence rate of NAGSC. Furthermore, a comparison between the Lyapunov function of NAGSC and that of the heavyball method will explain why the gradient correction yields acceleration in the former case. This is discussed in Section 3.1.
Step 3: Constructing Discrete Lyapunov Functions
Our framework make it possible to translate continuous Lyapunov functions into discrete Lyapunov functions via a phasespace representation (see, for example, [Arn13]). We illustrate the procedure in the case of NAGSC. The first step is formulate explicit position and velocity updates:
(2.5)  
where the velocity variable is defined as:
The initial velocity is . Interestingly, this phasespace representation has the
flavor of symplectic discretization, in the sense that the update for
is explicit (it only depends on the last iterate )
while the update for is implicit (it depends on the
current iterates and )
The representation (2.5) suggests translating the continuoustime Lyapunov function (2.4) into a discretetime Lyapunov function of the following form:
(2.6)  
by replacing continuous terms (e.g., ) by their discrete counterparts (e.g., ). Akin to the continuous (2.4), here , , and correspond to potential energy, kinetic energy, and mixed energy, respectively, from a mechanical perspective. To better appreciate this translation, note that the factor in results from the term in (2.5). Likewise, in is from the term in (2.5). The need for the final (small) negative term is technical; we discuss it in Section 3.2.
Step 4: Analyzing Algorithms Using Discrete Lyapunov Functions
The last step is to map properties of highresolution ODEs to corresponding properties of optimization methods. This step closely mimics Step 2 except that now the object is a discrete algorithm and the tool is a discrete Lyapunov function such as (2.6). Given that Step 2 has been performed, this translation is conceptually straightforward, albeit often calculationintensive. For example, using the discrete Lyapunov function (2.6), we will recover the optimal linear rate of NAGSC and gain insights into the fundamental effect of the gradient correction in accelerating NAGSC. In addition, NAGC is shown to minimize the squared gradient norm at an inverse cubic rate by a simple analysis of the decreasing rate of its discrete Lyapunov function.
3 Gradient Correction for Acceleration
In this section, we use our highresolution ODE framework to analyze NAGSC and the heavyball method. Section 3.1 focuses on the ODEs with an objective function , and in Section 3.2 we extend the results to the discrete case for . Finally, in Section 3.3 we offer a comparative study of NAGSC and the heavyball method from a finitedifference viewpoint.
Throughout this section, the strategy is to analyze the two methods in parallel, thereby highlighting the differences between the two methods. In particular, the comparison will demonstrate the vital role of the gradient correction, namely in the discrete case and in the ODE case, in making NAGSC an accelerated method.
3.1 The ODE Case
The following theorem characterizes the convergence rate of the highresolution ODE corresponding to NAGSC.
Theorem 1 (Convergence of NAGSc Ode).
Let . For any step size , the solution of the highresolution ODE (1.11) satisfies
The theorem states that the functional value tends to the minimum at a linear rate. By setting , we obtain .
The proof of Theorem 1 is based on analyzing the Lyapunov function for the highresolution ODE of NAGSC. Recall that defined in (2.4) is
The next lemma states the key property we need from this Lyapunov function
Lemma 3.1 (Lyapunov function for NAGSc Ode).
The proof of this theorem relies on Lemma 3.1 through the inequality . The term plays no role at the moment, but Section 3.2 will shed light on its profound effect in the discretization of the highresolution ODE of NAGSC.
Proof of Theorem 1.
Lemma 3.1 implies , which amounts to
By integrating out , we get
(3.2) 
Recognizing the initial conditions and , we write (3.2) as
Since , we have that and . Together with the Cauchy–Schwarz inequality, the two inequalities yield
which is valid for all . To simplify the coefficient of , note that can be replaced by in the analysis since . It follows that
Furthermore, a bit of analysis reveals that
since , and this step completes the proof of Theorem 1. ∎
We now consider the heavyball method (1.2). Recall that the momentum coefficient is set to . The following theorem characterizes the rate of convergence of this method.
Theorem 2 (Convergence of heavyball ODE).
Let . For any step size , the solution of the highresolution ODE (1.10) satisfies
As in the case of NAGSC, the proof of Theorem 2 is based on a Lyapunov function:
(3.3) 
which is the same as the Lyapunov function (2.4) for NAGSC except for the lack of the term. In particular, (2.4) and (3.3) are identical if . The following lemma considers the decay rate of (3.3).
Lemma 3.2 (Lyapunov function for the heavyball ODE).
The proof of Theorem 2 follows the same strategy as the proof of Theorem 1. In brief, Lemma 3.2 gives by integrating over the time parameter . Recognizing the initial conditions
in the highresolution ODE of the heavyball method and using the smoothness of , Lemma 3.2 yields
if the step size . Finally, since , the coefficient satisfies
The proofs of Lemma 3.1 and Lemma 3.2 share similar ideas. In view of this, we present only the proof of the former here, deferring the proof of Lemma 3.2 to Appendix B.1.
Proof of Lemma 3.1.
Along trajectories of (1.11) the Lyapunov function (2.4) satisfies
(3.4)  
Furthermore, is greater than or equal to both and due to the strong convexity of . This yields
which together with (3.4) suggests that the time derivative of this Lyapunov function can be bounded as
(3.5) 
Next, the Cauchy–Schwarz inequality yields
from which it follows that
(3.6) 
Combining (3.5) and (3.6) completes the proof of the theorem.
∎
Remark 3.3.
The only inequality in (3.4) is due to the term , which is discussed right after the statement of Lemma 3.1. This term results from the gradient correction in the NAGSC ODE. For comparison, this term does not appear in Lemma 3.2 in the case of the heavyball method as its ODE does not include the gradient correction and, accordingly, its Lyapunov function (3.3) is free of the term.
3.2 The Discrete Case
This section carries over the results in Section 3.1 to the two discrete algorithms, namely NAGSC and the heavyball method. Here we consider an objective since secondorder differentiability of is not required in the two discrete methods. Recall that both methods start with an arbitrary and