Stochastic control, entropic interpolation and gradient flows on Wasserstein product spaces*
Since the early nineties, it has been observed that the Schrödinger bridge problem can be formulated as a stochastic control problem with atypical boundary constraints. This in turn has a fluid dynamic counterpart where the flow of probability densities represents an entropic interpolation between the given initial and final marginals. In the zero noise limit, such entropic interpolation converges in a suitable sense to the displacement interpolation of optimal mass transport (OMT). We consider two absolutely continuous curves in Wasserstein space and study the evolution of the relative entropy on on a finite time interval. Thus, this study differs from previous work in OMT theory concerning relative entropy from a fixed (often equilibrium) distribution (density). We derive a gradient flow on Wasserstein product space. We find the remarkable property that fluxes in the two components are opposite. Plugging in the “steepest descent” into the evolution of the relative entropy we get what appears to be a new formula: The two flows approach each other at a faster rate than that of two solutions of the same Fokker-Planck. We then study the evolution of relative entropy in the case of uncontrolled-controlled diffusions. In two special cases of the Schrödinger bridge problem, we show that such relative entropy may be monotonically decreasing or monotonically increasing.
In the Schrödinger bridge problem (SBP) , one seeks the random evolution (a probability measure on path-space) which is closest in the relative entropy sense to a prior Markov diffusion evolution and has certain prescribed initial and final marginals and . As already observed by Schrödinger [34, 35], the problem may be reduced to a static problem which, except for the cost, resembles the Kantorovich relaxed formulation of the optimal mass transport problem (OMT). Considering that since  (OMT) also has a dynamic formulation, we have two problems which admit equivalent static and dynamic versions . Moreover, in both cases, the solution entails a flow of one-time marginals joining and . The OMT yields a displacement interpolation flow whereas the SBP provides an entropic interpolation flow.
Trough the work of Mikami, Mikami-Thieullen and Leonard [25, 27, 27, 22, 23], we know that the OMT may be viewed as a “zero-noise limit” of SBP when the prior is a sort of uniform measure on path space with vanishing variance. This connection has been extended to more general prior evolutions in [9, 10]. Moreover, we know that, thanks to a very useful intuition by Otto , the displacement interpolation flow may be viewed as a constant-speed geodesic joining and in Wasserstein space . What can be said from this geometric viewpoint of the entropic flow? It cannot be a geodesic, but can it be characterized as a curve minimizing a suitable action? In , we showed that this is indeed the case resorting to a time-symmetric fluid dynamic formulation of SBP. The action features an extra term which is a Fisher information functional. Moreover, this characterization of the Schrödinger bridge answers at once a question posed by Carlen [4, pp. 130-131].
It has been observed since the early nineties that SBP can be turned, thanks to Girsanov’s theorem, into a stochastic control problem with atypical boundary constraints, see [12, 3, 13, 31, 15]. The latter has a fluid dynamic counterpart. It is therefore interesting to compare the flow associated to the uncontrolled evolution (prior) to the optimal one. In particular, it is interesting to study the evolution of the relative entropy on the product Wasserstein space on a finite time interval. Thus, this study differs from previous work in OMT theory concerning relative entropy from an equilibrium distribution (density). We derive in Section IV a gradient flow on Wasserstein product space. We find the remarkable property that fluxes in the two components are opposite. Plugging in the“steepest descent” into the evolution of the relative entropy we get what appears to be a new formula (23): The two flows approach each other at a faster rate than that of two solutions of the same Fokker-Planck. We then study the evolution of relative entropy in the case of uncontrolled-controlled diffusions. We show by one special case of the Schrödinger bridge problem that such relative entropy may even be monotonically increasing.
The paper is outlined as follows. In Section II, we recall some fundamental facts and concepts from the theory of optimal transportation. In Section III, we review the variational formulation of the Fokker-Planck equation as a gradient flow on Wasserstein space. Section IV, we study the evolution of relative entropy on Wasserstein product space. In Section V, we recall some basic elements of the Nelson-Föllmer kinematics of finite-energy diffusions. Finally, in Section VI, we study the relative entropy change in the case of a controlled evolution. This is then specialized to the Schrödinger bridge.
Ii Elements of optimal mass transport theory
The literature on this problem is by now so vast and our degree of competence is such that we shall not even attempt here to give a reasonable and/or balanced introduction to the various fascinating aspects of this theory. Fortunately, there exist excellent monographs and survey papers on this topic, see [33, 14, 37, 1, 38, 30], to which we refer the reader. We shall only briefly review some concepts and results which are relevant for the topics of this paper.
Ii-a The static problem
Let and be probability measures on the measurable spaces and , respectively. Let be a measurable map with representing the cost of transporting a unit of mass from location to location . Let be the family of measurable maps such that , namely such that is the push-forward of under . Then Monge’s optimal mass transport problem (OMT) is
As is well known, this problem may be unfeasible, namely the family may be empty. This is never the case for the “relaxed” version of the problem studied by Kantorovich in the 1940’s
where are “couplings” of and , namely probability distributions on with marginals and . Indeed, always contains the product measure . Let us specialize the Monge-Kantorovich problem (2) to the case and . Then, if does not give mass to sets of dimension , by Brenier’s theorem [37, p.66], there exists a unique optimal transport plan (Kantorovich) induced by a a.e. unique map (Monge), , convex, and we have
Here denotes the identity map. Among the extensions of this result, we mention that to strictly convex, superlinear costs by Gangbo and McCann . The optimal transport problem may be used to introduce a useful distance between probability measures. Indeed, let be the set of probability measures on with finite second moment. For , the Wasserstein (Vasershtein) quadratic distance, is defined by
As is well known [37, Theorem 7.3], is a bona fide distance. Moreover, it provides a most natural way to “metrize” weak convergence111 converges weakly to if for every continuous, bounded function . in [37, Theorem 7.12], [1, Proposition 7.1.5] (the same applies to the case replacing with everywhere). The Wasserstein space is defined as the metric space . It is a Polish space, namely a separable, complete metric space.
Ii-B The dynamic problem
So far, we have dealt with the static optimal transport problem. Nevertheless, in [2, p.378] it is observed that “…a continuum mechanics formulation was already implicitly contained in the original problem addressed by Monge… Eliminating the time variable was just a clever way of reducing the dimension of the problem”. Thus, a dynamic version of the OMT problem was already in fieri in Gaspar Monge’s 1781 “Mémoire sur la théorie des déblais et des remblais” ! It was elegantly accomplished by Benamou and Brenier in  by showing that
Here the flow varies over continuous maps from to and over smooth fields. In , Villani states at the beginning of Chapter that two main motivations for the time-dependent version of OMT are
a time-dependent model gives a more complete description of the transport;
the richer mathematical structure will be useful later on.
We can add three further reasons:
Let and be optimal for (5). Then
with solving Monge’s problem, provides, in McCann’s language, the displacement interpolation between and . Then may be viewed as a constant-speed geodesic joining and in Wasserstein space (Otto). This formally endows with a “pseudo” Riemannian structure. McCann discovered  that certain functionals are displacement convex, namely convex along Wasserstein geodesics. This has led to a variety of applications. Following one of Otto’s main discoveries [21, 29], it turns out that a large class of PDE’s may be viewed as gradient flows on the Wasserstein space . This interpretation, because of the displacement convexity of the functionals, is well suited to establish uniqueness and to study energy dissipation and convergence to equilibrium. A rigorous setting in which to make sense of the Otto calculus has been developed by Ambrosio, Gigli and Savaré  for a suitable class of functionals. Convexity along geodesics in also leads to new proofs of various geometric and functional inequalities , [37, Chapter 9]. Finally, we mention that, when the space is not flat, qualitative properties of optimal transport can be quantified in terms of how bounds on the Ricci-Curbastro curvature affect the displacement convexity of certain specific functionals [38, Part II].
The tangent space of at a probability measure , denoted by  may be identified with the closure in of the span of , where is the family of smooth functions with compact support. It is naturally equipped with the scalar product of .
Iii The Fokker-Planck equation as a gradient flow on Wasserstein space
Let us review the variational formulation of the Fokker-Planck equation as a gradient flow on Wasserstein space [21, 37, 36]. Consider a physical system with phase space and with Hamiltonian . The thermodynamic states of the system are given by the family of probability distributions on admitting density . On , we define the internal energy as the expected value of the Energy observable in state
Let us also introduce the (differential) Gibbs entropy
where is Boltzmann’s constant. is strictly concave on . According to the Gibbsian postulate of classical statistical mechanics, the equilibrium state of a microscopic system at constant absolute temperature T and with Hamiltonian function H is necessarily given by the Boltzmann distribution law with density
where Z is the partition function222The letter was chosen by Boltzmann to indicate “zuständige Summe” (pertinent sum- here integral).. Let us introduce the Free Energy functional defined by
Since is strictly concave on and is linear, it follows that is strictly convex on the state space . By Gibbs’ variational principle, the Boltzmann distribution is a minimum point of the free energy on . Also notice that
Since does not depend on , we conclude that Gibb’s principle is a trivial consequence of the fact that minimizes on .
Consider now an absolutely continuous curve . Then [1, Chapter 8], there exist “velocity field” such that the following continuity equation holds on
Suppose , so that the continuity equation
holds. We want to study the free energy functional or, equivalently, , along the flow . Using (10), we get
Integrating by parts, if the boundary terms at infinity vanish, we get
Thus, the Wasserstein gradient of is
The corresponding gradient flow is
But this is precisely the Fokker-Planck equation corresponding to the diffusion process
where is a standard -dimensional Wiener process. The process (13) has the Boltzmann distribution (8) as invariant density. Recall that [1, p.220] or, equivalently, are displacement convex and have therefore a unique minimizer.
It seems worthwhile investigating to what extent the fundamental assumption of statistical mechanics that the variables with longer relaxation time form a vector Markov process having (8) as invariant density is equivalent to the requirement that the flow of one-time densities be a gradient flow in Wasserstein space for the free energy.
Iv Relative entropy as a functional on Wasserstein product spaces
Consider now two absolutely continuous curves and and their velocity fields and . Then, on
where the fields and satisfy
The differentiability of the Wasserstein distance has been studied [38, Theorem 23.9]. Consider instead the relative entropy functional on
Relative entropy functionals , where is a fixed probability measure (density), have been studied as geodesically convex functionals on , see [1, Section 9.4]. Our study of the evolution of is motivated by problems on a finite time interval such as the Schrödinger bridge problem and stochastic control problems (Section VI) where it is important to evaluate relative entropy on two flows of marginals.
After an integration by parts, assuming that the boundary terms at infinity vanish, we get
Notice that the last expression looks like
Thus, we identify the gradient of the functional on as
Let us now compute the gradient flow on corresponding to gradient (21). We get
we observe the remarkable property that in the “steepest descent” (22) on the product Wasserstein space the “fluxes” are opposite and, therefore, . If we plug the steepest descent (22) into (19), we get what appears to be a new formula
which should be compared to (14).
V Elements of Nelson-Föllmer kinematics of finite-energy diffusion processes
Let be a complete probability space. A stochastic process is called a finite-energy diffusion with constant diffusion coefficient if the paths belong to (-dimensional continuous functions) and
where is at each time a measurable function of the past and is a standard -dimensional Wiener process. Moreover, the drift satisfies the finite energy condition
In , Föllmer has shown that a finite-energy diffusion also admits a reverse-time Ito differential. Namely, there exists a measurable function of the future called backward drift and another Wiener process such that
Let us agree that always indicate a strictly positive variable. For any function let be the forward increment at time and let be the backward increment at time . For a finite-energy diffusion, Föllmer has also shown in  that forward and backward drifts may be obtained as Nelson’s conditional derivatives 
the limits being taken in . It was finally shown in  that the one-time probability density of (which exists for every ) is absolutely continuous on and the following duality relation holds
Let us introduce the fields
Then, Ito’s rule for the forward and backward differential of imply that satisfies the two Fokker-Planck equations
Following Nelson, let us introduce the current and osmotic drift of by
respectively. Clearly is similar to the classical velocity, whereas is the velocity due to the noise which tends to zero when tends to zero. Let us also introduce
which has the form of a continuity equation expressing conservation of mass. When is Markovian with and , (27) reduces to Nelson’s relation
Then (31) holds with
Vi Relative entropy production for controlled evolution
Consider on a finite-energy Markov process taking values in with forward Ito differential
Let be the probability density of . Consider also the feedback controlled process with forward differential
By (24), we now get
which shows that two solutions of the same Fokker-Plank equation tend to get closer.
Consider now the situation where represents a “prior” evolution on and the controlled evolution is the solution of the Schrödinger bridge problem for a pair of initial and final marginals and [17, 39]. Then
and the differential of is given by
where is space-time harmonic for the prior evolution, namely it satisfies
Let be the density of . Let us first consider the special case of the Schrödinger bridge problem where relative entropy on path space is minimized under the only constraint that the initial marginal density be . Then, the optimal control is identically zero and the evolution of the relative entropy is given by (37). Consider instead the case of the problem where only the final marginal density is imposed. In such case,
Then (36) gives
This shows that increases up to time . It represents the intuitive fact that the bridge evolution has to be as close as possible to the prior but the final value of the relative entropy must be the positive quantity . Thus, approaches this positive quantity from below. Result (40) may be viewed as a reverse-time H-theorem, as the bridge and the reference evolution have the same backward drift .
-  L. Ambrosio, N. Gigli and G. Savaré, Gradient Flows in Metric Spaces and in the Space of Probability Measures, Lectures in Mathematics ETH Zürich, Birkhäuser Verlag, Basel, 2nd ed. 2008.
-  J. Benamou and Y. Brenier, A computational fluid mechanics solution to the Monge-Kantorovich mass transfer problem, Numerische Mathematik, 84, no. 3, 2000, 375-393.
-  A. Blaquière, Controllability of a Fokker-Planck equation, the Schrödinger system, and a related stochastic optimal control, Dynamics and Control, vol. 2, no. 3, pp. 235-253, 1992.
-  E. Carlen, Stochastic mechanics: A look back and a look ahead, in Diffusion, quantum theory and radically elementary mathematics, W. G. Faris ed., Mathematical Notes 47, Princeton University Press, 2006, 117-139.
-  Y. Chen and T.T. Georgiou, Stochastic bridges of linear systems, preprint, arXiv1407.3421, IEEE Trans. Aut. Control, to appear.
-  Y. Chen, T.T. Georgiou and M. Pavon, Optimal steering of a linear stochastic system to a final probability distribution, , Part I, arXiv1408.2222, IEEE Trans. Aut. Control, to appear.
-  Y. Chen, T.T. Georgiou and M. Pavon, Optimal steering of a linear stochastic system to a final probability distribution, Part II, arXiv1410.3447, IEEE Trans. Aut. Control, to appear.
-  Y. Chen, T.T. Georgiou and M. Pavon, “Fast cooling for a system of stochastic oscillators” Nov. 2014, arXiv :1411.1323v2, J. Math. Phys., 56, n.11, 113302, 2015.
-  Y. Chen, T.T. Georgiou and M. Pavon, On the relation between optimal transport and Schrödinger bridges: A stochastic control viewpoint, arXiv :1412.4430v1, J. Optim. Theory and Applic., published online 2015, DOI 10.1007/s10957-015-0803-z.
-  Y. Chen, T.T. Georgiou and M. Pavon, “Optimal transport over a linear dynamical system”, Feb. 2015, arXiv:1502.01265v1, submitted for publication.
-  Y. Chen, T.T. Georgiou and M. Pavon, Entropic and displacement interpolation: a computational approach using the Hilbert metric, arXiv:1506.04255v1, submitted for publication.
-  P. Dai Pra, A stochastic control approach to reciprocal diffusion processes, Applied Mathematics and Optimization, 23 (1), 1991, 313-329.
-  P.Dai Pra and M.Pavon, On the Markov processes of Schroedinger, the Feynman-Kac formula and stochastic control, in Realization and Modeling in System Theory - Proc. 1989 MTNS Conf., M.A.Kaashoek, J.H. van Schuppen, A.C.M. Ran Eds., Birkaeuser, Boston, 1990, 497- 504.
-  L. C. Evans, Partial differential equations and Monge-Kantorovich mass transfer, in Current developments in mathematics, 1977, Int. Press, Boston, 1999, 65-126.
-  R. Fillieger, M.-O. Hongler and L. Streit, Connection between an exactly solvable stochastic optimal control problem and a nonlinear reaction-diffusion equation, J. Optimiz. Theory Appl. 137 (2008), 497-505.
-  H. Föllmer, in Stochastic Processes - Mathematics and Physics , Lecture Notes in Mathematics (Springer-Verlag, New York,1986), Vol. 1158, pp. 119-129.
-  H. Föllmer, Random fields and diffusion processes, in: Ècole d’Ètè de Probabilitès de Saint-Flour XV-XVII, edited by P. L. Hennequin, Lecture Notes in Mathematics, Springer-Verlag, New York, 1988, vol.1362,102-203.
-  W. Gangbo and R. McCann, The geometry of optimal transportation. Acta Mathematica, 177 (1996), no. 2, 113-161.
-  R. Graham, Path integral methods in nonequilibrium thermodynamics and statistics, in Stochastic Processes in Nonequilibrium Systems, L. Garrido, P. Seglar and P.J.Shepherd Eds., Lecture Notes in Physics 84, Springer-Verlag, New York, 1978, 82-138.
-  X. Jiang, Z. Luo, and T. Georgiou, Geometric methods for spectral analysis, IEEE Transactions on Signal Processing, 60, no. 3, 2012, 1064Ð1074.
-  R. Jordan, D. Kinderlehrer, and F. Otto, The variational formulation of the Fokker-Planck equation, SIAM J. Math. Anal. 29: 1-17 (1998).
-  C. Léonard, A survey of the Schroedinger problem and some of its connections with optimal transport, Discrete Contin. Dyn. Syst. A, 2014, 34 (4): 1533-1574.
-  C. Léonard, From the Schrödinger problem to the Monge-Kantorovich problem, J. Funct. Anal., 2012, 262, 1879-1920.
-  R. McCann, A convexity principle for interacting gases, Adv. Math., 128 (1997), 1, 153-179.
-  T. Mikami, Monge’s problem with a quadratic cost by the zero-noise limit of h-path processes, Probab. Theory Relat. Fields, 129, (2004), 245-260.
-  T. Mikami and M. Thieullen, Duality theorem for the stochastic optimal control problem., Stoch. Proc. Appl., 116, 1815?1835 (2006).
-  T. Mikami and M. Thieullen, Optimal Transportation Problem by Stochastic Optimal Control, SIAM Journal of Control and Optimization, 47, N. 3, 1127-1139 (2008).
-  E. Nelson, Dynamical Theories of Brownian Motion, Princeton University Press, Princeton, 1967.
-  F. Otto, The geometry of dissipative evolution equations: the porous medium equation, Comm. Partial Differential Equations 26, 1-2 (2001), 101-174.
-  Y. Ollivier, H. Pajot and C. Villani, Optimal Transportation. Theory and Applications, London Mathematical Society Lecture Notes Series 413, Cambridge University Press, 2014.
-  M.Pavon and A.Wakolbinger, On free energy, stochastic control, and Schroedinger processes, Modeling, Estimation and Control of Systems with Uncertainty, G.B. Di Masi, A.Gombani, A.Kurzhanski Eds., Birkauser, Boston, 1991, 334-348.
-  M. Pavon and F. Ticozzi, On entropy production for controlled Markovian evolution, J. Math. Phys., 47, 06330, doi:10.1063/1.2207716 (2006).
-  T. Rachev and L. Rüschendorf, Mass Transportation Problems, Vol. I: Theory, Vol. II: Applications. Probability and its applications. Springer- Verlag, New York, 1998.
-  E. Schrödinger, Über die Umkehrung der Naturgesetze, Sitzungsberichte der Preuss Akad. Wissen. Berlin, Phys. Math. Klasse (1931), 144-153.
-  E. Schrödinger, Sur la théorie relativiste de l’electron et l’interpretation de la mécanique quantique, Ann. Inst. H. Poincaré 2, 269 (1932).
-  E. Tannenbaum, T. Georgiou and A. Tannenbaum, Signals and Control Aspects of Optimal Mass Transport and the Boltzmann Entropy, Proc. 49th IEEE-CDC conference, Atlanta, GA, 2010, 1885-1890.
-  C. Villani, Topics in optimal transportation, AMS, 2003, vol. 58.
-  C. Villani, Optimal transport. Old and new. Grundlehren der Mathematischen Wissenschaften 338. Springer-Verlag, 2009.
-  A. Wakolbinger, Schroedinger bridges from 1931 to 1991, in Proc. of the 4th Latin American Congress in Probability and Mathematical Statistics, Mexico City 1990, Contribuciones en probabilidad y estadistica matematica, 3 (1992), 61-79.