Stochastic control, entropic interpolation and gradient flows on Wasserstein product spaces*

Stochastic control, entropic interpolation and gradient flows on Wasserstein product spaces*

Yongxin Chen, Tryphon Georgiou and Michele Pavon *Research partially supported by the NSF under Grant ECCS-1509387, the AFOSR under Grants FA9550-12-1-0319, and FA9550-15-1-0045 and by the University of Padova Research Project CPDA 140897.Yongxin Chen is with the Department of Mechanical Engineering, University of Minnesota, Minneapolis, Minnesota MN 55455, USA chen2468@umn.edu Tryphon Georgiou is with the Department of Electrical and Computer Engineering, University of Minnesota, Minneapolis, Minnesota MN 55455, USA tryphon@umn.edu Michele Pavon is with the Dipartimento di Matematica, Università di Padova, via Trieste 63, 35121 Padova, Italy pavon@math.unipd.it
Abstract

Since the early nineties, it has been observed that the Schrödinger bridge problem can be formulated as a stochastic control problem with atypical boundary constraints. This in turn has a fluid dynamic counterpart where the flow of probability densities represents an entropic interpolation between the given initial and final marginals. In the zero noise limit, such entropic interpolation converges in a suitable sense to the displacement interpolation of optimal mass transport (OMT). We consider two absolutely continuous curves in Wasserstein space and study the evolution of the relative entropy on on a finite time interval. Thus, this study differs from previous work in OMT theory concerning relative entropy from a fixed (often equilibrium) distribution (density). We derive a gradient flow on Wasserstein product space. We find the remarkable property that fluxes in the two components are opposite. Plugging in the “steepest descent” into the evolution of the relative entropy we get what appears to be a new formula: The two flows approach each other at a faster rate than that of two solutions of the same Fokker-Planck. We then study the evolution of relative entropy in the case of uncontrolled-controlled diffusions. In two special cases of the Schrödinger bridge problem, we show that such relative entropy may be monotonically decreasing or monotonically increasing.

I Introduction

In the Schrödinger bridge problem (SBP) [17], one seeks the random evolution (a probability measure on path-space) which is closest in the relative entropy sense to a prior Markov diffusion evolution and has certain prescribed initial and final marginals and . As already observed by Schrödinger [34, 35], the problem may be reduced to a static problem which, except for the cost, resembles the Kantorovich relaxed formulation of the optimal mass transport problem (OMT). Considering that since [2] (OMT) also has a dynamic formulation, we have two problems which admit equivalent static and dynamic versions [23]. Moreover, in both cases, the solution entails a flow of one-time marginals joining and . The OMT yields a displacement interpolation flow whereas the SBP provides an entropic interpolation flow.

Trough the work of Mikami, Mikami-Thieullen and Leonard [25, 27, 27, 22, 23], we know that the OMT may be viewed as a “zero-noise limit” of SBP when the prior is a sort of uniform measure on path space with vanishing variance. This connection has been extended to more general prior evolutions in [9, 10]. Moreover, we know that, thanks to a very useful intuition by Otto [29], the displacement interpolation flow may be viewed as a constant-speed geodesic joining and in Wasserstein space [37]. What can be said from this geometric viewpoint of the entropic flow? It cannot be a geodesic, but can it be characterized as a curve minimizing a suitable action? In [9], we showed that this is indeed the case resorting to a time-symmetric fluid dynamic formulation of SBP. The action features an extra term which is a Fisher information functional. Moreover, this characterization of the Schrödinger bridge answers at once a question posed by Carlen [4, pp. 130-131].

It has been observed since the early nineties that SBP can be turned, thanks to Girsanov’s theorem, into a stochastic control problem with atypical boundary constraints, see [12, 3, 13, 31, 15]. The latter has a fluid dynamic counterpart. It is therefore interesting to compare the flow associated to the uncontrolled evolution (prior) to the optimal one. In particular, it is interesting to study the evolution of the relative entropy on the product Wasserstein space on a finite time interval. Thus, this study differs from previous work in OMT theory concerning relative entropy from an equilibrium distribution (density). We derive in Section IV a gradient flow on Wasserstein product space. We find the remarkable property that fluxes in the two components are opposite. Plugging in the“steepest descent” into the evolution of the relative entropy we get what appears to be a new formula (23): The two flows approach each other at a faster rate than that of two solutions of the same Fokker-Planck. We then study the evolution of relative entropy in the case of uncontrolled-controlled diffusions. We show by one special case of the Schrödinger bridge problem that such relative entropy may even be monotonically increasing.

The paper is outlined as follows. In Section II, we recall some fundamental facts and concepts from the theory of optimal transportation. In Section III, we review the variational formulation of the Fokker-Planck equation as a gradient flow on Wasserstein space. Section IV, we study the evolution of relative entropy on Wasserstein product space. In Section V, we recall some basic elements of the Nelson-Föllmer kinematics of finite-energy diffusions. Finally, in Section VI, we study the relative entropy change in the case of a controlled evolution. This is then specialized to the Schrödinger bridge.

Ii Elements of optimal mass transport theory

The literature on this problem is by now so vast and our degree of competence is such that we shall not even attempt here to give a reasonable and/or balanced introduction to the various fascinating aspects of this theory. Fortunately, there exist excellent monographs and survey papers on this topic, see [33, 14, 37, 1, 38, 30], to which we refer the reader. We shall only briefly review some concepts and results which are relevant for the topics of this paper.

Ii-a The static problem

Let and be probability measures on the measurable spaces and , respectively. Let be a measurable map with representing the cost of transporting a unit of mass from location to location . Let be the family of measurable maps such that , namely such that is the push-forward of under . Then Monge’s optimal mass transport problem (OMT) is

(1)

As is well known, this problem may be unfeasible, namely the family may be empty. This is never the case for the “relaxed” version of the problem studied by Kantorovich in the 1940’s

(2)

where are “couplings” of and , namely probability distributions on with marginals and . Indeed, always contains the product measure . Let us specialize the Monge-Kantorovich problem (2) to the case and . Then, if does not give mass to sets of dimension , by Brenier’s theorem [37, p.66], there exists a unique optimal transport plan (Kantorovich) induced by a a.e. unique map (Monge), , convex, and we have

(3)

Here denotes the identity map. Among the extensions of this result, we mention that to strictly convex, superlinear costs by Gangbo and McCann [18]. The optimal transport problem may be used to introduce a useful distance between probability measures. Indeed, let be the set of probability measures on with finite second moment. For , the Wasserstein (Vasershtein) quadratic distance, is defined by

(4)

As is well known [37, Theorem 7.3], is a bona fide distance. Moreover, it provides a most natural way to “metrize” weak convergence111 converges weakly to if for every continuous, bounded function . in [37, Theorem 7.12], [1, Proposition 7.1.5] (the same applies to the case replacing with everywhere). The Wasserstein space is defined as the metric space . It is a Polish space, namely a separable, complete metric space.

Ii-B The dynamic problem

So far, we have dealt with the static optimal transport problem. Nevertheless, in [2, p.378] it is observed that “…a continuum mechanics formulation was already implicitly contained in the original problem addressed by Monge… Eliminating the time variable was just a clever way of reducing the dimension of the problem”. Thus, a dynamic version of the OMT problem was already in fieri in Gaspar Monge’s 1781 “Mémoire sur la théorie des déblais et des remblais” ! It was elegantly accomplished by Benamou and Brenier in [2] by showing that

(5a)
(5b)
(5c)

Here the flow varies over continuous maps from to and over smooth fields. In [38], Villani states at the beginning of Chapter that two main motivations for the time-dependent version of OMT are

  • a time-dependent model gives a more complete description of the transport;

  • the richer mathematical structure will be useful later on.

We can add three further reasons:

  • it opens the way to establish a connection with the Schrödinger bridge problem, where the latter appears as a regularization of the former [25, 27, 27, 22, 23, 9, 10];

  • it allows to view the optimal transport problem as an (atypical) optimal control problem [6]-[10].

  • In some applications such as interpolation of images [11] or spectral morphing [20], the interpolating flow is essential!

Let and be optimal for (5). Then

with solving Monge’s problem, provides, in McCann’s language, the displacement interpolation between and . Then may be viewed as a constant-speed geodesic joining and in Wasserstein space (Otto). This formally endows with a “pseudo” Riemannian structure. McCann discovered [24] that certain functionals are displacement convex, namely convex along Wasserstein geodesics. This has led to a variety of applications. Following one of Otto’s main discoveries [21, 29], it turns out that a large class of PDE’s may be viewed as gradient flows on the Wasserstein space . This interpretation, because of the displacement convexity of the functionals, is well suited to establish uniqueness and to study energy dissipation and convergence to equilibrium. A rigorous setting in which to make sense of the Otto calculus has been developed by Ambrosio, Gigli and Savaré [1] for a suitable class of functionals. Convexity along geodesics in also leads to new proofs of various geometric and functional inequalities [24], [37, Chapter 9]. Finally, we mention that, when the space is not flat, qualitative properties of optimal transport can be quantified in terms of how bounds on the Ricci-Curbastro curvature affect the displacement convexity of certain specific functionals [38, Part II].

The tangent space of at a probability measure , denoted by [1] may be identified with the closure in of the span of , where is the family of smooth functions with compact support. It is naturally equipped with the scalar product of .

Iii The Fokker-Planck equation as a gradient flow on Wasserstein space

Let us review the variational formulation of the Fokker-Planck equation as a gradient flow on Wasserstein space [21, 37, 36]. Consider a physical system with phase space and with Hamiltonian . The thermodynamic states of the system are given by the family of probability distributions on admitting density . On , we define the internal energy as the expected value of the Energy observable in state

(6)

Let us also introduce the (differential) Gibbs entropy

(7)

where is Boltzmann’s constant. is strictly concave on . According to the Gibbsian postulate of classical statistical mechanics, the equilibrium state of a microscopic system at constant absolute temperature T and with Hamiltonian function H is necessarily given by the Boltzmann distribution law with density

(8)

where Z is the partition function222The letter was chosen by Boltzmann to indicate “zuständige Summe” (pertinent sum- here integral).. Let us introduce the Free Energy functional defined by

(9)

Since is strictly concave on and is linear, it follows that is strictly convex on the state space . By Gibbs’ variational principle, the Boltzmann distribution is a minimum point of the free energy on . Also notice that

Since does not depend on , we conclude that Gibb’s principle is a trivial consequence of the fact that minimizes on .

Consider now an absolutely continuous curve . Then [1, Chapter 8], there exist “velocity field” such that the following continuity equation holds on

Suppose , so that the continuity equation

(10)

holds. We want to study the free energy functional or, equivalently, , along the flow . Using (10), we get

(11)

Integrating by parts, if the boundary terms at infinity vanish, we get

Thus, the Wasserstein gradient of is

The corresponding gradient flow is

(12)

But this is precisely the Fokker-Planck equation corresponding to the diffusion process

(13)

where is a standard -dimensional Wiener process. The process (13) has the Boltzmann distribution (8) as invariant density. Recall that [1, p.220] or, equivalently, are displacement convex and have therefore a unique minimizer.

Remark 1

It seems worthwhile investigating to what extent the fundamental assumption of statistical mechanics that the variables with longer relaxation time form a vector Markov process having (8) as invariant density is equivalent to the requirement that the flow of one-time densities be a gradient flow in Wasserstein space for the free energy.

Let us finally plug the “steepest descent” (12) into (11). We get, after integrating by parts, the well known formula [19]

(14)

The last integral in (14) is sometimes called the relative Fisher information of with respect to [37, p.278].

Iv Relative entropy as a functional on Wasserstein product spaces

Consider now two absolutely continuous curves and and their velocity fields and . Then, on

(15)
(16)

Let us suppose that and , for all . Then (15)-(16) become

(17)
(18)

where the fields and satisfy

The differentiability of the Wasserstein distance has been studied [38, Theorem 23.9]. Consider instead the relative entropy functional on

Relative entropy functionals , where is a fixed probability measure (density), have been studied as geodesically convex functionals on , see [1, Section 9.4]. Our study of the evolution of is motivated by problems on a finite time interval such as the Schrödinger bridge problem and stochastic control problems (Section VI) where it is important to evaluate relative entropy on two flows of marginals.

We get

(19)

After an integration by parts, assuming that the boundary terms at infinity vanish, we get

(20)

Notice that the last expression looks like

Thus, we identify the gradient of the functional on as

(21)

Let us now compute the gradient flow on corresponding to gradient (21). We get

(22)

Since

we observe the remarkable property that in the “steepest descent” (22) on the product Wasserstein space the “fluxes” are opposite and, therefore, . If we plug the steepest descent (22) into (19), we get what appears to be a new formula

(23)

which should be compared to (14).

Let us return to equation (20). By multiplying and dividing by in the last term of the middle expression, we get

(24)

which is precisely the expression obtained in [32, Theorem III.1].

V Elements of Nelson-Föllmer kinematics of finite-energy diffusion processes

Let be a complete probability space. A stochastic process is called a finite-energy diffusion with constant diffusion coefficient if the paths belong to (-dimensional continuous functions) and

(25)

where is at each time a measurable function of the past and is a standard -dimensional Wiener process. Moreover, the drift satisfies the finite energy condition

In [16], Föllmer has shown that a finite-energy diffusion also admits a reverse-time Ito differential. Namely, there exists a measurable function of the future called backward drift and another Wiener process such that

(26)

Moreover, satisfies

Let us agree that always indicate a strictly positive variable. For any function let be the forward increment at time and let be the backward increment at time . For a finite-energy diffusion, Föllmer has also shown in [16] that forward and backward drifts may be obtained as Nelson’s conditional derivatives [28]

the limits being taken in . It was finally shown in [16] that the one-time probability density of (which exists for every ) is absolutely continuous on and the following duality relation holds

(27)

Let us introduce the fields

Then, Ito’s rule for the forward and backward differential of imply that satisfies the two Fokker-Planck equations

(28)
(29)

Following Nelson, let us introduce the current and osmotic drift of by

(30)

respectively. Clearly is similar to the classical velocity, whereas is the velocity due to the noise which tends to zero when tends to zero. Let us also introduce

Then, combining (28) and (29), we get

(31)

which has the form of a continuity equation expressing conservation of mass. When is Markovian with and , (27) reduces to Nelson’s relation

(32)

Then (31) holds with

(33)

Vi Relative entropy production for controlled evolution

Consider on a finite-energy Markov process taking values in with forward Ito differential

(34)

Let be the probability density of . Consider also the feedback controlled process with forward differential

(35)

Here the control is adapted to the past and is such that is a finite-energy diffusion. Let be the probability density of . We are interested in the evolution of . By (33)-(31), the densities satisfy

By (24), we now get

(36)

Suppose now is also uncontrolled and differs from only because of the initial condition at . Then (36) gives the well known formula generalizing (14)

(37)

which shows that two solutions of the same Fokker-Plank equation tend to get closer.

Consider now the situation where represents a “prior” evolution on and the controlled evolution is the solution of the Schrödinger bridge problem for a pair of initial and final marginals and [17, 39]. Then

and the differential of is given by

(38)

where is space-time harmonic for the prior evolution, namely it satisfies

(39)

Let be the density of . Let us first consider the special case of the Schrödinger bridge problem where relative entropy on path space is minimized under the only constraint that the initial marginal density be . Then, the optimal control is identically zero and the evolution of the relative entropy is given by (37). Consider instead the case of the problem where only the final marginal density is imposed. In such case,

Then (36) gives

(40)

This shows that increases up to time . It represents the intuitive fact that the bridge evolution has to be as close as possible to the prior but the final value of the relative entropy must be the positive quantity . Thus, approaches this positive quantity from below. Result (40) may be viewed as a reverse-time H-theorem, as the bridge and the reference evolution have the same backward drift [17].

References

  • [1] L. Ambrosio, N. Gigli and G. Savaré, Gradient Flows in Metric Spaces and in the Space of Probability Measures, Lectures in Mathematics ETH Zürich, Birkhäuser Verlag, Basel, 2nd ed. 2008.
  • [2] J. Benamou and Y. Brenier, A computational fluid mechanics solution to the Monge-Kantorovich mass transfer problem, Numerische Mathematik, 84, no. 3, 2000, 375-393.
  • [3] A. Blaquière, Controllability of a Fokker-Planck equation, the Schrödinger system, and a related stochastic optimal control, Dynamics and Control, vol. 2, no. 3, pp. 235-253, 1992.
  • [4] E. Carlen, Stochastic mechanics: A look back and a look ahead, in Diffusion, quantum theory and radically elementary mathematics, W. G. Faris ed., Mathematical Notes 47, Princeton University Press, 2006, 117-139.
  • [5] Y. Chen and T.T. Georgiou, Stochastic bridges of linear systems, preprint, arXiv1407.3421, IEEE Trans. Aut. Control, to appear.
  • [6] Y. Chen, T.T. Georgiou and M. Pavon, Optimal steering of a linear stochastic system to a final probability distribution, , Part I, arXiv1408.2222, IEEE Trans. Aut. Control, to appear.
  • [7] Y. Chen, T.T. Georgiou and M. Pavon, Optimal steering of a linear stochastic system to a final probability distribution, Part II, arXiv1410.3447, IEEE Trans. Aut. Control, to appear.
  • [8] Y. Chen, T.T. Georgiou and M. Pavon, “Fast cooling for a system of stochastic oscillators” Nov. 2014, arXiv :1411.1323v2, J. Math. Phys., 56, n.11, 113302, 2015.
  • [9] Y. Chen, T.T. Georgiou and M. Pavon, On the relation between optimal transport and Schrödinger bridges: A stochastic control viewpoint, arXiv :1412.4430v1, J. Optim. Theory and Applic., published online 2015, DOI 10.1007/s10957-015-0803-z.
  • [10] Y. Chen, T.T. Georgiou and M. Pavon, “Optimal transport over a linear dynamical system”, Feb. 2015, arXiv:1502.01265v1, submitted for publication.
  • [11] Y. Chen, T.T. Georgiou and M. Pavon, Entropic and displacement interpolation: a computational approach using the Hilbert metric, arXiv:1506.04255v1, submitted for publication.
  • [12] P. Dai Pra, A stochastic control approach to reciprocal diffusion processes, Applied Mathematics and Optimization, 23 (1), 1991, 313-329.
  • [13] P.Dai Pra and M.Pavon, On the Markov processes of Schroedinger, the Feynman-Kac formula and stochastic control, in Realization and Modeling in System Theory - Proc. 1989 MTNS Conf., M.A.Kaashoek, J.H. van Schuppen, A.C.M. Ran Eds., Birkaeuser, Boston, 1990, 497- 504.
  • [14] L. C. Evans, Partial differential equations and Monge-Kantorovich mass transfer, in Current developments in mathematics, 1977, Int. Press, Boston, 1999, 65-126.
  • [15] R. Fillieger, M.-O. Hongler and L. Streit, Connection between an exactly solvable stochastic optimal control problem and a nonlinear reaction-diffusion equation, J. Optimiz. Theory Appl. 137 (2008), 497-505.
  • [16] H. Föllmer, in Stochastic Processes - Mathematics and Physics , Lecture Notes in Mathematics (Springer-Verlag, New York,1986), Vol. 1158, pp. 119-129.
  • [17] H. Föllmer, Random fields and diffusion processes, in: Ècole d’Ètè de Probabilitès de Saint-Flour XV-XVII, edited by P. L. Hennequin, Lecture Notes in Mathematics, Springer-Verlag, New York, 1988, vol.1362,102-203.
  • [18] W. Gangbo and R. McCann, The geometry of optimal transportation. Acta Mathematica, 177 (1996), no. 2, 113-161.
  • [19] R. Graham, Path integral methods in nonequilibrium thermodynamics and statistics, in Stochastic Processes in Nonequilibrium Systems, L. Garrido, P. Seglar and P.J.Shepherd Eds., Lecture Notes in Physics 84, Springer-Verlag, New York, 1978, 82-138.
  • [20] X. Jiang, Z. Luo, and T. Georgiou, Geometric methods for spectral analysis, IEEE Transactions on Signal Processing, 60, no. 3, 2012, 1064Ð1074.
  • [21] R. Jordan, D. Kinderlehrer, and F. Otto, The variational formulation of the Fokker-Planck equation, SIAM J. Math. Anal. 29: 1-17 (1998).
  • [22] C. Léonard, A survey of the Schroedinger problem and some of its connections with optimal transport, Discrete Contin. Dyn. Syst. A, 2014, 34 (4): 1533-1574.
  • [23] C. Léonard, From the Schrödinger problem to the Monge-Kantorovich problem, J. Funct. Anal., 2012, 262, 1879-1920.
  • [24] R. McCann, A convexity principle for interacting gases, Adv. Math., 128 (1997), 1, 153-179.
  • [25] T. Mikami, Monge’s problem with a quadratic cost by the zero-noise limit of h-path processes, Probab. Theory Relat. Fields, 129, (2004), 245-260.
  • [26] T. Mikami and M. Thieullen, Duality theorem for the stochastic optimal control problem., Stoch. Proc. Appl., 116, 1815?1835 (2006).
  • [27] T. Mikami and M. Thieullen, Optimal Transportation Problem by Stochastic Optimal Control, SIAM Journal of Control and Optimization, 47, N. 3, 1127-1139 (2008).
  • [28] E. Nelson, Dynamical Theories of Brownian Motion, Princeton University Press, Princeton, 1967.
  • [29] F. Otto, The geometry of dissipative evolution equations: the porous medium equation, Comm. Partial Differential Equations 26, 1-2 (2001), 101-174.
  • [30] Y. Ollivier, H. Pajot and C. Villani, Optimal Transportation. Theory and Applications, London Mathematical Society Lecture Notes Series 413, Cambridge University Press, 2014.
  • [31] M.Pavon and A.Wakolbinger, On free energy, stochastic control, and Schroedinger processes, Modeling, Estimation and Control of Systems with Uncertainty, G.B. Di Masi, A.Gombani, A.Kurzhanski Eds., Birkauser, Boston, 1991, 334-348.
  • [32] M. Pavon and F. Ticozzi, On entropy production for controlled Markovian evolution, J. Math. Phys., 47, 06330, doi:10.1063/1.2207716 (2006).
  • [33] T. Rachev and L. Rüschendorf, Mass Transportation Problems, Vol. I: Theory, Vol. II: Applications. Probability and its applications. Springer- Verlag, New York, 1998.
  • [34] E. Schrödinger, Über die Umkehrung der Naturgesetze, Sitzungsberichte der Preuss Akad. Wissen. Berlin, Phys. Math. Klasse (1931), 144-153.
  • [35] E. Schrödinger, Sur la théorie relativiste de l’electron et l’interpretation de la mécanique quantique, Ann. Inst. H. Poincaré 2, 269 (1932).
  • [36] E. Tannenbaum, T. Georgiou and A. Tannenbaum, Signals and Control Aspects of Optimal Mass Transport and the Boltzmann Entropy, Proc. 49th IEEE-CDC conference, Atlanta, GA, 2010, 1885-1890.
  • [37] C. Villani, Topics in optimal transportation, AMS, 2003, vol. 58.
  • [38] C. Villani, Optimal transport. Old and new. Grundlehren der Mathematischen Wissenschaften 338. Springer-Verlag, 2009.
  • [39] A. Wakolbinger, Schroedinger bridges from 1931 to 1991, in Proc. of the 4th Latin American Congress in Probability and Mathematical Statistics, Mexico City 1990, Contribuciones en probabilidad y estadistica matematica, 3 (1992), 61-79.
Comments 5
Request Comment
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
110078
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel
5

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description