Meanfield optimal control and optimality conditions in the space of probability measures
Abstract.
We derive a framework to compute optimal controls for problems with states in the space of probability measures. Since many optimal control problems constrained by a system of ordinary differential equations (ODE) modelling interacting particles converge to optimal control problems constrained by a partial differential equation (PDE) in the meanfield limit, it is interesting to have a calculus directly on the mesoscopic level of probability measures which allows us to derive the corresponding firstorder optimality system. In addition to this new calculus, we provide relations for the resulting system to the firstorder optimality system derived on the particle level, and the firstorder optimality system based on calculus under additional regularity assumptions. We further justify the use of the adjoint in numerical simulations by establishing a link between the adjoint in the space of probability measures and the adjoint corresponding to calculus. Moreover, we prove a convergence rate for the convergence of the optimal controls corresponding to the particle formulation to the optimal controls of the meanfield problem as the number of particles tends to infinity.
Corresponding author]totzeck@mathematik.unikl.de
Keywords. Optimal control with ODE/PDE constraints, interacting particle systems, meanfield limits.
AMS subject classifications. 49K15, 49K20
1. Introduction
In the past few years, the growing interest in the (optimal) control of interacting particle systems and their corresponding meanfield limits led to many contributions on their numerical behavior (see, e.g., [diss, sheep1]) as well as their analytical properties, e.g.,[FornasierSolombrino, FornasierPP]. They can be found in various fields of applications, for example physical or biological models like crowd dynamics [sheep1, Borzi, zuazua, bongini], consensus formation [Dante], or even global optimization [CBO1, CBO2]. Meanwhile, there are also first approaches for stochastic particle systems available [bonnet, pham].
Since there are several points of view on this subject, the analytical techniques vary from standard ODE and PDE theory over optimal transport to measurevalued solutions. This induces also different variants for the derivation of firstorder optimality conditions and/or gradient information, which clearly also has some impact on the design of appropriate numerical algorithms for the solution of the optimal control problems at hand.
Here, we present a calculus for the derivation of firstorder optimality conditions in the space of probability measures and link it to other different approaches discussed in [bonnet, FornasierPP, FornasierSolombrino] (cf. Section 4). As opposed to the firstorder optimality conditions found in those papers, our calculus yields vector fields as adjoint variables, which is consistent with the adjoint variables that appear on the level of particle systems. Furthermore, we show, in Section 5, how these new insights might be used for further analytical investigations. To get an idea of the different strategies, we begin with a simple example that displays all the main features of our calculus.
1.1. An illustrative example: Controlling a single particle
Let us start with an illustrative example from classical optimal control in order to illustrate the idea without the complication of a meanfield limit. We denote the dimension of the state space by and the time interval of interest is for some . We assume that the control variable acts on the velocity of a single particle with trajectory for and we want to optimize a given functional depending on the trajectory, i.e.,
(1) 
where and are given, sufficiently regular functions.
Then, the standard Pontryagin Maximum principle yields the existence of an adjoint variable satisfying
(2) 
with terminal condition .
Moreover, the control satisfies the optimality condition
These conditions can be translated into the calculation of a saddlepoint of the microscopic Lagrangian
(3) 
On the other hand, the discrete ODE can be translated into a macroscopic formulation via the method of characteristics: with initial value the concentrated measure is the unique solution of
(4) 
Since all measures are concentrated at we can reinterpret as the evaluation of a feedbackcontrol at and equivalently obtain
(5) 
Since
we can formulate an optimal control problem at the macroscopic level for the measure and the control variable , i.e.,
(6) 
This macroscopic optimal control problem is in fact equivalent to the microscopic one for a single particle, since we can choose the state space as the Banach space of Radon measures and the control space as an appropriate space of reasonably smooth functions on . The uniqueness of solutions to the transport equation and the special initial value will always yield a concentrated measure and the identification brings us back to the microscopic control.
However, with the macroscopic formulation we have another option to derive optimality conditions in these larger spaces, based on the Lagrangian
(7) 
Then, the macroscopic adjoint equation becomes
(8) 
and the optimality condition is given by
Due to the equivalence of the microscopic and macroscopic optimal control problem it is natural to ask for the relation between the adjoint variables and , which is not obvious at a first glance and yet only very little discussed. For first results in this direction see [herty18]. Using the special structure of the solution and the identification with the microscopic control we can rewrite the optimality condition as
which induces the identification
(9) 
Indeed, the method of characteristics confirms that satisfies the microscopic adjoint equation. This becomes more apparent if we consider only variations of that respect the nonnegativity and mass one condition of the probability measure, i.e.,
with a vectorvalued measure being absolutely continuous with respect to . Then, an integration by parts argument directly reveals the relation to .
By using variations of this kind we reinterpret the state space as a Riemannian manifold of Borel probability measures equipped with the 2Wasserstein distance instead of the flat Banach space of Radon measures. The analysis of particle systems and limiting nonlinear partial differential equations in the 2Wasserstein distance has been a quite fruitful field of study in the last years following the seminal papers [Otto, JKO]. It is hence highly overdue to study such an approach also in the optimal control setting.
We mention that the values of outside the trajectory are irrelevant for the specific control problem. Solving
we obtain the adjoints for all possible microscopic control problems with initial value in . This is just the wellknown HamiltonJacobiBellmann equation, usually derived with different arguments.
Remark 1.1.
The above arguments can also be extended to a stochastic control system (see, e.g., [roy2018]):
(10) 
with being a Wiener process and the solution to the stochastic differential equation with initial condition . In this case the state equation for the probability density becomes
(11) 
and does not necessarily remain a concentrated measure in time, which corresponds to the stochasticity of the model.
1.2. Control in the Meanfield Limit
Having understood the relation between microscopic and macroscopic formulations of the optimal control problem, it seems an obvious step to consider optimal control problems for a high number of particles and their meanfield limit as , which is also the motivation for this paper. However, in the meanfield limit there is no microscopic particle system and corresponding optimal control problem, hence an additional step is needed to understand the connection in the limit. The basis for such a step is to understand the characteristic flow, which replaces the particle dynamics and naturally leads to an analysis in the Wasserstein distance. We will further investigate this meanfield setting in the remainder of the paper.
Here, we restrict our considerations to first order dynamics, but the present paper can be seen as an analytical justification of the convergence shown numerically in [sheep1]. It is an additional contribution to the field of optimization of particle systems and their meanfield limits which is lively discussed in the recent years (e.g. [Borzi, Dante, FornasierSolombrino, FornasierPP, CBO1, CBO2, Giacomo]). Moreover, we would like to connect the fields of optimal control and gradient flows as well as optimal transport. In particular, we show relations between the adjoints derived by calculus and adjoints derived in the space of probability measures (adjoints).
The paper is organized as follows: in Section 2 the microscopic model for particles and the corresponding meanfield equation is introduced. Further, we formulate the optimal control problems under investigation. The first main contribution of the article is the derivation of the firstorder optimality conditions in the mesoscopic formulation given in Section 3. A discussion of the relation of this new calculus to the firstorder optimality systems on the particle level and the firstorder optimality condition based on calculus is the content of Section 4. In Section 5 we show the second main result which is the convergence rate for the optimal controls as .
2. Optimal Control Problems
First, we generalize the oneparticle case to interacting particles, modeling, e.g., crowd dynamics [sheep1]. Then, we derive its corresponding meanfield limit, i.e., the mesoscopic approximation. These two are the state systems for the respective optimal control problems. Further, we present the assumptions which are necessary for the wellposedness of the state systems.
2.1. The State Models
As before, denotes the dimension of the state space and with is the time interval of interest.
2.1.1. The particle system
The considered particle system consists of particles of the same type and controls represented by the functions
The vectors
denote the states of the particles and the controls, respectively.
The particle system reads explicitly
(12) 
with given defining the initial states of the particles. The operator on the righthand side strongly depends on the type of application. Here, we assume

Let be given, such that for all :
where the constant is independent of .
We further define via
where
is the empirical measure for the state .

For any two , there exists a constant , independent of and , such that
Remark 2.1.
By definition, assigns the probability of finding particles with states within a measurable set on the state space at time .
Standard results from ODE theory yield the existence and uniqueness of a global solution.
Remark 2.3.
In particular, for applications in the control of crowds we have that models interactions, i.e., particleparticle and particlecontrol interactions by means of forces (see [CarrilloSurvey] and the references therein). Then, is often given by
(13) 
for given interaction forces and modeling the interactions within the cloud of particles itself and of the particles with the controls, respectively.
2.1.2. The meanfield model
In order to define the limiting problem for an increasing number of particles explicitly, we consider the empirical measure .
Using the ideas from [Neunzert, BraunHepp, Dobrushin] we derive the corresponding PDE formally as
(14) 
which is the meanfield 1particle distribution evolution equation, supplemented with the initial condition , where denotes the space of Borel probability measures on with finite second moment, endowed with the 2Wasserstein distance, which makes a complete metric space. For the sake of completeness we recall the 2Wasserstein distance:
where denotes the set of all Borel probabililty measures on that have and as first and second marginals respectively, i.e.,
In the rest of the article we denote by the second moment of .
Remark 2.4.
Here denotes the meanfield representation of . In fact, for the structure given by (13), we obtain
(15) 
In the meanfield setting we consider the following notion of solution.
Definition 2.5.
We call a weak measure solution of (14) with initial condition iff for any test function we have
An existence and uniqueness result for solutions of (14) may be found, e.g., in [BraunHepp, WassersteinConvergence, Dobrushin, Golse], where the notion of solution is established in the Wasserstein space :
Proposition 2.6.
Further, for we have , where is the initial condition of (12).
Remark 2.7.
Under the assumptions 1 and 2 we have enough regularity to use the classical method of characteristics to deduce for any the existence of an unique global flow satisfying
(16) 
In particular, for we obtain the nonlinear flow with a random initial condition distributed according to , i.e., . The solution of (14) may then be explicitly expressed as for all We shall make use of this representation at several points in the remainder. For simplicity we set
The following stability statement will be useful in the coming results. Its proof may be found in Appendix A.
Lemma 2.8.
We end this section with an important observation:
Remark 2.9.
We emphasize that the particle problem is just a special case of the meanfield problem specified by the inital condition. Indeed, for the initial condition we have , where is the initial condition of (12). Strictly speaking, we have only one optimization problem to consider in the following. Whether the problem at hand is of microscopic or mesoscopic type is determined by the initial condition.
2.2. Optimal Control Problem
We define the set of admissible controls as
(17) 
This choice of ensures the continuity of the controls (compare also the previous existence results).
For the study of the respective optimal control problem we require:

The cost functional is of separable type, i.e.,
(18) where is continuously differentiable, weakly lower semicontinuous and coercive on . Further, is a cylindrical function of the form
where and such that , and
for some constant .

For the microscopic case, we define as well as
(19) and assume that is continuously differentiable.
Remark 2.10.
Note, that the differentiability properties in the previous assumptions are only necessary for the derivation of the optimality conditions in the next sections, and not for the existence of the respective optimal controls.
A direct consequence of assumption 3 is the continuity of in the Wasserstein metric.
Lemma 2.11.
Proof.
Let and be arbitrary. Then, for each , we have by 3, the meanvalue theorem and Hölder’s inequality that
where is the optimal coupling between and . In particular, the estimate above shows that the mapping is locally Lipschitz for every .
Denote and . The assumptions on and , and the previous estimate yields
where we used the fact that for all , . ∎
Remark 2.12.
The wellposedness of the state problem justifies the notation assigning the unique solution of the state equation to the control. Then, the optimal control problem we investigate in the following is given by
Problem 1:
Find such that
() 
For later use, we note that in the particle case, i.e., for discrete initial data (cf. Remark 2.9), we can rewrite the optimization problem as follows:
For fixed, find such that
() 
Using the standard argument based on the boundedness of a minimizing sequence in and continuity properties of stated in 3 and 4, we obtain the following existence result:
Remark 2.14.
The wellposedness of () follows directly from the above theorem, as the particle problem is a special case of (), see Remark 2.9. Nevertheless, one can prove the wellposedness of () also directly using classical techniques in the optimal control of ODEs.
3. Firstorder optimality conditions in the Wasserstein space
The main objective of this section is to derive the firstorder optimality conditions (FOC) for the optimal control problem () in the framework of probability measures with bounded second moment equipped with the 2Wasserstein distance. For the sake of a smooth presentation we restrict the interaction terms to the special ones defined in (13) and (15), respectively. This allows us to pose the following regularity assumption

.
For given initial condition we define the state space as
As the optimization in the setting is not wellknown, we begin by discussing known results (see [Ambrosio, Chapter 8.1]) regarding the constraint
(20) 
Recall Proposition 2.6 that provides for each a unique solution of (20). In particular, satisfies
(21) 
for all . Therefore, there is a welldefined solution operator , which allows us to recast the constrained minimization problem as
where is the socalled reduced functional.
Definition 3.2.
A pair is said to be admissible if for all .
Unfortunately, the reduced cost functional is not handy in deriving the firstorder optimality conditions for (). For this reason, we will take an extendedLagrangian approach. We begin by observing that () may be recast as
which may be further reformulated as
(22) 
Indeed, notice that , since implies for every . Therefore, if for some , the linearity in of yields for every , which consequently shows that .
Under the separation assumption on , i.e., , (22) becomes
with
In the following we derive a necessary condition for to be a stationary point. Let be an optimal pair, and be a perturbation of for an arbitrary smooth map such that and there exists a unique satisfying for all Then
and the directional derivative of at along h is given by
which requires us to know the relationship between and .
Remark 3.3.
Note, that Lemma 2.8 above provides a stability estimate of the form
for appropriate constants . Hence, for each , the curve starting from at is absolutely continuous w.r.t. the 2Wasserstein distance. In this case, there exists a vector field for each satisfying [Ambrosio, Proposition 8.4.6]
(23) 
Furthermore,
where the explicit coupling was used. In particular, we have that
The previous remark allows us to establish an explicit relationship between and h.
Lemma 3.4.
Let be an admissible pair, and such that

, and

there exists satisfying ,
for sufficiently small. Then, the velocity field satisfying (23) fulfills
(24) 
with
Proof.
Let . Then for any ,
On the other hand, since for all , we also have that
where
We prove that for : Let be an optimal coupling between and . Then