# Optimal consensus control of the Cucker-Smale model

###### Abstract:

We study the numerical realisation of optimal consensus control laws for agent-based models. For a nonlinear multi-agent system of Cucker-Smale type, consensus control is cast as a dynamic optimisation problem for which we derive first-order necessary optimality conditions. In the case of a smooth penalization fo the control energy, the optimality system is numerically approximated via a gradient-descent method. For sparsity promoting, non-smooth -norm control penalizations, the optimal controllers are realised by means of heuristic methods. For an increasing number of agents, we discuss the approximation of the consensus control problem by following a mean-field modelling approach.

First]Rafael Bailo Second]Mattia Bongini First]José A. Carrillo First]Dante Kalise

Department of Mathematics, Imperial College London, South Kensington Campus, London SW7 2AZ, United Kindgom (e-mail: {carrillo,dkaliseb,r.bailo}@ic.ac.uk)

CEREMADE, Université Paris Dauphine, Place du Maréchal de Lattre de Tassigny, 75775 Paris Cedex 16, France (email: bongini@ceremade.dauphine.fr)

Keywords: Agent-based models, Cucker-Smale model, consensus control, optimal control, first-order optimality conditions, sparse control, mean-field modelling.

This paper is concerns control problems for second-order, nonlinear, multi-agent systems (MAS). In such dynamics, the state of each agent is characterized by a pair , representing variables which we refer to as position and velocity respectively. The uncontrolled system consists of simple binary interaction rules between the agents, such as attraction, repulsion, and alignment forces. This commonly leads to self-organization phenomena, flocking or formation arrays. However, this behaviour strongly depends on the cohesion of the initial configuration of the system, and therefore control design is relevant in order to generate an external intervention able to steer the dynamics towards a desired configuration.

For second-order MAS, it is of particular interest the study of consensus emergence and control. In this context, we understand consensus as a traveling formation in which every agent has the same velocity. Self-organized consensus emergence for the Cucker-Smale model, see Cucker and Smale (2007), has been already characterized in Ha and Liu (2009); Ha et al. (2010); Carrillo et al. (2010a). The problem of consensus control for Cucker-Smale type models has been discussed in Caponigro et al. (2013); Bongini et al. (2015). A related problem is the design of controllers achieving a given formation, which has been previously addressed in Perea et al. (2009); Borzì and Wongkaew (2015).

In this work we focus on the design of control laws enforcing consensus emergence. For this, we cast the consensus control problem in the framework of optimal control theory, for which an ad-hoc computational methodology is presented. We begin by considering a finite horizon control problem, in which the deviation of the population with respect to consensus is penalised along a quadratic control term. In this smooth case we derive first-order optimality conditions, which are then numerically realised via a first-order, gradient descent method: the so-called Barzilai-Borwein (BB) method. While the use of gradient descent methods is a standard tool for the numerical approximation of optimal control laws, see Borzí and Schulz (2011), the use of the BB method for large-scale agent-based models is relatively recent, Deroo et al. (2012), and we report on its use as a reliable method for optimal consensus control problems (OCCP) in nonlinear MAS.

While the control performance is satisfactory, our setting allows the controller to act differently on every agent at every instant. The question of a more parsimonious control design remains open. As an extension of the proposed methodology, we address the finite horizon OCCP with a non-smooth, sparsity-promoting control penalisation. While this control synthesis is sparse, acting on a few agents over a finite time frame, its numerical realisation is far more demanding due to the lack of smoothness in the cost functional. To circumvent this difficulty, the numerical realisation of the control synthesis resorts to the use of heuristics related to particle swarm optimisation (PSO), and to nonlinear model predictive control (NMPC). Finally, based on the works Carrillo et al. (2010a, b); Fornasier and Solombrino (2014); Bongini et al. (2017); Albi et al. (2017a), we discuss the resulting mean-field optimal control problem: that obtained as the number of agents tends to and the micro-state of the population is replaced by an agent density function .

The rest of the paper is organized as follows: In Section 2 we revisit the Cucker-Smale model and results on consensus emergence and control. In Section 3 we address the OCCP via first-order necessary optimality conditions and its numerical realisation. Section 4 introduces the sparse OCCP and its approximation. Concluding, in Section 5 we present a mean-field modelling approach for OCCP when the number of agents is sufficiently large.

We consider a set of agents with state interacting under second-order Cucker-Smale type dynamics

(2.1) | ||||

(2.2) |

where is a communication kernel of the type

(2.3) |

and we use the notation , . Both and are indistinctly used for the -norm, while stands for the -norm. We will focus on the study of consensus emergence, i.e. the convergence towards a configuration in which

(2.4) |

Note that for a system of the type (Optimal consensus control of the Cucker-Smale model)-(Optimal consensus control of the Cucker-Smale model), a consensus configuration will remain as such without any external intervention, and positions will evolve in planar formation. The emergence of consensus as a self-organization phenomenon, either by a sufficiently cohesive initial configuration or a strong interaction , is a problem of interest in its own right. To study consensus emergence, it is useful to define the bilinear form

Note that for a population in consensus, . A solution to (Optimal consensus control of the Cucker-Smale model)-(Optimal consensus control of the Cucker-Smale model) tends to a consensus configuration if and only if

Analogously, we define . We briefly recall some well-known results on self-organized consensus emergence.

###### Theorem 1

Unconditional consensus emergence (Cucker and Smale (2007); Carrillo et al. (2010b)). Given an interaction kernel with and , the Cucker-Smale dynamics (Optimal consensus control of the Cucker-Smale model)-(Optimal consensus control of the Cucker-Smale model) convergence asymptotically to consensus, i.e. , for .

###### Theorem 2

then the Cucker-Smale dynamics (Optimal consensus control of the Cucker-Smale model)-(Optimal consensus control of the Cucker-Smale model) convergence asymptotically to consensus.

In this work we are concerned with inducing consensus through the synthesis of an external forcing term in the form

(2.5) | ||||

(2.6) | ||||

(2.7) |

where the control signals and a compact subset of .

In this section we entertain the problem of obtaining a forcing term which will either induce consensus on an initial configuration that would otherwise diverge, or which accelerates the rate of convergence for initial data that would naturally self-organise. Formally, for and given a set of admissible control signals for the entire population, we seek a solution to the minimisation problem

(3.8) |

with the running cost defined as

(3.9) |

While existence of a minimiser of (3.8) follows from the smoothness and convexity properties of the system dynamics and the cost, the Pontryagin Minimum Principle Pontryagin et al. (1962) yields first-order necessary conditions for the optimal control. Let be adjoint variables associated to , then the optimality system consists of a solution satisfying (2.5)-(2.7) along with the adjoint equations

(3.10) | ||||

(3.11) | ||||

(3.12) |

and the optimality condition

(3.13) |

The adjoint system (3.10)-(3.13) is used to implement a gradient descent method for the numerical realisation of the optimal control law. It can be readily verified that the gradient of the cost in (3.8) is given by

(3.14) |

obtained by differentiating (3.13) with respect to . With this expression for , the gradient iteration is presented in Algorithm 1.

(3.15) |

In Algorithm 1, the gradient is first obtained by integrating the foward-backward optimality system, and then the step in 4) is chosen as in the Barzilai-Borwein method, see Barzilai and Borwein (1988).

Figure 3.1 shows a comparison of the free two-dimensional dynamics of a sample initial condition and the system under an approximation to the optimal control found with this algorithm. A Runge-Kutta 4th order scheme was used to integrate the differential equations for the state and the adjoint with end time , time step (resulting on points for the time discretisation), and a stopping tolerance for the gradient norm of . The condition is chosen such that consensus would not be reached naturally; long-time numerics of the free system show that converges to an asymptotic value around . In the controlled setting, consensus is reached rapidly as can be seen from the trajectories themselves, as well as from the fast convergence of the functional to zero. Furthermore, the norm of the control also decreases in time as the system is steered towards and into the self-organisation region.

In this section, we address the problem of enforcing sparsity on the optimal consensus strategy. As shown in Albi et al. (2017b); Bongini and Fornasier (2014); Caponigro et al. (2013); Kalise et al. (2017), one way to do it is by using as control cost the -norm in the minimization problem (3.8), instead of the standard squared -norm . However, the choice of the non-differentiable control cost gives rise to a non-smooth cost functional , for which gradient-based numerical solvers like the one presented in Section Optimal consensus control of the Cucker-Smale model are not directly suitable.

To circumvent the nonsmoothness of , we shall resort to a metaheuristic procedure known as particle swarm optimization (PSO).

First introduced in Kennedy and Eberhart (1995); Shi and Eberhart (1998), PSO is a numerical procedure that solves a minimization problem by iteratively trying to improve a candidate solution. PSO solves the problem by generating a population of points in the discrete control state space of solutions called particles. Each particle is treated as a point in this -dimensional space with coordinates , and the cost functional is evaluated at each of these points. The best previous position (i.e., the one for which the cost functional is minimal) of any particle is recorded, together with the index of the best particle among all the particles. We let then the particles evolve according to the system

where are two constant parameters and are two random variables with support in .

PSO is a metaheuristic since it makes few or no assumptions about the problem being optimized and can search very large spaces of candidate solutions: in particular, PSO does not use the gradient of the problem being optimized, which means PSO does not require that the optimization problem be differentiable. However, it yields to a decrease of the cost function.

Notice that whenever the dimension of the control space is very large (that is, either , or are large), the problem suffers from the curse of dimensionality, as the evaluation of at all the particles and the subsequent search for the best position becomes prohibitively expensive. To mitigate this difficulty, we shall optimise within a nonlinear model predictive control (NMPC) loop with short prediction horizon.

For a prediction horizon of steps, for and a discrete time version of the dynamics (Optimal consensus control of the Cucker-Smale model)-(Optimal consensus control of the Cucker-Smale model), we minimise the following performance index

(4.16) |

for some -norm with , generating a sequence of controls from which only the first term is taken to evolve the dynamics from to . The system’ state is sampled again and the calculations are repeated starting from the new current state, yielding a new control and a new predicted state path. Although this approach is suboptimal with respect to the full time frame optimisation presented in Section 3, in practice it produces very satisfactory results.

For , the NMPC approach recovers an instantaneous controller, whereas for it solves the full time frame problem (3.8). Such flexibility is complemented with a robust behaviour, as the optimisation is re-initialized every time step, allowing to address perturbations along the optimal trajectory. For further references, see Mayne et al. (2000).

We now report the results of the numerical simulations of (3.8) with and together with the setup PSO-NMPC described above. The aim is to check whether the optimal control obtained with the -control cost is sparser than the one obtained with the -norm. To do so, we shall compare their norms at each time, since a sparse control will be equal to 0 most of the time.

Starting from an initial configuration that does not converge to consensus, we compare the effect of a different NMPC horizon and of a different -control cost for in (4.16) on the optimal control strategy.

In this case, the controller is optimizing periods ahead instead of the full time frame. Figure 4.2 shows the controlled dynamics of the agents with the optimal control obtained for (top) and (bottom). Both controls decisively improve the alignment behaviour with respect to the uncontrolled dynamics. Figure 4.3 shows the behaviour of the functional: for both controlled dynamics, the velocity spread goes steadily to 0.

To see how sparse the controls are, for each control strategy in Figure 4.4 we show the corresponding heat map, i.e., the matrix such that contains the norm of the control acting on the -th agent at step . The stronger the control, the brighter the entry shall be: we can notice that the heat map for is sparser than the one for , being concentrated on few bright spots. This corroborates the findings of Bongini and Fornasier (2014); Caponigro et al. (2013), where the sparsifying powers of an -control cost were shown.

We now consider the case . Since the effects of the controls on the dynamical systems are the same as for the case , we just show the heat map of the controls to compare their sparsity, see Figure 4.5. We notice that the heat map of the control is less bright than the one for , showing that the -cost is again able to enforce sparsity better than the standard -cost.

The final black strip in the heat map means that the system has reached the consensus region and the optimal control is just the zero vector. Hence, in this particular case the control obtained with the -cost is actually better than the control in steering the system towards a consensus configuration.

In this section we consider the continuous control problem that results in the limit of the discrete problem from Section Optimal consensus control of the Cucker-Smale model as .

Formally, the forced Cucker-Smale dynamics (2.5)-(2.7) can be written continuously as a Vlasov-type transport equation

(5.17) |

where and are respectively a probability density for the state and a forcing term. An equivalent minimisation problem can be posed:

(5.18) |

for fixed and a running cost given by the expression:

(5.19) |

The solutions of the discrete control problem (3.8) converge to that of the continuous problem (5.17) in some sense, see Fornasier and Solombrino (2014). This can be verified numerically by fixing an initial distribution and studying the sequence solutions of the discrete problem with initial conditions sampled from said distribution; a subsequence is known to converge as . Besides the solution, the optimal value of the objective functional is also expected to converge, which can be verified.

Initial conditions without natural consensus were constructed by sampling from a superposition of two Gaussian distributions and letting . The marginal distributions of on and in are thus given by\@footnotemark\@footnotetextHere stands for the bivariate Normal distribution with mean vector and positive semi-definite covariance matrix .:

(5.20) |

where is the identity matrix, see Figure 5.6.

A sequence of such discrete problems were solved for several values of . Figure 5.7 shows the comparison between the free and controlled trajectories for various values of . The Runge-Kutta scheme was used to solve the differential equations for the state and the adjoint with end time , time step (resulting on points for the time discretisation), and a stopping tolerance of .

Figure 5.8 shows the evolution of the optimal cost , which appears to be of order as expected for the convergence as . Figure 5.9 shows the marginal distribution of on for the free and forced settings with the same scale; the controlled case yields a singular distribution indicating consensus. Figure 5.10 shows a heat map of the optimal control . We observed that the average norm of the control is of as ; furthermore the time at which the control is nearly zero is roughly constant for large . Table 5.1 shows the evolution of the number of optimisation iterations (i.e. loops on Algorithm 1) as well as the computation CPU time in hours; notice that the number of iterations remains roughly constant, while the computation time scales quadratically in .

Agents () | 50 | 100 | 150 | 200 | 250 | 300 |
---|---|---|---|---|---|---|

Iterations () | 26 | 25 | 24 | 27 | 28 | 28 |

Time () | 0.1 | 0.1 | 0.3 | 0.7 | 1.1 | 1.4 |

350 | 400 | 450 | 500 | 550 | 600 | |

27 | 27 | 28 | 28 | 25 | 28 | |

1.8 | 2.5 | 3.3 | 4.3 | 4.9 | 6.8 | |

650 | 700 | 750 | 800 | 850 | 900 | |

28 | 27 | 28 | 28 | 28 | 28 | |

8.2 | 8.8 | 10.7 | 12.2 | 14.4 | 15.1 | |

950 | 1000 | 1050 | 1100 | 1150 | 1200 | |

28 | 29 | 27 | 28 | 28 | 27 | |

17.4 | 19.0 | 19.8 | 22.8 | 25.0 | 28.1 | |

1250 | 1300 | 1350 | 1400 | 1450 | 1500 | |

27 | 28 | 28 | 28 | 28 | 28 | |

28.7 | 33.5 | 35.6 | 39.5 | 41.3 | 44.7 | |

1550 | 1600 | 1650 | 1700 | 1750 | 1800 | |

28 | 28 | 28 | 28 | 28 | 28 | |

49.1 | 51.7 | 52.8 | 58.8 | 59.8 | 65.0 | |

1850 | 1900 | 1950 | 2000 | |||

27 | 29 | 28 | 28 | |||

65.4 | 75.3 | 77.4 | 82.8 |

## References

- Albi et al. (2017a) Albi, G., Choi, Y.P., Fornasier, M., and Kalise, D. (2017a). Mean-field control hierarchy. Appl. Math. Optim., 76(1), 93–175.
- Albi et al. (2017b) Albi, G., Fornasier, M., and Kalise, D. (2017b). A Boltzmann approach to mean-field sparse feedback control. IFAC-PapersOnLine, 50(1), 2898 – 2903.
- Barzilai and Borwein (1988) Barzilai, J. and Borwein, J.M. (1988). Two-Point Step Size Gradient Methods. IMA J. Numer. Anal., 8(1), 141–148.
- Bongini and Fornasier (2014) Bongini, M. and Fornasier, M. (2014). Sparse stabilization of dynamical systems driven by attraction and avoidance forces. Netw. Heterog. Media, 9(1), 1–31.
- Bongini et al. (2015) Bongini, M., Fornasier, M., and Kalise, D. (2015). (Un)conditional consensus emergence under perturbed and decentralized feedback controls. Discrete Contin. Dyn. Syst. Ser. A, 35(9), 4071–4094.
- Bongini et al. (2017) Bongini, M., Fornasier, M., Rossi, F., and Solombrino, F. (2017). Mean-field pontryagin maximum principle. J. Opt. Theory Appl., 175(1), 1–38.
- Borzí and Schulz (2011) Borzí, A. and Schulz, V. (2011). Computational Optimization of Systems Governed by Partial Differential Equations. Society for Industrial and Applied Mathematics.
- Borzì and Wongkaew (2015) Borzì, A. and Wongkaew, S. (2015). Modeling and control through leadership of a refined flocking system. Math. Models Methods Appl. Sci., 25(2), 255–282.
- Caponigro et al. (2013) Caponigro, M., Fornasier, M., Piccoli, B., and Trélat, E. (2013). Sparse stabilization and optimal control of the Cucker-Smale model. Math. Control Relat. Fields, 3, 447–466.
- Carrillo et al. (2010a) Carrillo, J.A., Fornasier, M., Rosado, J., and Toscani, G. (2010a). Asymptotic flocking dynamics for the kinetic Cucker-Smale model. SIAM J. Math. Anal., 42, 218–236.
- Carrillo et al. (2010b) Carrillo, J.A., Fornasier, M., Toscani, G., and Vecil, F. (2010b). Particle, Kinetic, and Hydrodynamic Models of Swarming. In G. Naldi, L. Pareschi, and G. Toscani (eds.), Mathematical Modeling of Collective Behavior in Socio-Economic and Life Sciences, Series: Modelling and Simulation in Science and Technology, 297–336. Birkhäuser, Boston, MA.
- Cucker and Smale (2007) Cucker, F. and Smale, S. (2007). Emergent behavior in flocks. IEEE Trans. Automat. Control, 52(5), 852–862.
- Deroo et al. (2012) Deroo, F., Ulbrich, M., Anderson, B.D.O., and Hirche, S. (2012). Accelerated iterative distributed controller synthesis with a barzilai-borwein step size. In 2012 IEEE 51st IEEE Conference on Decision and Control (CDC), 4864–4870.
- Fornasier and Solombrino (2014) Fornasier, M. and Solombrino, F. (2014). Mean-field optimal control. ESAIM Control Optim. Calc. Var., 20(4), 1123–1152.
- Ha et al. (2010) Ha, S.Y., Ha, T., and Kim, J.H. (2010). Emergent behavior of a Cucker-Smale type particle model with nonlinear velocity couplings. IEEE Trans. Automat. Control, 55(7), 1679–1683.
- Ha and Liu (2009) Ha, S.Y. and Liu, J.G. (2009). A simple proof of the Cucker-Smale flocking dynamics and mean-field limit. Commun. Math. Sci., 7(2), 297–325.
- Kalise et al. (2017) Kalise, D., Kunisch, K., and Rao, Z. (2017). Infinite horizon sparse optimal control. J. Opt. Theory Appl., 172(2), 481–517.
- Kennedy and Eberhart (1995) Kennedy, J. and Eberhart, R. (1995). Particle swarm optimization. In Proceedings of IEEE International Conference on Neural Networks IV, volume 1000, 1942–1948. IEEE.
- Mayne et al. (2000) Mayne, D.Q., Rawlings, J.B., Rao, C.V., and Scokaert, P.O. (2000). Constrained model predictive control: Stability and optimality. Automatica, 36(6), 789–814.
- Perea et al. (2009) Perea, L., Gómez, G., and Elosegui, P. (2009). Extension of the Cucker-Smale control law to space flight formations. AIAA Journal of Guidance, Control, and Dynamics, 32, 527–537.
- Pontryagin et al. (1962) Pontryagin, L.S., Boltyanskii, V.G., Gamkrelidze, R.V., and Mishchenko, E.F. (1962). The mathematical theory of optimal processes. Interscience Publishers John Wiley & Sons, Inc. New York-London.
- Shi and Eberhart (1998) Shi, Y. and Eberhart, R. (1998). A modified particle swarm optimizer. In Evolutionary Computation Proceedings, 1998. IEEE World Congress on Computational Intelligence., The 1998 IEEE International Conference on, 69–73. IEEE.