# Approximate Consensus Multi-Agent Control Under Stochastic Environment with Application to Load Balancing

###### Abstract

The paper is devoted to the approximate consensus problem for networks of nonlinear agents with switching topology, noisy and delayed measurements. In contrast to the existing stochastic approximation-based control algorithms (protocols), a local voting protocol with nonvanishing step size is proposed. Nonvanishing (e.g., constant) step size protocols give the opportunity to achieve better convergence rate (by choosing proper step sizes) in coping with time-varying loads and agent states. The price to pay is replacement of the mean square convergence with an approximate one. To analyze dynamics of the closed loop system, the so-called method of averaged models is used. It allows to reduce analysis complexity of the closed loop system. In this paper the upper bounds for mean square distance between the initial system and its approximate averaged model are proposed. The proposed upper bounds are used to obtain conditions for approximate consensus achievement.

The method is applied to the load balancing problem in stochastic dynamic networks with incomplete information about the current states of agents and with changing set of communication links. The load balancing problem is formulated as consensus problem in noisy model with switched topology. The conditions to achieve the optimal level of load balancing (in the sense that if no new task arrives, all agents will finish at the same time) are obtained.

The performance of the system is evaluated analytically and by simulation. It is shown that the performance of the adaptive multi-agent strategy with the redistribution of tasks among “connected” neighbors is significantly better than the performance without redistribution. The obtained results are important for control of production networks, multiprocessor, sensor or multicomputer networks, etc.

## I Introduction

The problems of control and distributed interaction in dynamical networks attracted much attention in the last decade. A number of survey papers [1, 2], monographs [3, 4, 5], special issues of journals [6, 7, 8] and edited volumes [9] have been published in this area. This interest has been driven by applications in various fields, including, for example, multiprocessor networks, transportation networks, production networks, coordinated motion for unmanned flying vehicles, submarines and mobile robots, distributed control systems for power networks, complex crystal lattices, and nanostructured plants [1, 2, 3, 4, 5, 6, 7, 8, 9, 10].

Despite a large number of publications, satisfactory solutions have been obtained mostly for a restricted class of problems (see [1, 2, 3, 4, 5, 6, 7, 8, 9] and references therein). Factors such as nonlinearity of agent dynamics, switching topology, noisy and delayed measurements may significantly complicate the solutions. Additional important factors can be the limited transmission rate in the channel and discretization phenomenon. In the presence of various disruptive factors, asymptotically exact consensus may be hard to achieve, especially in a time-varying environment. For such cases, approximate consensus problems should be examined.

In this paper, we investigate the approximate consensus problem in a multi-agent stochastic system with nonlinear dynamics, measurements with noise and delays, and uncertainties in the topology and in the control protocol. Such a problem is important for the control of production networks, multiprocessors, sensor or multicomputer networks, etc. As an example, the load balancing system in a network with noisy and delayed information about the load and with switched topology is studied. In contrast to the existing stochastic approximation-based control algorithms (protocols), local voting with nonvanishing step size is considered.

In the literature, the average consensus problem on graphs with noisy measurements of its neighbors’ states, under general imperfect communications is considered in [11, 12], where stochastic approximation-type algorithms with decreasing to zero step size are used. Noisy convergence with nonvanishing step-size was studied in [13], but the control step parameters were chosen differently for different agents and the considered network scenario is a specific one. The stochastic gradient-like (stochastic approximation) methods have also been used in the presence of stochastic disturbances and noise [14, 15, 16, 11, 17]. However, in these works [11, 13, 14, 15, 16, 17], the considered network scenarios are often specific ones, much simpler than the more general scenario considered in this paper.

In [18], the considered network scenario is most close to that in this paper. In [18], a stochastic approximation type algorithm was proposed for solving consensus problem and justified for the group of cooperating agents that communicate with imperfect information in discrete time, under the conditions of dynamic topology and delay. Under some general assumptions a necessary and sufficient condition was proved for the asymptotic mean square consensus when step size tends to zero and with a simple form of dynamic functions: in the paper in [18]. However, as to be shown in the results, under dynamic state changes for the agents (e.g., feeding new jobs), using step sizes that decrease to zero may greatly affect the convergence rate. In our paper, we consider a more general case of functions and step size nondecreasing to zero.

In [19, 20, 21, 22] the efficiency of stochastic approximation algorithms with constant step size was studied for some specific cases with different properties and constraints than these considered in this paper.

As for the load balancing problem, numerous articles are devoted to it (e.g., [23, 24, 25, 26, 27, 28, 29]), indicating the relevance of this topic. However, most of these articles do not consider noise or delays. While in a single computer this assumption could be rather realistic, if we consider networked systems, noise, delays and possible link-“breaks” need to be justified. The load balancing problem in centralized networks with noisy information about load and agent productivity was analyzed in [30, 31]. In such a centralized network, there is a load broker that redistributes jobs among agents. However, in case when each agent is not connected with every other agent, it is not possible to choose one of the agents as the load broker, who would distribute the jobs among the agents. In this case, it is necessary to consider decentralized networks. However, to the best of our knowledge, few results for load-balancing in such distributed networks are available.

In this paper, the results of our previous works [32, 33, 34, 35, 36] are summarized, extended and improved. In particular, we relax the assumption of the weights boundedness of the control protocol, replacing it by the boundedness of its variances. In addition, new and much larger size simulation experiments were performed and results added.

The contributions of the paper are several-fold. First, the approximate consensus problem for a general network scenario is investigated, which is a network of nonlinear agents with switching topology, noisy and delayed measurements. Second, in this approximate consensus problem, we specifically consider a more general state function and step size nondecreasing to zero in the local voting protocol. Third, to analyze the dynamics of the stochastic discrete systems, the method of averaged models (Derevitskii-Fradkov-Ljung (DFL)-scheme) [37, 38, 39] is adopted. Forth, the consensus conditions for the case without delays in measurements and for the case with delays are obtained. In addition, to demonstrate the use of the obtained results, the load balancing problem in a distributed network is studied. Furthermore, simulation results validating the analysis are presented.

The rest of the paper is organized as follows. In Section II, the basic concepts of graph theory are introduced, the consensus problem is posed, and some preliminary results for consensus conditions in non-stochastic system are considered. In Section III, the basic assumptions are described and the consensus conditions for the case without delays in measurements and for the case with delays are obtained. In Sections IV, the load balancing problem is considered, and analytical and simulation results are presented and discussed. Section V contains conclusion remarks.

## Ii Preliminaries

### Ii-a Concepts of Graph Theory

First we present the notation used in this article. The agent index is used as a superscript and not as an exponent.

Consider a network as a set of agents (nodes) .

A directed graph (digraph) consists of a set and a set of directed edges . Denote the neighbour set of node as .

We associate a weight with each edge . Matrix is called an adjacency or connectivity matrix of the graph. Denote as the corresponding graph. The in-degree of node is the number of edges having as head. The out-degree of node is the number of edges having as tail. If the in-degree equals to the out-degree for all nodes the graph is said to be balanced. Define the weighted in-degree of node as the -th row sum of : and is a corresponding diagonal matrix. The symbol stands for the Laplacian of graph .

A directed path from to is a sequence of nodes , such that . Node is said to be connected to node if a directed path from to exists. The distance from to is the length of the shortest path from to . The graph is said to be strongly connected if and are connected for all distinct nodes .

A directed tree is a digraph where each node , except the root, has exactly one parent node so that . We call a subgraph of if and . The digraph is said to contain a spanning tree if there exists a directed tree as a subgraph of .

The following fact from graph theory will be important.

The symbol denotes a maximal in-degree of the graph . In correspondence with the Gershgorin Theorem [41], we can deduce another important property of the Laplacian: all eigenvalues of the matrix have nonnegative real part and belong to the circle with center on the real axis at the point and with radius which equals to .

Let denote eigenvalues of the matrix . We arrange them in ascending order of real parts: . By virtue of Lemma 1, if the graph has a spanning tree then is a simple eigenvalue and all other eigenvalues of are in the open right half of the complex plane.

The second eigenvalue of matrix is important for analysis in many applications. It is usually called Fiedler eigenvalue. For undirected graphs it was shown in [3] that:

(1) |

and for the connected undirected graph

(2) |

where is the longest distance between two nodes and .

For all vectors the -norm will be used, i.e. a square root of the sum of all its elements squares.

For reader’s convenience, we provide a list of key notation used in this paper.

— the set of nodes | |

— the set of edges, | |

weight of edge | |

digraph with nodes and edges | |

neighbour set of node | |

: | adjacency or connectivity matrix |

graph defined by the adjacency matrix | |

— weighted in-degree of node (-th row sum of ) | |

maximal in-degree of the graph | |

— diagonal matrix of waighted in-degree of | |

— Laplacian of graph | |

eigenvalues of the matrix | |

diameter, the longest distance between two nodes | |

— volume, the sum of in-degrees | |

— maximal set of communication links | |

state of agent at time | |

noisy and delayed measurement agent obtains from agent at time | |

noise in at time | |

integer-valued delay in at time | |

maximal delay | |

control actions | |

protocol with topology | |

subset of at time | |

step sizes of the local voting protocol | |

weight parameter of the local voting protocol | |

matrix of the local voting protocol | |

identity matrix of size | |

vector consisting of units | |

left eigenvector of matrix : | |

time to -consensus | |

consensus value | |

mathematical expectation | |

conditional expectation under the condition | |

conditional expectation with respect to -algebra | |

probability that the delay equals | |

adjacency matrix of the averaged system | |

queue length of the atomic elementary jobs of the agent at time | |

productivity of the agent at time | |

new job received by agent at time | |

implementation time of jobs at time | |

load of agent at time | |

— average residual | |

maximum deviation from the average load on the network |

### Ii-B Problem Statement

#### Ii-B1 The network model

Consider a dynamic network of agents that collaborate to solve a problem that each cannot solve alone.

The concepts of graph theory will be used to describe the network topology. Let the dynamic network topology be modeled by a sequence of digraphs , where changes with time. The corresponding adjacency matrices are denoted as . The maximal set of communication links is .

We assume that a time-varying state variable corresponds to each agent of the graph at time . Its dynamics are described for the discrete time case as

(3) |

or for the continuous time case as

(4) |

with some functions , depending on states in the previous time and control actions .

Each agent uses its own state (possibly noisy) to form its control strategy

(5) |

and if , noisy and delayed measurements of its neighbors’ states

(6) |

where are noises, is an integer-valued delay, and is a maximal delay.

If then agent receives information from agent for the purposes of feedback control.

#### Ii-B2 The locol voting protocol

###### Definition 1

A feedback on observations

(7) |

where is called a protocol (control algorithm) with topology .

In this paper, we consider the local voting protocol:

(8) |

where are step sizes of control protocol, . We set for other pairs and denote as the matrix of the control protocol.

#### Ii-B3 Consensus concepts

In this paper, various consensus concepts will be employed, which are defined as follows.

###### Definition 2

Agents and are said to agree in a network at time if and only if .

###### Definition 3

The network is said to reach a consensus at time if .

###### Definition 4

The network is said to achieve asymptotic mean square consensus if there exists for all .

###### Definition 5

The network is said to reach an average consensus at time if all nodes’ states drive to the same constant steady-state value: , where is the average of the initial states of the agents

(9) |

Here, this value does not depend on the graph structure. The average consensus problem is important in many applications. For instance, in wireless sensor networks each agent measures some quantity (e.g., temperature, salinity content, etc.) and it is desired to determine the best estimate of the measured quantity, which is the average if all sensors have identical noise characteristics.

###### Definition 6

The network is said to achieve -consensus at time if there exists a variable such that for all .

###### Definition 7

is called time to -consensus, if the network achieves -consensus for all .

###### Definition 8

The network is said to achieve mean square -consensus at time if there exists a variable such that for all .

###### Definition 9

The network is said to achieve asymptotic mean square -consensus at time if and there exists a variable such that for all .

### Ii-C Preliminary Results

Consider the particular case of dynamic systems on graphs when the second term in (3) has a simple form: , for all agents , and all observations are made without noise and delays:

Denote and column vectors obtained by the vertical concatenation of corresponding variables. Control protocol (8) can be rewritten in a matrix form:

(10) |

The dynamics (3) for the discrete time case is described by:

(11) |

and for the continuous-time case by:

(12) |

With (10), the dynamics of the closed-loop system for the discrete time case takes the form:

(13) |

where is matrix of size of ones and zeros on the diagonal, and for the continuous time case the dynamics takes the form

(14) |

We will show that the control protocol (8) with and provides consensus asymptotically for both discrete and continuous-time models. Similar results can be found in [14, 42].

#### Ii-C1 The discrete-time case

###### Lemma 2

Indeed, for the discrete case the equation (13) turns into

(16) |

where the Perron matrix has one simple eigenvalue equal to one and all others are inside the unit circle if the condition (15) is satisfied. Since the sum of row elements of the Laplacian equals to zero, the sum of row elements of matrix equals to one, i.e. vector consisting of units is a right eigenvector of corresponding to the unit eigenvalue. The unit eigenvalue is simple if the graph has a spanning tree. All other eigenvalues are inside the unit circle. Let denote the left eigenvector of matrix which is orthogonal to . Consequently, if the graph has a spanning tree then in the limit of we get

(17) |

i.e. an asymptotic consensus is reached. The consensus value equals to the normalized linear combination of initial states with weights equal to elements of the left eigenvector of matrix

(18) |

This value depends on the graph topology and, consequently, on connection links between agents.

###### Lemma 3

If the graph is balanced then the sums of the rows of the Laplacian is equal to the sum of the corresponding columns, and this property is transferred to the matrix then , and the consensus value equals to the initial values average

and does not depend on the topology of the graph.

#### Ii-C2 The continuous-time case

###### Lemma 4

For the continuous-time case we have

(20) |

Let and be left and right orthonormal eigenvectors of the matrix corresponding to its ordered eigenvalues . If the graph has a spanning tree then is a simple eigenvalue and all other eigenvalues of are in the open right half of complex plane. Thus, the system (20) is partially stable with one pole at the origin and the rest are in the open left half plane.

For the first left eigenvector of matrix we have

(21) |

i.e. is invariant, that is constant and independent of the states of agents. Thus, .

We apply the modal expansion and rewrite the state vector in terms of eigenvalues and eigenvectors of the matrix . If all the eigenvalues of are simple (in fact, it is only important that is simple), then

(22) |

In the limit of we get or , i.e. an asymptotic consensus is reached.

In the continuous-time case, we focus on the problem of reaching an approximate -consensus ().

###### Lemma 5

From (22) by evaluating the square of the norm of the first term we can obtain

(24) |

From here we have the expression (23) for the time to -consensus in the system (20).

Here we highlight that, in contrast to the earlier results using in [42], we have considered instead of inside the argument of -function.

## Iii Main Results

In this section, we present the main results of this paper. All proofs are included in the Appendix.

### Iii-a Main Assumptions

Let be the underlying probability space corresponding to the sample space, the collection of all events, and the probability measure respectively.

For the remaining article, we assume that the following conditions are satisfied:

A1. functions are Lipschitz in and : , and for any fixed the function is such that . Note that, following from this Lipschitz condition, the growth rate is bounded: .

A2. a) the noises are centered, independent and have bounded variance .

b) appearances of variable edges in graph are independent random events.

c) weights in the control protocol are independent random variables with .

d) there exists a finite quantity : with probability 1 and integer-valued delays are independent, identically distributed random variables taking values with probabilities .

More over, all these random variables and matrices are mutually independent.

The next assumption is for a matrix constructed as follows. Specifically, if , we add new “fictitious” agents whose states at time equal to the corresponding states of the “real” agents at the previous time: . Then, is a matrix of size , where , with

(25) |

Here, the operation is a remainder of division, and is a division without remainder.

Note that if , this definition of network topology (of matrix of size ) is reduced to

(26) |

Also note that we have defined a matrix in such a way that . We assume that the following condition is satisfied for this network topology matrix:

A3. Graph has a spanning tree, and for any edge among the elements of the matrix , there exists at least one non-zero.

### Iii-B The Case without Delay in Measurement

We first consider the case where there is no delay in measurement, i.e. .

Rewrite the dynamics of the agents in the vector-matrix form:

(27) |

where is the vector of dimension :

(28) |

To analyze the stochastic system behavior at the particular choice of the coefficients (parameters), in the control protocol, it is common to use the method of averaged models [37], (also called ODE approach [38], or Derevitskii-Fradkov-Ljung (DFL)-scheme [43]), which we also adopt in this paper.

Specifically in our use, the method of averaged models consists on the approximate replacement of the initial stochastic difference equation (27) by an ordinary differential equation:

(29) |

where

(30) |

where is the adjacency matrix whose construction is introduced in the previous subsection.

Note, that if the last part of the condition A1 is not satisfied, then instead of (30) one can use the following definition

(31) |

According to [37], the trajectories of from (27)-(28) and of from (29)-(30) are close in a finite time interval. Here and below let .

In the following theorem the upper bounds for mean square distance between the initial system and its averaged continuous model will be given.

###### Theorem 1

If conditions A1, A2a–c are satisfied, function is smooth in , for any , and , then there exists such that for the following inequality holds:

(32) |

where , and are some constants.

We return to the problem of achieving consensus. Assume that, in the averaged continuous model (29)-(30), the -consensus is reached over time, i.e. all components of the vector become close to some common value for all . Then, we have the following result.

###### Theorem 2

Let the conditions A1, A2a–c be satisfied, functions are smooth by , for any , , for the continuous model (29)-(30) the -consensus is achieved for time , consensus protocol parameters are chosen so that and for some constants the following inequality holds

(33) |

then the mean square -consensus in the stochastic discrete system (27)-(28) at time is achieved.

Consider an important special case where . In this case, the time to -consensus in the averaged continuous model (29)-(30) can be obtained from Lemma 5:

(34) |

Then, based on Theorem 2, the following important consequence is obtained.

###### Corollary 1

If for any , conditions A2a–c, A3 are satisfied, functions are smooth in , for any , then for any arbitrarily small positive number for any denoted in (34), when is selected as sufficiently small

(35) |

at time in the stochastic discrete system (27)-(28), the mean square -consensus for agents is achieved, where are some constants and is the closest to the imaginary axis eigenvalue of matrix with nonzero real part.

### Iii-C The General Case with Delay in Measurement

We now consider the general case, where .

Let for , and denote as the extended state vector , where is a vector consisting of such that , i.e. this is a value with positive probability involved in the formation of at least one of the controls. To simplify, we assume that so introduced an extended state vector is , i.e. it includes all the components with all kinds of delays not exceeding .

Rewrite the dynamics of the agents in vector-matrix form:

(36) |

where is the following matrix of size :

(37) |

where is the identity matrix of size , and — vector function of the arguments:

(38) |

containing non-zero components only on the first places.