Secure and PrivacyPreserving Average Consensus
Abstract.
Average consensus is fundamental for distributed systems since it underpins key functionalities of such systems ranging from distributed information fusion, decisionmaking, to decentralized control. In order to reach an agreement, existing average consensus algorithms require each agent to exchange explicit state information with its neighbors. This leads to the disclosure of private state information, which is undesirable in cases where privacy is of concern. In this paper, we propose a novel approach that enables secure and privacypreserving average consensus in a decentralized architecture in the absence of any trusted thirdparties. By leveraging homomorphic cryptography, our approach can guarantee consensus to the exact value in a deterministic manner. The proposed approach is lightweight in computation and communication, and applicable to timevarying interaction topology cases. A hardware implementation is presented to demonstrate the capability of our approach.
1. Introduction
As a building block of distributed computing, average consensus has been an active research topic in computer science and optimization for decades (Degroot, 1974; Lynch, 1996). In recent years, with the advances of wireless communications and embedded systems, particularly the advent of wireless sensor networks and the InternetofThings, average consensus is finding increased applications in fields as diverse as automatic control, signal processing, social sciences, robotics, and optimization (OlfatiSaber et al., 2007).
Conventional average consensus approaches employ the explicit exchange of state values among neighboring nodes to reach agreement on the average computation. Such an explicit exchange of state information has two disadvantages. First, it results in breaches of the privacy of participating nodes who want to keep their data confidential. For example, a group of individuals using average consensus to compute a common opinion may want keep secret their individual personal opinions (Tsitsiklis, 1984). Another example is power systems where multiple generators want to reach agreement on cost while keeping their individual generation information private (Zhang and Chow, 2011). Secondly, storing or exchanging information in the plaintext form (without encryption) is vulnerable to attackers which try to steal information by hacking into the communication links or even the nodes. With the increased number of reported attack events and the growing awareness of security, keeping data encrypted in storage and communications has become the norm in many applications, particularly many realtime sensing and control systems such as the power systems and wireless sensor networks.
To address the pressing need for privacy and security, recently, several relevant average consensus approaches have been proposed. Most of these approaches use the idea of obfuscation to mask the true state values by adding carefullydesigned noise on the state. Such approaches usually exploit tools such as meansquare statistics (Mo and Murray, 2017) or “differential privacy” which is heavily used for database privacy in computer science (Nozari et al., 2015; Huang et al., 2015; Manitara and Hadjicostis, 2013). Although enhances privacy, such noisebased obfuscation also unavoidably affects the performance of average consensus, either directly preventing converging to the true value, or making convergence only achievable in the statistical meansquare sense. Furthermore, these approaches normally rely on the assumption of timeinvariant interaction graph, which is difficult to satisfy in many practical applications where the interaction patterns may vary due to node mobility or fading communication channels.
Neither can the above noisebased approaches protect nodes from attackers which try to steal information by hacking into the nodes or the communication channels. To improve resilience to such attacks, a common approach is to employ cryptography. However, it is worth noting that although cryptography based approaches can easily provide privacy and security when a trusted thirdparty is available, like in the multiparty computation (Lagendijk et al., 2013), their extension to completely decentralized average consensus without any trusted thirdparties are extremely difficult due to the difficulties in the decentralized management of keys. In fact, in the only reported result incorporating cryptography into decentralized average consensus (Lazzeretti et al., 2014), privacy is obtained by paying the price of depriving participating nodes from access to the final consensus value, although partial information such as a binary decision is still retrievable for participating nodes.
In this paper, we propose a homomorphic cryptography based approach that can guarantee privacy and security in decentralized average consensus even in the presence of a timevarying interaction graph. Different from existing noisebased privacypreserving approaches which can only achieve average consensus in the statistic case, our approach can guarantee convergence to the exact average value in a deterministic manner. Unlike the existing cryptography based average consensus approach in (Lazzeretti et al., 2014), this approach allows every participating nodes to access the exact final value. Furthermore, the approach is completely decentralized and lightweight in computation, which makes it easily applicable to resource restricted systems.
The outline of this paper is as follows. Section 2 reviews the protocol used for average consensus problem and the homomorphic cryptography, particularly the Paillier cryptosystem. Our encrypted protocol is introduced in Section 3. In Section 4 we provide a proof of convergence and bounds of critical parameter, followed by a systematic discussion of privacy guarantees as well as security enforcement mechanisms in Section 5. Implementation issues and a physical implementation example are presented in Section 6. The conclusion is drawn in Section 7.
2. Background
In this section we briefly review the average consensus problem and the homomorphic encryption.
2.1. Average Consensus
We follow the same convention as in (OlfatiSaber et al., 2007) where a network of nodes is represented by a graph with node set , edge set , and a weighted adjacency matrix which satisfies if and 0 otherwise. The set of neighbors of a node is denoted as
(1) 
Throughout this paper we assume that the graph is undirected and connected. Therefore, is symmetric
(2) 
Note that the superscript denotes that the weights are timevarying. Sometimes we drop for the sake of notation simplicity, but it is worth noting that all discussions in the paper are always applicable under timevarying weights. To achieve average consensus, namely converging of all states to the average of initial values, i.e., , one commonlyused update rule for the continuoustime (CT) domain is
(3) 
The counterpart for discrete time (DT) is
(4) 
where is a constant step size residing in the range .
2.2. Homomorphic Encryption
Our method to protect privacy and security is to encrypt the state. To this end, we briefly introduce a cryptosystem, more specifically the publickey cryptosystem which is applicable in open and dynamic networks without the assist of any trusted third party for key management. Many popular cryptosystems such as RSA (Rivest et al., 1978), ElGamal (ElGamal, 1985), and Paillier (Paillier, 1999) are publickey cryptosystems. In this paper we focus on the Pailler cryptosystem which provides the following basic functions:

Key generation:

Choose two large prime numbers and of equal bitlength and compute .

Let .

Let where is the Euler’s totient function.

Let which is the modular multiplicative inverse of .

The public key is then .

The private key is then .


Encryption ():
Recall the definitions of and where is the greatest common divisor of and .
Choose a random .

The ciphertext is given by , where .


Decryption ():

Define the integer division function .

The plaintext is .

A cryptosystem is homomorphic if it allows certain computations to be carried out on the encrypted ciphertext. The Paillier cryptosystem is additive homomorphic because the ciphertext of , i.e., , can be obtained from and directly:
(5)  
The dependency on random numbers and is explicitly shown in (5), yet they play no role in the decryption. For the sake of readability, the following shorthand notation will be used instead:
(6) 
Moreover, if we multiply the same ciphertext times, we can obtain
(7) 
Notice however, the Paillier cryptosystem is not multiplicative homomorphic because in (7) is in the plaintext form. Furthermore, the existence of the random number in Paillier cryptosystem gives it resistance to dictionary attacks (Goldreich, 2001) which infer a key to an encrypted message by systematically trying all possibilities, like exhausting all words in a dictionary. Moreover, since Paillier cryptography only works on numbers that can be represented by binary strings, we multiply a realvalued state by a large integer before converting it to a binary string so as to ensure small quantization errors. The details will be discussed in Sec. 6.1.
3. Confidential Interaction Protocol
In this section, we propose a completely decentralized, thirdparty free confidential interaction protocol that can guarantee average consensus while protecting the privacy of all participating nodes. Instead of adding noise to hide the states, our approach combines encryption with randomness in the system dynamics, i.e., the coupling weights , to prevent two communicating parties in a pairwise interaction from disclosing information to each other. In this way the states are free from being contaminated by covering noise, guaranteeing a deterministic convergence to the exact average value.
In this section we present details of our confidential interaction protocol based on (3) and (4). In particular we show how a node can obtain the weighted difference (8) between itself and any of its neighbor without disclosing each other’s state information:
(8)  
subject to 
Plugging the state difference (8) into (3) gives a new formulation of continuoustime average consensus
(9) 
Similarly, we can rewrite the discretetime consensus update rule as
(10) 
Notice that in a decentralized system it is impossible to protect the privacy of both nodes in a pairwise interaction if the protocol (8) is used without a third party distributing secret . This is due to the fact that even if we encrypt all the intermediate steps, if one node, for instance , has access to , it can still infer the value of through . From now on, for the sake of simplicity in bookkeeping, we omit the superscript in . But it is worth noting that all the results hold for timevarying weights.
We solve this problem by constructing each weight as the product of two random numbers, namely , with (resp. ) generated by and only known to node (resp. , here is a positive value denoting the range in implementations which will be explained in detail later). We will show later that this weight construction approach renders two interacting nodes unable to infer each other’s state while guaranteeing convergence to the average. Next, without loss of generality, we use a pair of connected nodes () to illustrate the idea (cf. Fig. 1). For simplicity, we assume that the states and are scalar. Each node maintains its own public and private key pairs .
Due to symmetry, we only show how node obtains the weighted state difference , i.e., the flow . Before starting the information exchange, node (resp. ) generates its new nonnegative random number (resp. ) which is within a certain range in implementation. First, node sends its encrypted negative state as well as the public key to node . Note that here the subscript in denotes encryption using the public key of node . Node then computes the encrypted weighted difference following the three steps below:

Encrypt with ’s public key : .

Compute the difference directly in ciphertext:
(11) 
Compute the weighted difference in ciphertext:
(12)
Then returns to . After receiving , decrypts it using the private key and multiplies the result with to get the weighted difference :
(13)  
In a similar manner, the exchange produces for who then decrypts the message and multiplies the result by its own multiplier to get
(14)  
After each node collects the weighted differences from all neighbors, it updates its state with (3) or (4) accordingly.
Several remarks are in order:

The construction of each as the product of two random numbers and is key to guarantee that the weights are symmetric, i.e., , which is crucial for average consensus (OlfatiSaber et al., 2007).

does not have the private key of and cannot see which is encrypted in .

Given , cannot solve for because is only known to .

At each iteration, realvalued states are converted to fixed point representation for encryption; the weighted differences are converted back to real values for update.

We encrypt because it is more difficult to compute subtraction in ciphertext. The issue regarding encrypting negative values using Paillier is discussed in Sec. 6.
4. Theoretical Analysis of Convergence
In this section, we discuss theoretically the convergence property under the confidential interaction protocol and how to set the appropriate values for the multiplier (and for DT).
4.1. Convergence for ContinuousTime Consensus
Let denote the augmented state vector of all nodes. The network dynamics in (3) can be rewritten as:
(15) 
where is the timevarying Laplacian matrix defined by
(16) 
Theorem 1 ().
Proof: It is already known that average consensus can be achieved if for all time , there exists a constant such that is true for some (OlfatiSaber et al., 2007; Moreau, 2005). Noting that the weights obtained from the confidential interaction protocol in Sec. 3 are random and independent of each other, the proof can be obtained by following the line of reasoning in (OlfatiSaber et al., 2007; Moreau, 2005).
4.2. Convergence for DiscreteTime Consensus
In discretetime domain (4) can be rewritten as
(18) 
where is the Perron matrix and is the timevarying Laplacian matrix defined by
(19) 
Theorem 2 ().
Proof: The proof can be obtained by following the similar line of reasoning of Corollary 2 in (OlfatiSaber et al., 2007).
Remark 1 ().
Since the framework allows timevarying weighted adjacency matrix for the discretetime domain or for the discretetime domain, it can easily be extended to the case with switching interaction graphs according to (Moreau, 2005).
5. Analysis of Privacy and Security
Privacy and security are often used interchangeably in the literature but here we make the distinction explicit. Among the control community privacy is equivalent to the concept of unobservability. Privacy is also closely related to the concept of semantic security from cryptography (Goldreich, 2001). Both concepts essentially concern with an honestbutcurious adversary which is interested in learning the states of the network but conforms to the rules of the system. Security, on the other hand, deals with a broader issue which includes learning the states as well as the possibilities of exploiting the system to cause damages.
5.1. Privacy Guarantees
Our protocol provides protection against an honestbutcurious adversary or an observer eavesdropping the communication. In the literature, an honestbutcurious adversary is usually defined as a node who follows all protocol steps correctly but is curious and collects received data in an attempt to learn some information about other participating parties. An observer eavesdropping the communication is usually defined as an adversary who is able to intercept exchanged messages and read the bits within. In this paper, for the sake of simplicity, we generally refer to both types of adversaries as honestbutcurious adversaries.
The Paillier encryption algorithm is known to provide semantic security, i.e., Indistinguishability under Chosen Plaintext Attack (INDCPA) (Paillier, 1999). As a result, the recipient of the first transmission cannot see the value of at any time. We now prove that an honestbutcurious adversary cannot infer the initial state of a neighbor even if it can accumulate and correlate the return messages in multiple steps (except in a trivial case that should always be avoided, as explained in Theorem 4).
As per the naming convention in cryptography, it is customary to name the legitimate sender and receiver participants as (Alice) and (Bob), and the adversary as (Eve).
Theorem 3 ().
Assume that all nodes follow the confidential interaction protocol. An honestbutcurious node Eve cannot learn the initial state of a neighboring node Alice if Alice is also connected to another legitimate node Bob.
Proof: Without loss of generality, we consider the connection configuration illustrated in Fig. 2 (a) where Eve can interact with both Alice and Bob. If Eve cannot infer the state of Alice or Bob in this configuration, neither can it when either the Alice—Eve connection or the Bob—Eve connection is removed which reduces the information accessible to Eve.
From the perspective of the honestbutcurious node Eve, the measurements seen at each time step are . In matrix form, define these observations as
where:
(21) 
It can be easily derived that after steps, the measurements collected by Eve are given by
(22) 
where the observability matrix is given by
(23) 
with being the Perron matrix defined in (18). Note that the entries of and are unknown to Eve because and are randomly chosen by Alice and Bob respectively. Therefore, the ability for Eve to infer the state of other nodes cannot be analyzed using conventional observability based approach in e.g., (Pequito et al., 2014; Alaeddini et al., 2017).
We propose a new analysis approach based on the solvability of systems of equations. From (22) it can be seen that Eve can establish equations based on received information from time instant 0 to . Given that after consensus, Eve can know the final state of other nodes which is equal to its own final state (represent it as ), it can establish one more equation
(24) 
which makes the number of employable equations to . If there are more than unknowns involved in these equations, then it is safe to say that Eve cannot solve the equations and get the initial states of and . In fact, the confidential interaction protocol introduces unknown parameters , which, in combination with unknown to Eve, will make the total number of unknowns to . Therefore, the honestbutcurious Eve cannot use the accessible system of equations in (22) to solve for the initial states of and .
Remark 2 ().
Following the same line of reasoning, it can be obtained that an honestbutcurious node Eve 1 cannot infer the initial state of a neighboring node Alice if Alice is also connected to another honestbutcurious node Eve 2 that does not collude with Eve 1.
Based on the analysis framework, we can also obtain a situation in which it is possible for Eve to infer other nodes’ states which should be avoided.
Theorem 4 ().
If a node Alice is connected to the rest of the network only through an (or a group of colluding) honestbutcurious node(s) Eve, then Alice’s initial state can be inferred by Eve.
Proof: If Alice is directly connected to multiple honestbutcurious nodes that collude with each other, then these nodes can share information with each other to cooperatively estimate Alice’s state, and hence can be regarded as one node. Therefore, we just consider the case where Alice is only connected to one honestbutcurious node Eve, as illustrated in Fig. 2 (b). In this case, from the perspective of the honestbutcurious node Eve, the measurement seen at each time step is . Similar to the proof of Theorem 3, we can write the measurements accessible to Eve in a matrix form , where
(25) 
After steps, the measurements collected by Eve are given by
(26) 
with the observability matrix having the same form as (23).
Now in the equations collected by Eve in (26), there are unknowns . However, after converging to average consensus, Eve will be able to know the final state of other nodes (the same as its final state), which enables it to construct another equation about the initial states like (24). This will make the total number of equations equal to the total number of involved unknowns and make solving initial state of possible.
Next we use an example to illustrate that it is indeed possible for Eve to infer the state of Alice if Eve is Alice’s only neighbor. Consider the configuration in Fig. 2 (b). Eve receives from Alice. In addition, when the protocol converges after K steps, Eve knows the final state which is identical for all the nodes. The initial value of Alice can be simply inferred by Eve through
(27)  
Therefore this single connection configuration should always be avoided, which is also required by other dataobfuscation based privacy protocols, for instance in (Manitara and Hadjicostis, 2013) and (Mo and Murray, 2017).
5.2. Security Solution
Due to the additive homomorphic property, the Paillier cryptosystem is vulnerable to active adversaries who are able to alter the message being sent through the channel. Although such adversaries cannot find out the exact states of the communicating nodes, they can still inflict significant damage to the system.
Consider the scenario where the communication from node Alice to Bob is intercepted by an active adversary Eve (cf. Fig. 3 (a)). Since Alice’s public key is sent along with , Eve may use the additive homomorphism to inject an arbitrary noise to the original message to sway it to . If Bob has no way to tell if the received message has been modified, Eve may exploit this vulnerability to make the network either converge to a wrong value or not converge at all.
In applications where security is of prime concern, it is imperative to be able to verify the integrity of any incoming message. We propose to attach a digital signature to the exchanged message in the confidential interaction protocol, based on which the recipient can verify possible modifications during communication. The signature requires an additional pair of public/private keys () and a hash function , and is represented as (, , where is an unforgeable certificate assigned by an authority. The additional private key is sent so that Bob can decrypt and check if the resulting matches the received in terms of the hash operation (cf. Fig. 3 (b)). Because without the public key , Eve cannot forge a valid signature (that can be decrypted by Bob), any Eve’s attempt to modify will cause a mismatch between received and decrypted in terms of the hash operation .
6. Implementation Details
In addition to the constraints imposed on and , there are other technical issues that must be addressed for the implementation of our confidential interaction protocol.
6.1. Quantization
Realworld applications typically have which are represented by floating point numbers in modern computing architectures. On the contrary, encryption algorithms only work on unsigned integers. Define the casting function and its inverse as
(28) 
where maps the input to the nearest integer in . For the Paillier cryptosystem, this mapping is equivalent to the rounding operation, hence the step size is which is uniform. Consequently the maximum quantization error is bounded by
(29) 
In practice we choose a sufficiently large value for so that the quantization error is negligible. This is exactly how we convert the state of a node from real value to a fixed length integer and back to a floating point number. The conversion is performed at each iteration of the protocol.
6.2. Subtraction and Negative Values
Another issue is how to treat the sign of an integer for encryption. (Kogiso and Fujita, 2015) solves this problem by mapping negative values to the end of the group where is given by the public key. We offer an alternative solution by taking advantages of the fact that encryption algorithms blindly treat bit strings as unsigned integers. In our implementation all integer values are stored in fixlength integers (i.e., long int in C) and negative values are left in two’s complement format. Encryption and intermediate computations are carried out as if the underlying data were unsigned. When the final message is decrypted, the overflown bits (bits outside the fixed length) are discarded and the remaining binary number is treated as a signed integer which is later converted back to a real value.
6.3. Implementation on Raspberry Pi
To confirm the effectiveness of the secure and privacypreserving average consensus approach in realworld cyberphysical systems, we implemented the algorithm on six Raspberry Pi boards with 64bit ARMv8 CPU and 1 GB RAM.
In the implementation, the communication was conducted through WiFi based on the “sys/socket.h” C library. Paillier encrption and decryption were realized using the “libpaillier0.8” library from (Pai, lier). To obtain in a pairwise interaction, a node employs a request message to initialize the interaction and the other node replies with a response message. In a multinode network, for a node to be able to simultaneously receive requests and responses from multiple neighbors, parallelism needed to be introduced. The “pthread” C library was used to generate multiple parallel threads to handle incoming requests and responses. Each time a node receives a request/response, it generates a new thread to handle it and immediately listens for more requests. Because in the implementation, it is impossible to start all nodes simultaneously, a counter is introduced on each node and its value is embedded in each request/response packet to help nodes make sure that they are on the same pace. For 64 byte encryption key, the size of the actual packet is 144 bytes, which includes all necessary headers and stuffing bytes. For each interaction, the average processing latency was 7.8 ms, which is acceptable for most realtime cyberphysical systems. The implementation result is given in Fig. 4, which shows that perfect consensus can be achieved.
7. Conclusions
In this paper we proposed a decentralized secure and privacypreserving protocol for the network average consensus problem. In contrast to previous approaches where the states are covered with random noise which unavoidably affects the convergence performance, we encode randomness to the system dynamics with the help of an additive homomorphic cryptosystem which allows the convergence to the exact average in a deterministic manner. The protocol also allows easy incorporation of active attacker defending mechanisms. Although our approach has higher computational complexity compared to the unencrypted alternatives, experimental results on Raspberry Pi confirm that the computational burden is manageable on resourcerestricted cyberphysical systems.
Acknowledgement
The authors would like to thank Christoforos Hadjicostis and Yilin Mo for their comments on an initial draft of this article. This work was supported in part by the National Science Foundation under Grant 1738902.
References
 (1)
 Pai (lier) http://acsc.cs.utexas.edu/libpaillier/. Paillier Library.
 Alaeddini et al. (2017) A. Alaeddini, K. Morgansen, and M. Mesbah. 2017. Adaptive Communication Networks with Privacy Guarantees. In Proceedings of 2017 American Control Conference. 4460–4465.
 Degroot (1974) M. H. Degroot. 1974. Reaching a Consensus. J. Amer. Statist. Assoc. 69, 345 (1974), 118–121.
 ElGamal (1985) T. ElGamal. 1985. A Public Key Cryptosystem and a Signature Scheme Based on Discrete Logarithms. Springer Berlin Heidelberg, Berlin, Heidelberg, 10–18. DOI:http://dx.doi.org/10.1007/3540395687_2
 Goldreich (2001) O. Goldreich. 2001. Foundations of Cryptography: volume 2, Basic Applications. Cambridge University Press.
 Huang et al. (2015) Z. Huang, S. Mitra, and N. Vaidya. 2015. Differentially Private Distributed Optimization. In Proceedings of the 2015 International Conference on Distributed Computing and Networking (ICDCN ’15). ACM, New York, NY, USA, Article 4, 10 pages. DOI:http://dx.doi.org/10.1145/2684464.2684480
 Kogiso and Fujita (2015) K. Kogiso and T. Fujita. 2015. Cybersecurity enhancement of networked control systems using homomorphic encryption. In Proceedings of the 54th IEEE International Conference on Decision and Control. 6836–6843.
 Lagendijk et al. (2013) R. L. Lagendijk, Z. Erkin, and M. Barni. 2013. Encrypted signal processing for privacy protection: Conveying the utility of homomorphic encryption and multiparty computation. IEEE Signal Processing Magazine 30, 1 (Jan 2013), 82–105.
 Lazzeretti et al. (2014) R. Lazzeretti, S. Horn, P. Braca, and P. Willett. 2014. Secure multiparty consensus gossip algorithms. In 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 7406–7410. DOI:http://dx.doi.org/10.1109/ICASSP.2014.6855039
 Lynch (1996) N. A. Lynch. 1996. Distributed Algorithms. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
 Manitara and Hadjicostis (2013) N. E. Manitara and C. N. Hadjicostis. 2013. Privacypreserving asymptotic average consensus. In 2013 European Control Conference (ECC). 760–765.
 Mo and Murray (2017) Y. Mo and R. M. Murray. 2017. Privacy Preserving Average Consensus. IEEE Trans. Automat. Control 62, 2 (Feb 2017), 753–765.
 Moreau (2005) L. Moreau. 2005. Stability of multiagent systems with timedependent communication links. IEEE Trans. Automat. Control 50, 2 (Feb 2005), 169–182. DOI:http://dx.doi.org/10.1109/TAC.2004.841888
 Nozari et al. (2015) E. Nozari, P. Tallapragada, and J. Cortés. 2015. Differentially Private Average Consensus with Optimal Noise Selection. IFACPapersOnLine 48, 22 (2015), 203 – 208.
 OlfatiSaber et al. (2007) R. OlfatiSaber, J. A. Fax, and R. M. Murray. 2007. Consensus and cooperation in networked multiagent systems. Proc. IEEE 95, 1 (2007), 215–233. DOI:http://dx.doi.org/10.1109/JPROC.2006.887293
 Paillier (1999) P. Paillier. 1999. PublicKey Cryptosystems Based on Composite Degree Residuosity Classes. Springer Berlin Heidelberg, Berlin, Heidelberg, 223–238. DOI:http://dx.doi.org/10.1007/354048910X_16
 Pequito et al. (2014) S. Pequito, S. Kar, S. Sundaram, and A. P. Aguiar. 2014. Design of communication networks for distributed computation with privacy guarantees. In In Decision and Control (CDC), 2014 IEEE 53rd Annual Conference on. 1370–1376.
 Rivest et al. (1978) R. L. Rivest, A. Shamir, and L. Adleman. 1978. A Method for Obtaining Digital Signatures and Publickey Cryptosystems. Commun. ACM 21, 2 (Feb. 1978), 120–126.
 Tsitsiklis (1984) J. N. Tsitsiklis. 1984. Problems in decentralized decision making and computation. Ph.D. Dissertation.
 Zhang and Chow (2011) Z. Zhang and M. Y. Chow. 2011. Incremental cost consensus algorithm in a smart grid environment. In 2011 IEEE Power and Energy Society General Meeting. 1–6. DOI:http://dx.doi.org/10.1109/PES.2011.6039422