Minimum cost distributed source coding over a network

Minimum cost distributed source coding over a network

Aditya Ramamoorthy Aditya Ramamoorthy is with the Department of Electrical and Computer Engineering, Iowa State University, Ames IA 50011, USA (email: adityar@iastate.edu). The material in this paper was presented in part at the IEEE International Symposium on Information Theory, Nice, France June 2007
Abstract

This work considers the problem of transmitting multiple compressible sources over a network at minimum cost. The aim is to find the optimal rates at which the sources should be compressed and the network flows using which they should be transmitted so that the cost of the transmission is minimal. We consider networks with capacity constraints and linear cost functions. The problem is complicated by the fact that the description of the feasible rate region of distributed source coding problems typically has a number of constraints that is exponential in the number of sources. This renders general purpose solvers inefficient. We present a framework in which these problems can be solved efficiently by exploiting the structure of the feasible rate regions coupled with dual decomposition and optimization techniques such as the subgradient method and the proximal bundle method.

{keywords}

minimum cost network flow, distributed source coding, network coding, convex optimization, dual decomposition.

I Introduction

In recent years the emergence of sensor networks [1] as a new paradigm has introduced a number of issues that did not exist earlier. Sensor networks have been considered among other things by the military for battlefields, by ecologists for habitat monitoring and even for extreme event warning systems. These networks consist of tiny, low-power nodes that are typically energy constrained. In general, they also have low-computing power. Thus, designing efficient sensor networks requires us to address engineering challenges that are significantly different from the ones encountered in networks such as the Internet. One unique characteristic of sensor networks is that the data that is sensed by different sensor nodes and relayed to a terminal is typically highly correlated. As an example consider a sensor network deployed to monitor the temperature or humidity levels in a forest. The temperature is not expected to vary significantly over a small area. Therefore we do expect that the readings corresponding to nearby sensors are quite correlated. It is well-known that the energy consumed in transmission by a sensor is quite substantial and therefore efficient low power methods to transfer the data across the network are of interest. This leads us to investigate efficient techniques for exploiting the correlation of the data while transmitting it across the network. There are multiple ways in which the correlation can be exploited.

• The sensor nodes can communicate amongst themselves to inform each other of the similarity of their data and then transmit only as much data as is required. This comes at the cost of the overhead of inter-sensor communication.

• The sensors can choose to act independently and still attempt to transmit the compressed data. This strategy is likely to be more complicated from the receiver’s point of view.

Usually the terminal to which the data is transmitted has significantly more resources (energy, computing power etc.). Thus, the latter solution is more attractive from a network resource efficiency point of view. The question of whether the distributed compression of correlated sources can be as efficient as their compression when the sources communicate with each other was first considered and answered in the affirmative by Slepian and Wolf in their famous paper [2]. A number of authors [3][4] have investigated the construction of coding techniques that achieve the Slepian-Wolf bounds and also proposed their usage in sensor networks [5].

New paradigms have also emerged recently in the area of network information transfer. Traditionally information transfer over networks has been considered via routing. Data packets from a source node are allowed to be replicated and forwarded by the intermediate nodes so that terminal nodes can satisfy their demands. However, network coding offers an interesting alternative where intermediate nodes in a network have the ability to forward functions of incoming packets rather than copies of the packets. The seminal work of Ahlswede et al. [6] showed that network coding achieves the capacity of single-source multiple-terminal multicast where all the terminals are interested in receiving the same set of messages from the source. This was followed by a number of works that presented constructions and bounds for multicast network codes [7][8]. More recently, there has been work [9][10] on characterizing rate regions for arbitrary network connections where the demands of the terminals can be arbitrary.

Given these developments in two different fields, a natural question to ask is how can one transmit compressible sources over a network using network coding and whether this can be done efficiently. This problem was considered by Song and Yeung [11] and Ho et al. [12]. They showed that as long as the minimum cuts between all nonempty subsets of sources and a particular terminal were sufficiently large, random linear network coding over the network followed by appropriate decoding at the terminals achieves the Slepian-Wolf bounds. The work of Ramamoorthy et al. [13] investigated the performance of separate source and network codes and showed that separation does not hold in general. Both these papers only considered capacity constraints on the edges of the network and did not impose any cost associated with edge usage.

In the networking context the problem of minimum cost network flow has been widely investigated. Here, every edge in the network has a cost per unit flow associated with it. The cost of a given routing solution is the sum of the costs incurred over all the links. The problem is one of finding network flows such that the demand of the terminals is satisfied at minimum cost. This problem has been very well investigated in the routing context [14]. The problem of minimum cost multicast using network coding was considered by Lun et al. [15] and they presented centralized and distributed solutions to it.

In this paper we consider the problem of minimum cost joint rate and flow allocation over a network that is utilized for communicating compressible sources. We consider the scenario when the compression is to be performed in a distributed manner. The sources are not allowed to communicate with each other. The main issue with joint rate and flow allocation is that typically the feasible rate region for the recovery of the sources (e.g. the Slepian-Wolf region) is described by a number of inequalities that is exponential in the number of sources. Thus, using a regular LP solver for solving the corresponding linear programming problem will be inefficient. In our work, we only consider networks where the links are independent and where transmission up to the link’s capacity is assumed to be error free. In general, the capacity region characterization of more complex networks such as wireless networks will need to take account issues such as interference. Moreover, it would introduce related issues such as scheduling. We do not consider these problems in this work.

I-a Main Contributions

The main contributions of this paper are as follows. We present a framework in which minimum cost problems that involve transmitting compressible sources over a network in a distributed manner can be solved efficiently. We consider general linear cost functions, allow capacity constraints on the edges of the network and consider the usage of network coding. The following problems are considered.

• Slepian-Wolf over a network. The sources are assumed to be discrete and memoryless and they need to be recovered losslessly [2] at the terminals of the network. We address the problem of jointly finding the operating rate vectors for the sources and the corresponding network flows that allow lossless recovery at the terminals at minimum cost.

• Quadratic Gaussian CEO over a network. A Gaussian source is observed by many noisy sensors and needs to be recovered at the terminal subject to a quadratic distortion constraint [16]. We present a solution to the problem of joint rate and network flow allocation that allows recovery at the terminal at minimum cost.

• Lifetime maximization of sensor networks with distortion constraints. A Gaussian source observed by many noisy sensors needs to be recovered at the terminal with a certain fidelity. We are interested in finding routing flows that would maximize the lifetime of the network.

We demonstrate that these problems can be solved efficiently by exploiting the structure of the feasible rate regions coupled with dual decomposition techniques and subgradient methods [17].

I-B Related Work

Problems of a similar flavor have been examined in several papers. Cristescu et al. considered the Networked Slepian-Wolf problem [18] and the case of lossy correlated data gathering over a network [19], but did not impose capacity constraints on the edges. Their solutions only considered very specific types of cost functions. The work of Li & Ramamoorthy [20, 21] and Roumy & Gesbert [22] considered a rate allocation under pairwise constraints on the distributed source code used for compression. The work of Liu et al. [23] and [24] considers a related problem, where they seek to minimize the total communication cost of a wireless sensor network with a single sink. They show that when the link communication costs are convex, then the usage of Slepian-Wolf coding and commodity flow routing is optimal. Moreover, they introduce the notion of distance entropy, and show that under certain situations the distance entropy is the minimum cost achieved by Slepian-Wolf coding and shortest path routing. They also propose hierarchical transmission schemes that exploit correlation among neighboring sensor nodes, and do not require global knowledge of the correlation structure. These schemes are shown to be order-optimal in specific situations. The main difference between our work and theirs, is the fact that we consider network coding and networks with multiple terminals. Moreover, in the case of general convex link cost functions, their focus is on showing that Slepian-Wolf coding and commodity flow routing is optimal. They do not consider the problem of actually finding the optimal flows and rates.

A problem formulation similar to ours was introduced by Barros et al. [25] but they did not present an efficient solution to it. The problem of exponentially many constraints has been noted by other authors as well [26][27].

The approach in our work is inspired by the work of Yu et al. [28]. However since our cost functions only penalize the usage of links in the network, we are effectively able to exploit the structure of the feasible rate region to make our overall solution efficient. In addition we explicitly derive the dual function and the corresponding update equations for maximizing it based on the specific structure of the rate region. Furthermore, we consider applications in network coding and lifetime maximization in sensor networks that have not been considered previously. In concurrent and independent work [29] presented some approaches similar to ours (see also [30], where the case of two sources is discussed). However our approach has been applied to the minimum cost quadratic Gaussian CEO problem over a network and lifetime maximization with distortion constraints that were not considered in [29].

A reviewer has pointed out that the problem of generalizing the Slepian-Wolf theorem to the network case was first considered by Han [31] in 1980. However, in [31] only networks with a single terminal were considered. In the single terminal case the corresponding flows can be supported by pure routing. Interestingly, in the same paper, Han references the work of Fujishige [32] that studies the optimal independent flow problem (this was also pointed by the same reviewer). Fujishige’s work considers a network flow problem that has polymatroidal [33] constraints for the source values and the terminal values. In particular, if there is only one terminal, then this algorithm provides an efficient solution to the minimum cost Slepian-Wolf problem over a network. However, it is unclear whether it can be extended to the case of multiple terminals and network coding. We discuss Fujishige’s work in more detail in Section III, after the precise problem has been formulated.

This paper is organized as follows. Section II overviews the notation and the broad setup under consideration in this paper. Section III formulates and solves the minimum cost Slepian-Wolf problem over a network, Section IV discusses the quadratic Gaussian CEO problem over a network and Section V presents and solves the problem of lifetime maximization of sensor networks when the source needs to be recovered under distortion constraints. Each of these sections also present simulation results that demonstrate the effectiveness of our method. Section VI discusses the conclusions and future work.

Ii Preliminaries

In this section we introduce the basic problem setup that shall be used in the rest of this paper. In subsequent sections we shall be presenting efficient algorithms for solving three different problems that fall under the umbrella of distributed source coding problems over a network. We shall present the exact formulation of the specific problem within those sections. We are given the following.

• A directed acyclic graph that represents the network. Here represents the set of vertices, the set of edges and is the capacity of the edge in bits/transmission. The edges are assumed to be error-free and the capacity of the edges is assumed to be rational. We are also given a set of source nodes where and a set of terminal nodes where . Without loss of generality we assume that the vertices are numbered so that the vertices correspond to the source nodes.

• A set of sources , such that the source is observed at source node . The values of the sources are drawn from some joint distribution and can be either continuous or discrete.

Based on these we can define the capacity region of the terminal with respect to as

 CTj={(R1,…,RNS):∀B⊆S,∑i∈BRi≤min-cut(B,Tj)}.

Thus, consists of inequalities that define the maximum flow (or minimum cut) from each subset of to the terminal . A rate vector can be transmitted from the source nodes to terminal via routing [14]. In the subsequent sections we shall consider different minimum cost problems involving the transmission of the sources over the network to the terminals.

Iii Minimum cost Slepian-Wolf over a network

Under this model, the sources are discrete and memoryless and their values are drawn i.i.d. from a joint distribution . The source node only observes for . The different source nodes operate independently and are not allowed to communicate. The source nodes want to transmit enough information using the network to the terminals so that they can recover the original sources, losslessly.

This problem was first investigated in the seminal paper of Slepian and Wolf [2] where they considered the sources to be connected to the terminal by a direct link and the links did not have capacity constraints. The celebrated result of [2] states that the independent source coding of the sources can be as efficient as joint coding when the sources need to be recovered error-free at the terminal.

Suppose that for the classical Slepian-Wolf problem, the rate of the source is . Let denote the vector of sources , for . The feasible rate region for this problem is given by

 RSW={(R1,…,RNS):∀B⊆S,∑i∈BRi≥H(XB|XBc)}

The work of Csiszár [34] showed that linear codes are sufficient to approach the Slepian-Wolf (henceforth S-W) bounds arbitrarily closely.

Note that the original S-W problem does not consider the sources to be communicating with the terminal (or more generally multiple terminals) over a network. Furthermore, there are no capacity constraints on the edges connecting the sources and the terminal. In situations such as sensor networks, where the sensor nodes are typically energy-constrained, we would expect the source nodes to be in communication with the terminal node over a network that is both capacity and cost-limited. Capacity constraints may be relatively strict since a significant amount of power is consumed in transmissions. The costs of using different links could be used to ensure that a certain part of the network is not overused resulting in non-uniform depletion of resources. Thus the problem of transmitting correlated data over a network with capacity constraints at minimum cost is of interest. We define an instance of the S-W problem over a network by .

The transmission schemes based on linear codes (such as those in [34]) are based on block-wise coding, i.e., each source encodes source symbols at a time. An edge with capacity bits/transmission can transmit bits per block. Conceptually, each edge can be regarded as multiple unit capacity edges, with each unit capacity edge capable of transmitting one bit per block. When communicating a block of length , we consider the graph , or equivalently the graph (where denotes a vector of ones) where splits each edge from into unit capacity edges.

To facilitate the problem formulation we construct an augmented graph where we append a virtual super source node to , so that

 V∗ = V∪{s∗}, E∗ = {(s∗,v)| v∈S}∪E,~{}and C∗ij = {Cij(i,j)∈E,H(Xj)~{}if~{}i=s∗~{}and~{}j∈S.

We let .

Definition 1

Feasibility. Consider an instance of the S-W problem over a network, . Let be the capacity region of each receiver with respect to . If

 RSW∩CTi≠∅,∀i=1,…,NR,

then the feasibility condition is said to be satisfied and is said to be feasible.

The next theorem (from [12]) implies that as long as the feasibility condition is satisfied, random linear network coding over followed by appropriate decoding at suffices to reconstruct the sources error-free at .

Theorem 1

Sufficiency of the feasibility condition [12]. Consider an instance of the S-W problem over a network, . If the feasibility condition (Definition 1) is satisfied, then random linear network coding over followed by minimum-entropy [34] or maximum-likelihood decoding can recover the sources at each terminal in with the probability of decoding error going to as .

The proof of the necessity of the feasibility condition can be found in [31].

It follows that if for all , it is sufficient to perform random linear network coding over a subgraph of where the feasibility condition continues to be satisfied. The question then becomes, how do we choose appropriate subgraphs?

For this purpose, we now present the formulation of the minimum cost S-W problem over a network.

Let represent the flow variable for edge over corresponding to the terminal for and represent the max-of-flows variable, for edge . Note that under network coding the physical flow on edge will be . The variable , represents the virtual flow variable over edge for terminal [15].

We introduce variables that represent the operating S-W rate variables for each terminal. Thus represents the rate vector for terminal . Let , represent the cost for transmitting at a unit flow over edge . We are interested in the following optimization problem that we call MIN-COST-SW-NETWORK.

 minimize~{}∑(i,j)∈Efijzijs. t.~{}~{}0≤x(Tk)ij≤zij≤C∗ij,~{}~{}(i,j)∈E∗,Tk∈T (1)
 ∑{j|(i,j)∈E∗}x(Tk)ij−∑{j|(j,i)∈E∗}x(Tk)ji=σ(Tk)i, (2) for i∈V∗,Tk∈T, x(Tk)s∗i≥R(Tk)i,~{}~{}for~{}i∈S,Tk∈T (3) R(Tk)∈RSW,~{}~{}for~{}Tk∈T (4)

where

 σ(Tk)i=⎧⎪⎨⎪⎩H(X1,X2,…,XNS)~{}~{}% if~{}i=s∗−H(X1,X2,…,XNS)~{}~{}if~{}i=Tk0~{}~{}otherwise (5)

The constraints in (1), (2) and (5) are precisely the formulation of the minimum cost single-source multiple terminal multicast with network coding for a total rate of . The constraint (3) enforces the flow (corresponding to terminal from through source ) to be at least . Constraint (4) ensures that the rate vectors belong to the Slepian-Wolf region . A proof that the total rate can be fixed to be exactly for each terminal can be found in Appendix-I.

Suppose there exists a feasible solution , for to MIN-COST-SW-NETWORK. Let

 V∗z = V, E∗z = {(i,j)∈E∗| zij>0},~{}and C∗zij = {zij~{}if~{}(i,j)∈E∗z0~{}otherwise.

We define the subgraph of induced by to be the graph and the corresponding graph over block length as . The subgraphs induced by can be defined analogously. We now show that if MIN-COST-SW-NETWORK is feasible then the subgraph induced by the feasible continues to satisfy the condition in definition 1 and therefore it suffices to perform random linear network coding over this subgraph followed by appropriate decoding at the terminals to recover the sources.

Lemma 1

Suppose that there exists a feasible solution , for to MIN-COST-SW-NETWORK. Then, random linear network coding over the subgraph induced by followed by maximum likelihood decoding at the terminals can recover the sources at each terminal in as .

Proof: To simplify the presentation we assume that all and are rational and the block length is large enough so that and are integral. For each terminal we shall show that over and then use Theorem 1.

Consider a terminal . We are given the existence of a feasible solution from which we can find the corresponding flow for denoted by . Now consider the subgraph of induced by . Since is feasible, it supports a rate of from to which implies (using Menger’s theorem [35]) that there exist edge-disjoint paths from to . Furthermore at least of those edge-disjoint paths connect source node (where ) to . It follows that if then the number of edge disjoint paths from to is greater than or equal to .

Now, note that which implies that for all

 N∑i∈BR(T1)i≥NH(XB|XBc).

This means that there exist at least edge-disjoint paths from to in the subgraph induced by which in turn implies that over the subgraph induced by . This holds for all , as we have a feasible for all the terminals. Finally induces a subgraph where this property continues to hold true for each terminal since , for all . Therefore for each terminal we have shown that over for all sufficiently large . Using Theorem 1 we have the required proof.

The formulation of MIN-COST-SW-NETWORK as presented above is a linear program and can potentially be solved by a regular LP solver. However the number of constraints due to the requirement that is that grows exponentially with the number of sources. For regular LP solvers the time complexity scales with the number of constraints and variables. Thus, using a regular LP solver is certainly not time-efficient. Moreover even storing the constraints consumes exponential space and thus using a regular LP solver would also be space-inefficient. In the sequel we present efficient techniques for solving this problem.

Iii-a Solving MIN-COST-SW-NETWORK via dual decomposition

Suppose that we are given an instance of the S-W problem over a network specified by . We assume that is feasible. The MIN-COST-SW-NETWORK optimization problem is a linear program and therefore feasibility implies that strong duality holds [36].

We shall refer to the variables , for as the primal variables. To simplify notation we let denote the vector of flow variables corresponding to terminal on the edges from the virtual super node to the source nodes in . We form the Lagrangian of the optimization problem with respect to the constraints . This is given by

 L(λ,z,x(T1),…,x(TNR),R(T1),…,R(TNR))=fTz+NR∑k=1λTk(R(Tk)−x(Tk)s∗),

where is the dual variable such that (where denotes component-wise inequality).

For a given , let denote the dual function obtained by minimizing over . Since strong duality holds in our problem we are guaranteed that the optimal value of MIN-COST-SW-NETWORK can be equivalently found by maximizing subject to [36]. Thus, if can be determined in an efficient manner for a given then we can hope to solve MIN-COST-SW-NETWORK efficiently.

Consider the optimization problem for a given .

 minimize~{}~{}fTz+NR∑k=1λTk(R(Tk)−x(Tk)s∗)s. t.~{}~{}0≤x(Tk)ij≤zij≤Cij,~{}~{}(i,j)∈E∗,Tk∈T∑{j|(i,j)∈E∗}x(Tk)ij−∑{j|(j,i)∈E∗}x(Tk)ji=σ(Tk)i, i∈V∗,Tk∈TR(Tk)∈RSW, Tk∈T.

We realize on inspection that this minimization decomposes into a set of independent subproblems shown below.

 minimize~{}~{}fTz−NR∑k=1λTkx(Tk)s∗s. t.~{}~{}0≤x(Tk)ij≤zij≤Cij,~{}~{}(i,j)∈E∗,Tk∈T∑{j|(i,j)∈E∗}x(Tk)ij−∑{j|(j,i)∈E∗}x(Tk)ji=σ(Tk)i, i∈V∗,Tk∈T (6)

and for each ,

 minimize~{}~{}λTkR(Tk)subject to~{}~{}R(Tk)∈RSW. (7)

The optimization problem in (6) is a linear program with variables and for and a total of constraints that can be solved efficiently by using a regular LP solver. It can also be solved by treating it as a minimum cost network flow problem with fixed rates for which many efficient techniques have been developed [14].

However each of the subproblems in (7) still has constraints and therefore the complexity of using an LP solver is still exponential in . However using the supermodularity property of the conditional entropy function , it can be shown that is a contra-polymatroid with rank function [37]. In addition, the form of the objective function is also linear. It follows that the solution to this problem can be found by a greedy allocation of the rates as shown in [33]. We proceed as follows.

1. Find a permutation such that .

2. Set

 R(Tk)π(1)=H(X{π(1)}|X{π(1)}c)~{}~{}andR(Tk)π(i)=H(X{π(1),…,π(i)}|X{π(1),…,π(i)}c)−H(X{π(1),…,π(i−1)}|X{π(1),…,π(i−1)}c)~{}for 2≤i≤NS. (8)

The previous algorithm presents us a technique for finding the value of efficiently. It remains to solve the maximization

 maxλ⪰0g(λ).

For this purpose we use the fact that the dual function is concave (possibly non-differentiable) and can therefore be maximized by using the projected subgradient algorithm [17]. The subgradient for can be found as [17].

Let represent the value of the dual variable at the iteration and be the step size at the iteration. A step by step algorithm to solve MIN-COST-SW-NETWORK is presented below.

1. Initialize .

2. For given solve

 minimize~{}~{}fTz−NR∑k=1(λik)Tx(Tk)s∗s. t.~{}~{}0≤x(Tk)ij≤zij≤Cij,~{}~{}(i,j)∈E∗,Tk∈T∑{j|(i,j)∈E∗}x(Tk)ij−∑{j|(j,i)∈E∗}x(Tk)ji=σ(Tk)i,for i∈V∗,Tk∈T

using an LP solver and for each ,

 minimize~{}~{}(λik)TR(Tk)subject to~{}~{}R(Tk)∈RSW (9)

using the greedy algorithm presented in (8).

3. Set for all . Goto step 2 and repeat until convergence.

While subgradient optimization provides a good approximation on the optimal value of the primal problem, a primal optimal solution or even a feasible, near-optimal solution is usually not available. In our problem, we seek to jointly find the flows and the rate allocations that support the recovery of the sources at the terminals at minimum cost. Thus, finding the appropriate flows and rates specified by the primal-optimal or near primal-optimal is important. Towards this end we use the method of Sherali and Choi [38].

We now briefly outline the primal recovery procedure of [38]. Let for be a set of convex combination weights for each . This means that

 k∑j=1μkj=1,~{}and~{}μkj≥0.

We define

 γjk=μkj/θk,~{}for~{}1≤j≤k,~{}and~% {}k≥1,

and let

 Δγmaxk≜max{γjk−γ(j−1)k:j=2,…,k}.

Let the primal solution returned by subgradient optimization at iteration be denoted by the vector and let the primal iterate be defined as

 (~z,~x,~R)k=k∑j=1μkj(z,x,R)j% ~{}~{}for k≥1. (10)

Suppose that the sequence of weights for and the sequence of step sizes are chosen such that

1. for all for each .

2. , as , and

3. as and for all , for some .

Then an optimal solution to the primal problem can be obtained from any accumulation point of the sequence of primal iterates .

Some useful choices for the step sizes and the convex combination weights that satisfy these conditions are given below (see [38]).

1. , for where , and and for all .

2. , for where and for all .

The strategy for obtaining a near-optimal primal solution for the MIN-COST-SW-NETWORK problem is now quite clear. We run the subgradient algorithm in the manner outlined above and keep computing the sequence of primal iterates and stop when the primal iterates have converged.

Iii-B Results

In this section we present results on the performance of our proposed algorithm. We generated graphs at random by choosing the position of the nodes uniformly at randomly from the unit square. Two nodes were connected with an edge of capacity if they were within a distance of of each other and were connected with an edge of capacity if they were within a distance of of each other. The orientation of the edges is from left to right. A certain number of nodes were declared to be sources, a certain number to be terminals and the remaining nodes were used for relaying.

Let the random vector at the sources be denoted by . As in [39], a jointly Gaussian model was assumed for the data sensed at the sources. Thus the pdf of the observations is assumed to be

 f(z1,z2,…,zNS)=1√2πNS√det(CZZ)×exp(−12(z−μ)TC−1ZZ(z−μ)),

where is the covariance of the observations. We assumed a correlation model where and when (where and are positive constants and is the distance between nodes and ). It is further assumed that the samples are quantized independently at all source nodes with the same quantization step that is sufficiently small. Under these conditions, the quantized random vector is such that

 H(X)≈h(Z)−NSlogΔ

as shown in [40] where represents the entropy of and represents the differential entropy of . We can also express the conditional entropy

 H(XB|XBc)≈12log((2πe)NS−|Bc|det(CZZ)det(CZBcZBc))−(NS−|Bc|)logΔ

We used these conditional entropies for the Slepian-Wolf region of the sources.

Figure 1 shows a network consisting of nodes, with source nodes, relay nodes and terminals. We chose , and for this example. The quantization step size was chosen to be . The cost of all the edges in the graph was set to .

For the subgradient algorithm, we chose for all and the step size . The averaging process ignored the first 50 primal solution due to their poor quality. We observe a gradual convergence of the cost of our solution to the optimal in Fig. 1.

Iii-B1 Remark 1

If one uses a regular LP solver to solve the problem, as noted above, the complexity would scale with the number of variables and constraints, that grow exponentially with the number of sources. However, one is guaranteed that the LP solver will terminate in a finite number of steps eventually. Our proposed algorithm uses the subgradient method with step sizes such that the recovered solution will converge to the optimal as the number of iterations go to infinity [17]. In general, it does not seem possible to claim convergence in a finite number of steps for this method. A discussion around convergence issues of the subgradient method can be found in Chap. 6 of [17]. We point out that in practice, we found the algorithms to converge well. Note also that even the description of the LP requires space that increases very quickly, therefore using the LP formulation becomes impractical even with a moderate number of sources.

We now discuss the work of Fujishige [32]. Towards this end we need to define the following quantities. A polymatroid is defined as a pair where is a finite set and is a function from to the positive reals, , that satisfies the axioms of a rank function. A vector , with entries indexed by the elements of is called an independent vector of if , for all .

Suppose that we have a directed graph , with linear costs , (as defined above) with two vertex subsets (source nodes) and (terminal nodes). Suppose that for each , a polymatroid is defined. An independent flow is a triple , such that: (i) is an independent vector of and is the vector of flows entering the network at the source nodes, (ii) is an independent vector of and is the vector of flows absorbed at the terminal nodes, and (iii) is a flow vector such that flow balance is satisfied at each node in . The algorithm in [32], returns an independent flow of maximum value, whose cost is minimum.

In the case of a single terminal, this algorithm can be used to solve the problem (as noted in [31]) as follows. The set of source nodes and the conditional entropy function, specify a contra-polymatroid. An equivalent polymatroidal representation can be found without much difficulty (see [31]). Thus, these specify and . If there is only one terminal, then one can simply define a (trivial) polymatroid on it. This specifies and .

The situation is different when one considers network coding and multiple terminals. The algorithm in [32], is only guaranteed to return one set of flows that satisfies flow balance at all source nodes, internal nodes and the terminals. It is possible to show the existence of instances where one set of flows will not simultaneously satisfy all the terminals, when one considers the problem. An example can be found in Figure 6 in [13]. Moreover, the objective function in , penalizes the maximum of the flows across the terminals at each node, which is different from the one in [32]. Thus, it is unclear whether this algorithm can be adapted to our problem in a straightforward manner.

Iv Quadratic Gaussian CEO over a network

In general, the problem of transmitting compressible sources over a network need not have the requirement of lossless reconstruction of the sources. This maybe due to multiple reasons. The terminal may be satisfied with a low resolution reconstruction of the sources to save on network resources or lossless reconstruction may be impossible because of the nature of sources. If a source is continuous then perfect reconstruction would theoretically require an infinite number of bits. Thus the problem of lossy reconstruction has also been an active area of research. In this section we shall consider the quadratic Gaussian CEO problem [16] over a network. We start by outlining the original problem considered by [16]. We then present the minimum cost formulation in the network context and present efficient solutions to it.

Consider a data sequence that cannot be observed directly. Instead, independent corrupted versions of the data sequence are available at a set of agents who are in communication with the Chief Estimation Officer (CEO) over different communication channels. The agents are not allowed to cooperate in any fashion. Suppose that the CEO requires the reconstruction of at an average distortion level of at most . Here, the distortion level is a metric of the fidelity of the reconstruction. Suppose agent communicates with the CEO at rate . The CEO problem [41] is one of studying the region of feasible rate vectors that allow the reconstruction of the data sequence under the prescribed distortion constraint. As in the Slepian-Wolf case there is a direct link between the agents and the terminal (or the CEO). The quadratic Gaussian CEO problem is the particular instance of the CEO problem when the data source is Gaussian and the distortion metric is mean squared error. A formal description of the problem follows.

Let represent a sequence of i.i.d. Gaussian random variables and where are i.i.d. Gaussian independent of with . Furthermore and are independent when .

Let be a small real number. The agent encodes a block of length from his observations (here, denotes a particular realization of the random variable ) using an encoding function of rate . The codewords from the sources are sent to the CEO who seeks to recover an estimate of the source message over time instants using a decoding function .

Definition 2

A rate vector is said to be achievable for a distortion level if for , there exists such that for all , there exist encoding functions and a decoding function such that where .

A complete characterization of the feasible rate region for a given distortion level , denoted by has been obtained in [42][43] and is given below.

 R(D)=⋃(r1,…,rNS)∈F(D)RD(r1,…,rNS) (11)

where

 RD(r1,…,rNS)≜{(R1,…,RNS):A⊆{1,…,NS},A≠ϕ,∑k∈ARk≥∑k∈Ark+12log(1σ2X+∑NSri=11−e−2riσ2i1σ2X+∑i∈Ac1−e−2riσ2i)},
 F(D)={(r1,…,rNS):ri≥0,1σ2X+NS∑i=11−e−2riσ2i≥1D}.

It is important to note that is convex [43]. Thus, in principle the minimization of a convex function of can be performed efficiently.

We are interested in the quadratic Gaussian CEO problem over a network. In line with our general setup presented in Section II, the source node in observes the process and encodes the observations at a rate . Once again we are interested in the minimum cost network flow problem with rates such that they permit the recovery of the source at the terminals with the desired level of fidelity, which in this case shall be measured by mean squared error.

We start by highlighting the differences between this problem and the minimum cost Slepian-Wolf over a network. In the previous subsection we observed that the work of Ho et al. shows that random linear network coding over a subgraph such that allows the lossless recovery of the sources at the terminals in and essentially the Slepian-Wolf theorem holds even in the network case with multiple terminals i.e. any rate vector that can be obtained by joint coding can be obtained by distributed coding even when there are multiple terminals. However, an analogous result in the case of the quadratic Gaussian CEO problem does not exist. Furthermore, the rate region for the quadratic Gaussian CEO problem over a general network is unknown. As a simple example, we may have a network where two source nodes are connected to a common intermediate node. The intermediate node can then combine the quantized observations from these source nodes to generate a new quantized observation such that a lower rate is possible. Thus the rate region given by the classical Gaussian CEO problem may not hold as the two codewords may be fused to produce a new codeword that enables lower rate transmission.

The first issue can be handled by assuming that there is only one terminal, i.e., and so that routing will suffice to transmit a rate vector belonging to to the terminal . Thus in this problem we shall not consider network coding. For the second issue we assume that the network operates in a separate compression and information transfer mode. The set of source nodes quantize their observations as they would in the original quadratic Gaussian CEO problem. After this source coding step, the network ignores any data correlations and routes the data as though it were incompressible. In general this separation of the compression and the information transfer is suboptimal, however it is likely to be a simple way of operating the network.

It is more convenient to cast this optimization in terms of the original graph rather than the augmented graph. The MIN-COST-QUAD-CEO-NETWORK problem becomes

 minimize~{}∑(i,j)∈Efijxijsubject to~{}~{}0≤xij≤Cij,~{}~{}~{}(i,j)∈E
 ∑{j|(i,j)∈E}xij−∑{j|(j,i)∈E}xji =0,i∈(S∪{T1})c (12) ∑{j|(i,j)∈E}xij−∑{j|(j,i)∈E}xji ≥Ri, i∈S (13) ∑{j|(i,j)∈E}xij−∑{j|(j,i)∈E}xji ≤−∑i∈SRi, i=T1 (14) R ∈R(D)

Here (12) enforces the flow balance at all nodes in except those in , (13) enforces the constraint that at least units of flow is injected at each source node and (14) ensures that at least is received at the terminal . For the MIN-COST-SW-NETWORK problem the total rate to be transmitted to could be fixed to as shown in the Appendix. However for the problem presented above fixing the total rate is not possible because of the nature of the inequalities specifying . A feasible solution to the optimization presented above would yield a routing solution such that the delivery of a rate vector belonging to is possible at terminal