Cut-Set Bounds on Network Information Flow
Explicit characterization of the capacity region of communication networks is a long standing problem. While it is known that network coding can outperform routing and replication, the set of feasible rates is not known in general. Characterizing the network coding capacity region requires determination of the set of all entropic vectors. Furthermore, computing the explicitly known linear programming bound is infeasible in practice due to an exponential growth in complexity as a function of network size. This paper focuses on the fundamental problems of characterization and computation of outer bounds for multi-source multi-sink networks. Starting from the known local functional dependencies induced by the communications network, we introduce the notion of irreducible sets, which characterize implied functional dependencies. We provide recursions for computation of all maximal irreducible sets. These sets act as information-theoretic bottlenecks, and provide an easily computable outer bound for networks with correlated sources. We extend the notion of irreducible sets (and resulting outer bound) for networks with independent sources. We compare our bounds with existing bounds in the literature. We find that our new bounds are the best among the known graph theoretic bounds for networks with correlated sources and for networks with independent sources.
The network coding approach introduced in [2, 3] generalizes routing by allowing intermediate nodes to forward coded combinations of all received data packets. This yields many benefits that are by now well documented [4, 5, 6, 7]. One fundamental open problem is to characterize the capacity region and the classes of codes that achieve capacity. The single session multicast problem is well understood. Its capacity region is characterized by max-flow/min-cut bounds and linear codes are optimal .
Significant complications arise in more general scenarios, involving multiple sessions. A computable characterization of the capacity region is still unknown. One approach is to develop bounds as the intersection of a set of linear constraints (specified by the network topology and sink demands) and the set of entropy functions (inner bound), or its closure (outer bound) [8, 9, 4]. An exact expression for the capacity region does exist, again in terms of . Unfortunately, this expression, or even the bounds [8, 9, 4] cannot be computed in practice, due to the lack of an explicit characterization of the set of entropy functions for three or more random variables. The difficulties arising from the structure of are not simply an artifact of the way the capacity region and bounds are written. It has been shown that the problem of determining the capacity region for multi-source network coding is completely equivalent to characterization of .
One way to resolve this difficulty is via relaxation of the bound, replacing the set of entropy functions with the set of polymatroids (which has a finite, polyhedral characterization). This results in a geometric bound that is in principle computable using linear programming . In practice however, the number of variables and constraints in this linear program both increase exponentially with the number of links in the network. This prevents numerical computation for any meaningful case of interest. An alternative approach is to seek graphical bounds based on functional dependence properties and cut sets in graphs derived from the original communications network.
The more difficult problems of characterization and computation of bounds for networks with correlated sources has received less attention than networks with independent sources. For a few special cases, necessary and sufficient conditions for reliable transmission have been found. In particular, it was recently showed  that the minimum cut is a necessary and sufficient condition for reliable transmission of multiple correlated sources to all sinks. This result includes the necessary and sufficient condition ,  for networks in which every source is demanded by single sink as a special case. However, the correlated source problem is an uncharted area in general. A related important problem is that of separation of distributed source coding and network coding. It has been shown  that separation holds for two-source two-sink networks. However it has also been shown by example that separation fails for two-source three-sink and three-source two-sink networks.
In this paper we develop new outer bounds for the capacity region of general multicast networks with correlated sources. We further develop the main concepts to also give tighter bounds for networks with independent sources. The main idea of these bounds is to find subsets of random variables in the network that act as information blockers, or information-theoretic cut sets. These are sets of variables that determine all other variables in the network. We develop the properties of these sets, which leads to recursive algorithms for their enumeration. These algorithms can be thought of as operating on a specially constructed functional dependence graph that encodes the local functional dependencies imposed by encoding and decoding constraints.
I-a Organization and Main Results
Section II provides required background, including a review of regions in the entropy space. These regions are used to describe a family of geometric bounds on the capacity region for network coding. We also describe existing graphical bounds. Section III presents main results of the paper. In Section III-A, we generalize the concept of a functional dependence graph (FDG), Definition 7, to handle polymatroidal variables (a wider class of objects than random variables). This gives us a single framework that supports both geometric and graphical bounds. Following on from this, we introduce the notion of irreducible sets and maximal irreducible sets for functional dependence graphs, which are our key ingredients for characterization and computation of capacity bounds. Recursive algorithm finding all maximal irreducible sets for cyclic FDGss is developed using the structural properties of maximal irreducible sets. In Section III-B, we describe construction of a cyclic FDG, called network FDG, from a given multi-source multi-sink network. Maximal irreducible sets in a network FDG are information bottlenecks, and provide Theorem 2111A simpler version of this bound was presented at IEEE International Symposium on Information Theory, Seoul, South Korea, June/July 2009  which outer bounds the capacity region for networks with correlated sources. It is established that Theorem 2 is the best known graph theoretic bound for multi-source multi-sink networks with correlated sources. In Section III-C we adapt our approach to take advantage of the additional constraints introduced when sources are mutually independent. This results in an improved bound, Theorem 3. In Appendix V we give an algorithm to enumerate all maximal irreducible sets for acyclic FDGs. In Section IV, we compare our new bounds with previously known results: cut-set bound , network sharing bound , the notion of information dominance  and progressive d-separating edge-set bound .
Sets will be denoted with calligraphic typeface, e.g. . Set complement is denoted by the superscript (where the universal set will be clear from context). Set subscripts identify the set of objects indexed by the subscript: . Collections of sets are denoted in bold, e.g., . The power set is the collection of all subsets of . Where no confusion will arise, set union will be denoted by juxtaposition, , and singletons will be written without braces.
We start with a brief review on classes of polymatroids which are used to derived a framework to characterize outer bounds on the network coding capacity region. The framework will also enable us to understand the connection between some geometric bounds and graphical bounds. As we shall see, some of these graphical bounds can be interpreted as relaxations of geometric bounds.
Let be a set of variables and be a real-valued function such that . Each function can also be viewed as a column vector in (or in knowing that is always ) often called the entropy space .
Definition 1 (Polymatroidal function or polymatroids)
Definition 2 (Entropic function)
A function is called entropic if there exists a set of discrete random variables such that
for all . Here, is the Shannon entropy function.
It is well known that all entropic functions are polymatroids. In the context of entropy functions, those polymatroid axioms are equivalent to the basic, or Shannon-type inequalities . For these reasons, an element in a ground set of a polymatroid may also be called “variable”. Note that the chain rule for polymatroids also directly follows from the definition of . Functional dependency and independence in polymatroids can also be similarly defined as in random variables. Specifically, with respect to a polymatroid ,
a subset of variables is a function of another subset of variables if
a subset of variables is conditionally independent of another subset of variables given if
Definition 3 (Almost entropic function)
A function is almost entropic if there exists a sequence of entropic functions such that .
Let , and be respectively the set of all polymatroidal, entropic and almost entropic functions. It is clear that
In general, the region is not closed and hence strictly contains . While is convex , it is still extremely hard to characterize (and hence also ). In fact, is not even a polyhedron for . On the contrary, its outer bound is a much simpler polyhedron in the non-negative orthant and in fact is the intersection of
half spaces induced by the following elemental inequalities 
where and .
Ii-B Network Coding
Let the directed acyclic graph serve as a simplified model of a communication network with error-free point-to-point communication links. We use and to respectively denote tail and the head of the directed edge . For nodes and edge , we write as a shorthand for and for . Also, for , we write if . A path in a directed graph is a sequence of nodes such that there exists edges with and . Such a path is said to have length . Node is reachable from node if there exist a path from node to . Furthermore, node is connected to if there exist nodes and edges with and , and/or and . In other words, is connected to if the two nodes are connected, by ignoring the direction of the edges.
Let be an index set for multicast sessions and be the set of sources. The source is available at the set of nodes and is demanded by multiple sink nodes . We call the tuple the connection requirement.
In this paper, we assume that the sources are i.i.d. sequences
so that copies of generated at different time will be independent of each other. However, within the same time instance , the sources may be correlated among different sources. In the special case when is also mutually independent, we will say the sources are independent. Also, the distribution of and hence entropies of any subset of sources are assumed to be known. For notation simplicity, we will use to denote a generic copy of the sources at any particular time instance.
For a network subject to a connection requirement and , a deterministic network code (of block length ) is a collection of source and edge random variables where is the message transmitted on the edge and is the block of source symbols . Unlike which is a collection of i.i.d. random variable, the superscript in is only used to indicate the block length of the code. It does not mean that is a collection of i.i.d. random variables.
Clearly, these random variables cannot be arbitrarily but must satisfy some constraint. In particular, it is required that 1) an edge random variable must be a function of incident edge random variables and source random variables, and 2) for any , a sink node must be able to reconstruct the demanded source. More precisely, we have the following definition.
Definition 4 (Network code)
A network code of block length is described by a set of local encoding functions and decoding functions
Here, the alphabets of the block of source random variables and edge random variables are denoted by and respectively.
With respect to a given network code, the joint distribution for the set of all source and edge random variables will become well-defined. Furthermore, for any , one can construct a global encoding function such that
Definition 5 (Achievability)
An edge capacity tuple is called achievable if there exists a sequence of network codes
(and also the corresponding induced source and edges random variables ) such that
for all and .
When sources are correlated, it is natural to assume a fixed joint distribution of the sources. In that case, the network coding capacity region is the set of all achievable edge capacity tuples that support the transmission of the sources. When sources are independent, only the entropies but not the joint distribution matter (as one can always compress the sources independently before transmission). Therefore, as in some existing literature, one may instead focus on finding the set of source rates or entropies that a network can transport, subject to a fixed edge capacity tuple.
Ii-C Network Coding Bounds
Definition 6 below provides a standard framework to formulate “geometric” bounds on the set of achievable edge capacity tuples (denoted by ).
Consider any network coding problem (with an underlying network and connection requirement ). For any non-empty subset of polymatroids on the ground set , let be the set of tuples for which there exists satisfying
Note that, in (9) can be viewed as a generic source random variable and also as an element in the ground set .
When sources are independent, i.e., , and are respectively inner and outer bounds for [8, Chapter 15]. In Yan et al. , an exact characterization of for multi-source multi-sink network coding was also obtained.
When sources are correlated, using arguments similar to those used in the proof of [8, Theorem 15.9], one can prove that is still an outer bound for . Note that in the bound , only the joint entropies of the sources but not their joint probability distribution are used to derive the bound. Therefore, one can tighten the bound by incorporating additional information about the joint distribution in characterizing bounds (see ,  and ).
Since entropy functions and almost entropic functions are polymatroidal (4) and the regions , , , , , are closed and convex, it follows that is an outer bound for the set of achievable rates. The relation of these capacity bounds is summarized below.
Weighted sum-rate bounds induced by can in principle be computed using linear programming. One practical difficulty with numerical computation of such bounds is that the number of variables and the number of constraints due to both increase exponentially with (refer to (5)). Attempts to simplify these bounds using direct application of Fourier-Motzkin  may prove fruitless. In , the authors have proposed a graph based approach to simplify the bound by exploiting the abundant set of functional dependencies in a network coding problem.
In addition to above bounds, there are also many “graphical” bounds (i.e., bounds that rely on a graph representation of the network coding system) in existing literatures. We will review and compare these bounds, such as cut-set bound , network sharing bound  and progressive -separating edge-set bound  in Section IV.
Iii Main Results
The main results of this paper are graphical bounds for networks with correlated or independent sources. In Section III-A we will define a functional dependence graph, which represents a set of local functional dependencies between polymatroidal variables. Our definition extends  to accommodate cycles containing source nodes, and polymatroidal variables in place of random variables. This section also provides the main technical ingredients for our new bounds. In particular, we describe a test for functional dependence, and give a basic result relating local and global dependence. Section III-B describes our new bound for general multicast networks with correlated sources, based on the implications of local functional dependence. Section III-C considers source independence implications to further strengthen the proposed bound.
The main ingredient of most graph based outer bounds is the following theorem:
Theorem 1 (Bottleneck Bound)
Let be a set such that
for any polymatroid . Then,
and the theorem is proved. \qed
As a consequence, one may identify various subsets satisfying (18) and use them to derive bounds for the network coding rate region. The question however is how to find such bottleneck subsets. Finding all bottlenecks can be a very challenging and computing intensive task. In the remaining of the section, we will derive various graph based technique to find such bottlenecks.
Iii-a Functional Dependence Graphs
Definition 7 (Functional Dependence Graph)
Let be a set of polymatroids on a ground set . A directed graph is called a functional dependence graph for if and only if for all
Alternatively, a function is said to satisfy the FDG if it satisfies (20). An FDG is called cyclic if every node is a member of a directed cycle.
Definition 7 is more general than the FDG of [29, Chapter 2]: Firstly, in our definition there is no distinction between source and non-source random variables. The graph simply characterizes functional dependence between variables. In fact, our definition admits cyclic directed graphs with cycles containing source nodes, and there may be no nodes with in-degree zero (which are source nodes in ). We also do not require independence between sources (when they exist), which is implied by the acyclic constraint in . Our definition admits functions with additional functional dependence relationships that are not represented by the graph. It only specifies a certain set of conditional functions which must be zero. Our definition holds for a wider class of objects (variables in polymatroids) rather than only random variables. Clearly an FDG in the sense of  satisfies the conditions of Definition 7, but the converse is not true. For clarity, a functional dependence graph (FDG) is defined according to our Definition 7.
Definition 7 specifies an FDG in terms of local dependence structure. Given such local dependence constraints, it is of great interest to determine all implied functional dependence relations. In other words, given an FDG, we wish to find all sets and such that for all satisfying the FDG.
Definition 8 ( determines )
Consider a directed graph . For any sets , we say that determines (with respect to Procedure A) if there are no elements of remaining after the following procedure:
Remove all the edges outgoing from the nodes in and subsequently remove all nodes and edge with no incoming edges and nodes respectively.
We will use to denote that determines .
Definition 9 (Blanket)
For a given set , let be the set of nodes deleted by the procedure of Definition 8 together with the nodes in . We will call the blanket of (with respect to Procedure A).
Clearly is the largest set of nodes with . To this end, define for
to be the set of parents of node . Where it does not cause confusion, we will abuse notation and identify variables and nodes in the FDG, e.g. (20) will be written or simply .
Lemma 1 (Grandparent lemma)
Let be an FDG for a polymatroid . For any with
By hypothesis, for any . Furthermore, note that for any , conditioning cannot increase the function 222This is a direct consequence of submodularity (3). and hence for any . Now using this property, and the chain rule for polymatroids,
We emphasize that in the proof of Lemma 1, we have only used the submodular property of polymatroids, together with the hypothesized local dependence structure specified by the FDG.
Let be an FDG for a polymatroid . Then for disjoint subsets ,
Let . Then, by Definition 8 there must exist directed paths from some nodes in to each node in , and there must not exist a directed path to any node in which does not also intersect . In other words, apart from the paths from nodes in and their sub-paths, any other path leading to must have an element of as its member. Recursively invoking Lemma 1, the lemma is proved. \qed
Definition 10 (Irreducible set)
A set of nodes is irreducible (with respect to Procedure A) if there is no such that . Furthermore, an irreducible set is maximal if .
In this paper, we are mainly interested in cyclic FDGs to characterize cut-set bounds on network capacity. However, for other applications, acyclic FDGs may also be of interest. In Appendix V we define maximal irreducible sets for acyclic network and give an algorithm to compute them. For cyclic graphs, every subset of a maximal irreducible set is irreducible. In contrast to acyclic graphs the converse is not true, that is, there can be irreducible sets that are not maximal and are not subsets of any maximal irreducible set.
If and are both maximal irreducible sets, then for any polymatorids satisfying the FDG .
As we shall see, the corollary, together with Theorem 1, can be used to derive capacity bounds for network coding. Therefore, we are interested in finding every maximal irreducible set. This may be accomplished via AllMaxSetsC() in Algorithm 1, which recursively finds all maximal irreducible sets. In the algorithm, the graph , where , is isomorphic to via some bijection and hence iff . For set we define .
The actual number of operations (or the time complexity) to execute the function call depends on the topology of the FDG. The recursion tree is described in Figure 1. We make the following observations: (1) the leaf nodes of the recursion tree (such nodes are represented within circles) are subsets containing and/or complement of maximal irreducible sets (denote by a maximal irreducible set and by the set of all such sets), (2) any leaf node which is not a complement of any is a subset of some and (3) each node of the recursion tree represents a unique set. Hence the number of nodes are upper bounded by the cardinality of the set . Using the union bound, the total number of calls of the function AllMaxSetsC() can be upper bounded by
Due to the recursive nature, the algorithm is easy to implement. The number of recursive calls can be further reduced, for example, by providing all cut-sets separating subsets of sources and corresponding sinks and using complement of the cut-sets as input to Algorithm 1 while replacing by (this is important for input other than ). We will see in Section IV that the maximal irreducible sets are subsets of such cut-sets.
Iii-B A Bound for Network with Correlated Sources
So far, we have defined functional dependence graphs, developed some of their properties, and given algorithm for finding all maximal irreducible sets. In order to apply these results to find bounds on network coding capacity, we need to construct FDGs from multi-source communications networks with multicast constraints.
Definition 11 (Network FDG)
For a given network coding problem (defined by the network topology and connection requirement ), its induced network FDG is a directed graph defined as follows
The set of nodes is equal to
is a directed edge in if it satisfies one of the following conditions
, and ;
, and ;
, , and ;
, , and ;
, , and .
In the above definition, the physical meaning of is the decoded estimates of at the sink node . Note that the decoding constraints (11) require that for all .
Example 1 (Network FDG of the butterfly network)
In network FDGs, there are nodes for auxiliary variables which represent decoding estimate and are the same as the source variables demanded at the sink. Accordingly, the following procedure finds functional dependency in network FDG taking multicasting into consideration.
Definition 12 (Procedure B)
Consider a network FDG as defined in Definition 11. For any sets , we say determines (with respect to Procedure B) if there are no elements of remaining after the following procedure:
Remove all edges outgoing from nodes in and subsequently remove all nodes and edges with no incoming edges and nodes respectively.
If any is removed, (a) remove all for and (b) subsequently remove all edges and nodes with no incoming edges and nodes, go to Step 2. Else terminate.
We will use to denote that determines with respect to Procedure B.
As before, concepts such as blanket and irreducibility can be similarly defined with respect to Procedure B. Specifically, for a given set , its blanket (with respect to Procedure B) is denoted by and is defined as the largest set of nodes with . A set of nodes is called irreducible (with respect to Procedure B) if there is no such that . An irreducible set is maximal if . In addition, if and are maximal irreducible sets, then
for all polymatroid satisfying the network FDG.
Furthermore, the recursion described earlier in Algorithm 1 can also be used to find maximal irreducible sets for multi-source multi-sink networks with correlated sources, replacing by .
Example 2 (Butterfly network)
The maximal irreducible sets for the butterfly network in Figure 2(a) are
The construction of is as follows. For any subset of , let
It can then be verified directly that satisfies the network FDG, Definition 11. \qed
We can now state our first main result, an easily computable outer bound for the capacity region of a network coding system.
Theorem 2 (Functional Dependence Bound)
Consider a network coding problem and its induced network FDG . If is a maximal irreducible set (with respect to Procedure B) in and is achievable, then
In the special case when sources are independent, then inequality (26) is reduced to
Let be the set of all maximal irreducible set with respect to Procedures B and let
Example 3 (Butterfly network)
If the sources and are instead independent, we obtain
Note that the first two bounds on the sum rate in (31) follow from the maximal irreducible sets described in Example 2. The last nine bounds are consequences of the individual rate bounds in (29) and (30).
The capacity region for the special case of multicast networks in which all correlated sources are demanded by all sinks was established by Han  using a simple cut-set based characterization. The cut-sets used by Han  are in fact the maximal irreducible sets, yielding the following corollary.
Corollary 2 (When every sink node demands all sources)
For multicast networks in which all correlated sources are demanded by all sinks,
Iii-C When Sources are Independent
In this subsection, we further consider the special case when sources are independent. Unlike the case when sources are correlated, the problem of characterizing graphical bounds for networks with independent sources has been well investigated [16, 2, 30, 17, 19, 18]. A source independence constraint may imply additional functional dependencies beyond those implied by the network coding and decoding constraints alone. These additional functional dependencies may in turn be used to improve of our characterization of the set of achievable rate region.
To understand the new bound, we first begin with a review of some basic graph concepts. The -separation criterion  is a tool to infer certain conditional independence relationships amongst a set of random variables where (some of) their local conditional independence relations are represented by a Bayesian Network (directed acyclic graph). It has also been shown that the -separation criterion is valid for finding certain conditional independence in cyclic functional dependence graphs  (see Definition 14). The fd-separation criterion  is an extension of -separation finding certain conditional independence relationships in FDGs. In this section, we generalize this result by showing that fd-separation can be used to find conditional independence relationships for polymatroidal variables represented by an FDG.
Definition 13 (Ancestral graph)
Consider a directed graph induced by a network coding problem. For any subset , let denote the set of all nodes in such that for every node ), there is a directed path from to some node in in the subgraph .
The ancestral graph with respect to (denoted by ) is a subgraph of consisting of nodes and edges such that .
Definition 14 (-separation)
A set -separates and in a network FDG if the nodes in and the nodes in are disconnected in what remains of after removing all edges outgoing from nodes in .
Definition 15 (fd-separation )
Let be a network FDG. A set fd-separates and in if the nodes in and the nodes in are disconnected in what remains of after removing all edges outgoing from nodes in and subsequently, recursively removing all edges that have no source nodes as ancestors.
Now we show that fd-separation is valid for polymatroidal variables represented by the subgraph of network FDG (Definition 11). First, note that the subgraph of network FDG is a functional dependence graph in the sense of  (with random variables replaced by polymatroidal variables) since the vertices in represent source and edge variables, the edges in represent functional dependencies between the variables and the vertices representing the source variables have no incoming edges.
If the subset of nodes fd-separates and in the subgraph of a network FDG for , then .
In the following, we will give a tighter graphical bound for networks when sources are independent. We will follow a similar approach used to derive Theorem 1 by finding maximal irreducible sets induced by fd-separation in subgraph of network FDG.
Definition 16 (Procedure C)
Consider a network FDG as defined in Definition 11. For any sets , we say determines (with respect to Procedure C) if there are no elements of remaining after the following procedure:
Remove all edges outgoing from and subsequently recursively remove all nodes and edges with no incoming edges and nodes respectively and all nodes and edges with no source nodes as ancestors. Call the resulting graph .
If there exists any disconnected from any in then from (a) remove for all and (b) subsequently recursively remove all nodes and edges with no incoming edges and nodes respectively. Call the resulting graph , go to Step 2. Else terminate.
We will use to denote that determines with respect to Procedure C.
Note that Step 2 of Definition 16 uses fd-separation. The concepts for blanket, irreducibility are similarly defined with respect to Procedure C. Specifically, for a given set , its blanket (with respect to Procedure C) is denoted by and is defined as the largest set of nodes with . A set of nodes is called irreducible (with respect to Procedure C) if there is no such that . An irreducible set is maximal if . In addition, if and are maximal irreducible sets, then
Furthermore, the recursion described earlier in Algorithm 1 can now be used to find maximal irreducible sets for multi-source multi-sink networks with independent sources, replacing by .
For any given network FDG ,
We remark that, there may exist some in such that (refer to Example 4 in which ).
If and in a network FDG , then .
Suppose . Let be the set of variables removed by Step 1 in Definition 16. Then by Lemma 2, . Now, let be the set of all nodes representing the estimates removed by Step 2(a) in Definition 16. Then, by the definition of fd-separation in the subgraph and Lemma 4, . But the decoding constraints (11) imply then for ,
Example 4 (Butterfly Network, Independent Sources)
The sets , , , , are new maximal irreducible sets found by replacing by in Algorithm 1. Source independence is an essential ingredient to find these new maximal irreducible sets. However independence is not necessary to find the other maximal irreducible sets. Also note that the maximal irreducible sets , , , previously found by Algorithm 1 with