Entropy distribution and condensation in random networks
with a given degree distribution
The entropy of network ensembles characterizes the amount of information encoded in the network structure, and can be used to quantify network complexity, and the relevance of given structural properties observed in real network datasets with respect to a random hypothesis. In many real networks the degrees of individual nodes are not fixed but change in time, while their statistical properties, such as the degree distribution, are preserved. Here we characterize the distribution of entropy of random networks with given degree sequences, where each degree sequence is drawn randomly from a given degree distribution. We show that the leading term of the entropy of scale-free network ensembles depends only on the network size and average degree, and that entropy is self-averaging, meaning that its relative variance vanishes in the thermodynamic limit. We also characterize large fluctuations of entropy that are fully determined by the average degree in the network. Finally, above a certain threshold, large fluctuations of the average degree in the ensemble can lead to condensation, meaning that a single node in a network of size can attract links.
One reason why network science has recently attracted significant research attention is that network structure efficiently encodes the complexity of a large variety of systems, from the brain to different techno-social infrastructures Barabasi (); Newman_book (); Dorogovtsev_book (); dyn (). In the last fifteen years or so there has been significant progress in characterizing not only universal properties of complex networks, e.g., scale-free degree distributions or small-world properties, but also their specific features that distinguish one network from another—degree correlations, community structure Santo (), or motif distributions Alon (); Loops (); Cliques (); massimo ().
More recently, considerable effort has focused on quantifying network complexity using new network entropy measures Entropy (); AB2009 (); BC2008 (); AB2010 (); PNAS (); Munoz (); Peixoto_BM (); Peixoto_PRL (); DG () borrowed from information theory, statistical mechanics Newman1 (); Garlaschelli1 (); Garlaschelli2 (); Coolen2 (), and quantum information AB2011 (); Garnerone (). The entropy of a network ensemble evaluates the total number of networks belonging to the ensemble Entropy (); AB2009 (). The more complex, sophisticated, and unique the network structure, the smaller the number of networks in the ensemble having these peculiar properties, the smaller the entropy. The entropy measures have proven useful for solving inference problems involving real-world networks PNAS (); Munoz (); Peixoto_BM (); Peixoto_PRL (). The statistical mechanics treatment of network ensembles can be used to characterize the likelihood that a real dataset is generated by a model Garl_l (). Some real networks have been shown to belong with high likelihood to ensembles of random geometric graphs in hyperbolic spaces, modelling trade-offs between popularity and similarity in network evolution, and casting preferential attachment as an emergent phenomenon Krioukov1 (); Krioukov2 (). More recently, the entropy of multiplex ensembles has been proposed to characterize the complexity of multilayer networks Multiplex ().
By definition, a network ensemble is a set of graphs with probability measure . It is important to make a distinction between microcanonical and canonical ensembles AB2009 (). In microcanonical ensembles, some structural network properties are fixed to given values. For example, the total number of links in graphs of size can be fixed to , or the degrees of all nodes can be fixed to degree sequence , . In this case, the ensemble consists of all graphs of size , and the probability measure is uniform: if the number of graphs satisfying the constraints is , then for all such graphs, , and for all other graphs that do not satisfy the constraints. In the canonical counterparts of these ensembles, the same structural constraints are fixed only on average—the resulting ensembles are maximum-entropy ensembles under the constraints that the expected values of the number of edges or node degrees in the ensemble are or . The probability measure in this case is not uniform—the closer the to satisfying the constraints, the larger the . In random graph ensembles with a fixed exact or expected number of links, the canonical distribution converges to the microcanonical distribution in the thermodynamic limit . However, as soon as the number of imposed constraints is extensive, the canonical and microcanonical network ensembles are not equal even in the thermodynamic limit, and neither are their entropies. For example, the entropy of the microcanonical ensemble with a fixed degree sequence is not equal to the entropy of the canonical ensemble where only the expected degree sequence is fixed AB2009 ().
In network theory there is usually no question of how precisely we know the node degrees: given a graph, its degree sequence is uniquely defined. However when a network practitioner works with real network data, she is typically given a collection of network measurements. Does she have to treat the measured degrees of nodes as precisely defined as well, given that measurements are always imprecise and ever-changing, and so is the network itself? The answer is usually ‘no’—the relevant information is not the exact degree sequence, but its statistical properties, such as the distribution of these degrees. The ensembles of networks with a given exact or even expected degree sequence do not account for possible statistical fluctuations of node degrees in a given dataset, motivating us to consider here ensembles of random networks whose exact or expected degree sequences are independently sampled from a given distribution . This approach is a way to explore only the statistical properties of networks, and not their specific linking diagrams that might be affected by false or missing links, almost always present in real data.
Specifically, we study the distribution of entropy in network ensembles with a given exact or expected degree distribution . We find that in both cases (hard/exact and soft/expected), if the network ensemble is sparse, the average entropy is well-defined and self-averaging. We also evaluate the probability that ensemble entropy is equal to a particular value, conditioned on the total number of links in the network, and show that this conditional entropy distribution is always well-behaved. Characterizing large entropy deviations in the ensemble with a given degree distribution and average degree, we observe that a condensation phenomena can occur in the network. This phenomena occurs only if the average degree in the network exceeds the degree distribution average . For there is a symmetry under permutation of the labels of the nodes of the network, meaning that if we fix the average degree of the network, then the degrees of all nodes are , where is the network size. Instead, if , we observe a spontaneous breaking of this symmetry, with a single node having an degree. These results hold in both hard and soft ensembles, i.e., ensembles with fixed exact or expected degree distributions.
We begin with reviewing in Section II what is known about the entropy of network ensembles with a fixed expected or exact degree sequence. We then move to Section III and IV where we analyze some properties of the entropy distributions in the ensemble with a given expected and exact degree distribution, respectively. Final remarks are in Section V.
Ii Entropy of network ensembles with a given degree sequence
A network ensemble is specified once probability is assigned to every network of size . We denote nodes by . The set of simple undirected unweighted labeled networks of size is bijective to the set of symmetric boolean adjacency matrices having zeroes on the diagonal. Depending on whether nodes and are connected or not in network , element of ’s adjacency matrix is either or . We next impose the constraint that the degree of each node is fixed to some . We can treat this constraint as hard or soft. If it is hard, we impose it exactly. The resulting ensemble is a microcanonical network ensemble with given degree sequence , known as the configuration model MR (); Chung (). In the soft case, we relax the constraint, and demand that the degree of each node , averaged over all networks in the ensemble, is , which no longer has to be integer but can be any non-negative real number. The resulting ensemble is a canonical network ensemble with a given expected degree sequence , belonging to the class of random graphs known as exponential random graphs Caldarelli (); Boguna (); Newman1 (); dyn (). In what follows we denote by the probability of in the canonical () or micro-canonical () ensembles. The entropy of these ensemble evaluates the typical number of networks in the ensemble and is given by
where the sum is performed over all networks in the ensemble.
ii.1 Entropy of the canonical ensemble
The probability distribution in the canonical ensemble is defined as the distribution that maximizes entropy
subject to the following constraints:
By summing over all networks , we sum over their adjacency matrices. Introducing Lagrangian multipliers to enforce the conditions in Eq. , and Lagrangian multiplier to normalize the probability measure , we solve the system of equations
to find the expression for distribution :
where the normalization constant
is called the “partition function.” Since the probability distribution in Eq. has an exponential form, this ensemble is called exponential random graphs.
In this ensemble, we can relate entropy in Eq. to partition function :
We call the entropy of the canonical ensemble the Shannon entropy.
The probability of a link between node and node in the ensemble is given by
where are called “hidden variables”. Upon this change of variables, the constraints in Eq. translate to
This system of equations can be solved for yielding the values of Lagrangian multipliers . Using in Eq. , a simpler expression for distribution reads
Therefore probability is actually the probability to generate network with hidden variables by connecting node pairs and with probability given by Eq. (8), and not connecting them with probability . Using these link existence probabilities , the entropy of the ensemble in Eq. can be written as
By inserting the explicit dependence of probabilities on hidden variables , we can extract the leading term of the entropy that depends only on the average degree in the network, and the subleading term that increases linearly with :
where and in any sparse network are given by
If all expected degrees , where is the average expected degree , then hidden variables are proportional to expected degrees , , and we can approximate probabilities in Eq. by
which corresponds to the case where links in the network are uncorrelated.
In this case the expression for the extensive entropy term in Eq. simplifies to
ii.2 Entropy of the microcanonical ensemble
In the microcanonical ensemble, all networks satisfy the hard constraint that the degree sequence is exactly. We assume here that the degree sequence is graphical DelGenio1 (); DelGenio2 (), meaning that it can be realized by at least one network. This condition is obviously satisfied if the degree sequence is read off from a real network. The probability distribution in the ensemble is uniform—all networks satisfying this constraint have the same probability
where stands for the Kronecker delta, and where “partition function” is given by
This partition function simply counts the number of networks with degree sequence .
The definition of the network ensemble entropy in Eq. applied to the microcanonical distribution in Eq. yields
where we call the Gibbs entropy of the network ensemble. The Gibbs entropy of the microcanonical ensemble is related to the Shannon entropy of the conjugate canonical ensemble via
where is equal to the logarithm of the probability that in the conjugate canonical ensemble the hard constraints are satisfied:
The relation between entropies in Eq. can be obtained by substituting the canonical distribution given by Eq. into Eq. , yielding
It does not vanish in the thermodynamic limit . Therefore in view of the relation between the microcanonical and canonical entropies in Eq. , the microcanonical and conjugate canonical ensembles are not equivalent even in the large- limit.
Iii Entropy distribution in the network ensemble with a given distribution of expected degrees
In this section we consider the network ensemble in which the expected degree sequence is not fixed but sampled in each network realization from a fixed distribution . Drawing from the field of disordered systems, we make a distinction between quenched and annealed disorder. If the disorder is annealed the degree of the nodes are not fixed and they are continuously drawn form a degree distribution . If the disorder is quenched, then the expected degree sequence in each realization is assumed to be fixed but unknown, and for each expected degree sequence, the ensemble probability distribution is obtained by maximizing the entropy. Below we consider the quenched case only.
For a fixed expected degree sequence , the maximum-entropy distribution is given by Eq. (11), while the entropy of this distribution is given by Eq. (12). If this degree sequence has probability in a larger ensemble, then the probability and entropy distributions in this larger ensemble are
Therefore the distribution of the entropy in this ensemble gives a very important indication on how the number of possible network realization with expected degree sequence changes if the sequence realization is drawn randomly from a degree distribution . If we cannot compute the full distribution exactly, we may still characterize its average, variance, and relative error
iii.1 Entropy distribution in scale-free networks
We assume that each expected degree sequence has probability , where is the expected degree distribution, and consider the specific case of power-law with . We first focus on the leading term of entropy given by , where is the sum of expected degrees .
We distinguish between two cases.
When , distribution is Gaussian, the average of is well defined in the network and its relative error vanishes in the large network limit. Indeed, since , we have
For large and , due to the structural degree cutoff , we observe that the average of is also well defined in the network and its relative error also vanishes in the large network limit, but with a different exponent. Indeed, since we have
These results are important because they imply that for every value of the average of the leading entropy term is well defined, with vanishing relative error.
Since the leading entropy term depends only on the average degree , we can further analyze entropy fluctuations in the ensemble with a fixed . Therefore we next evaluate the conditional probability distribution that depends only on the distribution of the subleading entropy term, since average degree determines uniquely the leading term.
iii.2 Conditional entropy distribution
If is the probability of degree sequence , then
Since entropy is a function of hidden variables only, , we can perform the following change of variables in the last equation:
where is the probability of hidden variables sequence . After this transformation, and expressing delta functions in Eq. (28) via exponentials, we obtain,
We next make the simplifying assumption that the hidden variables are i.i.d. distributed, with some distribution . In the large network limit we can then transform the multiplex integral over variables to a functional integral over density function
imposing constraint by Lagrangian multiplier . The distribution defined in Eq. (30) becomes
The integrals in Eq. (32) can be evaluated at the saddle point given by
where we have performed the Wick rotation of parameters and . Denoting by the -dependent solution of the above saddle point equations, we obtain the following simple expression for distribution :
is the Kullback-Leibler distance between distributions and . Therefore conditional distribution is well behaved, and depends only on KL-distance .
iii.3 Condensation as a large deviation event
The saddle point Eqs. (35) have a solution only if the average degree is equal to or less than the expected degree of the degree distribution, i.e. only if
In fact the Lagrangian multipliers must be real and greater than zero to guarantee that given by Eq. (35) is well defined. Following Marsili (), to explore large deviation properties of a fat-tailed distribution, we use the following ansatz:
This ansatz accounts for a spontaneous breaking of permutation symmetry between the hidden variables in the ensemble, and reflects our expectation to detect condensation in some large-deviation realization in the ensemble. With this ansatz, probability becomes
where functions and are as in Eqs. (34). The problem of minimizing function with respect to all its parameters has a non-trivial solution only if , in which case we have
with , and for any . In Figure 1 we show the phase diagram , where is the exponent of the hidden variable distribution , and for . Above the curve , i.e., in the shaded region of parameter values, we observe condensation.
Iv Entropy distribution in the network ensemble with a given distribution of exact degrees
The results presented in the previous section concerning the soft ensembles remain qualitatively unchanged if we consider the hard ensembles of networks with a given degree distribution of exact degrees. Note that we are treating here always quenched disorder. In fact here we consider ensembles of networks of fixed degree sequence, where each degree sequence is drawn randomly from a given degree distribution. As in the soft case, in this hard case we assume that the disorder is quenched, and that the exact degree sequence is fixed but unknown and drawn from degree distribution . The probability of degree sequence is thus , and for each we consider the microcanonical ensemble of networks with the fixed sequence of exact degrees . In this ensemble, network has probability defined in Eq. (17), so that the probability distribution in the ensemble is
where the leading term of is . In what follows we analyze the entropy distribution in the ensemble
If we cannot compute the full distribution exactly, we may still characterize its average, variance, and relative error
iv.1 Entropy distribution in scale-free networks
We first consider the probability distribution of the leading term of entropy , defined as
where depends only on the average degree in the network. According to the Generalized Central Limit theorem Bouchaud (), and similarly to the soft case, we have the following two cases:
The distribution converges to a Gaussian distribution and the relative error on the average of is vanishing in the large network limit. In fact we find that
Due to the structural degree cutoff , the entropy distribution has a vanishing relative error given by
In the both cases, the average is well defined with a relative error vanishing in the large network limit.
iv.2 Conditional entropy distribution
Similarly to the soft case, we next show that the large entropy fluctuations are due exclusively to the fluctuations of the total number of links in the ensemble. We note that these fluctuations are necessarily present since the exact degree sequence in each network in the ensemble is independently sampled from the given distribution. Following a similar procedure, we evaluate the probability of conditioned to a fixed value of the average degree in the network , . This conditional entropy distribution depends only on the distribution of the subleading contributions to , , because the average degree determines uniquely the leading term .
If is the probability of degree sequence , then
Since entropy is a function of hidden variables only, , we can change variables
where is the probability of hidden variables sequence , and obtain,
Assuming next that our hidden variables are i.i.d. distributed with some distribution , we transform the multiplex integral over variables in the large network limit to a functional integral over density function
imposing constraint by Lagrangian multiplier . The distribution defined in Eq. (51) becomes
The integrals in Eq. (53) can be evaluated at the saddle point given by