# Multilayer Spectral Graph Clustering via Convex Layer Aggregation

###### Abstract

Multilayer graphs are commonly used for representing different relations between entities and handling heterogeneous data processing tasks. New challenges arise in multilayer graph clustering for assigning clusters to a common multilayer node set and for combining information from each layer. This paper presents a theoretical framework for multilayer spectral graph clustering of the nodes via convex layer aggregation. Under a novel multilayer signal plus noise model, we provide a phase transition analysis that establishes the existence of a critical value on the noise level that permits reliable cluster separation. The analysis also specifies analytical upper and lower bounds on the critical value, where the bounds become exact when the clusters have identical sizes. Numerical experiments on synthetic multilayer graphs are conducted to validate the phase transition analysis and study the effect of layer weights and noise levels on clustering reliability.

Multilayer Spectral Graph Clustering via Convex Layer Aggregation

Pin-Yu Chen
^{†}^{†}thanks: This work was partially supported by the US Army Research Office, grant W911NF-15-1-0479.
Alfred O. Hero III, Fellow, IEEE |

Department of Electrical Engineering and Computer Science, University of Michigan, Ann Arbor, USA |

{pinyu, hero}@umich.edu |

## 1 Introduction

Multilayer graphs are useful for representing different relations between entities and handing heterogeneous multilayer data processing tasks, where each layer describes a specific type of connectivity pattern among a common node set across layers. For example, in multi-relational social networks, each layer corresponds to one type of social relation. In temporal networks, each layer corresponds to the snapshot of the entire network at a sampled time instance. Multilayer graphs have been applied to many signal processing and data mining techniques, including inference of mixture models [1, 2], tensor decomposition [3], information extraction [4], multi-view learning and processing [5], graph wavelet transform [6], principal component analysis and dictionary learning [7, 8], anomaly detection [9], and community detection [10, 11, 12], among others.

In particular, the task of multilayer graph clustering is to find a consensus cluster assignment on each node in the common node set by inspecting the connectivity pattern in each layer. Different from clustering in single-layer graphs, clustering in multilayer graph faces new challenges due to (1) information aggregation from multiple layers, and (2) lack of a theoretical framework on clustering reliability assessment. By viewing the connectivity pattern in each layer as a signal plus noise model, this paper aims to provide a theoretical framework for analyzing the performance of multilayer spectral graph clustering (SGC) via convex layer aggregation, where spectral clustering is implemented on an aggregated graph via convex combination of each layer. Specifically, fixing the within-cluster edges (signals) and varying the parameters governing the between-cluster edges (noises), we show that the accuracy of multilayer SGC can be separated into two regimes: a reliable regime where high clustering accuracy can be guaranteed, and an unreliable regime where high clustering accuracy is impossible. Moreover, we show that the upper and lower bounds on the critical noise level that separates these two regimes are closed-form functions of the signal strength, the number of clusters, the cluster size distributions, and the layer weight vector for convex layer aggregation. In addition, the bounds become exact in the case of identical cluster sizes. Numerical experiments on synthetic multilayer graphs are conducted to validate the phase transition analysis and study the effect of layer weights and noise levels on clustering reliability.

## 2 Related Work

Layer aggregation has been a principal method for processing and mining multilayer graphs [13, 14, 15, 16, 17, 18], as it transforms a multilayer graph into a single aggregated graph, facilitating application of data analysis techniques designed for single-layer graphs. Extending from the stochastic block model (SBM) for graph clustering in single-layer graphs [19, 20], multilayer SBM has been proposed for graph clustering on multilayer graphs [21, 22, 23, 24, 18, 25]. Under the assumption of two equally-sized clusters, the authors in [18] show that if each layer is an independent realization of a common SBM, the inferential limit for cluster detectability decays with , where is the number of layers. In [25], a layer selection method based on a multilayer SBM is proposed to improve the performance of graph clustering. However, the multilayer SBM assumes homogeneous connectivity structure for within-cluster and between-cluster edges in each layer, and it assumes layer-wise independence. The multilayer signal plus noise model considered in this paper is a general model that includes the multilayer SBM, as it does not impose any distributional assumption on the within-cluster connectivity for each layer. More details on multilayer graph models for graph clustering can be found in the recent survey papers [10, 11].

## 3 Multilayer Signal Plus Noise Model

We consider the multilayer graph model of layers representing different relationships among a common node set of nodes. The graph in the -th layer is an undirected graph with nonnegative edge wights, which is denoted by , where is the set of weighted edges in the -th layer. The binary symmetric adjacency matrix is used to represent the connectivity structure of . The entry if nodes and are connected in the -th layer, and otherwise. Similarly, the nonnegative symmetric weight matrix is used to represent the edge weights in , where and have the same zero structure.

We assume each layer in the multilayer graph is a (possibly correlated) representation of common clusters that partitions the node set , where the -th cluster has cluster size such that . and denote the largest and smallest cluster size, respectively. Specifically, the adjacency matrix of in the -th layer can be represented as

(1) |

where is an binary symmetric matrix denoting the adjacency matrix of within-cluster edges of the -th cluster in the -th layer, and is an binary rectangular matrix denoting the adjacency matrix of between-cluster edges of clusters and in the -th layer, , , and .

Similarly, the edge weight matrix can be represented as

(2) |

where is an nonnegative symmetric matrix denoting the edge weights of within-cluster edges of the -th cluster in the -th layer, and is an nonnegative rectangular matrix denoting the edge weights of between-cluster edges of clusters and in the -th layer, , , and .

Using the cluster-wise block representations of the adjacency and edge weight matrices for the multilayer graph model described in (1) and (2), we propose a signal plus noise model for and to analyze the effect of convex layer aggregation on graph clustering. Specifically, for each layer we assume the connectivity structure and edge weight distributions follow the random interconnection model (RIM) [26]. In RIM the signal of the -th cluster in the -th layer is the connectivity structure and weights of the within-cluster edges represented by the matrices and , respectively. In particular, analogous to the formulation of many detection problems in signal processing, the signal can be arbitrary in the sense that we impose no distributional assumption for the within-cluster edges. The noise between clusters and in the -th layer is the connectivity structure and weights of the between-cluster edges represented by the matrices and , respectively.

Throughout this paper, we assume the connectivity of a between-cluster edge (i.e., the noise) in each layer is independently drawn from a layer-wise and block-wise independent common Bernoulli distribution. Specifically, each entry in representing the existence of an edge between clusters and in the -layer is an independent realization of a Bernoulli random variable with edge connection probability that is layer-wise and block-wise independent. In addition, given the existence of an edge between clusters and in the -layer, the entry representing the corresponding edge weight is independently drawn from a nonnegative distribution with mean and bounded fourth moment that is layer-wise and block-wise independent.

For the -th layer, the noise accounting for the between-cluster edges is said to be block-wise identical if the noise parameters and for every cluster pair and , . Otherwise it is said to be block-wise non-identical.

## 4 Multilayer Spectral Graph Clustering via Convex Layer Aggregation

### 4.1 Notations and mathematical formulations

Let be an column vector representing the layer weight vector for convex layer aggregation, where is the set of feasible layer weight vectors. The single-layer graph obtained via convex layer aggregation with layer weight vector is denoted by . The (weighted) adjacency matrix and the edge weight matrix of satisfy and . The graph Laplacian matrix of is defined as , where is a diagonal matrix, is the vector of nodal strength of , is the column vector of ones, and is the graph Laplacian matrix of . Similarly, the graph Laplacian matrix accounting for the within-cluster edges of the -th cluster in is defined as , where , , and . The -th smallest eigenvalue of is denoted by . Based on the definition of , the smallest eigenvalue of is 0, since , where is the column vector of zeros.

Spectral graph clustering (SGC) [27] partitions the nodes in into () clusters based on the eigenvectors associated with the smallest eigenvalues of . Specifically, SGC first transforms a node in to a -dimensional vector in the subspace spanned by these eigenvectors, and then implements K-means clustering [28] on the -dimensional vector space representation to group the nodes in into clusters based on their distances. For analysis purposes, throughout this paper we assume is a connected graph. If is connected, it is known that for all [29]. Furthermore, the eigenvector associated with the smallest eigenvalue provides no information about graph clustering since it is proportional to .

Let denote the eigenvector matrix where its -th column is the -th eigenvector associated with , . By the Courant-Fischer theorem [30], is the solution to the minimization problem

(3) |

where the optimal value in (4.1) is the partial eigenvalue sum , is the identity matrix, and the constraints in (4.1) impose orthonormality and centrality on the eigenvectors. In summary, multilayer SGC via convex layer aggregation works by computing the eigenvector matrix from of , and implementing K-means clustering on the rows of to group the nodes into clusters.

### 4.2 Phase transitions under block-wise identical noise

Under the multilayer signal plus noise model, if we further assume block-wise identical noise, then the noise level in the -th layer can be characterized by the parameter , where is the edge connection parameter and is the mean of the between-cluster edge weights in the -th layer. Given a layer weight vector , let denote the aggregated noise level of the graph . Theorem 1 below establishes phase transitions in the eigendecomposition of the graph Laplacian matrix of . We show that there exists a critical value such that the smallest eigenpairs of that are used for multilayer SGC have different characteristics when and . In particular, we show that the solution to the minimization problem in (4.1), the eigenvector matrix represented by the cluster partitioned form, where with its rows indexing the nodes in cluster , has cluster-wise separability when in the sense that the matrices are row-wise identical and cluster-wise distinct, whereas when the row-wise average of each matrix is a zero vector and hence the clusters are not separable by inspecting the rows of .

###### Theorem 1 (block-wise identical noise).

Given a layer weight vector , and assuming the block-wise identical noise model with aggregated noise level , let , where .
There exists a critical value such that the following holds almost surely as and :

(a)

Furthermore,

(b)

where .

In particular, when , has the following properties:

(b-1) The columns of are constant vectors.

(b-2) Each column of has at least two nonzero cluster-wise constant components, and these constants have alternating signs such that their weighted sum equals (i.e., ).

(b-3) No two columns of have the same sign on the cluster-wise nonzero components.

Finally, satisfies:

(c) , where

In particular, when .

Theorem 1 (a) establishes a phase transition in the increase of the normalized partial eigenvalue sum with respect to the aggregated noise level . When the quantity is exactly . When the slope in of changes and the intercept depends on the cluster having the smallest aggregated partial eigenvalue sum given a layer weight vector . In particular, when all clusters have the same size (i.e., ) so that , undergoes a slope change from to at the critical value .

Theorem 1 (b) establishes a phase transition in cluster-wise separability of the eigenvector matrix for multilayer SGC. When , the conditions (b-1) to (b-3) imply that the rows of the cluster-wise components are coherent, and hence the row vectors in possess cluster-wise separability. On the other hand, when , the row sum of each is a zero vector, making incoherent. This means that the entries of each column in have alternating signs and hence K-means clustering on the rows of yields incorrect clusters.

Theorem 1 (c) establishes upper and lower bounds on the critical threshold value of the aggregated noise level given a layer weight vector . These bounds are determined by the cluster having the smallest aggregated partial eigenvalue sum , the number of clusters , and the largest and smallest cluster size ( and ). When all cluster sizes are identical (i.e., ), these bounds become tight (i.e., ). Moreover, by the nonnegativity of the layer weights we can obtain a universal lower bound on for any , which is

(4) |

Since is a measure of connectivity for cluster in the -th layer, the lower bound of in (4) implies that the performance of multilayer SGC is indeed affected by the least connected cluster among all clusters and across layers. Specifically, if the graph in each layer is unweighted and , then reduces to the algebraic connectivity [29, 31] of cluster in the -th layer.

### 4.3 Phase transitions under block-wise non-identical noise

Under the block-wise non-identical noise model, the noise level of between-cluster edges between clusters and in the -th layer is characterized by the parameter , , , and . Let be the maximum noise level in the -th layer and let denote the aggregated maximum noise level given a layer weight vector .

Let be the eigenvector matrix of under the block-wise non-identical noise model, and let be the eigenvector matrix of the graph Laplacian of another random graph generated under the block-wise identical noise model with aggregated noise level , which is independent of . Theorem 2 below specifies the distance between the subspaces spanned by the columns of and by inspecting their principal angles [27]. Specifically, since and both have orthonormal columns, the vector of principal angles between their column spaces is , where is the -th largest singular value of a real rectangular matrix . Let , and let be defined entrywise. When , Theorem 2 provides an upper bound on the Frobenius norm of , which is denoted by . Moreover, if , where is the critical threshold value for the block-wise identical noise model as specified in Theorem 1, then can be further bounded.

###### Theorem 2 (block-wise non-identical noise).

Given a layer weight vector , and assuming the block-wise non-identical noise model with maximum noise level for each layer, let be be the critical threshold value for the block-wise identical noise model specified by Theorem 1, and define . For a fixed , if and as , the following statement holds almost surely as and :

(5) |

Furthermore, let . If ,

(6) |

Theorem 2 shows that the subspace distance is upper bounded by (5), where is the eigenvector matrix of under the block-wise identical noise model when its aggregated noise level . Furthermore, if the aggregated maximum noise level , then a tight upper bound on can be obtained by (6). Therefore, using the cluster-wise separability of as established in Theorem 1 (b), when , cluster-wise separability in can be expected provided that is small. The proofs of Theorems 1 and 2 are given in the supplementary file.

## 5 Numerical Results

To validate the phase transitions in the accuracy of multilayer SGC via convex layer aggregation, we generate synthetic multilayer graphs from a two-layer correlated multilayer graph model. Specifically, we generate edge connections within and between equally-sized ground-truth clusters on layers and . The two layers and are correlated since their edge connections are generated in the following manner. For every node pair () of the same cluster, with probability there is a within-cluster edge () in and , with probability there is a within-cluster edge () in but not in , with probability there is a within-cluster edge () in but not in , and with probability there is no edge () in and . These four parameters are nonnegative and sum to . For between-cluster edges, we adopt the block-wise identical noise model such that for each layer , the edge connection between every node pair from different clusters is an i.i.d. Bernoulli random variable with parameter .

### 5.1 Phase transitions incurred by noise levels

By varying the noise level , Fig. 1 shows the accuracy of multilayer SGC with respect to different layer weight vector , where the accuracy is evaluated in terms of cluster detectability, i.e., the fraction of correctly identified nodes in the same cluster. Given a fixed , as proved in Theorem 1, there is indeed a phase transition in cluster detectability that separates the noise level into two regimes: a reliable regime where high clustering accuracy is guaranteed, and an unreliable regime where high clustering accuracy is impossible. Furthermore, the critical value of that separates these two regimes are successfully predicted by Theorem 1 (c), which validates the phase transition analysis. The rightmost plot in Fig. 1 shows the geometric mean of cluster detectability from different layer weight vectors. There is an universal region of perfect cluster detectability that includes the region specified by the universal phase transition lower bound in (4).

### 5.2 Phase transitions incurred by layer weights

Next we investigate the effect of layer weight vector on multilayer SGC via convex layer aggregation given fixed noise levels. In the two-layer graph setting, since by definition , it suffices to study the effect of on clustering accuracy. Fig. 2 shows the clustering accuracy by varying under the two-layer correlated graph model. As shown in Fig. 2, if each layer has low noise level (left plot), then any layer weight vector can lead to correct clustering result. If one layer has high noise level (middle plots), then there exists a critical value that separates the cluster detectability into a reliable regime and an unreliable regime. In particular, Theorem 1 implies that the critical value , if existed, satisfies the condition when , which is equivalent to

(7) |

It is observed that the empirical critical value matches the predicted value from (5.2). Lastly, if each layer has high noise level (right plot), then no layer weight vector can lead to correct clustering result, and the corresponding cluster detectability is similar to random guessing of clustering accuracy 33.33%.

## 6 Conclusion

This paper establishes a phase transition analysis on multilayer spectral graph clustering (SGC) via convex layer aggregation under a novel multilayer signal plus noise model. By varying the noise level, we specify the critical value that separates the clustering performance of multilayer (SGC) into a reliable regime and an unreliable regime. Numerical experiments validate the phase transitions incurred by noise levels and layer weights, which are successfully predicted by the developed analytical results.

## References

- [1] B. Oselio, A. Kulesza, and A. O. Hero, “Multi-layer graph analysis for dynamic social networks,” IEEE J. Sel. Topics Signal Process., vol. 8, no. 4, pp. 514–523, Aug 2014.
- [2] K. S. Xu and A. O. Hero, “Dynamic stochastic blockmodels for time-evolving social networks,” IEEE J. Sel. Topics Signal Process., vol. 8, no. 4, pp. 552–562, 2014.
- [3] M. De Domenico, A. Solé-Ribalta, E. Cozzo, M. Kivelä, Y. Moreno, M. A. Porter, S. Gómez, and A. Arenas, “Mathematical formulation of multilayer networks,” Phys. Rev. X, vol. 3, p. 041022, Dec 2013.
- [4] B. Oselio, A. Kulesza, and A. Hero, “Information extraction from large multi-layer social networks,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5451–5455.
- [5] D. Xiaowen, “Multi-view signal processing and learning on graphs,” Ph.D. dissertation, ÉCOLE POLYTECHNIQUE FÉDÉRALE DE LAUSANNE, 2014.
- [6] N. Leonardi and D. Van De Ville, “Tight wavelet frames on multislice graphs,” IEEE Trans. Signal Process., vol. 61, no. 13, pp. 3357–3367, 2013.
- [7] K. Benzi, B. Ricaud, and P. Vandergheynst, “Principal patterns on graphs: Discovering coherent structures in datasets,” IEEE Trans. Signal Inf. Process. Netw., vol. 2, no. 2, pp. 160–173, 2016.
- [8] P.-Y. Chen, S. Choudhury, and A. O. Hero, “Multi-centrality graph spectral decompositions and their application to cyber intrusion detection,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2016, pp. 4553–4557.
- [9] Y. Park, C. E. Priebe, and A. Youssef, “Anomaly detection in time series of graphs using fusion of graph invariants,” IEEE J. Sel. Topics Signal Process., vol. 7, no. 1, pp. 67–75, 2013.
- [10] M. Kivelä, A. Arenas, M. Barthelemy, J. P. Gleeson, Y. Moreno, and M. A. Porter, “Multilayer networks,” Journal of complex networks, vol. 2, no. 3, pp. 203–271, 2014.
- [11] J. Kim and J.-G. Lee, “Community detection in multi-layer graphs: A survey,” ACM SIGMOD Record, vol. 44, no. 3, pp. 37–48, 2015.
- [12] P.-Y. Chen and A. Hero, “Deep community detection,” IEEE Trans. Signal Process., vol. 63, no. 21, pp. 5706–5719, Nov. 2015.
- [13] D. Cai, Z. Shao, X. He, X. Yan, and J. Han, “Community mining from multi-relational networks,” in European Conference on Principles of Data Mining and Knowledge Discovery. Springer, 2005, pp. 445–452.
- [14] L. Tang, X. Wang, and H. Liu, “Uncoverning groups via heterogeneous interaction analysis,” in IEEE International Conference on Data Mining. IEEE, 2009, pp. 503–512.
- [15] Z. Wu, Z. Bu, J. Cao, and Y. Zhuang, “Discovering communities in multi-relational networks,” in User Community Discovery. Springer, 2015, pp. 75–95.
- [16] L. Tang, X. Wang, and H. Liu, “Community detection via heterogeneous interaction analysis,” Data Mining and Knowledge Discovery, vol. 25, no. 1, pp. 1–33, 2012.
- [17] M. De Domenico, V. Nicosia, A. Arenas, and V. Latora, “Structural reducibility of multilayer networks,” Nature Communications, vol. 6, 2015.
- [18] D. Taylor, S. Shai, N. Stanley, and P. J. Mucha, “Enhanced detectability of community structure in multilayer networks through layer aggregation,” Phys. Rev. Lett., vol. 116, p. 228301, Jun 2016.
- [19] P. W. Holland, K. B. Laskey, and S. Leinhardt, “Stochastic blockmodels: First steps,” Social Networks, vol. 5, no. 2, pp. 109–137, 1983.
- [20] P.-Y. Chen and A. O. Hero, “Universal phase transition in community detectability under a stochastic block model,” Phys. Rev. E, vol. 91, p. 032804, Mar 2015.
- [21] Q. Han, K. Xu, and E. Airoldi, “Consistent estimation of dynamic and multi-layer block models,” in International Conference on Machine Learning, 2015, pp. 1511–1520.
- [22] S. Paul and Y. Chen, “Community detection in multi-relational data with restricted multi-layer stochastic blockmodel,” arXiv preprint arXiv:1506.02699, 2015.
- [23] P. Barbillon, S. Donnet, E. Lazega, and A. Bar-Hen, “Stochastic block models for multiplex networks: an application to a multilevel network of researchers,” Journal of the Royal Statistical Society: Series A (Statistics in Society), 2016.
- [24] T. Vallès-Català, F. A. Massucci, R. Guimerà, and M. Sales-Pardo, “Multilayer stochastic block models reveal the multilayer structure of complex networks,” Phys. Rev. X, vol. 6, p. 011036, Mar 2016.
- [25] N. Stanley, S. Shai, D. Taylor, and P. J. Mucha, “Clustering network layers with the strata multilayer stochastic block model,” IEEE Transactions on Network Science and Engineering, vol. 3, no. 2, pp. 95–105, Apr 2016.
- [26] P.-Y. Chen and A. O. Hero, “Phase transitions and a model order selection criterion for spectral graph clustering,” arXiv preprint arXiv:1604.03159, 2016.
- [27] U. Luxburg, “A tutorial on spectral clustering,” Statistics and Computing, vol. 17, no. 4, pp. 395–416, Dec. 2007.
- [28] J. A. Hartigan and M. A. Wong, “Algorithm AS 136: A k-means clusterin algorithm,” Applied statistics, pp. 100–108, 1979.
- [29] M. Fiedler, “Algebraic connectivity of graphs,” Czechoslovak Mathematical Journal, vol. 23, no. 98, pp. 298–305, 1973.
- [30] A. Jennings and J. J. McKeown, Matrix computation. John Wiley & Sons Inc, 1992.
- [31] P.-Y. Chen and A. Hero, “Phase transitions in spectral community detection,” IEEE Trans. Signal Process., vol. 63, no. 16, pp. 4339–4347, Aug 2015.