Joint CLT for top eigenvalues of sample covariance matrices of separable high dimensional long memory processes

# Joint CLT for top eigenvalues of sample covariance matrices of separable high dimensional long memory processes

Peng TIAN111Department of Statistics and Actuarial Science, The University of Hong Kong. Email: tianpeng83@gmail.com
July 1, 2019
###### Abstract

For , consider the sample covariance matrix

 SN(T)=1NXX∗

from a data set , where is a matrix having i.i.d. entries with mean zero and variance one, and are deterministic positive semi-definite Hermitian matrices of dimension and , respectively. We assume that is bounded in spectral norm, and is a Toeplitz matrix with its largest eigenvalues diverging to infinity. The matrix can be viewed as a data set of an -dimensional long memory stationary process having separable dependence structure.

As and , we establish the asymptotics and the joint CLT for where denotes the th largest eigenvalue of , and is a fixed integer. For the CLT, we first study the case where the entries of are Gaussian, and then we generalize the result to some more generic cases. This result substantially extends our previous result in [27], where we studied , the case where and with having Gaussian entries.

In order to establish this CLT, we are led to study the first order asymptotics of the largest eigenvalues and the associated eigenvectors of deterministic Toeplitz matrices. In particular, we prove multiple spectral gap properties for these largest eigenvalues and a delocalization property for their associated eigenvectors.

## 1 Introduction

#### Motivation.

For , let be a matrix having i.i.d. entries with mean zero and variance one. Let be two deterministic positive semi-definite Hermitian matrices of dimension and , respectively. We define the separable empirical covariance matrix by

 SN(Γ):=1NC1/2NZΓnZ∗C1/2N. (1)

If is identity, the matrix is the classical sample covariance matrix. Let be a positive definite Toeplitz matrix with covariance kernel function

 γ(h)=(1+|h|)ρL(|h|), (2)

where , and is a slowly varying function at infinity. We say that a real function is slowly varying at infinity, or slowly varying for short, if it is asymptotically positive, and for any ,

 limx→∞L(cx)L(x)=1.

Let be a fixed integer. In this article we study the asymptotics and joint CLT of the largest eigenvalues of in the regime and . In the sequel, this regime will be simply denoted as . We assume that the sequence is bounded in spectral norm and the empirical spectral distributions (ESD) converge weakly to a probability measure different from , where denotes the dirac measure at . We recall that the ESD of a Hermitian matrix is defined by

 μA:=1NN∑i=1δλi(A),

where are the eigenvalues of .

If we set , then can be viewed as a sample data set of an -dimensional long memory stationary process with a separable dependence structure. Recall that a multidimensional process , where , is (second order) stationary if

 E∥Xt∥2<∞,EXt=EX0andCov(Xt+h,Xt)=Cov(Xh,X0)=R(h)∀t,h∈Z (3)

where is a positive definite function from to the set of matrices. Here we say that a matrix valued function is positive definite if it satisfies , and for any , the matrix is nonnegative definite. Note that the matrix is the covariance matrix of the concatenate vector

If in particular

 (R(hi−hj))ni,j=1=(γ(hi−hj))ni,j=1⊗CN,

for some nonnegative definite and some positive definite function , where denotes the Kronecker product, then we say that the process has a separable dependence structure, or simply that the process is separable in covariance. Equivalently, the separability condition can also be stated as , for some nonnegative definite matrix and positive definite function .

The classical empirical covariance model has been intensively studied in the last decades. These studies are mostly concentrated on the global behaviors of the spectrum, including limiting spectral distribution (LSD) ([26, 36, 20, 39, 34, 33]) and CLT for linear spectral statistics ([3, 4, 17, 28]); and also concentrated on the local behaviors of individual eigenvalues ([7, 18, 19, 15, 23, 8, 22, 5, 6]). Among these works, we mention that Baik et al. [7], Paul [29] and Bai and Yao [5, 6] studied the CLT of the spiked eigenvalues when all eigenvalues of are one except several who exceed the so-called BBP phase transition threshold, and they established the Gaussian-type fluctuations for these spiked eigenvalues of .

Recently several models of matrices with having a small number of divergent eigenvalues have been considered, in the context of principal component analysis (PCA) [21, 32, 37, 12] and long memory processes [27]. Although the assumptions in these various works differ, many results coincide with the degenerated case of Bai and Yao [6] after normalization (see for example [27]).

The model assumes that the columns of the data matrix are i.i.d. The separable model introduces a special type of correlations between columns, or different weights on columns, achieving certain balance between generality and simplicity. So it attracts more and more attention nowadays. A first result on the model is due to L. Zhang [42] on the LSD of . She proved that if and , then as , the ESD will converge weakly to a nonrandom probability measure for which if or , then ; otherwise for each , the Cauchy-Stieltjes transform of , together with another two functions, denoted by and , is the unique solution in the set

 U={(s(z),gΓ(z),gC(z)):Is(z)<0,I(zgΓ(z))<0,I(zgC(z))<0}

to the following system of equations

 ⎧⎪ ⎪ ⎪⎨⎪ ⎪ ⎪⎩s(z)=z−1(1−r)+z−1r∫11−gC(z)xdνΓ(x),s(z)=z−1∫11−gΓ(z)xdνC(x),s(z)=z−1+gΓ(z)gC(z). (4)

Note that we use Cauchy-Stieltjes transform, opposite of the usual Stieltjes transform. For any probability measure on , its Cauchy-Stieltjes transform is defined as

Later, D. Paul and J.W. Silverstein [30] proved in the case where is diagonal, that almost surely, for large enough , there is no eigenvalue of in any closed interval outside the support of the limiting spectral distribution (LSD). This is an extension of the results of [2] for . R. Couillet and W. Hachem [13] studied the analytical properties of the LSD when and , including the determination of the support of , extending the work of S. Choi and J.W. Silverstein [35] for to the separable model . The CLT for linear spectral statistics has also been studied by Z. Bai et al. [1] and H. Li et al. [24].

As for the extreme eigenvalues of , far less is known compared to the classical model . In [38], F. Yang proved the edge universality under the condition that the densities of the LSD’s have a regular square-root behavior at the rightmost edge (soft edge). With this result, if we find the fluctuations of the largest eigenvalue at a soft edge in the case where the entries are Gaussian, then the fluctuations at a soft edge in general cases will be determined. However, even in the Gaussian case, these fluctuations are still unknown.

The general spiked eigenvalues of have not been studied either. In [41], a very particular case of this problem has been touched. One can refer to Remark 2.4 for the relations between the concerned results of [41] and the ours. This paper does not aim at fully filling this blank, but to initialize by studying the largest eigenvalues of . From [27] we know that defined by (2) has unbounded spectral norm, and after normalization by its largest eigenvalue, the spectrum of concentrates near zero. Then by [42], we know that almost surely as . In [27] we have proved that the distance between the largest two eigenvalues is bounded away from zero (the so-called spectral gap property). So when is identity, the largest eigenvalue of behaves analogously to a classical spiked model. In this paper, we will prove the multiple spectral gap property, that is, for any fixed integer , the distance between the th and the th largest eigenvalues is bounded away from zero. So we can expect that the largest eigenvalues of are spiked, whether or not is identity.

Another contribution from this paper is the study of lagest eigenvalues of Toeplitz matrix defined by (2). Especially we prove the multiple spectral gap property for its lagest eigenvalues. Precisely, in [27] we have proved that the top eigenvalues of normalized Toeplitz matrix converge to the corresponding eigenvalues of a compact operator . In this paper, by proving that all the nonzero eigenvalues of are simple, we prove the multiple spectral gap property. We also study the asymptotic behavior of the eigenvectors associated with these eigenvalues, and prove that they are delocalized, that is, if is a normalized eigenvector of associated with , then there exists a constant independent of such that . These two results are important to our CLT for the largest eigenvalues of , and may have independent interests.

#### Introduction to our results.

As in [27], we normalize by its spectral norm and consider . Then we study the asymptotics and fluctuations of largest eigenvalues of .

We first study the asymptotics of the largest eigenvalues of . In [27], the No eigenvalues outside the support of LSD and the exact separation property of the classical sample covariance matrix play important roles in the proof. For the separable model, analogous results are still unknown. However, from the proof of Theorem 2.3 in [27], we know that the largest eigenvalues of converge to the corresponding eigenvalues of a compact operator. Thus for any small , there is only a finite number of eigenvalues of outside . This allows to obtain the following: assume that the entries of have finite fourth moment, then for any , as , in probability,

 λj(SN(T))∼λj(Tn)trCNN. (5)

Moreover, if the entries ’s are Gaussian, the above convergence holds almost surely. Note that if is identity, this result is consistent with Proposition 2.1 in [27].

Then we study the fluctuations of the largest eigenvalues . But before this, we will study the largest eigenvalues of and prove a multiple spectral gap property: for any fixed integer , the distance between and is bounded away from zero for large .

On this basis, we study the fluctuations of largest eigenvalues of and consider the Gaussian case first. Let and be the eigenvalues of and respectively. Note that if the entries of are Gaussian, then the eigenvalues of have the same distribution as the eigenvalues of

 1Ndiag(√c1,…,√cN)Z∗diag(t1,…,tn)Z∗diag(√c1,…,√cN).

So we can study the generic model , and it is enough to assume that and are diagonal with diagonal elements and , respectively. We also assume that and are bounded in , and , and for any , the number of eigenvalues of outside is bounded. Moreover we assume that satisfies the multiple spectral gap condition, and assume that the entries of have finite sixth moment. Denote

 Λm(Γ):=√N⎛⎜ ⎜ ⎜ ⎜ ⎜⎝λ1(SN(Γ))λ1(Γn)−θ1⋮λm(SN(Γ))λm(Γn)−θm⎞⎟ ⎟ ⎟ ⎟ ⎟⎠ (6)

where is the largest solution of the equation

 1=1NN∑i=1ciθj−(N−1∑k≠jtktj−tk)ci. (7)

We claim that for large enough , the equation (7) admits positive solutions. Also note that under the conditions on and , we have and as . See Remark 2.3 for details. Then, we have

 Λm(Γ)D−−−−−→N,n→∞N(0,[σ2∫x2dνC(x)]Im) (8)

where . As a corollary, if the entries of is standard complex Gaussian, or if they are standard real Gaussian and , are also real, then no matter whether and are diagonal, the CLT (8) holds. And we have if is real Gaussian, and if is complex Gaussian.

This result is consistent with Theorem 2.2 in [27] because if , then the equation (7) becomes

 θj=1+1N∑k≠jtktj−tk. (9)

We also note that in this particular case has a closed formula. This is no longer the case for general . However we can express within a power series of the quantity who tends to as : Let

 mk:=1NN∑i=1cki,

then

 θj=m1+m2m1⎛⎝1N∑k≠jtktj−tk⎞⎠+(m3m22−m22m31)⎛⎝1N∑k≠jtktj−tk⎞⎠2+⋯. (10)

We will also see that if , then , and the above expression of is consistent with (9).

Then we generalize the CLT (8) to some nondiagonal . We preserve the other conditions on , and add new assumptions that the first or the second moment of the ESD of converges to in the speed of , i.e.

 trΓnn=o(1/√n)ortrΓ2nn=o(1/√n).

Intuitively, these two conditions ensure certain concentration of the eigenvalues near zero. Indeed we have assumed that all but a finite number of eigenvalues of are smaller than any , so the eigenvalues of are concentrated near , and asymptotically

 trΓ2nn≤trΓnn,

Therefore the condition expresses a higher concentration of the eigenvalues near than the condition .

If where is the parameter in the definition (2), the normalized Toeplitz matrix satisfies ; and if , we have . It is also known that is a parameter measuring the decay of the autocovariance function of the process. Note that the value is the threshold between short memory process and long memory process. If , then the process has short memory, and is uniformly bounded. In this case we could expect Tracy-Widom fluctuations for the largest eigenvalues of .

We assume that has enough order of moments (the order of moments may vary depending on the conditions). We still assume that is diagonal. Let be a normalized eigenvector (with ) associated with . Denote the Lévy-Prokhorov distance. Then, we prove

 dLP(Λm(Γ),N(0,Σ(N)m))→0

as , where is defined in (6) and with

 σi,j=(E|Z1,1|4−|EZ21,1|2−2)n∑k=1|ui,k|2|uj,k|2+∣∣ ∣∣n∑k=1ui,kuj,k∣∣ ∣∣2|EZ21,1|2.

We state our result in the form of Lévy-Prokhorov distance because in general we do not assume the convergence of . But if we have the convergence , then the CLT can be stated in the usual form

 Λm(Γ)D→N(0,Σm).

Two particular cases are of special interest. First assume that and are both diagonal. From the formula of , we have . Thus

 Λm(Γ)D→N(0,(E|Z1,1|4−1)∫x2dνC(x)Im).

This coincides with (8). Secondly, assume that is diagonal, is real, and as for any , then one can check that

 Σ(N)m−−−−−→N,n→∞|EZ21,1|2∫x2dνC(x)Im.

Thus in this case we have

 Λm(Γ)D→N(0,(1+E|Z21,1|2)∫x2dνC(x)Im). (11)

From Proposition 2.2 we know that the Toeplitz matrices are in this case. Note that in some other lectures are often assumed to be real with variance one, for which (11) is the same as the real Gaussian case; or are assumed to be complex with and , for which (11) is the same as the complex Gaussian case. In this sense the CLT result (11) is universal, though we do not have such kind of assumptions. This explains the phenomenon in Simulation 3(b) of [27].

The above result can be applied to for those whose parameter is in . The case where remains unsolved.

#### Organizations.

This paper is organized as follows. In Section 2 we state our main theorems. This section is divided in three parts. In 2.1, we state the results on Toeplitz matrices ; in 2.2, we state the asymptotics of the largest eigenvalues of ; in 2.3 we state the CLT for largest eigenvalues of in the case where are diagonal, or where are Gaussian; in 2.4, we present some generalizations of the CLT with nondiagonal . The other sections contains the proofs of these results.

#### Notations.

For a Hermitian operator or matrix , we denote its real eigenvalues by decreasing order as

 λ1(A)≥λ2(A)≥…

We also denote the largest eigenvalue of by . For a matrix or a vector , we use to denote the transpose of , and the conjugate transpose of .

The kernel of a linear operator , is denoted by and defined by

 kerA:={u∈X:Au=0}.

The spectrum of is denoted by .

For a matrix , we denote the th row of , and the submatrix of obtained by deleting the th row. Similarly denotes the submatrix of obtained by deleting the th row and the th column. When using these subscripts, we do not indicate the dependence on . By convention, these subscripts have higher priority than the transpose or conjugate transpose, for example is the conjugated transposition of the submatrix .

We denote the or norm by . For a matrix or a linear operator , the norm of induced by vector norm is denoted by , and we recall that . The or norm will be abbreviated as . We say that a function or a vector is "normalized" or "unit length" when or . When functions or vectors are said to be "orthonormal", they will be implicitly considered as elements of a Hilbert space. The inner product of two elements of a Hilbert space is denoted by .

For two probability measures and on , we denote their Lévy-Prokhorov distance by which is defined by

 dLP(P,Q):=inf{ε:P(A)≤Q(Aε)+ε,Q(A)≤P(Aε)+ε,∀A∈B(Rm)}, (12)

where is defined by

 Aε:={x∈Rm:∃y∈A, s.t. ∥x−y∥<ε}.

It is well known that this distance metrizes the weak convergence. For two random variables with distributions , respectively, we sometimes write , or which all mean .

Given , we denote by the integer satisfying . Given two sequences of nonnegative numbers , we denote

 xn∼yn ⇔ limn→∞(xnyn)=1.

The notations and mean and , respectively. If are random variables, the notation means that in probability. The notations and denote convergence in distribution and in probability, respectively. If are measures, we denote with a slight abuse of notation for the weak convergence of to .

###### Definition 1.

We say that a sequence of events hold with high probability, if ; with low probability, if ; with overwhelming probability, if for any , ; with tiny probability, if for any , .

The cardinal of a set is denoted by . In the proofs we use to denote a constant that may take different values from one place to another.

## 2 Main theorems

### 2.1 Spectral properties of Toeplitz matrices

We collect our results on Toeplitz matrices in this section. Let be a Toeplitz matrix defined by

 Tn:=(γ(i−j))ni,j=1 (13)

with a function in the form

 γ(h)=(1+|h|)ρL(|h|),∀h∈Z, (14)

where and is a slowly varying function at infinity. Let be the operator defined on by

 (K(ρ)f)(x)=∫10|x−y|ρf(y)dy, for f∈L2(0,1). (15)

In [27] We have established the relation between the eigenvalues of and the eigenvalues of . From the proof of Theorem 2.3 of [27] we know that the operator is compact and positive semi-definite. It has infinitely many positive eigenvalues. And for any , we have

 limn→∞λj(Tn)nγ(n)=λj(K(ρ))>0. (16)

Using the min-max formula for the largest eigenvalue and an argument by absurd, we have also proved that is simple, so that we proved the spectral gap property for the largest two eigenvalues of :

 limn→∞λ2(Tn)λ1(Tn)<1.

In this paper, using a different method, we will prove that all nonzero eigenvalues of are simple. As a consequence we prove the multiple spectral gap property for any th largest eigenvalue of :

 limn→∞λj+1(Tn)λj(Tn)<1.
###### Proposition 2.1.

All nonzero eigenvalues of the operator defined by (15) with are simple, and the associated eigenfunctions are continuous in .

We note that is self-adjoint, so for any nonzero eigenvalue , its algebraic multiplicity equals to its geometric multiplicity, which is defined as . For more information about algebraic multiplicity, see [25]. So here we say that a nonzero eigenvalue is simple, it means that

 dimker(λI−K(ρ))=1.

In the next proposition, we provide a quantitative description of the eigenvectors associated with for any fixed .

###### Proposition 2.2.

For any , let be the normalized eigenfunction of associated with , and be a normalized eigenvector of associated with . Then, up to a change of sign, we have

 limn→∞sup1≤k≤n{∣∣∣√nuj,k−fj(kn)∣∣∣}=0. (17)

From this proposition we deduce the delocalization of eigenvector associated with for any fixed . Indeed by (17), for large enough , we have

 ∥uj∥∞≤1+∥fj∥∞√n,

and because is continuous on , we have . Thus we conclude that

 ∥uj∥∞=Oj(1/√n)→0.

Let . The following proposition provides the decay of moments of the ESD .

###### Proposition 2.3.

Let be defined as above and .

1. If , then

 ∫xdμ~Tn=tr~Tnn=o(1/√n). (18)
2. If , then

 ∫x2dμ~Tn=tr~T2nn=o(1/√n). (19)

### 2.2 Convergence of largest eigenvalues of separable sample covariance matrix

For , let

 SN(Γ):=1NC1/2NZΓnZ∗C1/2N, (20)

where are respectively and deterministic positive semi-definite Hermitian matrices, and is a matrix having i.i.d. entries . Let

 c1≥c2≥⋯≥cNandt1≥t2≥⋯≥tn

be the eigenvalues of and respectively. Let be a fixed integer. We assume that the following assumptions hold:

1. The entries satisfy

 EZi,j=0,E|Zi,j|2=1andE|Zi,j|4<∞,
2. The spectral norm is bounded in , and the ESD of converges weakly as , to a probability measure .

3. There exists a decreasing sequance of nonzero positive numbers

 a1≥a2≥⋯

converging to such that for any , we have

 limn→∞tj=aj.

For further use, we will prove a concentration lemma for the largest eigenvalues of which will assume the following conditions.

1. The Hermitian matrices and are diagonal:

 CN=diag(c1,…,cN),Γn=diag(t1,…,tn).
2. (Bound condition) There exists a sequence of positive numbers such that almost surely for large enough ,

 |Zi,j|≤εn√n,

Note that under 3, we have , and for any ,

 supn#{j:tj>ε}<∞.
###### Remark 2.1.

We take two examples for which the bound condition 5 holds. The first case is where with some . In this case, we have

 ∞∑n=1NnP(|Zi,j|>εn√n)≤∞∑n=1Kε6+ϵnn1+ϵ/2<∞,

where we have assumed that the convergence rate of to is slower than any preassigned rate. Then by Borel-Cantelli’s Lemma, the bound condition holds.

The second case is where , and does not depend on for any fixed . In other words, are all from an infinite double array . In this case, by the truncation lemma 2.2 of [40], the bound condition holds.

Recall that we use to denote .

###### Proposition 2.4.

Let be defined as (20). Under 1, 2 and 3, for any , we have

 λj(SN(Γ))P−−−−−→N,n→∞aj∫xdνC(x).

Moreover, if ’s are real or complex Gaussian, or if 4 and 5 hold, then the above convergence holds almost surely.

###### Remark 2.2.

The almost sure convergence under 4 and 5 is in fact a byproduct of Lemma 4.1 which is needed by the proof of CLT 2.6. However this does not allow to conclude the a.s. convergence when ’s are Gaussian. Indeed if the entries of are i.i.d real Gaussian variables, and if or are complex and non diagonal, then we cannot diagonalize or because the real Gaussian vectors are not unitary invariant. Thus we will proceed an independent proof for Gaussian case with help of a Gaussian concentration inequality.

Applying the above generic result to the special case of , and combining with (16), we obtain the following result:

###### Corollary 2.5.

Let , be defined as before. Then if 1, 2 hold, we have

 λj(SN(T))nγ(n)P−−−−−→N,n→∞λj(K(ρ))∫xdνC(x),

or equivalently,

 λj(SN(T))∼λj(Tn)trCNN

in probability. Moreover, if ’s are standard real or complex Gaussian, then the above convergence holds almost surely.

### 2.3 CLT for largest eigenvalues: Diagonal & Gaussian case

In this section, we assume that , are diagonal, and study the CLT for largest eigenvalues of . As a corollary, we obtain the result for Gaussian case.

1. The sixth moment of the entries is finite:

 E|Z1,1|6<∞.
2. The largest eigenvalues of satisfy the multiple spectral gap property:

 ¯¯¯¯¯¯¯¯limn→∞tj+1tj<1for j=1,…,m.

For we define

 G(x,z):=1NN∑i=1cix−zci. (21)

For , let be the largest solution of the equation

 G⎛⎝θj,1N∑k≠jtktj−tk⎞⎠=1NN∑i=1ciθj−(1N∑k≠jtktj−tk)ci=1. (22)
###### Remark 2.3.

Note that if not all ’s are , and if , then from the graph of the function , we see that the equation on admits real solutions.

Moreover, we prove that the largest solution of (22) tends to . Indeed, we know that under the assumptions 3 and 7, we have

 N−1∑k≠jtktj−tk−−−−−→N,n→∞0,

and the assumption 2 ensures that

 1NN∑i=1ci−−−−→N→∞∫xdνC(x)≠0.

Also note that for every fixed ,

 G⎛⎝θ,1N∑k≠jtktj−tk