Empirical likelihood approach to goodness of fit testing

# Empirical likelihood approach to goodness of fit testing

[ [    [ [ Department of Mathematical Sciences, Indiana University Purdue University at Indianapolis, Indianapolis, IN 46202, USA. \printeade1 Department of Mathematical Sciences, Binghamton University, Binghamton, NY 13902, USA.
\smonth10 \syear2010\smonth3 \syear2012
\smonth10 \syear2010\smonth3 \syear2012
\smonth10 \syear2010\smonth3 \syear2012
###### Abstract

Motivated by applications to goodness of fit testing, the empirical likelihood approach is generalized to allow for the number of constraints to grow with the sample size and for the constraints to use estimated criteria functions. The latter is needed to deal with nuisance parameters. The proposed empirical likelihood based goodness of fit tests are asymptotically distribution free. For univariate observations, tests for a specified distribution, for a distribution of parametric form, and for a symmetric distribution are presented. For bivariate observations, tests for independence are developed.

\kwd
\aid

0 \volume19 \issue3 2013 \firstpage954 \lastpage981 \doi10.3150/12-BEJ440 \newremarkexampleExample \newremarkremarkRemark[section] \newremarkdetailsDetails

\runtitle

Goodness of Fit

{aug}

1]\fnmsHanxiang \snmPeng\thanksref1label=e1]hpeng@math.iupui.edu and 2]\fnmsAnton \snmSchick\corref\thanksref2label=e2]anton@math.binghamton.edu

estimated constraint functions \kwdinfinitely many constraints \kwdnuisance parameter \kwdregression model \kwdtesting for a parametric model \kwdtesting for a specific distribution \kwdtesting for independence \kwdtesting for symmetry

## 1 Introduction

The empirical likelihood approach was introduced by Owen O88 (), O90 () to construct confidence intervals in a nonparametric setting, see also Owen O01 (). As a likelihood approach possessing nonparametric properties, it does not require us to specify a distribution for the data and often yields more efficient estimates of the parameters. It allows data to decide the shape of confidence regions and is Bartlett correctable (DiCiccio, Hall and Romano DH91 ()). The approach has been developed to various situations, for example, to generalized linear models (Kolaczyk K94 ()), local linear smoother (Chen and Qin CQ00 ()), partially linear models (Shi and Lau SL00 (), Wang and Jing WJ03 ()), parametric and semiparametric models in multiresponse regression (Chen and Van Keilegom CK09 ()), linear regression with censored data (Zhou and Li ZL08 ()), and plug-in estimates of nuisance parameters in estimating equations in the context of survival analysis (Qin and Jing QJ01a (), Wang and Jing WJ01 (), Li and Wang LC01 ()). Algorithms, calibration and higher-order precision of the approach can be found in Hall and La Scala HS90 (), Emerson and Owen EO09 () and Liu and Chen LC10 () among others. It is especially convenient to incorporate side information expressed through equality constraints. Qin and Lawless QL94 () linked empirical likelihood with finitely many estimating equations. These estimating equations serve as finitely many equality constraints.

In semiparametric settings, information on the model can often be expressed by means of infinitely many constraints which may also depend on parameters of the model. In goodness of fit testing, the null hypothesis can typically be expressed by infinitely many such constraints. This is the case when testing for a fixed distribution (see Example 2 below), when testing for a given parametric model (Example 2), when testing for symmetry about a fixed point (Example 2), and when testing for independence (Example 2). Modeling conditional expectations can also be done by means of infinitely many constraints. This has applications to heteroscedastic regression models (Section 3) and to conditional moment restriction models treated by Tripathi and Kitamura TK03 () using a smoothed empirical likelihood approach.

Recently Hjort, McKeague and Van Keilegom HMV09 () extended the scope of the empirical method. In particular, they developed a general theory for constraints with nuisance parameters and considered the case with infinitely many constraints. Their results for infinitely many constraints, however, do not allow for nuisance parameters. In this paper we will fill this gap and in the process improve on their results. Let us now discuss some of our results in the following special case.

Let be independent copies of a random vector with distribution . Let be orthonormal elements of

 L2,0(Q)={u∈L2(Q)\dvt∫udQ=0}.

Then the random variables have mean zero, variance one and are uncorrelated. Now consider the empirical likelihood based on the first of these functions,

 Rn=sup{n∏j=1nπj\dvtπ∈Pn,n∑j=1πjuk(Zj)=0,k=1,…,m},

where denotes the closed probability simplex in dimension . For fixed , it follows from Owen’s work that has asymptotically a chi-square distribution with degrees of freedom. In other words,

 P(−2logRn>χ21−α(m))→α,0<α<1, (1)

where denotes the -quantile of the chi-square distribution with degrees of freedom. Hjort et al. HMV09 () have shown that (1) holds under some additional assumptions even if tends to infinity with by proving the asymptotic normality result

 (−2logRn−m)/√2m⟹N(0,1). (2)

This result requires higher moment assumptions on the functions and restrictions on the rate at which can tend to infinity. For example, if the functions are uniformly bounded, then the rate suffices for (2). They also state in their Theorem 4.1, that if is finite for some , then suffices for (2). A gap in their argument was fixed by Peng and Schick PS12 (). We shall show that larger are allowed in some cases. In particular, for , it suffices that holds (instead of their ) and if , then is enough (instead of their ), see our Theorems 7.2 and 7.3 below.

Our rate for matches the rate given in Theorem 2 of Chen, Peng and Qin CPQ09 (). These authors obtain asymptotic normality for larger than in Hjort et al. HMV09 () by imposing additional structural assumptions. These assumptions, however, are typically not met in the applications we have in mind.

One of the key points in our proof is a simple condition for the convex hull of some vectors to have the origin as an interior point. Our condition is that the smallest eigenvalue of exceeds . Here, denotes the euclidean norm of a vector . This sufficient condition ties in nicely with the other requirements used to establish the asymptotic behavior of the empirical likelihood and is typically implied by these. For example, conditions (A1)–(A3) in Theorem 2.1 of Hjort et al. HMV09 () already imply their (A0). Thus, the conclusion of their theorem is valid under (A1)–(A3) only, see our Theorem 6.1.

Let us now look at the case when the functions are unknown. Then we can work with the empirical likelihood

 ^Rn=sup{n∏j=1nπj\dvtπ∈Pn,n∑j=1πj^uk(Zj)=0,k=1,…,m},

where is an estimator of such that

 m∑k=11nn∑j=1(^uk(Zj)−uk(Zj))2=op(m−1). (3)

Now, we have the conclusion under the condition

 m∑k=1(n−1/2n∑j=1(^uk(Zj)−uk(Zj)))2=op(1) (4)

and mild additional conditions such as

1. [(ii)]

2. for some constant and all and , or

3. and .

Our results, however, go beyond this simple result. If (4) is replaced by

 m∑k=1(n−1/2n∑j=1(^uk(Zj)−uk(Zj)+E[uk(Z)ψ⊤(Z)]ψ(Zj)))2=op(1) (5)

with a measurable function into which is standardized under in the sense that and , the identity matrix, then the conclusion holds under (i) or (ii).

Our paper is organized as follows. In Section 2, we give four examples that motivate our research. The emphasis in these examples is on goodness of fit testing. The proposed empirical likelihood based goodness of fit tests are asymptotically distribution free. For univariate observations, tests for a specified distribution, for a distribution of parametric form, and for a symmetric distribution are presented. For bivariate observations, tests for independence are discussed. Another example is given in Section 3 with a small simulation study. This example considers tests for the regression parameters in simple linear heteroscedastic regression. The simulations compare our new procedure based on infinitely many constraints with the classical empirical likelihood procedure and illustrate improvements by the new procedures. In Section 4, we introduce notation and recall some results on the spectral norm of matrices. In Section 5, we derive a lemma that extracts the essence from the proofs of Owen (O01 (), Chapter 11) and also obtains the aforementioned sufficient condition for a convex hull of vectors to contain the origin as interior point. The results are derived for non-stochastic vectors and formulated as inequalities. The inequalities are used in Section 6 to obtain the behavior of the empirical likelihood with random vectors whose dimension may increase. The results are formulated abstractly and do not require independence. In Section 7, we specialize our results to the case of independent observations with infinitely many constraints, both known and unknown. We also briefly discuss the behavior under contiguous alternatives. The details for our examples are given in Section 8.

## 2 Motivating examples

In this section, we give examples that motivated the research in this paper.

{example}

[(Testing for a fixed distribution)] Let be independent copies of a random variable . Suppose we want to test whether their common distribution function equals a known continuous distribution function . Under the null hypothesis, we have for every , and has a uniform distribution. An orthonormal basis of is thus given by for any orthonormal basis of , where is the uniform distribution on . We shall work with the trigonometric basis defined by

 ϕk(x)=√2cos(k\uppix),x∈[0,1],k=1,2,…, (6)

as these basis functions are uniformly bounded by . As test statistic, we take

which uses the first of the trigonometric functions. Under the null hypothesis, we have for every as both and tend to infinity and tends to zero. Thus, the test has asymptotic size . Here, we are still in the framework of Hjort et al. HMV09 () with infinitely many known constraints.

{example}

[(Testing for a parametric model)] Let be again independent and identically distributed random variables. But now suppose we want to test whether their common distribution function belongs to a model indexed by an open subset of . Suppose that the distribution functions have densities such that the map is continuously differentiable in with derivative and the matrix is invertible for each . In this case we set . Let now be an estimator of the parameter in the model. We require it to satisfy the stochastic expansion

 ^θ=θ+1nn∑j=1J(θ)−1˙ℓθ(Xj)+oPθ(n−1/2) (7)

for each , where is the measure for which . Such estimators are efficient in the parametric model. Candidates are maximum likelihood estimators. As test statistic we take , the test statistic from the previous example with replaced by . Here, we are no longer in the framework of Hjort et al. HMV09 () as we now have infinitely many unknown constraints. We shall show that under the null hypothesis for every as both and tend to infinity and tends to zero. In view of this result, the test has asymptotic size . It is crucial for our result that we have chosen an estimator satisfying (7).

{example}

[(Testing for symmetry)] Let be independent copies of a random variable with a continuous distribution function . We want to test whether is symmetric about zero in the sense that for all real . Under the null hypothesis of symmetry, the random variables and are independent, and takes values and with probability one half. This is equivalent to for every , where is the distribution function of . Since is continuous, an orthonormal system of is given by where and are given in (6). This suggests the test statistic

 \alignedRn=sup{n∏j=1nπj\dvtπ∈Pn,∑nj=1πjsign(Xj)ϕk(Rj)=0,k=0,…,m},

where and is the empirical distribution function based on . We shall show that under symmetry one has for every as and tend to infinity and tends to zero. From this, we derive that the test has asymptotic size .

{example}

[(Testing for independence)] Let be independent copies of a bivariate random vector . We assume that the marginal distribution functions and are continuous. We want to test whether and are independent. Independence is equivalent to for all and and thus equivalent to for all positive integers and .

{longlist}

[(b)]

Assume first that and are known. This is for example the case in an actuarial setting where and denote residual lifetimes and their distribution functions are available from life tables. Motivated by the above, we take as test statistics

 Rn(F,G)=sup{n∏j=1nπj\dvtπ∈Pn,n∑j=1πjϕk(F(Xj))ϕl(G(Yj))=0,k,l=1,…,r}.

Under the null hypothesis, one has for every as and tend to infinity and tends to zero. Here, we are in the framework of Hjort, McKeague and Van Keilegom HMV09 (). The above shows that the test has asymptotic size .

Now assume that and are unknown. In this case, we replace both marginal distribution functions by their empirical distribution functions. The resulting test statistic is , where denotes the empirical distribution based on and the one based on . We shall show that under the null hypothesis for every as and tend to infinity and tends to zero. Thus the test has asymptotic size .

{remark}

Suppose that form a simple linear homoscedastic regression model, , with and independent. We can use the test statistic from case (b) to test the hypothesis whether the slope parameter is zero. Indeed, is equivalent to the independence of and .

{remark}

The asymptotic distributions of the above tests under contiguous alternatives are linked to non-central chi-square distributions; see Remark 7 for details. As the non-centrality parameters are bounded, the local asymptotic power along such a contiguous alternative coincides with the level. Our tests are asymptotically equivalent to Neyman’s smooth tests N39 () with increasing dimensions. In view of the optimality results of Inglot and Ledwina IL96 (), for those tests under moderate deviations, we expect similar results for our tests. Of course, this needs to be explored more carefully.

## 3 Another example and simulations

Let be independent copies of , where , with , bounded and bounded away from zero, and . Assume that has a finite variance and a continuous distribution function . We are interested in testing whether the regression parameter equals some specific value . We could proceed as in Owen O91 () and use the test based on the empirical likelihood

 Rn0(θ)=sup{n∏j=1nπj\dvtπ∈Pn,n∑j=1πj(1Xj)(Yj−θ1−θ2Xj)=0}.

But this empirical likelihood does not use all the information of the model. Here we have for every . Since is continuous (but unknown), we work with the empirical likelihood

 ^Rn1(θ)=sup{n∏j=1nπj\dvtπ∈Pn,n∑j=1πjur(G(Xj))(Yj−θ1−θ2Xj)=0},

where and is the empirical distribution function based on the covariate observations . It follows from Corollary 7.6 and Lemma 8.1 below that if . The resulting test is Both tests have asymptotic size .

We performed a small simulation study to compare the procedures. For our simulation, we chose and and took . We modeled the error as , with and independent of . As distributions for , we chose the exponential distribution with mean 5 ((5)) and the -distribution with three degrees of freedom ((3)), while for we chose the standard normal distribution ((0, 1)) and the double exponential distribution with location 0 and scale 0.5 ((0, 0.5)).

Table 1 reports simulated powers of the tests and (with several choices of ) and for some values of . The reported values are based on 1000 repetitions. The column labeled 0 corresponds to Owen’s test , while the columns labeled 2, 3, 4, 5 correspond to our tests with , respectively. Clearly our new test is more powerful than the traditional test. The values in the rows corresponding to the parameter values are the observed significance levels of the nominal significance level . Our new test overall has closer observed significance levels than the traditional one except for .

## 4 Notation

In this section, we introduce some of the notation we use throughout. We write for the euclidean norm and for the operator (or spectral) norm of a matrix which are defined by

 |A|2=trace(A⊤A)=∑i,jA2ijand|A|o=sup|u|=1|Au|=sup|u|=1(u⊤A⊤Au)1/2.

In other words, the squared euclidean norm equals the sum of the eigenvalues of , while the squared operator norm equals the largest eigenvalue of . Consequently, the inequality holds. Thus, we have

 |Ax|≤|A|o|x|≤|A||x|

for compatible vectors . We should also point out the identity

 |A|o=sup|u|=1sup|v|=1u⊤Av.

If is a nonnegative definite symmetric matrix, this simplifies to

 |A|o=sup|u|=1u⊤Au.

Using this and the Cauchy–Schwarz inequality, we obtain

 ∣∣∣∫fg⊤dμ∣∣∣2o ≤ ∣∣∣∫ff⊤dμ∣∣∣o∣∣∣∫gg⊤dμ∣∣∣o, (8) ∣∣∣∫ff⊤dμ∣∣∣o ≤ ∫|f|2dμ, (9)

whenever is a measure and and are measurable functions into and such that and are finite. As a special case, we derive the inequality

 |Sx+y−Sx|o≤|Sy|o+2|Sx|1/2o|Sy|1/2o

and therefore

 |Sx+y−Sx|o≤1nn∑j=1|yi|2+2|Sx|1/2o(1nn∑j=1|yi|2)1/2 (10)

with

 Sx+y=1nn∑j=1(xj+yj)(xj+yj)⊤,Sx=1nn∑j=1xjx⊤j,Sy=1nn∑j=1yjy⊤j

for vectors of the same dimension.

## 5 A maximization problem

Let be -dimensional vectors. Set ,

 ¯x=1nn∑j=1xj,S=1nn∑j=1xjx⊤j,x(ν)=sup|u|=1∣∣∣1nn∑j=1(u⊤xj)ν∣∣∣,ν=3,4,

and let and denote the smallest and largest eigenvalue of the matrix ,

 λ=inf|u|=1u⊤SuandΛ=sup|u|=1u⊤Su.

Using Lagrange multipliers, Owen O88 (), O01 () obtained the identity

 R=sup{n∏j=1nπj\dvtπ∈Pn,n∑j=1πjxj=0}=n∏j=111+ζ⊤xj

if there exists a in such that , , and

 n∑j=1xj1+ζ⊤xj=0. (11)

He also showed that such a vector exists and is unique if (i) the origin is an interior point of the convex hull of and (ii) the matrix is invertible. Let us now show that the inequality implies these two conditions. Indeed, the matrix is then positive definite and hence invertible as its smallest eigenvalue is positive. To show (i), we will rely on the following lemma.

###### Lemma 5.1

A random variable with and for some positive obeys the inequality

 P(Y>a)≥E[Y2]−2Ka2K2,0≤a
{pf}

Fix in . By the properties of , we obtain and .

The origin is an interior point of the convex hull of if for every unit vector there is at least one such that . This latter condition is equivalent to

 N=inf|u|=1n∑j=11[u⊤xj>0]≥1.

For a unit vector , we have and thus

 n∑j=11[u⊤xj>0]≥n∑j=11[u⊤(xj−¯x)>|¯x|]=N(u).

It follows from the triangle inequality that for . Note that is positive if is positive definite. Thus, Lemma 5.1 yields the lower bound with

 σ2(u)=1nn∑j=1(u⊤(xj−¯x))2=uTSu−(u⊤¯x)2≥λ−|¯x|2≥λ−x∗|¯x|.

Thus, we have . This shows that the inequality implies and hence the desired condition (i).

Assume now that the inequality holds. We proceed as on page 220 of Owen O01 (). Let be a unit vector such that . Then we have the identity

 0=1nn∑j=1u⊤xj(1+ζ⊤xj−ζ⊤xj)1+ζ⊤xj=u⊤¯x−|ζ|1nn∑j=1(u⊤xj)21+ζ⊤xj

and the inequality

 λ≤u⊤Su=1nn∑j=1(u⊤xj)2≤1nn∑j=1(u⊤xj)2(1+|ζ|x∗)1+ζ⊤xj.

Consequently, we find and obtain the bound

 |ζ|≤|¯x|λ−|¯x|x∗. (12)

From this, one immediately derives

 |ζ|x∗ ≤ |¯x|x∗λ−|¯x|x∗<14, (13) max1≤j≤n11+ζ⊤xj ≤ 11−|ζ|x∗<43, (14) 1nn∑j=1(ζ⊤xj)2 = ζ⊤Sζ≤Λ|ζ|2≤Λ|¯x|2(λ−|¯x|x∗)2. (15)

The identity and (14) yield

 ∣∣∣1nn∑j=1(rj1+ζ⊤xj−rj+rjx⊤jζ)∣∣∣≤∣∣∣1nn∑j=1rj(ζ⊤xj)2∣∣∣+431nn∑j=1|rj|∣∣ζ⊤xj∣∣3

for vectors of the same dimension. Taking , we derive with the help of (11)

 ∣∣ζ−S−1¯x∣∣≤∣∣∣1nn∑j=1S−1xj(ζ⊤xj)2∣∣∣+431nn∑j=1∣∣S−1xj∣∣∣∣ζ⊤xj∣∣3.

Using , the Cauchy–Schwarz inequality, (13) and (15) we bound the square of the first summand of the right-hand side by

 1nn∑j=1(ζ⊤xj)4sup|v|=1v⊤S−1v≤1λ|ζ|4x(4)

and the square of the second summand by

 169λ2x2∗ζ⊤Sζ1nn∑j=1(ζ⊤xj)4≤Λ9λ2|ζ|4x(4).

Combining the above, we obtain

 ∣∣ζ−S−1¯x∣∣2≤2(1λ+Λ9λ2)|ζ|4x(4). (16)

Using the inequality valid for , and then (14) we derive

 ∣∣∣2n∑j=1log(1+ζ⊤xj)−2nζ⊤¯x+nζ⊤Sζ∣∣∣≤23∣∣∣n∑j=1(ζ⊤xj)3∣∣∣+12(43)4n∑j=1∣∣ζ⊤xj∣∣4.

With , we can write and , and obtain the identity . Using this and (16), we arrive at the bound

 ∣∣∣2n∑j=1log(1+ζ⊤xj)−n¯x⊤S−1¯x∣∣∣≤n|ζ|3x(3)+n(169+2Λλ+2Λ29λ2)|ζ|4x(4).

In view of (12) and , this becomes

 ∣∣∣2n∑j=1log(1+ζ⊤xj)−n¯x⊤S−1¯x∣∣∣≤n|¯x|3x(3)(λ−|¯x|x∗)3+Λ2λ24n|¯x|4x(4)(λ−|¯x|x∗)4. (17)

If we bound by and by and use (13), we obtain the bound

 ∣∣∣2n∑j=1log(1+ζ⊤xj)−n¯x⊤S−1¯x∣∣∣≤(Λ+Λ3λ2)nx∗|¯x|3(λ−|¯x|x∗)3. (18)

Thus, we have proved the following result.

###### Lemma 5.2

The inequality implies that there is a unique in satisfying , , and (11) to (18).

## 6 Applications with random vectors

We shall now discuss implications of Lemma 5.2 to the case when the vectors are replaced by random vectors. We are interested in the case when the dimension of the random vectors increases with .

Let be -dimensional random vectors. With these random vectors we associate the empirical likelihood

 Rn=sup{n∏j=1nπj\dvtπ∈Pn,n∑j=1πjTnj=0}.

To study the asymptotic behavior of , we introduce

 T∗n=max1≤j≤n|Tnj|,¯Tn=1nn∑j=1Tnj,T(ν)n=sup|u|=1∣∣∣1nn∑j=1(u⊤Tnj)ν∣∣∣,ν=3,4

and the matrix

 Sn=1nn∑j=1TnjT⊤nj,

and let and denote the smallest and largest eigenvalues of ,

 λn=inf|u|=1u⊤SnuandΛn=sup|u|=1u⊤Snu.