Two New Entropy Estimators for Testing Exponentiality with Type-II Censored Data

# Two New Entropy Estimators for Testing Exponentiality with Type-II Censored Data

A.Kohansal., S.Rezakhah 111Faculty of Mathematics and Computer Science, Amirkabir University of Technology, Tehran, Iran. Email: rezakhah@aut.ac.ir, ak_kohansal@aut.ac.ir
###### Abstract

This paper proposes two estimators of the joint entropy of the Type-II censored data. Consistency of both estimators is proved. Simulation results show that the second one shows less bias and root of mean square error (RMSE) than leading estimator. Also, two goodness of fit test statistics based on the Kullback-Leibler information with the Type-II censored data are established and their performances with the leading test statistics are compared. We provide a Monte Carlo simulation study which shows that the test statistics and show better powers than leading test statistics against the alternatives with monotone decreasing and monotone increasing hazard functions, respectively.

Keywords: Entropy, Monte Carlo simulation, Kullback-Leibler information, Moving average method, Hazard function.

Mathematics Subject Classification: 62G10, 62G30.

## 1 Introduction

Suppose that a random variable has a distribution function , with a continuous density function . The differential entropy of the random variable is defined by Shannon [22] to be

 H(f)=−∫∞−∞f(x)lnf(x)dx. (1.1)

The entropy difference has been considered in [8] and [11] for establishing the goodness of fit tests for the class of the maximum entropy distributions.

The Kullback-Leibler (KL) information in favor of against is defined to be

 I(g;f)=∫∞−∞g(x)lng(x)f(x)dx.

Because has the property that , and the equality holds if and only if , different estimators of the KL information has been also considered as a goodness of fit test statistic in some papers including [2], [9], [20] and [25]. For complete samples, some of these test statistics perform very well for exponentiality [9], and some others of them perform very well for normality, see [23], [26] and [1].

For the censored data, some authors studied the problem of goodness of fit test and discussed some test statistics. Brain and Shapiro [6] proposed two test statistics and show that these test statistics perform better than other test statistics for the censored data. Samanta and Schwarz [21] proposed a test statistic and showed that the proposed test statistic has competing performance with the test statistics which introduced by Brain and Shapiro [6] for the censored data. Recently Park [19] obtained an estimator for entropy of Type-II censored data and proposed a test statistic based on KL information. He showed that the power of the proposed test statistic is greater than the power of the test statistics which proposed by Brain and Shapiro [6], and Samanta and Schwarz [21] against the alternatives with monotone increasing hazard functions. In the case of progressively censored data, Balakrishnan et al. [4] studied the testing exponentiality based on KL information with progressively Type-II censored data. Habibi Rad et al. [12] studied goodness of fit test based on KL information for progressively Type-II censored data. Pakyari and Balakrishnan [17] proposed several goodness of fit methods for location-scale families of distributions under progressively Type-II censored data. They [18] also investigated a general purpose approximate goodness of fit test for progressively Type-II censored data.

In this paper, we enhance the estimator which was introduced by Park [19] and obtain two new entropy estimators of Type-II censored data. Simulation results show that the second one shows less bias and RMSE than leading estimator. Also, we provide two new test statistics. The first one achieves higher power than the previous test statistics against the alternatives with monotone decreasing hazard functions and the other one achieves higher power than the previous test statistics against the alternatives with monotone increasing hazard functions.

The rest of the article is arranged as follows: In Section 2, we introduce two estimators of the joint entropy of the Type-II censored data. Also, we show that both are consistent. Scale invariance property of variances and mean squared errors of the proposed estimators is studied in the same section. In Section 3, we use the KL information with the Type-II censored data and obtain two new test statistics. In Section 4, we introduce goodness-of-fit tests for exponentiality based on the proposed test statistics and then compare their powers with the powers of other test statistics. Also, by using the new test statistics, we compare biases and RMSEs of the new entropy estimators with the leading entropy estimator.

## 2 New entropy estimators

In this section, we introduce two entropy estimators and prove some of their properties.

### 2.1 Entropy estimator for monotone decreasing hazard function alternatives

In this subsection, we obtain one entropy estimator which provides a new test statistic that achieves higher power than the previous test statistics against the alternatives with monotone decreasing hazard functions.

Vasicek [23] expressed (1.1) in the form,

 H=∫10ln(dF−1pdpdp)

and provided its estimator as:

 H(m,n)=1nn∑i=1lnx(i+m:n)−x(i−m:n)2mn,

where the window size is a positive integer, which is less than ; and for , and for . Recently Park [19] expressed the joint entropy of , , in the form where

 ¯H1⋯r:n=−E(∫U(r:n−1)0ln(dF−1(p)dp)dp)−E((1−U(r:n−1))ln(1−U(r:n−1))),

and provided its estimator as:

 ¯Hm,n,r=1nr∑i=1ln(x(i+m:n)−x(i−m:n)2mn)−(1−rn)ln(1−rn). (2.2)

By approximating with

 −∫rn0ln(dF−1(p)dp)dp−(1−rn)ln(1−rn), (2.3)

we obtain an estimator for (2.3) as:

 ¯H(1)m,n,r=1nr∑i=1ln(¯xhi−1+m−¯xhi−1−mmn)−(1−rn)ln(1−rn), (2.4)

where is the harmonic mean of and is the harmonic mean of and the window size is a positive integer, which is less than ; and for , and for We expect that the performance of this estimator is better than (2.2), because we use more information for its calculation.

We can easily prove that the scale of the random variable has no effect on the accuracy of in estimating .

###### Property 2.1

Let and denote entropies of the distribution of continuous random variables and , respectively, and , where . It is easy to see that for . So we have Then the following properties hold

where the superscript and refer to the corresponding distribution.

###### Lemma 2.1

If and , then , which is defined in (2.2).

Proof: If we prove that then by Squeeze theorem, . So we establish as follows:

 0≤∣∣¯H(1)m,n,r−¯Hm,n,r∣∣ = ∣∣ ∣∣1nr∑i=1ln¯xhi−1+m−¯xhi−1−m2(x(i+m:n)−x(i−m:n))∣∣ ∣∣ ≤ 1nr∑i=1∣∣ ∣∣ln¯xhi−1+m−¯xhi−1−m2(x(i+m:n)−x(i−m:n))∣∣ ∣∣ ≤ 1nr∑i=1∣∣∣lnx(i−1+m:n)−x(i−1−m:n)2(x(i+m:n)−x(i−m:n))∣∣∣ → 0, as m,n→0 and mn→0.

The first inequality arises by using the Triangle inequality, and the second inequality is true because

 ¯xhi−1+m≤x(i−1+m:n) and ¯xhi−1+m≥x(i:n),

therefore, . This completes the proof.

###### Theorem 2.1

If and , then is a consistent estimator of .

Proof: Park [19] showed that is a consistent estimator of . So

 E[¯Hm,n,r] → ¯H1⋯r:n, (2.5) Var[¯Hm,n,r] → 0, (2.6)

as , and . According to the previous Lemma, , so (Billingsley [5])

 E[¯H(1)m,n,r−¯Hm,n,r] → 0, (2.7) E[¯H(1)m,n,r−¯Hm,n,r]2 → 0. (2.8)

Now, by using (2.5) and (2.7), we conclude (Billingsley [5]).
On the other hand, using (2.7) and (2.8), we have

 Var[¯H(1)m,n,r−¯Hm,n,r]=E[¯H(1)m,n,r−¯Hm,n,r]2−E2[¯H(1)m,n,r−¯Hm,n,r]→0, (2.9)

Also,

 Var[¯H(1)m,n,r−¯Hm,n,r]=Var[¯H(1)m,n,r]+Var[¯Hm,n,r]−2Cov(¯H(1)m,n,r,¯Hm,n,r), (2.10)

and by using (2.6)

 Var[¯Hm,n,r]−2Cov(¯H(1)m,n,r,¯Hm,n,r)→0. (2.11)

So, by applying (2.9), (2.10) and (2.11), we deduce . Therefore,

 E[¯H(1)m,n,r] → ¯H1⋯r:n, Var[¯H(1)m,n,r] → 0,  as  m,n→∞, mn→0,

so is a consistent estimator of

### 2.2 Entropy estimator for monotone increasing hazard function alternatives

#### 2.2.1 Moving average method

In statistics, smoothing a data set is to create an approximating function that attempts to capture important patterns in the data, while leaving out noise phenomena. One of the most common smoothing methods is moving average. This method is a technique that can be applied for the time series analysis, either to produce smoothed periodogram of data, or to make better estimation and forecasts [7].

A moving average (MA) method is the unweighted mean of the previous datum points. Suppose individual observations, are collected. The moving average of width at time is defined by Montgomery [16]

 Yi=Xi+Xi−1+⋯+Xi−w+1w=∑ij=i−w+1Xjwi≥w.

For periods , we do not have observations to calculate a moving average of width .

Now, we develop the construction of the moving average method. For this aim, we defined the moving average of width at time as:

 Yi=Xi+Xi+1+⋯+Xi+w−1w (2.12)
 =∑i+w−1j=iXjwi≤n−w+1.

From Equation (2.12), the moving average statistic is the average of the most recent observations. However, for , the moving average at time is defined as the average of all observations that are equal or greater than , i.e.

 Yi=∑nj=iXjn−i+1i>n−w+1. (2.13)

One characteristic of the MA is that if the data have an uneven path, applying the MA will eliminate abrupt variation and cause the smooth path. In the next subsection, this characteristic of the MA method is used and a new entropy estimator is presented.

#### 2.2.2 Entropy estimator

In this subsection, we use the MA method and obtain an entropy estimator which provides a new test statistic that achieves higher power than the previous test statistics against the alternatives with monotone increasing hazard functions. According to the subsection 2.1, we know that

 ¯H1⋯r:n=−E(∫U(r:n−1)0ln(dF−1(p)dp)dp)−E((1−U(r:n−1))ln(1−U(r:n−1))),

and the approximated of it, is defined in (2.3).

as a function of quantiles in (2.3) is the sample path of order statistics, but usually it is not smooth. So we propose to imply the MA method of proper order, say , to smooth this sample path and define the new variables from the equation (2.12) and (2.13) as follows:

 y1 = x(1:r)+⋯+x(k:r)k, y2 = x(2:r)+⋯+x(k+1:r)k, ⋮ (2.14) yr−k+1 = x(r−k+1:r)+⋯+x(r:r)k, yr−k+2 = x(r−k+2:r)+⋯+x(r:r)k−1, ⋮ yr−1 = x(r−1:r)+x(r:r)2, yr = x(r:r).

By this method, we obtain an estimator for (2.3) as:

 ¯H(2)m,n,r=1nr∑i=1lny(i+m:n)−y(i−m:n)^Fn(y(i+m:n))−^Fn(y(i−m:n))−(1−rn)ln(1−rn), (2.15)

where the window size of is a positive integer, which is less than ; and for , and for Also was introduced by Yousefzadeh and Arghami [24] as:

 ^Fn(y(i:n))=r−1r(n+1)(i+1r−1+y(i:n)−y(i−1:n)y(i+1:n)−y(i−1:n)),i=1,⋯,r,

for is less than and for is more than .

We can prove that the scale of the random variable has no effect on the accuracy of in estimating .

###### Property 2.2

Let and denote entropies of the distribution of continuous random variables and , respectively, and , where . It is easy to see that

 ^FYn(w(i:n))=r−1r(n+1)(i+1r−1+w(i:n)−w(i−1:n)w(i+1:n)−w(i−1:n))=^FWn(y(i:n))

for . So we have

 ¯H(2)Wm,n,r = 1nr∑i=1lnky(i+m:n)−ky(i−m:n)^Fn(ky(i+m:n))−^Fn(ky(i−m:n)) − (1−rn)ln(1−rn)=rnlnk+¯H(2)Ym,n,r.

Then the following properties hold

where the superscript and refer to the corresponding distribution.

###### Example 2.1

For the explanation of the proposed method, we simulate 30 samples from the exponential distribution with mean , consider their order statistics and censor 5 of them from the right, and plot the sample path of 25 points in Figure with .

The sample path of order statistics is smoothed by MA of order . New variables are defined from (2.2.2) and the smoothed path of new variables is plotted in Figure 1 with . This plot shows that the new sample path is smoother than the sample path of the original order statistics. Also, with considering MA of order 5, we define new variables from (2.2.2) and plot them in Figure 1 with . Even though the smoothing sample path of order statistics by using MA of order 3 is not as smooth as using MA of order 5, the resulting powers, which are demonstrated in section , are the same up to two digits of decimals. So without loss of generality, we just consider MA of order in (2.2.2).

###### Lemma 2.2

If and , then .

Proof: If we prove that then by Squeeze theorem, . So we establish as follows:

 0≤∣∣¯H(2)m,n,r−¯Hm,n,r∣∣ = ∣∣ ∣ ∣∣1nr∑i=1ln2mn(y(i+m:n)−y(i−m:n))(^Fn(y(i+m:n))−^Fn(y(i−m:n)))(x(i+m:n)−x(i−m:n))∣∣ ∣ ∣∣ ≤ 1nr∑i=1∣∣ ∣ ∣∣ln2mn(y(i+m:n)−y(i−m:n))(^Fn(y(i+m:n))−^Fn(y(i−m:n)))(x(i+m:n)−x(i−m:n))∣∣ ∣ ∣∣ ≤ 1nr∑i=1∣∣∣ln[n+12m−12mny(i+m:n)−y(i−m:n)x(i+m:n)−x(i−m:n)]∣∣∣ ≤ 1nr∑i=1∣∣∣ln[n+12m−12mnx(i+m+k:n)−x(i−m:n)x(i+m:n)−x(i−m:n)]∣∣∣ → 0, as m,n→0 and mn→0.

The first inequality arises by using the Triangle inequality and the second inequality is true, for more details see Yousefzadeh and Arghami [24]. Also, the third inequality is true because

 y(i+m:n)=1k(x(i+m:n)+⋯+x(i+m+k:n)) ⇒ y(i+m:n)≤x(i+m+k:n) y(i−m:n)=1k(x(i−m:n)+⋯+x(i−m+k:n)) ⇒ y(i−m:n)≥x(i−m:n),

therefore, . This completes the proof.

###### Theorem 2.2

If and , then is a consistent estimator of .

Proof: We should prove that

 E[¯H(2)m,n,r] → ¯H1⋯r:n, Var[¯H(2)m,n,r] → 0,  as  m,n→∞, mn→0

These equation obtain from the consistency of for . Proof of this theorem is quite similar to the proof of Theorem 2.1.

## 3 Test statistics

For a null distribution function the KL information for the Type-II censored data is defined to be:

 I1⋯r:n(f,f0)=∫∞−∞f1⋯r:n(z;θ)lnf1⋯r:n(z;θ)f01⋯r:n(z;θ)dz.

Then the KL information can be approximated with

 I1⋯r:n(f,f0)=−n¯H1⋯r:n−r∑i=1lnf0(z(i:n);θ)−(n−r)ln(1−F0(z(r:n);θ)). (3.16)

Thus the test statistic based on can be written as:

 T(j)m,n,r=−¯H(j)m,n,r−1n(r∑i=1lnf0(z(j)(i:n);^θ)+(n−r)ln(1−F0(z(j)(r:n);^θ))),    j=1,2,

where

 z(j)(i:n)={x(i:n)j=1y(i:n)j=2,    i=1⋯r (3.17)

and is an estimator of and and is defined in (2.4) and (2.15), respectively.

## 4 Testing exponentiality based on the Kullback-Leibler information

### 4.1 Test statistics

Suppose that we are interested in a goodness of fit test for

 ⎧⎨⎩H0:f0(x)=1θexp(−xθ),H1:f0(x)≠1θexp(−xθ),

where is unknown. Then the KL information for the Type-II censored data can be approximated in view of (3.16) with

 I1⋯r:n(f;f0)=−n¯H1⋯r:n+rlnθ+1θ(r∑i=1Z(j)(i:n)+(n−r)Z(j)r:n),

If we estimate the unknown with the maximum likelihood estimator,

 ^θ=(r∑i=1Z(j)(i:n)+(n−r)Z(j)(r:n))/r,

then we have two estimators of as:

where the random variable takes the value which is defined in (3.17). Since is non-negative and is zero if and only if , a.e., we reject the null hypothesis for large values of and .

### 4.2 Implementation of the test

Because the sampling distributions of the test statistics are intractable, we determine the percentage points using Monte Carlo samples from an exponential distribution. In determining the window size which depends on and the , we define the optimal window size to be one which gives minimum critical points in the sense of Ebrahimi et al. [9]. However, we find from the simulated percentage points, the optimal window size . In view of these results, our recommended values of for different and test statistic are listed in Table 1 and the critical values of corresponding to the optimum values of , are given in Table 2. Also, our recommended values of for different and test statistic are listed in Table 3, where and the critical values of corresponding to the optimum values of , are given in Table 4.

### 4.3 Power results

There are lots of test statistics for exponentiality concerning uncensored data including [3], [10], [13]-[15], but only some of them can be extended to the censored data. We consider here the test statistics of [6], and [19] among them. Brain and Shapiro [6] proposed two test statistics as:

 z = (12r−2)12∑r−1i=1(i−r2)Yi+1∑r−1i=1Yi+1 Z = z2+(54(r+1)(r−2)(r−3))12 ×12∑r−1i=1(i−r2)2Yi+1−r(r−2)∑r−1i=1Yi+1∑r−1i=1Yi+1

where , and , ; and show that and perform better than other test statistics for the censored data. Recently Park [19] proposed a test statistic as:

 Tm,n,r=−¯Hm,n,r+rn{ln[1r(r∑i=1X(i:n)+(n−r)X(r:n))]+1},

where is presented in (2.2). He showed that the power of the proposed test statistic is greater than the power of the test statistics which was introduced by Brain and Shapiro [6] against the alternatives with monotone increasing hazard functions.

Because the proposed test statistics are essentially related to the hazard function, the alternatives are considered according to the type of hazard functions as follows:

• Monotone decreasing hazard: Chi-square with degree of freedom 1 (A1), Gamma with shape parameter 0.5 (A2), Weibull with shape parameter 0.5 (A3) and Generalized Exponential with shape 0.5 (A4).

• Monotone increasing hazard: Uniform (B1), Weibull with shape parameter 2 (B2), Gamma with shape parameter 1.5, 2 (B3, B4 respectively), Chi-square with degree of freedom 3, 4 (B5, B6 respectively), Beta with shape parameters 1 and 2, 2 and 1 (B7, B8 respectively).

• Non-monotone hazard: Log normal with shape parameter 0.6, 1.0, 1.2 (C1, C2, C3 respectively), Beta with shape parameters 0.5 and 1.0 (C4).

We consider here the sample size to be , and draw conclusions. We made 10000 Monte Carlo simulations for to estimate the powers of our proposed test statistics and the competing test statistics, for . The simulation results are summarized in Figures . We can see from these figures that any test statistics does not beat others against all alternatives, but it is notable that the first proposed test statistic, , shows better powers than the competing test statistics against the alternatives with monotone decreasing hazard functions, see Figure . Also, against the alternatives with monotone increasing hazard functions, the second proposed test statistic, , shows better powers than the competing test statistics, see Figure .

### 4.4 RMSE comparisons

In this subsection, we report the results of a simulation study which compares the performances of the introduced entropy estimators with the estimator proposed by Park [19] in terms of their biases and RMSEs. We consider here the sample size to be , and draw conclusions. We made 10000 Monte Carlo simulations for and different to obtain the , , , their biases and RMSEs. The simulation results are summarized in Table . The results show that has the smallest bias and RMSE among them. Also, the bias and RMSE of is smaller than . We plot the empirical density of the test statistics based on these estimators for other and in Figure . This figure confirms the simulation results.

## 5 Conclusion

In this paper, the entropy estimator of the Type-II censored data which was introduced by Park [19] is modified and two new entropy estimators are obtained. Simulation results showed that the second proposed entropy estimator compared favourably with their competitors in terms of bias and RMSE, as it is expected of the structure of . Also, we provided two new test statistics for testing exponentiality with the Type-II censored data. The first one was quite powerful when compared to the existing goodness of fit tests proposed against the alternatives with monotone decreasing hazard functions. Moreover, the second one showed better powers than the available test statistics against the alternatives with monotone increasing hazard functions.

This work has the potential to be applied in the context of censored data and goodness of fit tests. This paper can elaborate further researches by extending such modifications for other censoring schemes such as progressive censoring schemes. Finally, this area of research can be expanded by considering other distributions besides the exponential distribution such as Pareto, Log normal and Weibull distributions.

## References

• [1] Alizadeh Noughabi H (2010) A new estimator of entropy and its application in testing normality, Journal of Statistical Computation and Simulation, 80, 1151-1162
• [2] Arizono I, Ohta H (1989) A test for normality based on Kullback-Leibler information, The American Statistician, 43, 20-23.
• [3] Ascher S (1990) A survey of tests for exponentiality, Communications in Statistics-Theory and Methods, 19, 1811-1825.
• [4] Balakrishnan N, Habibi Rad A, and Arghami N.R (2007) Testing exponentiality based on Kullback-Leibler information with progressively type-II censored data, IEEE Transactions on Reliability, 56, 301-307.
• [5] Billingsley P (1995) Probability and measure, Wiley, New York.
• [6] Brain C.W, Shapiro S.S (1983) A regression test for exponentiality: censored, complete samples, Technometrics, 25, 69-76.
• [7] Brockwell P.J, Davis R.A (1991) Time series: theory and methods, springer, New York.
• [8] Dudewicz E.J, van der Meulen E.C (1981) Entropy-based tests of uniformity, Journal of the American Statistical Association, 76, 967-974.
• [9] Ebrahimi N, Habibullah M, and Soofi E.S (1992) Testing exponentiality based on Kullback-Leibler information, Journal of the Royal Statistical Society: Series B, 54, 739-748.
• [10] Gan F.F, Koehler K.J (1990) Goodness-of-fit tests based on P-P probability plots, Technometrics, 32, 289-303.
• [11] Gokhale D.V (1983) On entropy-based goodness-of-fit tests, Computational Statistics and Data Analysis, 1, 157-165.
• [12] Habibi Rad A, Yousefzadeh F, and Balakrishnan N (2011) Goodness of fit test based on Kullback-Leibler information for progressively type-II censored data, IEEE Transactions on Reliability, 60, 570-579.
• [13] Henze N (1993) A new flexible class of omnibus tests for exponentiality, Communications in Statistics-Theory and Methods, 22, 115-133.
• [14] Kallenberg W.C.M, Ledwina T (1997) Data driven smooth tests for composite hypothesis: comparisons of powers, Journal of Statistical Computation and Simulation, 59, 101-121.
• [15] LaRiccia V (1991) Smooth goodness of fit tests: a quantile function approach, Journal of the American Statistical Association, 86, 427-431.
• [16] Montgomery D.C (2001) Introduction to Statistical Quality Control (4th edn), Wiley, New York.
• [17] Pakyari R, Balakrishnan N (2011) Goodness-of-fit tests for progressively Type-II censored data from location-scale distributions, Journal of Statistical Computation and Simulation, iFirst, 1-12.
• [18] Pakyari R, Balakrishnan N (2012) A general purpose approximate goodness-of-fit test for progressively type-II censored data, IEEE Transactions on Reliability, 61, 238-244.
• [19] Park S (2005) Testing exponentiality based on the Kullback-Leibler information with the type II censored data, IEEE Transactions on Reliability, 54, 22-26.
• [20] Park S, Park D (2003) Correcting moments for goodness of fit tests based on two entropy estimates, Journal of Statistical Computation and Simulation, 73, 685-694.
• [21] Samanta M, Schwarz C.J (1988) The Shapiro-Wilk test for exponentiality based on censored data, Journal of the American Statistical Association, 83, 528-531.
• [22] Shannon C.E (1948) A mathematical theory of communications, The Bell System Technical Journal, 27, 379-423.
• [23] Vasicek O (1976) A test for normality based on sample entropy, Journal of the Royal Statistical Society: Series B, 38, 54-59.
• [24] Yousefzadeh F, Arghami N.R (2008) Testing exponentiality based on Type-II censored data and a new cdf estimator, Communications in Statistics-Simulation and Computation, 37, 1479-1499.
• [25] Zamanzadeh E, Arghami N.R (2011) Goodness-of-fit test based on correcting moments of modified entropy estimator, Journal of Statistical Computation and Simulation, 81, 2077-2093.
• [26] Zamanzadeh E, Arghami N.R (2012) Testing normality based on new entropy estimators, Journal of Statistical Computation and Simulation, 82, 1701-1713.
• [27]