Two New Entropy Estimators for Testing Exponentiality with TypeII Censored Data
Abstract
This paper proposes two estimators of the joint entropy of the TypeII censored data. Consistency of both estimators is proved. Simulation results show that the second one shows less bias and root of mean square error (RMSE) than leading estimator. Also, two goodness of fit test statistics based on the KullbackLeibler information with the TypeII censored data are established and their performances with the leading test statistics are compared. We provide a Monte Carlo simulation study which shows that the test statistics and show better powers than leading test statistics against the alternatives with monotone decreasing and monotone increasing hazard functions, respectively.
Keywords: Entropy, Monte Carlo simulation, KullbackLeibler information, Moving average method, Hazard function.
Mathematics Subject Classification: 62G10, 62G30.
1 Introduction
Suppose that a random variable has a distribution function , with a continuous density function . The differential entropy of the random variable is defined by Shannon [22] to be
(1.1) 
The entropy difference has been considered in [8] and [11] for establishing the goodness of fit tests for the class of the maximum entropy distributions.
The KullbackLeibler (KL) information in favor of against is defined to be
Because has the property that , and the equality holds if and only if , different estimators of the KL information has been also considered as a goodness of fit test statistic in some papers including [2], [9], [20] and [25]. For complete samples, some of these test statistics perform very well for exponentiality [9], and some others of them perform very well for normality, see [23], [26] and [1].
For the censored data, some authors studied the problem of goodness of fit test and discussed some test statistics. Brain and Shapiro [6] proposed two test statistics and show that these test statistics perform better than other test statistics for the censored data. Samanta and Schwarz [21] proposed a test statistic and showed that the proposed test statistic has competing performance with the test statistics which introduced by Brain and Shapiro [6] for the censored data. Recently Park [19] obtained an estimator for entropy of TypeII censored data and proposed a test statistic based on KL information. He showed that the power of the proposed test statistic is greater than the power of the test statistics which proposed by Brain and Shapiro [6], and Samanta and Schwarz [21] against the alternatives with monotone increasing hazard functions. In the case of progressively censored data, Balakrishnan et al. [4] studied the testing exponentiality based on KL information with progressively TypeII censored data. Habibi Rad et al. [12] studied goodness of fit test based on KL information for progressively TypeII censored data. Pakyari and Balakrishnan [17] proposed several goodness of fit methods for locationscale families of distributions under progressively TypeII censored data. They [18] also investigated a general purpose approximate goodness of fit test for progressively TypeII censored data.
In this paper, we enhance the estimator which was introduced by Park [19] and obtain two new entropy estimators of TypeII censored data. Simulation results show that the second one shows less bias and RMSE than leading estimator. Also, we provide two new test statistics. The first one achieves higher power than the previous test statistics against the alternatives with monotone decreasing hazard functions and the other one achieves higher power than the previous test statistics against the alternatives with monotone increasing hazard functions.
The rest of the article is arranged as follows: In Section 2, we introduce two estimators of the joint entropy of the TypeII censored data. Also, we show that both are consistent. Scale invariance property of variances and mean squared errors of the proposed estimators is studied in the same section. In Section 3, we use the KL information with the TypeII censored data and obtain two new test statistics. In Section 4, we introduce goodnessoffit tests for exponentiality based on the proposed test statistics and then compare their powers with the powers of other test statistics. Also, by using the new test statistics, we compare biases and RMSEs of the new entropy estimators with the leading entropy estimator.
2 New entropy estimators
In this section, we introduce two entropy estimators and prove some of their properties.
2.1 Entropy estimator for monotone decreasing hazard function alternatives
In this subsection, we obtain one entropy estimator which provides a new test statistic that achieves higher power than the previous test statistics against the alternatives with monotone decreasing hazard functions.
Vasicek [23] expressed (1.1) in the form,
and provided its estimator as:
where the window size is a positive integer, which is less than ; and for , and for . Recently Park [19] expressed the joint entropy of , , in the form where
and provided its estimator as:
(2.2) 
By approximating with
(2.3) 
we obtain an estimator for (2.3) as:
(2.4) 
where is the harmonic mean of and is the harmonic mean of and the window size is a positive integer, which is less than ; and for , and for We expect that the performance of this estimator is better than (2.2), because we use more information for its calculation.
We can easily prove that the scale of the random variable has no effect on the accuracy of in estimating .
Property 2.1
Let and denote entropies of the distribution of continuous random variables and , respectively, and , where . It is easy to see that for . So we have Then the following properties hold
where the superscript and refer to the corresponding distribution.
Lemma 2.1
If and , then , which is defined in (2.2).
Proof: If we prove that then by Squeeze theorem, . So we establish as follows:
The first inequality arises by using the Triangle inequality, and the second inequality is true because
therefore, . This completes the proof.
Theorem 2.1
If and , then is a consistent estimator of .
Proof: Park [19] showed that is a consistent estimator of . So
(2.5)  
(2.6) 
as , and . According to the previous Lemma, , so (Billingsley [5])
(2.7)  
(2.8) 
Now, by using (2.5) and (2.7), we conclude (Billingsley [5]).
On the other hand, using (2.7) and (2.8), we have
(2.9) 
Also,
(2.10) 
and by using (2.6)
(2.11) 
So, by applying (2.9), (2.10) and (2.11), we deduce . Therefore,
so is a consistent estimator of
2.2 Entropy estimator for monotone increasing hazard function alternatives
2.2.1 Moving average method
In statistics, smoothing a data set is to create an approximating function that attempts to capture important patterns in the data, while leaving out noise phenomena. One of the most common smoothing methods is moving average. This method is a technique that can be applied for the time series analysis, either to produce smoothed periodogram of data, or to make better estimation and forecasts [7].
A moving average (MA) method is the unweighted mean of the previous datum points. Suppose individual observations, are collected. The moving average of width at time is defined by Montgomery [16]
For periods , we do not have observations to calculate a moving average of width .
Now, we develop the construction of the moving average method. For this aim, we defined the moving average of width at time as:
(2.12) 
From Equation (2.12), the moving average statistic is the average of the most recent observations. However, for , the moving average at time is defined as the average of all observations that are equal or greater than , i.e.
(2.13) 
One characteristic of the MA is that if the data have an uneven path, applying the MA will eliminate abrupt variation and cause the smooth path. In the next subsection, this characteristic of the MA method is used and a new entropy estimator is presented.
2.2.2 Entropy estimator
In this subsection, we use the MA method and obtain an entropy estimator which provides a new test statistic that achieves higher power than the previous test statistics against the alternatives with monotone increasing hazard functions. According to the subsection 2.1, we know that
and the approximated of it, is defined in (2.3).
as a function of quantiles in (2.3) is the sample path of order statistics, but usually it is not smooth. So we propose to imply the MA method of proper order, say , to smooth this sample path and define the new variables from the equation (2.12) and (2.13) as follows:
(2.14)  
By this method, we obtain an estimator for (2.3) as:
(2.15) 
where the window size of is a positive integer, which is less than ; and for , and for Also was introduced by Yousefzadeh and Arghami [24] as:
for is less than and for is more than .
We can prove that the scale of the random variable has no effect on the accuracy of in estimating .
Property 2.2
Let and denote entropies of the distribution of continuous random variables and , respectively, and , where . It is easy to see that
for . So we have
Then the following properties hold
where the superscript and refer to the corresponding distribution.
Example 2.1
For the explanation of the proposed method, we simulate 30 samples from the exponential distribution with mean , consider their order statistics and censor 5 of them from the right, and plot the sample path of 25 points in Figure with .
The sample path of order statistics is smoothed by MA of order . New variables are defined from (2.2.2) and the smoothed path of new variables is plotted in Figure 1 with . This plot shows that the new sample path is smoother than the sample path of the original order statistics. Also, with considering MA of order 5, we define new variables from (2.2.2) and plot them in Figure 1 with . Even though the smoothing sample path of order statistics by using MA of order 3 is not as smooth as using MA of order 5, the resulting powers, which are demonstrated in section , are the same up to two digits of decimals. So without loss of generality, we just consider MA of order in (2.2.2).
Lemma 2.2
If and , then .
Proof: If we prove that then by Squeeze theorem, . So we establish as follows:
The first inequality arises by using the Triangle inequality and the second inequality is true, for more details see Yousefzadeh and Arghami [24]. Also, the third inequality is true because
therefore, . This completes the proof.
Theorem 2.2
If and , then is a consistent estimator of .
Proof: We should prove that
These equation obtain from the consistency of for . Proof of this theorem is quite similar to the proof of Theorem 2.1.
3 Test statistics
4 Testing exponentiality based on the KullbackLeibler information
4.1 Test statistics
Suppose that we are interested in a goodness of fit test for
where is unknown. Then the KL information for the TypeII censored data can be approximated in view of (3.16) with
If we estimate the unknown with the maximum likelihood estimator,
then we have two estimators of as:
where the random variable takes the value which is defined in (3.17). Since is nonnegative and is zero if and only if , a.e., we reject the null hypothesis for large values of and .
4.2 Implementation of the test
Because the sampling distributions of the test statistics are intractable, we determine the percentage points using Monte Carlo samples from an exponential distribution. In determining the window size which depends on and the , we define the optimal window size to be one which gives minimum critical points in the sense of Ebrahimi et al. [9]. However, we find from the simulated percentage points, the optimal window size . In view of these results, our recommended values of for different and test statistic are listed in Table 1 and the critical values of corresponding to the optimum values of , are given in Table 2. Also, our recommended values of for different and test statistic are listed in Table 3, where and the critical values of corresponding to the optimum values of , are given in Table 4.
r  519  2040  4150 

m  3  4  5 
n  r  

10  5  0.5962  0.6855  0.7692 
6  0.6155  0.7185  0.8039  
7  0.6398  0.7333  0.8185  
8  0.6676  0.7607  0.8648  
9  0.7152  0.8075  0.9025  
20  10  0.3148  0.3640  0.4061 
11  0.3188  0.3689  0.4148  
12  0.3285  0.3727  0.4183  
13  0.3374  0.3825  0.4329  
14  0.3442  0.3911  0.4371  
15  0.3613  0.4113  0.4587  
16  0.3677  0.4157  0.4634  
17  0.3830  0.4333  0.4795  
18  0.4022  0.4521  0.5024  
19  0.4223  0.4717  0.5172  
30  15  0.2239  0.2599  0.2904 
16  0.2293  0.2625  0.2922  
17  0.2320  0.2659  0.2945  
18  0.2376  0.2707  0.3009  
19  0.2425  0.2757  0.3077  
20  0.2470  0.2800  0.3108  
21  0.2537  0.2834  0.3121  
22  0.2568  0.2918  0.3259  
23  0.2634  0.2949  0.3272  
24  0.2691  0.3017  0.3318  
25  0.2736  0.3090  0.3398  
26  0.2822  0.3136  0.3488  
27  0.2910  0.3239  0.3538 
r  45  67  89  1011  1213  r(r+1) 

m  5  6  7  8  9  (for even ) 
n  r  

10  5  0.3445  0.4253  0.5087 
6  0.3251  0.4128  0.5026  
7  0.3136  0.4099  0.4929  
8  0.3104  0.4046  0.4915  
9  0.3101  0.4038  0.4902  
20  10  0.1474  0.1913  0.2310 
11  0.1426  0.1830  0.2229  
12  0.1408  0.1828  0.2197  
13  0.1369  0.1805  0.2181  
14  0.1322  0.1773  0.2168  
15  0.1294  0.1740  0.2160  
16  0.1281  0.1742  0.2161  
17  0.1280  0.1739  0.2073  
18  0.1268  0.1649  0.2072  
19  0.1200  0.1620  0.1988  
30  15  0.0865  0.1168  0.1500 
16  0.0859  0.1152  0.1430  
17  0.0843  0.1124  0.1383  
18  0.0829  0.1090  0.1380  
19  0.0824  0.1083  0.1355  
20  0.0815  0.1079  0.1311  
21  0.0806  0.1043  0.1294  
22  0.0777  0.1038  0.1281  
23  0.0759  0.1027  0.1280  
24  0.0741  0.0988  0.1275  
25  0.0699  0.0976  0.1265  
26  0.0662  0.0977  0.1219  
27  0.0635  0.0910  0.1200 
4.3 Power results
There are lots of test statistics for exponentiality concerning uncensored data including [3], [10], [13][15], but only some of them can be extended to the censored data. We consider here the test statistics of [6], and [19] among them. Brain and Shapiro [6] proposed two test statistics as:
where , and , ; and show that and perform better than other test statistics for the censored data. Recently Park [19] proposed a test statistic as:
where is presented in (2.2). He showed that the power of the proposed test statistic is greater than the power of the test statistics which was introduced by Brain and Shapiro [6] against the alternatives with monotone increasing hazard functions.
Because the proposed test statistics are essentially related to the hazard function, the alternatives are considered according to the type of hazard functions as follows:

Monotone decreasing hazard: Chisquare with degree of freedom 1 (A1), Gamma with shape parameter 0.5 (A2), Weibull with shape parameter 0.5 (A3) and Generalized Exponential with shape 0.5 (A4).

Monotone increasing hazard: Uniform (B1), Weibull with shape parameter 2 (B2), Gamma with shape parameter 1.5, 2 (B3, B4 respectively), Chisquare with degree of freedom 3, 4 (B5, B6 respectively), Beta with shape parameters 1 and 2, 2 and 1 (B7, B8 respectively).

Nonmonotone hazard: Log normal with shape parameter 0.6, 1.0, 1.2 (C1, C2, C3 respectively), Beta with shape parameters 0.5 and 1.0 (C4).
We consider here the sample size to be , and draw conclusions. We made 10000 Monte Carlo simulations for to estimate the powers of our proposed test statistics and the competing test statistics, for . The simulation results are summarized in Figures . We can see from these figures that any test statistics does not beat others against all alternatives, but it is notable that the first proposed test statistic, , shows better powers than the competing test statistics against the alternatives with monotone decreasing hazard functions, see Figure . Also, against the alternatives with monotone increasing hazard functions, the second proposed test statistic, , shows better powers than the competing test statistics, see Figure .
4.4 RMSE comparisons
In this subsection, we report the results of a simulation study which compares the performances of the introduced entropy estimators with the estimator proposed by Park [19] in terms of their biases and RMSEs. We consider here the sample size to be , and draw conclusions. We made 10000 Monte Carlo simulations for and different to obtain the , , , their biases and RMSEs. The simulation results are summarized in Table . The results show that has the smallest bias and RMSE among them. Also, the bias and RMSE of is smaller than . We plot the empirical density of the test statistics based on these estimators for other and in Figure . This figure confirms the simulation results.
Bias  RMSE  

n  r  
30  15  0.1626  0.0100  0.1370  0.2159  0.1426  0.1953 
16  0.1691  0.0102  0.1508  0.2245  0.1478  0.2078  
17  0.1717  0.0035  0.1521  0.2284  0.1511  0.2108  
18  0.1760  0.0019  0.1540  0.2348  0.1557  0.2156  
19  0.1788  0.0095  0.1543  0.2386  0.1594  0.2176  
20  0.1902  0.0150  0.1579  0.2494  0.1638  0.2233  
21  0.1957  0.0189  0.1616  0.2565  0.1702  0.2294  
22  0.1954  0.0316  0.1588  0.2575  0.1752  0.2291  
23  0.2019  0.0384  0.1630  0.2660  0.1821  0.2359  
24  0.2042  0.0529  0.1626  0.2683  0.1882  0.2363  
25  0.2145  0.0613  0.1687  0.2792  0.1960  0.2442  
26  0.2169  0.0843  0.1665  0.2823  0.2106  0.2440  
27  0.2300  0.0980  0.1745  0.2941  0.2204  0.2514 
5 Conclusion
In this paper, the entropy estimator of the TypeII censored data which was introduced by Park [19] is modified and two new entropy estimators are obtained. Simulation results showed that the second proposed entropy estimator compared favourably with their competitors in terms of bias and RMSE, as it is expected of the structure of . Also, we provided two new test statistics for testing exponentiality with the TypeII censored data. The first one was quite powerful when compared to the existing goodness of fit tests proposed against the alternatives with monotone decreasing hazard functions. Moreover, the second one showed better powers than the available test statistics against the alternatives with monotone increasing hazard functions.
This work has the potential to be applied in the context of censored data and goodness of fit tests. This paper can elaborate further researches by extending such modifications for other censoring schemes such as progressive censoring schemes. Finally, this area of research can be expanded by considering other distributions besides the exponential distribution such as Pareto, Log normal and Weibull distributions.
References
 [1] Alizadeh Noughabi H (2010) A new estimator of entropy and its application in testing normality, Journal of Statistical Computation and Simulation, 80, 11511162
 [2] Arizono I, Ohta H (1989) A test for normality based on KullbackLeibler information, The American Statistician, 43, 2023.
 [3] Ascher S (1990) A survey of tests for exponentiality, Communications in StatisticsTheory and Methods, 19, 18111825.
 [4] Balakrishnan N, Habibi Rad A, and Arghami N.R (2007) Testing exponentiality based on KullbackLeibler information with progressively typeII censored data, IEEE Transactions on Reliability, 56, 301307.
 [5] Billingsley P (1995) Probability and measure, Wiley, New York.
 [6] Brain C.W, Shapiro S.S (1983) A regression test for exponentiality: censored, complete samples, Technometrics, 25, 6976.
 [7] Brockwell P.J, Davis R.A (1991) Time series: theory and methods, springer, New York.
 [8] Dudewicz E.J, van der Meulen E.C (1981) Entropybased tests of uniformity, Journal of the American Statistical Association, 76, 967974.
 [9] Ebrahimi N, Habibullah M, and Soofi E.S (1992) Testing exponentiality based on KullbackLeibler information, Journal of the Royal Statistical Society: Series B, 54, 739748.
 [10] Gan F.F, Koehler K.J (1990) Goodnessoffit tests based on PP probability plots, Technometrics, 32, 289303.
 [11] Gokhale D.V (1983) On entropybased goodnessoffit tests, Computational Statistics and Data Analysis, 1, 157165.
 [12] Habibi Rad A, Yousefzadeh F, and Balakrishnan N (2011) Goodness of fit test based on KullbackLeibler information for progressively typeII censored data, IEEE Transactions on Reliability, 60, 570579.
 [13] Henze N (1993) A new flexible class of omnibus tests for exponentiality, Communications in StatisticsTheory and Methods, 22, 115133.
 [14] Kallenberg W.C.M, Ledwina T (1997) Data driven smooth tests for composite hypothesis: comparisons of powers, Journal of Statistical Computation and Simulation, 59, 101121.
 [15] LaRiccia V (1991) Smooth goodness of fit tests: a quantile function approach, Journal of the American Statistical Association, 86, 427431.
 [16] Montgomery D.C (2001) Introduction to Statistical Quality Control (4th edn), Wiley, New York.
 [17] Pakyari R, Balakrishnan N (2011) Goodnessoffit tests for progressively TypeII censored data from locationscale distributions, Journal of Statistical Computation and Simulation, iFirst, 112.
 [18] Pakyari R, Balakrishnan N (2012) A general purpose approximate goodnessoffit test for progressively typeII censored data, IEEE Transactions on Reliability, 61, 238244.
 [19] Park S (2005) Testing exponentiality based on the KullbackLeibler information with the type II censored data, IEEE Transactions on Reliability, 54, 2226.
 [20] Park S, Park D (2003) Correcting moments for goodness of fit tests based on two entropy estimates, Journal of Statistical Computation and Simulation, 73, 685694.
 [21] Samanta M, Schwarz C.J (1988) The ShapiroWilk test for exponentiality based on censored data, Journal of the American Statistical Association, 83, 528531.
 [22] Shannon C.E (1948) A mathematical theory of communications, The Bell System Technical Journal, 27, 379423.
 [23] Vasicek O (1976) A test for normality based on sample entropy, Journal of the Royal Statistical Society: Series B, 38, 5459.
 [24] Yousefzadeh F, Arghami N.R (2008) Testing exponentiality based on TypeII censored data and a new cdf estimator, Communications in StatisticsSimulation and Computation, 37, 14791499.
 [25] Zamanzadeh E, Arghami N.R (2011) Goodnessoffit test based on correcting moments of modified entropy estimator, Journal of Statistical Computation and Simulation, 81, 20772093.
 [26] Zamanzadeh E, Arghami N.R (2012) Testing normality based on new entropy estimators, Journal of Statistical Computation and Simulation, 82, 17011713.
 [27]