Bounded Influence Propagation -Estimation:
A New Robust Method for ARMA Model Estimation
A new robust and statistically efficient estimator for ARMA models called the bounded influence propagation (BIP) -estimator is proposed. The estimator incorporates an auxiliary model, which prevents the propagation of outliers. Strong consistency and asymptotic normality of the estimator for ARMA models that are driven by independently and identically distributed (iid) innovations with symmetric distributions are established. To analyze the infinitesimal effect of outliers on the estimator, the influence function is derived and computed explicitly for an AR(1) model with additive outliers. To obtain estimates for the AR() model, a robust Durbin-Levinson type and a forward-backward algorithm are proposed. An iterative algorithm to robustly obtain ARMA(,) parameter estimates is also presented. The problem of finding a robust initialization is addressed, which for orders is a non-trivial matter. Numerical experiments are conducted to compare the finite sample performance of the proposed estimator to existing robust methodologies for different types of outliers both in terms of average and of worst-case performance, as measured by the maximum bias curve. To illustrate the practical applicability of the proposed estimator, a real-data example of outlier cleaning for R-R interval plots derived from electrocardiographic (ECG) data is considered. The proposed estimator is not limited to biomedical applications, but is also useful in any real-world problem whose observations can be modeled as an ARMA process disturbed by outliers or impulsive noise.
Autoregressive moving-average (ARMA) models are amongst the most popular models for characterizing dependent data and they have a long tradition in numerous real-world applications, e.g. in speech processing , biomedicine [2, 3], radar , electricity consumption forecasting [5, 6, 7], system identification  and econometry . Numerous extensions of the ARMA model, such as Seasonal Integrated ARMA (SARIMA) , Periodic ARMA (PARMA) , Controlled ARMA , and Time-Varying ARMA (TV-ARMA) models  have been proposed.
This paper focusses on robust parameter estimation for ARMA models associated with random processes for which the majority of samples are appropriately modeled by a stationary and invertible ARMA model and a minority consists of outliers with respect to the ARMA model. For such cases and, in general, classical estimators are unreliable and may break down completely [13, 14, 15, 16, 17, 18, 19, 20, 21, 5, 6, 7, 22, 23]. The nature of the outliers depends on the application. For example, motion artifacts are often evident in biomedical signals such as intracranial pressure (ICP), electrocardiographic (ECG) and photoplethysmographic (PPG) signals [24, 25, 26, 27, 28] while in electricity consumption forecasting outliers are associated with holidays, major sporting events and strikes [5, 21]. For a discussion on how outliers affect ARMA parameter estimation, the reader is referred, e.g. to [14, 16, 19, 29, 30] and there is a clear need for robust methods that can, to some extent, resist outliers. First contributions to robust estimation for dependent data were made in the 1980’s [31, 32, 33, 34], and in recent years, research in this area has increased significantly (e.g. [35, 18, 36, 37, 38, 19, 39, 40, 27, 22, 23, 41, 20, 5, 42, 43, 44, 45, 6, 7, 26, 24, 46, 47]).
Research on robust ARMA parameter estimation may be loosely grouped into two categories which are associated with the diagnostic approach (e.g. [13, 14, 15, 16, 17, 38, 39, 23, 47]) and the statistically robust approach (e.g. [18, 19, 20, 21, 5, 6, 7, 27, 22, 48, 49]). Diagnostic approaches enhance robustness via detection and hard rejection of outliers, followed by a classical parameter estimation method that handles missing values. Statistically robust methods utilize the entire data set and accommodate the outliers by bounding their influence on the parameter estimates. Robust statistical theory also provides measures, such as the influence function (IF), the breakdown point and the maximum bias curve [19, 50, 21], which characterize quantitative and qualitative robustness and allow for an analytical comparison of different estimators.
The main contributions of this paper is to propose and analyze a new estimator for ARMA model parameters called the bounded influence propagation (BIP) -estimator which is simultaneously robust and possesses a controllable statistical efficiency. Robustness and high efficiency are jointly achieved by incorporating an auxiliary model which prevents the propagation of outliers into the -estimator. The term ’propagation of outliers’ means that one outlier in the observations creates multiple outliers in the reconstructed innovation series. The BIP -estimate minimizes a robust and efficient scale of the reconstructed innovation series. In Theorem 1, strong consistency of the -estimator of the ARMA parameters is established. In Lemma 1, Fisher consistency of the -estimator of the ARMA parameters is shown, given all past observations. In Lemma 2, almost sure convergence of the -estimator of the innovations scale to the population value based on the expectation operator is proven. In Theorem 2, under an ARMA model, it is established that the BIP -estimator is asymptotically equivalent to a -estimator. Theorems 1 and 2 together prove the strong consistency of the proposed estimator under general conditions, which include the Gaussian ARMA model as a special case. In Theorem 3, asymptotic normality of the estimator for the ARMA model is proven by deriving the asymptotic equivalence to an M-estimator. To analyze the infinitesimal robustness of the BIP -estimator in the asymptotic case, its IF is derived. The IF is explicitly computed for an autoregressive process of order one, AR(1), in the case of additive outliers. To compute the estimates for the AR() model, a computationally efficient robust Durbin-Levinson type algorithm is proposed that incorporates the BIP model. Here the parameters are recursively found for increasing orders. In this way, searching for a robust starting point to minimize a non-convex cost function is avoided, which is a key-difficulty in robust estimation. A forward-backward algorithm to recursively compute the AR() parameters is also proposed. In the search for ARMA parameter estimates, a Marquard algorithm is used to find the parameters that minimize the -scale of the innovations. For this case, an algorithm to find a robust starting point is presented. The starting point algorithm uses a BIP-AR model based outlier cleaning operation. Numerical experiments to evaluate the estimator in terms of the maximum bias curve in order to assess its quantitative robustness and also to compare it to existing benchmark estimators are conducted. In particular, Monte Carlo experiments for ARMA models of orders are performed. This is unusual in robust ARMA parameter estimation, which usually is limited to ARMA models of lower orders. Patchy and independent replacement and additive outliers of different types are considered in the simulations. Finally, the proposed estimator is applied to a real-data example of artifact cleaning for R-R interval plots derived from electrocardiographic (ECG) data. R-R intervals denote the time intervals between consecutive heart beats and are used in heart rate and heart rate variability analysis.
Relation to existing work: In the analysis of our estimator, we build upon theoretical results that were established for the BIP MM-estimator . As for the classical regression setting, the  and MM  are alternative estimators with similar statistical and robustness properties. In the context of AR parameter estimation, a key advantage of the -estimator is its definition via the -scale. Based on this definition, a robust Durbin-Levinson type procedure is proposed. Further, the starting point for the BIP MM, especially for is difficult to find and expressions for the IF are not available for the BIP MM-estimator. Our estimator is also conceptually related to the filtered -estimator , which uses a robust filter to prevent outlier propagation. A disadvantage of the filtered estimators is that they are intractable in terms of robustness and asymptotic statistical analysis.
The paper is organized as follows. Section II introduces the signal and outlier models and discusses the propagation of outliers. Section III introduces the BIP -estimator and details associated statistical and robustness analysis. Section IV presents an algorithm for computing the stationary and invertible BIP -estimates. Section V compares the performance of the proposed BIP -estimator with existing ARMA parameter estimators via Monte Carlo simulations. Section VI provides a real-data example of artifact cleaning for R-R interval plots derived from ECG data. Conclusions, and possible extensions of this research are presented in Section VII.
Notation. Vectors (matrices) are denoted by bold-faced lowercase (uppercase letters), e.g. (). The th column vector of a matrix is denoted by . is the transpose operator. Sets are denoted by calligraphic letters, e.g. . refers to the estimator (or estimate) of the parameter vector , , and are, respectively, the probability density function (pdf) and cumulative distribution function (cdf) of , and are, respectively, the joint pdf and joint cdf of the random variables and , is the pdf of conditioned on and given . is the probability that . is the expectation operator, while denotes convergence to the normal distribution with mean vector and covariance matrix . Given a function , is the -dimensional column vector whose th element is . Finally, denotes the grid of equidistant points in , ranging from to with a step size of .
Ii Signal and Outlier Models
The ARMA and Bounded Innovation Propagation (BIP)-ARMA signal models, as well as some important outlier models, are briefly revisited. Attention is drawn to the fact that estimators, which are computed based on the innovations, require a mechanism that prevents the propagation of outliers.
Ii-a Signal model
denote a sequence of observations that was generated by a stationary and invertible ARMA() process up to time according to
where the true parameter vector , and .
(A1) Assume that are independent and identically distributed (iid) random variables with a symmetric distribution and further assume that .
To restrict the parameter space in a manner which is consistent with a stationary and invertible ARMA model, let be a parameter vector defined by the polynomials
which have all their roots outside the unit circle. Then, by defining
the following recursion follows
(A2) Assume that and do not have common roots.
Ii-B Outlier models
In real-world applications, the observations may not exactly follow (2). There exist several statistical models for outliers in dependent data (see e.g. [13, 14, 15, 16, 17, 23, 19, 21]). The following provides a brief review of important models.
The additive outlier (AO) model defines contaminated observations according to
where follows an ARMA model, as given in (2), defines the contaminating process that is independent of and is a stationary random process for which
For the replacement outlier (RO) model
Outliers may also differ in their temporal structure. For isolated outliers, takes the value 1, such that at least one non-outlying observation is between two outliers (e.g. follows an independent Bernoulli distribution). For patchy outliers, on the other hand, takes the value 1 for subsequent samples.
Ii-C Bounded innovation propagation (BIP)-ARMA model
ARMA parameter estimation, i.e., determining , is often based on minimizing some function of the reconstructed innovation sequence. However, as can be seen from (5), one AO or RO in can propagate onto multiple innovations . In the extreme case, all entries of the innovations sequence are disturbed by a single outlier. Thus, robust estimators are only applicable if they are combined with a mechanism to prevent outlier propagation. An auxiliary model to do this, is the BIP-ARMA model :
Here, , where if , , while if , . ARMA models are included by setting . Thus, by choosing to be one of the well-known monotone or redescending nonlinearities (e.g., Huber’s or Tukey’s) , all innovations that lie within some region around are left untouched and, on the other hand, the effect of a single AO or RO is bounded to a single corrupted innovation. In (II-C), is a robust M-scale of [50, 21], i.e., it solves
where is defined as
To make the M-estimator consistent in scale with the standard deviation when the data is Gaussian, in (12), is the expectation operator with respect to the standard normal distribution.
(A3) Assume that is a real-valued function with the following properties: , and is continuous, non-constant and non-decreasing in . is bounded and continuous.
(A4) Assume that is an odd, bounded and continuous function.
From (II-C), the innovations sequence can be recursively obtained for according to
Fig. 1 illustrates the influence of for an ARMA(2,1) model with parameters , . The red crosses mark the AO positions in the observations. When reconstructing the innovations with an ARMA model (6) that uses , multiple innovation samples are contaminated. This effect is suppressed when applying the BIP-ARMA (13).
Iii Proposed Estimator
We next define an estimator that is based on the idea of minimizing a robust and efficient scale of the reconstructed innovations, the -scale. The estimator is defined for the case when the sample size exceeds the number of model parameters, i.e., . It computes the -scale both for innovations reconstructed from the ARMA in (6) and from the BIP-ARMA in (13), and chooses as a final estimate , which provides the smaller -scale. We show, for iid innovations with a symmetric pdf, that the proposed estimator is strongly consistent with the ARMA model (Theorem 1 and Theorem 2). Further, the estimator is asymptotically normal for the ARMA model case with a controllable efficiency with respect to the maximum-likelihood estimator (Theorem 3). Finally, an expression for the IF which measures robustness against infinitesimal contamination is provided.
Iii-a Definition of the -estimator under the ARMA model
Let be an M-estimate of the scale of based on which satisfies A3, i.e.,
(A5) Assume that .
The -estimate of under the ARMA model is defined according to
where is the -estimate  of the scale of and is defined as
Here where for some small .
(A6) Assume that satisfies A3, and additionally, , where .
Iii-B Definition of the -estimator under the BIP ARMA model
The -estimate of under the BIP-ARMA model is defined according to
and is recursively obtained from (13). To compute , the MA-infinity representation of the BIP-ARMA model is used
where are the coefficients of . It then follows that
where is the standard deviation of and
The estimate of in Eq. (20) can then be computed according to
with , and where is chosen sufficiently large to approximate the MA-infinity representation.
Iii-C Definition of the proposed -estimator
The final -estimate of the innovations scale is
and the final parameter estimate becomes
It is shown in Sec. III-D that when the data follows an ARMA model without outliers, the result that is asymptotically obtained for . This implies that the asymptotic efficiency of is independent of . However, this does not hold in the finite sample size case.
Iii-D Statistical analysis
Theorem 1. establishes strong consistency of the -estimator of the ARMA parameters.
Assume that follows from Eq. (2) with satisfying A1. Further, assume that satisfies A3 and A5 and that satisfies A6. Then, the -estimator defined in Eq. (15) is strongly consistent for .
Proving this theorem requires Lemmas 1-3.
Lemma 1. provides the Fisher consistency of the -estimator of the ARMA parameters given all past observations111For visual clarity, let , and .
Let be an observation from an ARMA(p,q), as in Eq. (2). Assume that is bounded and satisfies A3 and A5. It then holds, with denoting the true innovations scale, that if and . This implies that the estimate , as defined in Eq. (15), is Fisher consistent for .
Consider the assumptions made in Theorem 1 which are the same assumptions made in Lemma 2 in . This lemma states: if and it holds, for an M-estimate of scale defined by
that . Since for
it follows by defining
Using Lemma 3.1 (i) from  it then follows that
for all . Then, using Lemma 3.1 (ii) from , and assuming that is continuously differentiable, it is sufficient to show, for , that
is nondecreasing with respect to , since A6 implies that
Lemma 2. states the almost sure convergence of the -estimator of the innovations scale to the population value based on the expectation operator.
Under the assumptions of Theorem 1, for any , it follows that
Then and . According to Lemma 5 of , it holds, for any , that
From Lemma 2 of , it holds, under the assumptions A3, A6 on , , that
Under the assumptions of Theorem 1, there exists , such that
Proof of Theorem 1.
Take arbitrarily small and let be as in Lemma 3. The continuity of the M-scale functional defined in (25) follows from Lebesgue’s dominated convergence theorem. The continuity of follows from (30) as long as satisfies A3. By Lemma 1 of this paper, there exists such that
By Lemma 2 of this paper, there exists , such that for
By Lemma 3, there exists , such that for
Therefore, for it holds that , which proves the theorem. ∎
Theorem 2. establishes, under an ARMA model, that the BIP - is asymptotically equivalent to a -estimator.
Assume that follows (2) with satisfying A1. Further, assume that and are bounded, that satisfies A3 and A5, that satisfies A6, that for any compact set , and, finally, that satisfies A4. Then, if is not white noise, with probability 1, there exists , such that for all and then a.s..
Theorem 3. establishes the asymptotic normality of the estimator for the ARMA model.
Let be as in (2), let A1, A2, A3 be fulfilled and let . Further, assume that and are continuous and bounded functions. Then, the -estimator is asymptotically normally distributed with
and being the matrix of dimensions with elements
Here and .
According to Theorem 5 of , an M-estimator, under the same assumptions that are made in this theorem, is asymptotically normally distributed with
To prove Theorem 3, it must be shown that the -estimator of the ARMA parameters satisfies an -estimating equation. Differentiating (15) yields the following system of equations:
with , where
if satisfies A6, the -estimate satisfies an M-estimating equation
with data adaptive given by
Special cases are (i) which results in and the -estimator being equivalent to an LS estimator, (ii) which results in the -estimator being equivalent to an S-estimator. The asymptotic value of the estimator is defined by
and under suitable regularity conditions, i.e., ergodicity, the interchange of limits is justified (e.g. by dominated convergence) to yield