# Gravitational-Wave Data Analysis.

Formalism and Sample Applications: The Gaussian Case

###### Abstract

The article reviews the statistical theory of signal detection in application to analysis of deterministic gravitational-wave signals in the noise of a detector. Statistical foundations for the theory of signal detection and parameter estimation are presented. Several tools needed for both theoretical evaluation of the optimal data analysis methods and for their practical implementation are introduced. They include optimal signal-to-noise ratio, Fisher matrix, false alarm and detection probabilities, -statistic, template placement, and fitting factor. These tools apply to the case of signals buried in a stationary and Gaussian noise. Algorithms to efficiently implement the optimal data analysis techniques are discussed. Formulas are given for a general gravitational-wave signal that includes as special cases most of the deterministic signals of interest.

update (26 July 2007)

Section 2 and 4.4.1 have been extended, some notations were changed and Figure 1 was updated. Equations (28) and (36) have been corrected. 15 references were added.

## 1 Introduction

In this review we consider the problem of detection of deterministic gravitational-wave signals in the noise of a detector and the question of estimation of their parameters. The examples of deterministic signals are gravitational waves from rotating neutron stars, coalescing compact binaries, and supernova explosions. The case of detection of stochastic gravitational-wave signals in the noise of a detector is reviewed in [5]. A very powerful method to detect a signal in noise that is optimal by several criteria consists of correlating the data with the template that is matched to the expected signal. This matched-filtering technique is a special case of the maximum likelihood detection method. In this review we describe the theoretical foundation of the method and we show how it can be applied to the case of a very general deterministic gravitational-wave signal buried in a stationary and Gaussian noise.

Early gravitational-wave data analysis was concerned with the detection of bursts originating from supernova explosions [99]. It involved analysis of the coincidences among the detectors [52]. With the growing interest in laser interferometric gravitational-wave detectors that are broadband it was realized that sources other than supernovae can also be detectable [92] and that they can provide a wealth of astrophysical information [85, 59]. For example the analytic form of the gravitational-wave signal from a binary system is known in terms of a few parameters to a good approximation. Consequently one can detect such a signal by correlating the data with the waveform of the signal and maximizing the correlation with respect to the parameters of the waveform. Using this method one can pick up a weak signal from the noise by building a large signal-to-noise ratio over a wide bandwidth of the detector [92]. This observation has led to a rapid development of the theory of gravitational-wave data analysis. It became clear that the detectability of sources is determined by optimal signal-to-noise ratio, Equation (24), which is the power spectrum of the signal divided by the power spectrum of the noise integrated over the bandwidth of the detector.

An important landmark was a workshop entitled Gravitational Wave Data Analysis held in Dyffryn House and Gardens, St. Nicholas near Cardiff, in July 1987 [86]. The meeting acquainted physicists interested in analyzing gravitational-wave data with the basics of the statistical theory of signal detection and its application to detection of gravitational-wave sources. As a result of subsequent studies the Fisher information matrix was introduced to the theory of the analysis of gravitational-wave data [40, 58]. The diagonal elements of the Fisher matrix give lower bounds on the variances of the estimators of the parameters of the signal and can be used to assess the quality of astrophysical information that can be obtained from detections of gravitational-wave signals [32, 57, 18]. It was also realized that application of matched-filtering to some sources, notably to continuous sources originating from neutron stars, will require extraordinary large computing resources. This gave a further stimulus to the development of optimal and efficient algorithms and data analysis methods [87].

A very important development was the work by Cutler et al. [31] where it was realized that for the case of coalescing binaries matched filtering was sensitive to very small post-Newtonian effects of the waveform. Thus these effects can be detected. This leads to a much better verification of Einstein’s theory of relativity and provides a wealth of astrophysical information that would make a laser interferometric gravitational-wave detector a true astronomical observatory complementary to those utilizing the electromagnetic spectrum. As further developments of the theory methods were introduced to calculate the quality of suboptimal filters [9], to calculate the number of templates to do a search using matched-filtering [74], to determine the accuracy of templates required [24], and to calculate the false alarm probability and thresholds [50]. An important point is the reduction of the number of parameters that one needs to search for in order to detect a signal. Namely estimators of a certain type of parameters, called extrinsic parameters, can be found in a closed analytic form and consequently eliminated from the search. Thus a computationally intensive search needs only be performed over a reduced set of intrinsic parameters [58, 50, 60].

Techniques reviewed in this paper have been used in the data analysis of prototypes of gravitational-wave detectors [73, 71, 6] and in the data analysis of presently working gravitational-wave detectors [90, 15, 3, 2, 1].

We use units such that the velocity of light .

## 2 Response of a Detector to a Gravitational Wave

There are two main methods to detect gravitational waves which have been implemented in the currently working instruments. One method is to measure changes induced by gravitational waves on the distances between freely moving test masses using coherent trains of electromagnetic waves. The other method is to measure the deformation of large masses at their resonance frequencies induced by gravitational waves. The first idea is realized in laser interferometric detectors and Doppler tracking experiments [82, 65] whereas the second idea is implemented in resonant mass detectors [13].

Let us consider the response to a plane gravitational wave of a freely falling configuration of masses. It is enough to consider a configuration of three masses shown in Figure 1 to obtain the response for all currently working and planned detectors. Two masses model a Doppler tracking experiment where one mass is the Earth and the other one is a distant spacecraft. Three masses model a ground-based laser interferometer where the masses are suspended from seismically isolated supports or a space-borne interferometer where the three masses are shielded in satellites driven by drag-free control systems.

In Figure 1 we have introduced the following notation: denotes the origin of the TT coordinate system related to the passing gravitational wave, () are 3-vectors joining and the masses, and () are, respectively, 3-vectors of unit Euclidean length along the lines joining the masses and the coordinate Euclidean distances between the masses, where is the label of the opposite mass. Let us also denote by the unit 3-vector directed from the origin to the source of the gravitational wave. We first assume that the spatial coordinates of the masses do not change in time.

Let be the frequency of the coherent beam used in the detector (laser light in the case of an interferometer and radio waves in the case of Doppler tracking). Let be the relative change of frequency induced by a transverse, traceless, plane gravitational wave on the coherent beam travelling from the mass to the mass , and let be the relative change of frequency induced on the beam travelling from the mass 3 to the mass 1. The frequency shifts and are given by [37, 10, 83]

(1) |

(2) |

where (here T denotes matrix transposition)

(3) |

In Equation (3) is the three-dimensional matrix of the spatial metric perturbation produced by the wave in the TT coordinate system. If one chooses spatial TT coordinates such that the wave is travelling in the direction, then the matrix is given by

(4) |

where and are the two polarizations of the wave.

Real gravitational-wave detectors do not stay at rest with respect to the TT coordinate system related to the passing gravitational wave, because they also move in the gravitational field of the solar system bodies, as in the case of the LISA spacecraft, or are fixed to the surface of Earth, as in the case of Earth-based laser interferometers or resonant bar detectors. Let us choose the origin of the TT coordinate system to coincide with the solar system barycenter (SSB). The motion of the detector with respect to the SSB will modulate the gravitational-wave signal registered by the detector. One can show that as far as the velocities of the masses (modelling the detector’s parts) with respect to the SSB are nonrelativistic, which is the case for all existing or planned detectors, the Equations (1) and (2) can still be used, provided the 3-vectors and () will be interpreted as made of the time-dependent components describing the motion of the masses with respect to the SSB.

It is often convenient to introduce the proper reference frame of the detector with coordinates . Because the motion of this frame with respect to the SSB is nonrelativistic, we can assume that the transformation between the SSB-related coordinates and the detector’s coordinates has the form

(5) |

where the functions describe the motion of the origin of the proper reference frame with respect to the SSB, and the functions account for the different orientations of the spatial axes of the two reference frames. One can compute some of the quantities entering Equations (1) and (2) in the detector’s coordinates rather than in the TT coordinates. For instance, the matrix of the wave-induced spatial metric perturbation in the detector’s coordinates is related to the matrix of the spatial metric perturbation produced by the wave in the TT coordinate system through equation

(6) |

where the matrix has elements . If the transformation matrix is orthogonal, then , and Equation (6) simplifies to

(7) |

For a standard Michelson, equal-arm interferometric configuration is given in terms of one-way frequency changes and (see Equations (1) and (2) with , where we assume that the mass 1 corresponds to the corner station of the interferometer) by the expression [93]

(8) |

In the long-wavelength approximation Equation (8) reduces to

(9) |

The difference of the phase fluctuations measured, say, by a photo detector, is related to the corresponding relative frequency fluctuations by

(10) |

By virtue of Equation (9) the phase change can be written as

(11) |

where the function ,

(12) |

is the response of the interferometer to a gravitational wave in the long-wavelength approximation. In this approximation the response of a laser interferometer is usually derived from the equation of geodesic deviation (then the response is defined as the difference between the relative wave-induced changes of the proper lengths of the two arms, i.e., ). There are important cases where the long-wavelength approximation is not valid. These include the space-borne LISA detector for gravitational-wave frequencies larger than a few mHz and satellite Doppler tracking measurements.

In the case of a bar detector the long-wavelength approximation is very accurate and the detector’s response is defined as , where is the wave-induced change of the proper length of the bar. The response is given by

(13) |

where is the unit vector along the symmetry axis of the bar.

In most cases of interest the response of the detector to a gravitational wave can be written as a linear combination of four constant amplitudes ,

(14) |

where the four functions depend on a set of parameters but are independent of the parameters . The parameters are called extrinsic parameters whereas the parameters are called intrinsic. In the long-wavelength approximation the functions are given by

(15) |

where is the phase modulation of the signal and , are slowly varying amplitude modulations.

Equation (14) is a model of the response of the space-based detector LISA to gravitational waves from a binary system [60], whereas Equation (15) is a model of the response of a ground-based detector to a continuous source of gravitational waves like a rotating neutron star [50]. The gravitational-wave signal from spinning neutron stars may consist of several components of the form (14). For short observation times over which the amplitude modulation functions are nearly constant, the response can be approximated by

(16) |

where and are constant amplitude and initial phase, respectively, and is a slowly varying function of time. Equation (16) is a good model for a response of a detector to the gravitational wave from a coalescing binary system [92, 22]. We would like to stress that not all deterministic gravitational-wave signals may be cast into the general form (14).

## 3 Statistical Theory of Signal Detection

The gravitational-wave signal will be buried in the noise of the detector and the data from the detector will be a random process. Consequently the problem of extracting the signal from the noise is a statistical one. The basic idea behind the signal detection is that the presence of the signal changes the statistical characteristics of the data , in particular its probability distribution. When the signal is absent the data have probability density function (pdf) , and when the signal is present the pdf is .

A full exposition of the statistical theory of signal detection that is outlined here can be found in the monographs [102, 56, 98, 96, 66, 44, 77]. A general introduction to stochastic processes is given in [100]. Advanced treatment of the subject can be found in [64, 101].

The problem of detecting the signal in noise can be posed as a statistical hypothesis testing problem. The null hypothesis is that the signal is absent from the data and the alternative hypothesis is that the signal is present. A hypothesis test (or decision rule) is a partition of the observation set into two sets, and its complement . If data are in we accept the null hypothesis, otherwise we reject it. There are two kinds of errors that we can make. A type I error is choosing hypothesis when is true and a type II error is choosing when is true. In signal detection theory the probability of a type I error is called the false alarm probability, whereas the probability of a type II error is called the false dismissal probability. is the probability of detection of the signal. In hypothesis testing the probability of a type I error is called the significance of the test, whereas is called the power of the test.

The problem is to find a test that is in some way optimal. There are several approaches to find such a test. The subject is covered in detail in many books on statistics, for example see references [54, 41, 62].

### 3.1 Bayesian approach

In the Bayesian approach we assign costs to our decisions; in particular we introduce positive numbers , , where is the cost incurred by choosing hypothesis when hypothesis is true. We define the conditional risk of a decision rule for each hypothesis as

(17) |

where is the probability distribution of the data when hypothesis is true. Next we assign probabilities and to the occurrences of hypothesis and , respectively. These probabilities are called a priori probabilities or priors. We define the Bayes risk as the overall average cost incurred by the decision rule :

(18) |

Finally we define the Bayes rule as the rule that minimizes the Bayes risk .

### 3.2 Minimax approach

Very often in practice we do not have the control over or access to the mechanism generating the state of nature and we are not able to assign priors to various hypotheses. In such a case one criterion is to seek a decision rule that minimizes, over all , the maximum of the conditional risks, and . A decision rule that fulfills that criterion is called minimax rule.

### 3.3 Neyman–Pearson approach

In many problems of practical interest the imposition of a specific cost structure on the decisions made is not possible or desirable. The Neyman–Pearson approach involves a trade-off between the two types of errors that one can make in choosing a particular hypothesis. The Neyman–Pearson design criterion is to maximize the power of the test (probability of detection) subject to a chosen significance of the test (false alarm probability).

### 3.4 Likelihood ratio test

It is remarkable that all three very different approaches – Bayesian, minimax, and Neyman–Pearson – lead to the same test called the likelihood ratio test [34]. The likelihood ratio is the ratio of the pdf when the signal is present to the pdf when it is absent:

(19) |

We accept the hypothesis if , where is the threshold that is calculated from the costs , priors , or the significance of the test depending on what approach is being used.

#### 3.4.1 Gaussian case – The matched filter

Let be the gravitational-wave signal and let be the detector noise. For convenience we assume that the signal is a continuous function of time and that the noise is a continuous random process. Results for the discrete time data that we have in practice can then be obtained by a suitable sampling of the continuous-in-time expressions. Assuming that the noise is additive the data can be written as

(20) |

In addition, if the noise is a zero-mean, stationary, and Gaussian random process, the log likelihood function is given by

(21) |

where the scalar product is defined by

(22) |

In Equation (22) denotes the real part of a complex expression, the tilde denotes the Fourier transform, the asterisk is complex conjugation, and is the one-sided spectral density of the noise in the detector, which is defined through equation

(23) |

where E denotes the expectation value.

From the expression (21) we see immediately that the likelihood ratio test consists of correlating the data with the signal that is present in the noise and comparing the correlation to a threshold. Such a correlation is called the matched filter. The matched filter is a linear operation on the data.

An important quantity is the optimal signal-to-noise ratio defined by

(24) |

We see in the following that determines the probability of detection of the signal. The higher the signal-to-noise ratio the higher the probability of detection.

An interesting property of the matched filter is that it maximizes the signal-to-noise ratio over all linear filters [34]. This property is independent of the probability distribution of the noise.

## 4 Parameter Estimation

Very often we know the waveform of the signal that we are searching for in the data in terms of a finite number of unknown parameters. We would like to find optimal procedures of estimating these parameters. An estimator of a parameter is a function that assigns to each data the “best” guess of the true value of . Note that because depends on the random data it is a random variable. Ideally we would like our estimator to be (i) unbiased, i.e., its expectation value to be equal to the true value of the parameter, and (ii) of minimum variance. Such estimators are rare and in general difficult to find. As in the signal detection there are several approaches to the parameter estimation problem. The subject is exposed in detail in reference [63]. See also [103] for a concise account.

### 4.1 Bayesian estimation

We assign a cost function of estimating the true value of as . We then associate with an estimator a conditional risk or cost averaged over all realizations of data for each value of the parameter :

(25) |

where is the set of observations and is the joint probability distribution of data and parameter . We further assume that there is a certain a priori probability distribution of the parameter . We then define the Bayes estimator as the estimator that minimizes the average risk defined as

(26) |

where E is the expectation value with respect to an a priori distribution , and is the set of observations of the parameter . It is not difficult to show that for a commonly used cost function

(27) |

the Bayesian estimator is the conditional mean of the parameter given data , i.e.,

(28) |

where is the conditional probability density of parameter given the data .

### 4.2 Maximum a posteriori probability estimation

Suppose that in a given estimation problem we are not able to assign a particular cost function . Then a natural choice is a uniform cost function equal to over a certain interval of the parameter . From Bayes theorem [20] we have

(29) |

where is the probability distribution of data . Then from Equation (26) one can deduce that for each data the Bayes estimate is any value of that maximizes the conditional probability . The density is also called the a posteriori probability density of parameter and the estimator that maximizes is called the maximum a posteriori (MAP) estimator. It is denoted by . We find that the MAP estimators are solutions of the following equation

(30) |

which is called the MAP equation.

### 4.3 Maximum likelihood estimation

Often we do not know the a priori probability density of a given parameter and we simply assign to it a uniform probability. In such a case maximization of the a posteriori probability is equivalent to maximization of the probability density treated as a function of . We call the function the likelihood function and the value of the parameter that maximizes the maximum likelihood (ML) estimator. Instead of the function we can use the function (assuming that ). is then equivalent to the likelihood ratio [see Equation (19)] when the parameters of the signal are known. Then the ML estimators are obtained by solving the equation

(31) |

which is called the ML equation.

#### 4.3.1 Gaussian case

For the general gravitational-wave signal defined in Equation (14) the log likelihood function is given by

(32) |

where the components of the column vector and the matrix are given by

(33) |

with , and where is a zero-mean Gaussian random process. The ML equations for the extrinsic parameters can be solved explicitly and their ML estimators are given by

(34) |

Substituting into we obtain a function

(35) |

that we call the -statistic. The -statistic depends (nonlinearly) only on the intrinsic parameters .

Thus the procedure to detect the signal and estimate its parameters consists of two parts. The first part is to find the (local) maxima of the -statistic in the intrinsic parameter space. The ML estimators of the intrinsic parameters are those for which the -statistic attains a maximum. The second part is to calculate the estimators of the extrinsic parameters from the analytic formula (34), where the matrix and the correlations are calculated for the intrinsic parameters equal to their ML estimators obtained from the first part of the analysis. We call this procedure the maximum likelihood detection. See Section 4.8 for a discussion of the algorithms to find the (local) maxima of the -statistic.

### 4.4 Fisher information

It is important to know how good our estimators are. We would like our estimator to have as small variance as possible. There is a useful lower bound on variances of the parameter estimators called Cramèr–Rao bound. Let us first introduce the Fisher information matrix with the components defined by

(36) |

The Cramèr–Rao bound states that for unbiased estimators the covariance matrix of the estimators . (The inequality for matrices means that the matrix is nonnegative definite.)

A very important property of the ML estimators is that asymptotically (i.e., for a signal-to-noise ratio tending to infinity) they are (i) unbiased, and (ii) they have a Gaussian distribution with covariance matrix equal to the inverse of the Fisher information matrix.

#### 4.4.1 Gaussian case

In the case of Gaussian noise the components of the Fisher matrix are given by

(37) |

For the case of the general gravitational-wave signal defined in Equation (14) the set of the signal parameters splits naturally into extrinsic and intrinsic parameters: . Then the Fisher matrix can be written in terms of block matrices for these two sets of parameters as

(38) |

where the top left block corresponds to the extrinsic parameters, the bottom right block corresponds to the intrinsic parameters, the superscript T denotes here transposition over the extrinsic parameter indices, and the dot stands for the matrix multiplication with respect to these parameters. Matrix is given by Equation (33), and the matrices and are defined as follows:

(39) |

The covariance matrix , which approximates the expected covariances of the ML parameter estimators, is defined as . Using the standard formula for the inverse of a block matrix [67] we have

(40) |

where

(41) |

We call (the Schur complement of ) the projected Fisher matrix (onto the space of intrinsic parameters). Because the projected Fisher matrix is the inverse of the intrinsic-parameter submatrix of the covariance matrix , it expresses the information available about the intrinsic parameters that takes into account the correlations with the extrinsic parameters. Note that is still a function of the putative extrinsic parameters.

We next define the normalized projected Fisher matrix

(42) |

where is the signal-to-noise ratio. From the Rayleigh principle [67] follows that the minimum value of the component is given by the smallest eigenvalue (taken with respect to the extrinsic parameters) of the matrix . Similarly, the maximum value of the component is given by the largest eigenvalue of that matrix. Because the trace of a matrix is equal to the sum of its eigenvalues, the matrix

(43) |

where the trace is taken over the extrinsic-parameter indices, expresses the information available about the intrinsic parameters, averaged over the possible values of the extrinsic parameters. Note that the factor 1/4 is specific to the case of four extrinsic parameters. We call the reduced Fisher matrix. This matrix is a function of the intrinsic parameters alone. We see that the reduced Fisher matrix plays a key role in the signal processing theory that we review here. It is used in the calculation of the threshold for statistically significant detection and in the formula for the number of templates needed to do a given search.

For the case of the signal

(44) |

the normalized projected Fisher matrix is independent of the extrinsic parameters and , and it is equal to the reduced matrix [74]. The components of are given by

(45) |

where is the Fisher matrix for the signal .

Fisher matrix has been extensively used to assess the accuracy of estimation of astrophysically interesting parameters of gravitational-wave signals. First calculations of Fisher matrix concerned gravitational-wave signals from inspiralling binaries in quadrupole approximation [40, 58] and from quasi-normal modes of Kerr black hole [38]. Cutler and Flanagan [32] initiated the study of the implications of higher PN order phasing formula as applied to the parameter estimation of inspiralling binaries. They used the 1.5PN phasing formula to investigate the problem of parameter estimation, both for spinning and non-spinning binaries, and examined the effect of the spin-orbit coupling on the estimation of parameters. The effect of the 2PN phasing formula was analyzed independently by Poisson and Will [76] and Królak, Kokkotas and Schäfer [57]. In both of these works the focus was to understand the new spin-spin coupling term appearing at the 2PN order when the spins were aligned perpendicular to the orbital plane. Compared to [57], [76] also included a priori information about the magnitude of the spin parameters, which then leads to a reduction in the rms errors in the estimation of mass parameters. The case of 3.5PN phasing formula was studied in detail by Arun et al. [12]. Inclusion of 3.5PN effects leads to an improved estimate of the binary parameters. Improvements are relatively smaller for lighter binaries.

Various authors have investigated the accuracy with which LISA detector can determine binary parameters including spin effects. Cutler [30] determined LISA’s angular resolution and evaluated the errors of the binary masses and distance considering spins aligned or anti-aligned with the orbital angular momentum. Hughes [46] investigated the accuracy with which the redshift can be estimated (if the cosmological parameters are derived independently), and considered the black-hole ring-down phase in addition to the inspiralling signal. Seto [89] included the effect of finite armlength (going beyond the long wavelength approximation) and found that the accuracy of the distance determination and angular resolution improve. This happens because the response of the instrument when the armlength is finite depends strongly on the location of the source, which is tightly correlated with the distance and the direction of the orbital angular momentum. Vecchio [97] provided the first estimate of parameters for precessing binaries when only one of the two supermassive black holes carries spin. He showed that modulational effects decorrelate the binary parameters to some extent, resulting in a better estimation of the parameters compared to the case when spins are aligned or antialigned with orbital angular momentum. Hughes and Menou [47] studied a class of binaries, which they called “golden binaries,” for which the inspiral and ring-down phases could be observed with good enough precision to carry out valuable tests of strong-field gravity. Berti, Buonanno and Will [21] have shown that inclusion of non-precessing spin-orbit and spin-spin terms in the gravitational-wave phasing generally reduces the accuracy with which the parameters of the binary can be estimated. This is not surprising, since the parameters are highly correlated, and adding parameters effectively dilutes the available information.

### 4.5 False alarm and detection probabilities – Gaussian case

#### 4.5.1 Statistical properties of the -statistic

We first present the false alarm and detection pdfs when the intrinsic parameters of the signal are known. In this case the statistic is a quadratic form of the random variables that are correlations of the data. As we assume that the noise in the data is Gaussian and the correlations are linear functions of the data, is a quadratic form of the Gaussian random variables. Consequently -statistic has a distribution related to the distribution. One can show (see Section III B in [49]) that for the signal given by Equation (14), has a distribution with 4 degrees of freedom when the signal is absent and noncentral distribution with 4 degrees of freedom and non-centrality parameter equal to signal-to-noise ratio when the signal is present.

As a result the pdfs and of when the intrinsic parameters are known and when respectively the signal is absent and present are given by

(46) | |||||

(47) |

where is the number of degrees of freedom of distributions and is the modified Bessel function of the first kind and order . The false alarm probability is the probability that exceeds a certain threshold when there is no signal. In our case we have

(48) |

The probability of detection is the probability that exceeds the threshold when the signal-to-noise ratio is equal to :

(49) |

The integral in the above formula can be expressed in terms of the generalized Marcum -function [94, 44], . We see that when the noise in the detector is Gaussian and the intrinsic parameters are known, the probability of detection of the signal depends on a single quantity: the optimal signal-to-noise ratio .

#### 4.5.2 False alarm probability

Next we return to the case when the intrinsic parameters are not known. Then the statistic given by Equation (35) is a certain generalized multiparameter random process called the random field (see Adler’s monograph [4] for a comprehensive discussion of random fields). If the vector has one component the random field is simply a random process. For random fields we can define the autocovariance function just in the same way as we define such a function for a random process:

(50) |

where and are two values of the intrinsic parameter set, and is the expectation value when the signal is absent. One can show that for the signal (14) the autocovariance function is given by

(51) |

where

(52) |

We have .

One can estimate the false alarm probability in the following way [50]. The autocovariance function tends to zero as the displacement increases (it is maximal for ). Thus we can divide the parameter space into elementary cells such that in each cell the autocovariance function is appreciably different from zero. The realizations of the random field within a cell will be correlated (dependent), whereas realizations of the random field within each cell and outside the cell are almost uncorrelated (independent). Thus the number of cells covering the parameter space gives an estimate of the number of independent realizations of the random field. The correlation hypersurface is a closed surface defined by the requirement that at the boundary of the hypersurface the correlation equals half of its maximum value. The elementary cell is defined by the equation

(53) |

for at cell center and on cell boundary. To estimate the number of cells we perform the Taylor expansion of the autocorrelation function up to the second-order terms:

(54) |

As attains its maximum value when , we have

(55) |

Let us introduce the symmetric matrix

(56) |

Then the approximate equation for the elementary cell is given by

(57) |

It is interesting to find a relation between the matrix and the Fisher matrix. One can show (see [60], Appendix B) that the matrix is precisely equal to the reduced Fisher matrix given by Equation (43).

Let be the number of the intrinsic parameters. If the components of the matrix are constant (independent of the values of the parameters of the signal) the above equation is an equation for a hyperellipse. The -dimensional Euclidean volume of the elementary cell defined by Equation (57) equals

(58) |

where denotes the Gamma function. We estimate the number of elementary cells by dividing the total Euclidean volume of the -dimensional parameter space by the volume of the elementary cell, i.e. we have

(59) |

The components of the matrix are constant for the signal when the phase is a linear function of the intrinsic parameters .

To estimate the number of cells in the case when the components of the matrix are not constant, i.e. when they depend on the values of the parameters, we write Equation (59) as

(60) |

This procedure can be thought of as interpreting the matrix as the metric on the parameter space. This interpretation appeared for the first time in the context of gravitational-wave data analysis in the work by Owen [74], where an analogous integral formula was proposed for the number of templates needed to perform a search for gravitational-wave signals from coalescing binaries.

The concept of number of cells was introduced in [50] and it is a generalization of the idea of an effective number of samples introduced in [36] for the case of a coalescing binary signal.

We approximate the probability distribution of in each cell by the probability when the parameters are known [in our case by probability given by Equation (46)]. The values of the statistic in each cell can be considered as independent random variables. The probability that does not exceed the threshold in a given cell is , where is given by Equation (48). Consequently the probability that does not exceed the threshold in all the cells is . The probability that exceeds in one or more cell is thus given by

(61) |

This by definition is the false alarm probability when the phase parameters are unknown. The number of false alarms is given by

(62) |

A different approach to the calculation of the number of false alarms using the Euler characteristic of level crossings of a random field is described in [49].

It was shown (see [29]) that for any finite and , Equation (61) provides an upper bound for the false alarm probability. Also in [29] a tighter upper bound for the false alarm probability was derived by modifying a formula obtained by Mohanty [68]. The formula amounts essentially to introducing a suitable coefficient multiplying the number of cells .

#### 4.5.3 Detection probability

When the signal is present a precise calculation of the pdf of is very difficult because the presence of the signal makes the data random process non-stationary. As a first approximation we can estimate the probability of detection of the signal when the parameters are unknown by the probability of detection when the parameters of the signal are known [given by Equation (49)]. This approximation assumes that when the signal is present the true values of the phase parameters fall within the cell where has a maximum. This approximation will be the better the higher the signal-to-noise ratio is.

### 4.6 Number of templates

To search for gravitational-wave signals we evaluate the -statistic on a grid in parameter space. The grid has to be sufficiently fine such that the loss of signals is minimized. In order to estimate the number of points of the grid, or in other words the number of templates that we need to search for a signal, the natural quantity to study is the expectation value of the -statistic when the signal is present. We have

(63) |

The components of the matrix are given in Equation (52). Let us rewrite the expectation value (63) in the following form,

(64) |

where is the signal-to-noise ratio. Let us also define the normalized correlation function

(65) |

From the Rayleigh principle [67] it follows that the minimum of the normalized correlation function is equal to the smallest eigenvalue of the normalized matrix , whereas the maximum is given by its largest eigenvalue. We define the reduced correlation function as

(66) |

As the trace of a matrix equals the sum of its eigenvalues, the reduced correlation function is equal to the average of the eigenvalues of the normalized correlation function . In this sense we can think of the reduced correlation function as an “average” of the normalized correlation function. The advantage of the reduced correlation function is that it depends only on the intrinsic parameters , and thus it is suitable for studying the number of grid points on which the -statistic needs to be evaluated. We also note that the normalized correlation function precisely coincides with the autocovariance function of the -statistic given by Equation (51).

Like in the calculation of the number of cells in order to estimate the number of templates we perform a Taylor expansion of up to second order terms around the true values of the parameters, and we obtain an equation analogous to Equation (57),

(67) |

where is given by Equation (56). By arguments identical to those in deriving the formula for the number of cells we arrive at the following formula for the number of templates:

(68) |

When the above formula coincides with the formula for the number of cells, Equation (60). Here we would like to place the templates sufficiently closely so that the loss of signals is minimized. Thus needs to be chosen sufficiently small. The formula (68) for the number of templates assumes that the templates are placed in the centers of hyperspheres and that the hyperspheres fill the parameter space without holes. In order to have a tiling of the parameter space without holes we can place the templates in the centers of hypercubes which are inscribed in the hyperspheres. Then the formula for the number of templates reads

(69) |

For the case of the signal given by Equation (16) our formula for number of templates is equivalent to the original formula derived by Owen [74]. Owen [74] has also introduced a geometric approach to the problem of template placement involving the identification of the Fisher matrix with a metric on the parameter space. An early study of the template placement for the case of coalescing binaries can be found in [84, 35, 19]. Applications of the geometric approach of Owen to the case of spinning neutron stars and supernova bursts are given in [24, 11].

The problem of how to cover the parameter space with the smallest possible number of templates, such that no point in the parameter space lies further away from a grid point than a certain distance, is known in mathematical literature as the covering problem [28]. The maximum distance of any point to the next grid point is called the covering radius . An important class of coverings are lattice coverings. We define a lattice in -dimensional Euclidean space to be the set of points including such that if and are lattice points, then also and are lattice points. The basic building block of a lattice is called the fundamental region. A lattice covering is a covering of by spheres of covering radius , where the centers of the spheres form a lattice. The most important quantity of a covering is its thickness defined as

(70) |

In the case of a two-dimensional Euclidean space the best covering is the hexagonal covering and its thickness . For dimensions higher than 2 the best covering is not known. We know however the best lattice covering for dimensions . These are so-called lattices which have a thickness equal to

(71) |

where is the volume of the -dimensional sphere of unit radius.

For the case of spinning neutron stars a 3-dimensional grid was constructed consisting of prisms with hexagonal bases [16]. This grid has a thickness around 1.84 which is much better than the cubic grid which has thickness of approximately 2.72. It is worse than the best lattice covering which has the thickness around 1.46. The advantage of an lattice over the hypercubic lattice grows exponentially with the number of dimensions.

### 4.7 Suboptimal filtering

To extract signals from the noise one very often uses filters that are not optimal. We may have to choose an approximate, suboptimal filter because we do not know the exact form of the signal (this is almost always the case in practice) or in order to reduce the computational cost and to simplify the analysis. The most natural and simplest way to proceed is to use as our statistic the -statistic where the filters are the approximate ones instead of the optimal ones matched to the signal. In general the functions will be different from the functions used in optimal filtering, and also the set of parameters will be different from the set of parameters in optimal filters. We call this procedure the suboptimal filtering and we denote the suboptimal statistic by .

We need a measure of how well a given suboptimal filter performs. To find such a measure we calculate the expectation value of the suboptimal statistic. We get

(72) |

where

(73) |

Let us rewrite the expectation value in the following form,

(74) |

where is the optimal signal-to-noise ratio. The expectation value reaches its maximum equal to when the filter is perfectly matched to the signal. A natural measure of the performance of a suboptimal filter is the quantity FF defined by

(75) |

We call the quantity FF the generalized fitting factor.

In the case of a signal given by

(76) |

the generalized fitting factor defined above reduces to the fitting factor introduced by Apostolatos [9]:

(77) |

The fitting factor is the ratio of the maximal signal-to-noise ratio that can be achieved with suboptimal filtering to the signal-to-noise ratio obtained when we use a perfectly matched, optimal filter. We note that for the signal given by Equation (76), FF is independent of the value of the amplitude . For the general signal with 4 constant amplitudes it follows from the Rayleigh principle that the fitting factor is the maximum of the largest eigenvalue of the matrix over the intrinsic parameters of the signal.

For the case of a signal of the form

(78) |

where is a constant phase, the maximum over in Equation (77) can be obtained analytically. Moreover, assuming that over the bandwidth of the signal the spectral density of the noise is constant and that over the observation time oscillates rapidly, the fitting factor is approximately given by

(79) |

In designing suboptimal filters one faces the issue of how small a fitting factor one can accept. A popular rule of thumb is accepting . Assuming that the amplitude of the signal and consequently the signal-to-noise ratio decreases inversely proportional to the distance from the source this corresponds to 10% loss of the signals that would be detected by a matched filter.

### 4.8 Algorithms to calculate the -statistic

#### 4.8.1 The two-step procedure

In order to detect signals we search for threshold crossings of the -statistic over the intrinsic parameter space. Once we have a threshold crossing we need to find the precise location of the maximum of in order to estimate accurately the parameters of the signal. A satisfactory procedure is the two-step procedure. The first step is a coarse search where we evaluate on a coarse grid in parameter space and locate threshold crossings. The second step, called fine search, is a refinement around the region of parameter space where the maximum identified by the coarse search is located.

There are two methods to perform the fine search. One is to refine the grid around the threshold crossing found by the coarse search [70, 68, 91, 88], and the other is to use an optimization routine to find the maximum of [49, 60]. As initial value to the optimization routine we input the values of the parameters found by the coarse search. There are many maximization algorithms available. One useful method is the Nelder–Mead algorithm [61] which does not require computation of the derivatives of the function being maximized.

#### 4.8.2 Evaluation of the -statistic

Usually the grid in parameter space is very large and it is important to calculate the optimum statistic as efficiently as possible. In special cases the -statistic given by Equation (35) can be further simplified. For example, in the case of coalescing binaries can be expressed in terms of convolutions that depend on the difference between the time-of-arrival (TOA) of the signal and the TOA parameter of the filter. Such convolutions can be efficiently computed using Fast Fourier Transforms (FFTs). For continuous sources, like gravitational waves from rotating neutron stars observed by ground-based detectors [49] or gravitational waves form stellar mass binaries observed by space-borne detectors [60], the detection statistic involves integrals of the general form

(80) |

where are the intrinsic parameters excluding the frequency parameter , is the amplitude modulation function, and the phase modulation function. The amplitude modulation function is slowly varying comparing to the exponential terms in the integral (80). We see that the integral (80) can be interpreted as a Fourier transform (and computed efficiently with an FFT), if and if does not depend on the frequency . In the long-wavelength approximation the amplitude function does not depend on the frequency. In this case Equation (80) can be converted to a Fourier transform by introducing a new time variable [87],