Iterated fractional Tikhonov regularization

# Iterated fractional Tikhonov regularization

Davide Bianchi    Alessandro Buccini    Marco Donatelli    Stefano Serra-Capizzano111Dipartimento di Scienza e Alta Tecnologia, Università dell’Insubria, 22100 Como, Italy
###### Abstract

Fractional Tikhonov regularization methods have been recently proposed to reduce the oversmoothing property of the Tikhonov regularization in standard form, in order to preserve the details of the approximated solution. Their regularization and convergence properties have been previously investigated showing that they are of optimal order. This paper provides saturation and converse results on their convergence rates. Using the same iterative refinement strategy of iterated Tikhonov regularization, new iterated fractional Tikhonov regularization methods are introduced. We show that these iterated methods are of optimal order and overcome the previous saturation results. Furthermore, nonstationary iterated fractional Tikhonov regularization methods are investigated, establishing their convergence rate under general conditions on the iteration parameters. Numerical results confirm the effectiveness of the proposed regularization iterations.

## 1 Introduction

We consider linear operator equations of the form

 Kx=y, (1.1)

where is a compact linear operator between Hilbert spaces and . We assume to be attainable, i.e., that problem (1.1) has a solution of minimal norm. Here denotes the (Moore-Penrose) generalized inverse operator of , which is unbounded when is compact, with infinite dimensional range. Hence problem (1.1) is ill-posed and has to be regularized in order to compute a numerical solution; see [4].

We want to approximate the solution of the equation (1.1), when only an approximation of is available with

 ∥yδ−y∥≤δ, (1.2)

where is called the noise level. Since is not a good approximation of , we approximate with where is a family of continuous operators depending on a parameter that will be defined later. A classical example is the Tikhonov regularization defined by , where denotes the identity and the adjoint of , cf. [6].

Using the singular values expansion of , filter based regularization methods are defined in terms of filters of the singular values, cf. Proposition 3. This is a useful tool for the analysis of regularization techniques [10], both for direct and iterative regularization methods [8, 11]. Furthermore, new regularization methods can be defined investigating new classes of filters. For instance, one of the contributes in [13] is the proposal and the analysis of the fractional Tikhonov method. The authors obtain a new class of filtering regularization methods adding an exponent, depending on a parameter, to the filter of the standard Tikhonov method. They provide a detailed analysis of the filtering properties and the optimality order of the method in terms of such further parameter. A different generalization of the Tikhonov method has been recently proposed in [12] with a detailed filtering analysis. Both generalizations are called “fractional Tikhonov regularization” in the literature and they are compared in [5], where the optimality order of the method in [12] is provided as well. To distinguish the two proposals in [13] and [12], we will refer in the following as “fractional Tikhonov regularization” and “weighted Tikhonov regularization”, respectively. These variants of the Tikhonov method have been introduced to compute good approximations of non-smooth solutions, since it is well known that the Tikhonov method provides over-smoothed solutions.

In this paper, we firstly provide a saturation result similar to the well-known saturation result for Tikhonov regularization [4]: let be the range of and let be the orthogonal projector onto , if

 sup{∥xδα−x†∥:∥Q(y−yδ)∥≤δ}=o(δ23),

then , as long as is not closed. Such result motivated us to introduce the iterated version of fractional and weighted Tikhonov in the same spirit of the iterated Tikhonov method. We prove that those iterated methods can overcome the previous saturation results. Afterwards, inspired by the works [1, 7] we introduce the nonstationary variants of our iterated methods. Differently from the nonstationary iterated Tikhonov, we have two nonstationary sequences of parameters. In the noise free case, we give sufficient conditions on these sequences to guarantee the convergence providing also the corresponding convergence rates. In the noise case, we show the stability of the proposed iterative schemes proving that they are regularization methods. Finally, few selected examples confirm the previous theoretical analysis, showing that a proper choice of the nonstationary sequences of parameters can provide better restorations compared to the classical iterated Tikhonov with a geometric sequence of regularizzation parameter according to [7].

The paper is organized as follows. Section 2 recalls the basic definition of filter based regularization methods and of optimal order of a regularization method. Fractional Tikhonov regularization with optimal order and converse results are studied in Section 3. Section 4 is devoted to saturation results for both variants of fractional Tikhonov regularization. New iterated fractional Tikhonov regularization methods are introduced in Section 5, where the analysis of their convergence rate shows that their are able to overcome the previous saturation results. A nonstationary iterated weighted Tikhonov regularization is investigated in detail in Section 6, while a similar nonstationary iterated fractional Tikhonov regularization is discussed in Section 7. Finally, some numerical examples are reported in Section 8.

## 2 Preliminaries

As described in the Introduction, we consider a compact linear operator between Hilbert spaces and (over the field or ) with given inner products and , respectively. Hereafter we will omit the subscript for the inner product as it will be clear in the context. If denotes the adjoint of (i.e., ), then we indicate with the singular value expansion (s.v.e.) of , where and are a complete orthonormal system of eigenvectors for and , respectively, and are written in decreasing order, with being the only accumulating point for the sequence . If is not finite dimensional, then , the spectrum of , namely . Finally, denotes the closure of , i.e., .

Let now be the spectral decomposition of the self-adjoint operator . Then from well-known facts from functional analysis [16] we can write , where is a bounded Borel measurable function and is a regular complex Borel measure for every . The following equalities hold

 Kx=+∞∑m=1σm⟨x,vm⟩um,x∈X, (2.1) K∗y=+∞∑m=1σm⟨y,um⟩vm,y∈Y, (2.2) f(K∗K)x:=∫σ(K∗K)f(σ2)dEσ2x=∞∑m=1f(σ2m)⟨x,vm⟩vm, (2.3) ⟨f(K∗K)x1,x2⟩=∫σ(K∗K)f(σ2)d⟨Eσ2x1,x2⟩=∞∑m=1f(σ2m)¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯¯⟨y,vm⟩⟨x,vm⟩, (2.4) ∥f(K∗K)∥≤sup{|f(σ2)|:σ2∈σ(K∗K)}, (2.5)

where the series (2.1) and (2.2) converge in the norms induced by the scalar products of and , respectively. If is a continuous function on then equality holds in (2.5).

###### Definition 1

We define the generalized inverse of a compact linear operator as

 K†y=∑m:σm>0σ−1m⟨y,um⟩vm,y∈D(K†), (2.6)

where

 D(K†)={y∈Y:∑m:σm>0σ−2m|⟨y,um⟩|2<∞}.

With respect to problem (1.1), we consider the case where only an approximation of satisfying the condition (1.2) is available. Therefore , , cannot be approximated by , due to the unboundedness of , and hence in practice the problem (1.1) is approximated by a family of neighbouring well-posed problems [4].

###### Definition 2

By a regularization method for we call any family of operators

 {Rα}α∈(0,α0):Y→X,α0∈(0,+∞],

with the following properties:

• is a bounded operator for every .

• For every there exists a mapping (rule choice) , , such that

 limsupδ→0{α(δ,yδ):yδ∈Y,∥y−yδ∥≤δ}=0,

and

 limsupδ→0{∥Rα(δ,yδ)yδ−K†y∥:yδ∈Y,∥y−yδ∥≤δ}=0.

Throughout this paper is a constant which can change from one instance to the next. For the sake of clarity, if more than one constant will appear in the same line or equation we will distinguish them by means of a subscript.

###### Proposition 3

Let be a compact linear operator and its generalized inverse. Let be a family of operators defined for every as

 Rαy:=∑m:σm>0Fα(σm)σ−1m⟨y,um⟩vm, (2.7)

where is a Borel function such that

 supm:σm>0|Fα(σm)σ−1m|=c(α)<∞, (2.8a) |Fα(σm)|≤c<∞,where c % does not depend on (α,m), (2.8b) limα→0Fα(σm)=1 point-wise in σm. (2.8c)

Then is a regularization method, with , and it is called filter based regularization method.

Proof. See [14] and [4].

For the sake of notational brevity, we fix the following notation

 xα :=Rαy,y∈D(K†), (2.9) xδα :=Rαyδ,yδ∈Y. (2.10)

We report hereafter the definition of optimal order, under the same a-priori assumption given in [4].

###### Definition 4

For every given , let

 Xν,ρ:={x∈X:∃ω∈X,∥ω∥≤ρ,x=(K∗K)ν2ω}⊂X.

A regularization method is called of optimal order under the a-priori assumption if

 Δ(δ,Xν,ρ,Rα)≤c⋅δνν+1ρ1ν+1, (2.11)

where for any general set , and for a regularization method , we define

 Δ(δ,M,Rα):=sup{∥x†−xδα∥:x†∈M,∥y−yδ∥≤δ}.

If is not known, as it will be usually the case, then we relax the definition introducing the set

 Xν:=⋃ρ>0Xν,ρ

and saying that a regularization method is called of optimal order under the a-priori assumption if

 Δ(δ,Xν,Rα)≤c⋅δνν+1. (2.12)
###### Remark 5

Since we are concerned with the rate that converges to zero as , the a-priori assumption is usually sufficient for the optimal order analysis, requiring that (2.12) is satisfied.

Hereafter we cite a theorem which states sufficient conditions for order optimality, when filtering methods are employed, see [14, Proposition 3.4.3, pag. 58].

###### Theorem 6

[14] Let be a compact linear operator, and , and let be a filter based regularization method. If there exists a fixed such that

 sup0<σ≤σ1|Fα(σ)σ−1|≤c⋅α−β, (2.13a) sup0≤σ≤σ1|(1−Fα(σ))σν|≤cν⋅αβν, (2.13b)

then is of optimal order, under the a-priori assumption , with the choice rule

 α=α(δ,ρ)=O(δρ)1β(ν+1).

If we are concerned just about the rate of convergence with respect to only , the preceding theorem can be applied under the a-priori assumption , fitting the proof to the latter case without any effort. On the contrary, below we present a converse result.

###### Theorem 7

Let be a compact linear operator with infinite dimensional range and let be a filter based regularization method with filter function . If there exist and such that

 (1−Fα(σ))σν≥cαβνfor σ∈[c′αβ,σ1] (2.14)

and

 ∥x†−xα∥=O(αβν), (2.15)

then .

Proof. By (2.6) and (2.7), it holds

 ∥x†−xα∥2 =∑σm>0(1−Fα(σm))2σ−2m|⟨y,um⟩|2 =∑σm>0(1−Fα(σm))2|⟨x†,vm⟩|2 =∑σm>0[(1−Fα(σm))σνm]2σ−2νm|⟨x†,vm⟩|2 ≥(cαβν)2∑σm≥c′αβσ−2νm|⟨x†,vm⟩|2.

thanks to the assumption (2.14). From (2.15) we deduce that

 limαβ→0∑σm≥c′αβσ−2νm|⟨x†,vm⟩|2<+∞.

Finally, if we define , then is well defined and , i.e., .

## 3 Fractional variants of Tikhonov regularization

In this section we discuss two recent types of regularization methods that generalize the classical Tikhonov method and that were first introduced and studied in [12] and [13].

### 3.1 Weighted Tikhonov regularization

###### Definition 8 ([12])

We call Weighted Tikhonov method the filter based method

 Rα,ry:=∑m:σm>0Fα,r(σm)σ−1m⟨y,um⟩vm,

where the filter function is

 Fα,r(σ)=σr+1σr+1+α, (3.1)

for and .

According to (2.9) and (2.10), we fix the following notation

 xα,r :=Rα,ry,y∈D(K†), (3.2) xδα,r :=Rα,ryδ,yδ∈Y. (3.3)
###### Remark 9

The Weighted Tikhonov method can also be defined as the unique minimizer of the following functional,

 Rα,ry:=argminx∈X{∥Kx−y∥W+α∥x∥}, (3.4)

where the semi-norm is induced by the operator . For , is to be intended as the Moore-Penrose (pseudo) inverse. Developing the calculations, it follows that

 Rα,ry=[(K∗K)r+12+αI]−1(K∗K)r−12K∗y. (3.5)

That is the reason that motivated us to rename the original method of Hochstenbach and Reichel, that appeared in [12], into weighted Tikhonov method. In this way it would be easier to distinguish from the fractional Tikhonov method introduced by Klann and Ramlau in [13].

The optimal order of the weighted Tikhonov regularization was proved in [5]. The following proposition restates such result, putting in evidence the dependence on of , and provides a converse result.

###### Proposition 10

Let be a compact linear operator with infinite dimensional range. For every given the weighted Tikhonov method, , is a regularization method of optimal order, under the a-priori assumption , with . The best possible rate of convergence with respect to is , that is obtained for with . On the other hand, if then .

Proof. For weighted Tikhonov the left-hand side of condition (2.13a) becomes

 sup0<σ≤σ1∣∣∣σrσr+1+α∣∣∣.

By derivation, if then it is straightforward to see that the quantity above is bounded by , with . Similarly, the left-hand side of condition (2.13b) takes the form

 sup0≤σ≤σ1∣∣∣ασνσr+1+α∣∣∣,

and it is easy to check that it is bounded by if and only if . From Theorem 6, as long as , with , if then we find order optimality (2.11) and the best possible rate of convergence obtainable with respect to is , for .

On the contrary, with and , we deduce that

 |(1−Fα,r(σ))σν|=ασνσr+1+α≥12α,for σ∈[αβ,σ1].

Therefore, if then by Theorem 7.

### 3.2 Fractional Tikhonov regularization

Here we introduce the fractional Tikhonov method defined and discussed in [13].

###### Definition 11 ([13])

We call Fractional Tikhonov method the filter based method

 Rα,γy:=∑m:σm>0Fα,γ(σm)σ−1m⟨y,um⟩vm,

where the filter function is

 Fα,γ(σ)=σ2γ(σ2+α)γ, (3.6)

for and .

Note that is well-defined also for , but the condition (2.8a) requires to guarantee that is a filter function.

We use the notation for and like in equations (3.2) and (3.3), respectively. The optimal order of the fractional Tikhonov regularization was proved in [13, Proposition 3.2]. The following proposition restates such result including also and provides a converse result.

###### Proposition 12

The extended fractional Tikhonov filter method is a regularization method of optimal order, under the a-priori assumption , for every and . The best possible rate of convergence with respect to is , that is obtained for with . On the other hand, if then .

Proof. Condition (2.8a) is verified for and the same holds for conditions (2.8b) and (2.8c). Deriving the filter function, it is immediate to see that equation (2.13a) is verified for , with . It remains to check equation (2.13b):

 (1−Fα,γ(σ))σν =(σ2+α)γ−σ2γ(σ2+α)γσν =(σ2α+1)γ−(σ2α)γ(σ2α+1)γ−1⋅ασνσ2+α =h(σ2α)⋅(1−Fα,1(σ))σν,

where is monotone, for every , and . Namely for and for . Therefore we deduce that

 γ(1−Fα,1(σ))≤(1−Fα,γ(σ))≤(1−Fα,1(σ)),for % 0≤γ≤1, (3.7) (1−Fα,1(σ))≤(1−Fα,γ(σ))≤γ(1−Fα,1(σ)),for γ≥1, (3.8)

from which we infer that

 supσ∈[0,σ1]∣∣(1−Fα,γ(σ))σν∣∣≤max{1,γ}supσ∈[0,σ1]|(1−Fα,1(σ))σν|≤cαν2, (3.9)

since is standard Tikhonov, that is of optimal order, with and for every , see [4]. On the contrary, with and , and by equations (3.7) and (3.8), we deduce that

 (1−Fα,γ(σ))σ2≥min{1,γ}(1−Fα,1(σ))σ2≥12α,for σ∈[α12,σ1]. (3.10)

Therefore, if then by Theorem 7.

## 4 Saturation results

The following proposition deals with a saturation result similar to a well known result for classic Tikhonov, cf. [4, Proposition 5.3].

###### Proposition 13 (Saturation for weighted Tikhonov regularization)

Let be a compact linear operator with infinite dimensional range and be the corresponding family of weighted Tikhonov regularization operators in Definition 8. Let be any parameter choice rule. If

 sup{∥xδα,r−x†∥:∥Q(y−yδ)∥≤δ}=o(δr+1r+2), (4.1)

then , where we indicated with the orthogonal projector onto .

Proof. Define

 δm :=σr+2m, yδm :=y+δmum so that ∥y−yδm∥≤δm, αm :=α(δm,yδm), xm :=xαm,r,xδm:=xδmαm,r.

By the assumption that has not finite dimensional range, then for every and . According to Remark 9, from equation (3.5) we have

 xδm−x†=Rαm,ryδm−x†=Rαm,ry+δmRαm,rum−x†=xm−x†+δmFαm,r(σm)σ−1mvm

and hence by (3.1)

 ∥xδm−x†∥2=∥xm−x†∥2+2δmσrmσr+1m+αm⟨xm−x†,vm⟩+(δmσrmσr+1m+αm)2.

From the choice of follows that

 (δ−r+1r+2m∥xδm−x†∥)2 ≥2δr+1r+2m+αm⟨xm−x†,vm⟩+⎛⎜⎝δr+1r+2mδr+1r+2m+αm⎞⎟⎠2 =21+δ−r+1r+2mαmδ−r+1r+2m⟨xm−x†,vm⟩+⎛⎜⎝11+δ−r+1r+2mαm⎞⎟⎠2. (4.2)

By (3.5),

 ((K∗K)r+12+αmI)(x†−xδm) =(K∗K)r+12x†+αmx†−(K∗K)r−12K∗yδm =αmx†−δm(K∗K)r−12K∗um, (4.3)

so that

 αm∥x†∥=O(δm+∥x†−xδm∥). (4.4)

Since, by assumption, , it follows from (4.4) that if , then

 limm→∞αmδ−r+1r+2m=0. (4.5)

Now, by (4.1) and (4.5) applied to inequality (4) it follows that ,which is a contradiction. Hence .

Note that for (classical Tikhonov) the previous proposition gives exactly Proposition 5.3 in [4]. On the other hand, taking a large , it is possible to overcome the saturation result of classical Tikhonov obtaining a convergence rate arbitrary close to .

A similar saturation result can be proved also for the fractional Tikhonov regularization in Definition 11.

###### Proposition 14 (Saturation for fractional Tikhonov regularization)

Let be a compact linear operator with infinite dimensional range and let be the corresponding family of fractional Tikhonov regularization operators in Definition 11, with fixed . Let be any parameter choice rule. If

 sup{∥xδα,γ−x†∥:∥Q(y−yδ)∥≤δ}=o(δ23), (4.6)

then , where we indicated with the orthogonal projector onto .

Proof. If , the thesis follows from the saturation result for standard Tikhonov [4, Proposition 5.3]. For , recalling that

 xα,γ−x†=∑σm>0(Fα,γ(σm)−1)σ−1m⟨y,um⟩vm,

by equations (3.7) and (3.8), we obtain

 ∥xα,γ−x†∥>c∥xα,1−x†∥, (4.7)

where and is standard Tikhonov. Let us define

 ϕγ(y):=∥xα,γ−x†∥.

Then, by the continuity of , there exists such that, for every , we find

 ϕγ(yδ)>c⋅ϕ1(yδ),

with being the closure of the ball of center and radius . Passing to the we obtain that

 sup{∥xδα,γ−x†∥:∥Q(y−yδ)∥≤δ}≥c⋅sup{∥xδα,1−x†∥:∥Q(y−yδ)∥≤δ}. (4.8)

Therefore, using relation (4.6), we deduce

 sup{∥xδα,1−x†∥:∥y−yδ∥≤δ}=o(δ23), (4.9)

and the thesis follows again from the saturation result for standard Tikhonov, cf. [4, Proposition 5.3].

Differently from the weighted Tikhonov regularization, for the fractional Tikhonov method, it is not possible to overcome the saturation result of classical Tikhonov, even for a large .

## 5 Stationary iterated regularization

We define new iterated regularization methods based on weighed and fractional Tikhonov regularization using the same iterative refinement strategy of iterated Tikhonov regularization [1, 4]. We will show that the iterated methods go beyond the saturation results proved in the previous section. In this section the regularization parameter will still be with the iteration step, , assumed to be fixed. On the contrary, in Section 6, we will analyze the nonstationary counterpart of this iterative method, in which will be replaced by a pre-fixed sequence and we will be concerned on the rate of convergence with respect to the index .

### 5.1 Iterated weighted Tikhonov regularization

We propose now an iterated regularization method based on weighted Tikhonov

###### Definition 15 (Stationary iterated weighted Tikhonov)

We define the stationary iterated weighted Tikhonov method (SIWT) as

 ⎧⎪⎨⎪⎩x0α,r:=0;((K∗K)r+12+αI)xnα,r:=(K∗K)r−12K∗y+αxn−1α,r, (5.1)

with and , or equivalently

 {x0α,r:=0xnα,r:= \emph{argmin}x∈X{∥Kx−y∥W+α∥x−xn−1α,r∥}, (5.2)

where is the semi-norm introduced in (3.4). We define as the -th iteration of weighted Tikhonov if .

###### Proposition 16

For any given and , the SIWT in (5.1) is a filter based regularization method, with filter function

 F(n)α,r(σ)=(σr+1+α)n−αn(σr+1+α)n. (5.3)

Moreover, the method is of optimal order, under the a-priori assumption , for and , with best convergence rate , that is obtained for , with . On the other hand, if , then .

Proof. Multiplying both sides of (5.1) by and iterating the process, we get

 ((K∗K)r+12+αI)nxnα,r ={n−1∑j=0αj((K∗K)r+12+αI)n−1−j}(K∗K)r−12K∗y =[((K∗K)r+12+αI)n−αnI](K∗K)−1K∗y.

Therefore, the filter function in (2.7) is equal to

 F(n)α,r(σ)=(σr+1+α)n−αn(σr+1+α)n,

as we stated. Condition (2.8c) is straightforward to verify. Moreover, note that

 F(n)α,r(σ) =(σr+1+α)n−αn(σr+1+α)n =σr+1σr+1+α⋅(∑n−1j=0αj(σr+1+α)n−1−j)(σr+1+α)n−1 =Fα,r(σ)⋅(1+(ασr+1+α)+⋯+(ασr+1+α)n−1),

from which it follows that

 Fα,r(σ)≤F(n)α,r(σ)≤nFα,r(σ). (5.4)

Therefore, conditions (2.8a), (2.8b) and (2.13a) follows immediately by the regularity of the weighted Tikhonov filter method for and by the order optimality for . Finally, condition (2.13b) becomes

 supσ∈[0,σ1]∣∣∣αnσν(σr+1+α)n∣∣∣,

and deriving one checks that it is bounded by , with , if and only if . Applying now Proposition 6 the rest of the thesis follows.

On the contrary, if we define and , then we deduce that

 (1−F(n)α,r(σ))σν=αnσν(σr+1+α)n≥12nαnfor σ∈[αβ,σ1].

Therefore, if