Ergodic Control of Infinite Dimensional SDEswith Degenerate Noise

Ergodic Control of Infinite Dimensional SDEs
with Degenerate Noise

Andrea COSSO111Department of Mathematics, University of Bologna, Piazza di Porta S. Donato 5, 40126 Bologna, Italy, andrea.cosso@unibo.it.    Giuseppina GUATTERI222Department of Mathematics, Politecnico di Milano, Piazza Leonardo da Vinci 32, 20133 Milano, Italy, giuseppina.guatteri@polimi.it.    Gianmario TESSITORE333Department of Mathematics and Applications, University of Milano-Bicocca, Via R. Cozzi 53 - Building U5, 20125 Milano, Italy, gianmario.tessitore@unimib.it.
Abstract

The present paper is devoted to the study of the asymptotic behavior of the value functions of both finite and infinite horizon stochastic control problems and to the investigation of their relation with suitable stochastic ergodic control problems. Our methodology is based only on probabilistic techniques, as for instance the so-called randomization of the control method, thus avoiding completely analytical tools from the theory of viscosity solutions. We are then able to treat with the case where the state process takes values in a general (possibly infinite dimensional) real separable Hilbert space and the diffusion coefficient is allowed to be degenerate.

Keywords: Ergodic control; infinite dimensional SDEs; BSDEs; randomization of the control method.

2010 Mathematics Subject Classification: 60H15, 60H30, 37A50.

1 Introduction

In the present paper we study the asymptotic behavior of the value functions both for finite horizon stochastic control problems (as the horizon diverges) and for discounted infinite horizon control problems (as the discount vanishes) and investigate their relation with suitable stochastic ergodic control problems. We will refer to such limits as ergodic limits. The main novelty of this work is that we deal with ergodic limits for control problems in which the state process is allowed to take values in a general (possibly infinite dimensional) real separable Hilbert space and the diffusion coefficient is allowed to be degenerate.

On the one side, the infinite dimensional framework imposes the use of purely probabilistic techniques essentially based on backward stochastic differential equations (BSDEs for short, see for instance the introduction of [12]), on the other, the degeneracy of the noise prevents the use of standard BSDEs techniques as they are, for instance, implemented, for similar problems, in [9]. Indeed, see again [12], the identification between solutions of BSDEs and value functions of stochastic optimal control problems can be easily obtained, by a Girsanov argument, as far as the so called structure condition, imposing large enough image of diffusion operator, holds. Here we wish to avoid such a requirement.

To our knowledge, the only paper that deals, by means of BSDEs, with ergodic limits in the degenerate case is [5] where authors use the same tool of randomized control problems and related constrained BSDEs that we will eventually employ here. Notice however that in [5] the state process lives in a finite-dimensional Euclidean space and probabilistic methods are combined with PDE techniques, relying on powerful tools from the theory of viscosity solutions. Here, as already mentioned, we have to completely avoid these arguments. As a matter of fact viscosity solutions require, in the infinite dimensional case, additional artificial assumptions that we can not impose here (see for instance the theory of -continuous viscosity solutions for second order PDEs, [8], [19]). On the other side to separate difficulties we consider, as in [9] but differently from [5], only additive and uncontrolled noise. This in particular considerably simplifies the proof of estimate (4.3).

Let us now give a more precise idea of the results obtained in the paper. Consider the following infinite and finite horizon stochastic control problems:

and

where the discount coefficient can be any positive real number, as well as the time horizon , while the controlled state process takes values in some real separable Hilbert space and is a (mild) solution to the time-homogenous stochastic differential equation

Here is a cylindrical Wiener process and is a possibly unbounded linear operator on . We assume that both and are dissipative. The control process is progressively measurable and takes values in some real separable Hilbert space (actually, can be taken more general, see Remark 2.2). Notice that the diffusion coefficient is only assumed to be a bounded linear operator, so that it can be degenerate. Our aim is to study the asymptotic behavior of

(1.1)

as and . In order to do it, we find non-linear Feynman-Kac representations for both and in terms of suitable backward stochastic differential equations (which can be seen as the probabilistic counterparts of the Hamilton-Jacobi-Bellman equations). Since can be degenerate we adopt the recently introduced so-called randomization method, see e.g. [16], [10], [2], which was also implemented in [5]. Here we use it in a rather different way, as we will explain below. The idea of the randomization (of the control) method is to introduce a new control problem (called the randomized infinite/finite horizon stochastic control problem), where we replace the family of control processes by a particular class of processes (depending on a control parameter ), here denoted by , which is, roughly speaking, dense in . More precisely, focusing for simplicity only on the infinite horizon case, we define the value function of the randomized infinite horizon stochastic control problem as follows:

where is the set of progressively measurable and uniformly bounded processes taking values in , while the state process is the pair satisfying

with being a trace class injective linear operator with dense image, while is a cylindrical Wiener process independent of . We prove by a density argument (Proposition 3.2) that , for every , so, in particular, does not depend on its second argument . Notice that we randomize the control by means of an independent cylindrical Wiener process , instead of using an independent Poisson random measure on (as it is usually the case in the literature on the randomization method). Taking a Poisson random measure has the advantage that can be any Borel space, while here we have to impose some restrictions on (see Remark 2.2). However, randomizing the control by means of a cylindrical Wiener process is simpler and more natural in an infinite dimensional setting, as many important fundamental results on SDEs and BSDEs can only be found for the case where the driving noise is of Wiener type. Moreover, with this choice the results presented here can receive more attention in the infinite dimensional literature. Furthermore, the Wiener type randomization has not been enough investigated in the literature, since it was implemented only in [3], where however the proof of the fundamental equality was based on PDE techniques (in particular, viscosity solutions’ arguments) adapted from [16], instead of using purely probabilistic arguments, as it was done in [10] and [2] for the Poisson type randomization. So, in particular, this is the first time that the equality is proved in purely probabilistic terms for the Wiener type randomization.

Once we know that , it is fairly standard in the framework of the randomization approach, to derive a non-linear Feynman-Kac formula for , see for instance [16]. As a matter of fact, notice that, for each positive integer , the control problem with value function

is a dominated problem. Therefore, by standard BSDE techniques, admits a non-linear Feynman-Kac representation in terms of some BSDE depending on the parameter . Passing to the limit as goes to infinity, we find, as in [5], a non-standard BSDE for (see Propositions 4.1 and 4.3). As a matter of fact such a BSDE, that we shall call ‘constrained’, involves a reflection term and is characterized by its maximality, see again [16]. We eventually exploit the probabilistic representation of (and similarly of ) to study the limits in (1.1). In particular, in Section 5 we prove that, up to a subsequence, the limit of (resp. ) exists and is given by a function (resp. constant ). Moreover we prove that and are related to a suitable constrained ergodic backward stochastic differential equation, again of the non standard, constrained, type see Theorem 5.1. This is the the most technical result of the paper. In the previous literature, see [5], PDE techniques are indeed of great help at this level. In the present context we have to prove in a direct way that the candidate solution to the ergodic constrained BSDE enjoys the required maximality property. To do that we exploit the extra regularity of the trajectories of the state equation implied by Assumption (A.2) (see estimate (3.3) in Proposition 3.1).

Concerning the long time asymptotics of , we show in Theorem 6.1 that this quantity converges to the same constant . We end Section 5 proving that, under suitable assumptions, coincides with the value function of an ergodic control problem. This latter result is again proved using only probabilistic techniques, while in [5] the proof is based on PDE arguments (see Remark 6.2 for more details on this point).

The rest of the paper is organized as follows. In Section 2 we firstly introduce the notations used throughout the paper, then we formulate both the infinite and finite horizon stochastic optimal control problems on a generic probabilistic setting; afterwards, we formulate both control problems on a specific probabilistic, product-space, setting and we prove (Proposition 2.3) that, even if the probabilistic setting has changed, value functions are still the same. Section 3 is devoted to the formulation of the randomized control problems; we prove (Proposition 3.2) that the value functions of the randomized problems coincide with the value functions of the original control problems. In Section 4 we find non-linear Feynman-Kac representation formulae for the value functions in terms of constrained backward stochastic differential equations. In Section 5 we introduce an ergodic BSDE and study the asymptotic behavior of the infinite horizon problem. Finally, in Section 6 we introduce an ergodic control problem and study the long-time asymptotics of the finite horizon problem.

2 Infinite/finite horizon optimal control problems

In the present section we introduce both an infinite and a finite horizon stochastic optimal control problem, first on a generic probability space and then on an enlarged probability space in product form (this latter will be then use throughout the paper). Firstly we fix some notations.

2.1 General notation

Let , and be real separable Hilbert spaces. In the sequel, we use the notations , and to denote the norms on , and respectively; if no confusion arises, we simply write . We use similar notation for the scalar products. We denote the dual spaces of , and by , , and respectively. We also denote by the space of bounded linear operators from to , endowed with the operator norm. Moreover, we denote by the space of Hilbert-Schmidt operators from to . Finally, we denote by the Borel -algebra of any topological space .

Given a complete probability space together with a filtration (satisfying the usual conditions of -completeness and right-continuity) and an arbitrary real separable Hilbert space we define the following classes of processes for fixed and :

  • denotes the set of (equivalence classes) of -predictable processes such that the following norm is finite:

  • denotes the set of processes defined on , whose restriction to an arbitrary time interval belongs to .

  • denotes the set of -predictable processes on with continuous paths in , such that the norm

    is finite. The elements of are identified up to indistinguishability.

  • denotes the set of processes defined on , whose restriction to an arbitrary time interval belongs to .

  • denotes the set of real-valued -adapted nondecreasing continuous processes on such that and .

  • denotes the set of processes defined on , whose restriction to an arbitrary time interval belongs to .

2.2 Formulation of the control problems

We formulate here both the discounted, infinite horizon, control problem and the finite horizon one whose asymptotic behavior is the main focus of the present paper. The notation chosen here may seem a bit artificial but this is done in order to keep the notation simple in the product space and randomized setting (see Sections 2 and 3) where the technical arguments are developed.

We fix a complete probability space on which a cylindrical Wiener process with values in is defined. By , or simply , we denote the natural filtration of , augmented with the family of -null sets of . Obviously, the filtration satisfies the usual conditions of right-continuity and -completeness.

In this section the notion of measurability and progressive measurability will always refer to the filtration .

Let be the family of -progressively measurable processes taking values in (see Remark 2.2 below for the case where the space of control actions is not necessarily a Hilbert space).

State process.

Given and , we consider the controlled stochastic differential equation

(2.1)

On the coefficients , , we impose the following assumptions.

  • is a linear, possibly unbounded operator generating an analytic semigroup . We assume that is dissipative i.e. , for all .

  • is a bounded linear operator. Moreover, there exist positive constants and such that

  • Fixed 0 we denote by , the domain of the fractional power of the operator , see [17]. We assume that there exists a such that the domain of the fractional power is compactly embedded in .

  • is continuous and there exists such that

    for all and .

    Moreover there exists such that

    for all and .

  • is assumed to be strongly dissipative: there exists such that

    for all and .

Remark 2.1

Assumption (A.4) can be balanced with the dissipativity of replacing by and by (). In particular we can always think that both and are strongly dissipative.

Proposition 2.1

Assume (A.1)(A.5). Then, for any and , there exists a unique (up to indistinguishability) process that belongs to for all and is a mild solution of (2.1), that is:

Moreover the following estimates hold:

  • for every and there exists a positive constant , independent of and , such that

  • there exists a positive constant , independent of , , , such that

Proof. This is a standard result, see [9, Proposition 3.6] for the proof in a general Banach space context. Notice that the presence of the control process does not causes any additional difficulty since assumptions (A.4) and (A.5) hold uniformly with respect to the control variable .

Finally, we fix a running cost and we impose the following assumption.

  • is continuous and bounded, moreover there exists such that

    for all and .

Infinite horizon control problem.

Given a positive discount , the cost corresponding to control and initial condition is defined as

where denotes the expectation with respect to . Moreover the value function is given by

Finite horizon control problem.

Fix a function satisfying:

  • is continuous and there exists such that

The cost with (finite) horizon and discount relative to the control and initial condition is defined as

Finally, the value function is given by

Remark 2.2

The request on the space of control actions to be an Hilbert space can be relaxed. As a matter of fact, suppose that the space of control actions is a certain set , so that, in the formulation of the stochastic optimal control problem, drift and running cost are defined on :

Suppose that has the following property: there exists a continuous surjection , for some real separable Hilbert space . This holds true, for instance, if is a compact, connected, locally connected subset of , for some positive integer (in this case, the existence of a continuous surjection , with , follows from the Hahn-Mazurkiewicz theorem, see for instance Theorem 6.8 in [18]). Then, we define and as

for every . Notice that, if (resp. ) satisfies assumptions (A.4) and (A.5) (resp. (A.6)) then (resp. ) still satisfies the same assumptions. Replacing , , by , , we find a stochastic control problem of the form studied in the present work, which has the same value function of the original control problem.

2.3 Formulation of the control problems on a product space

For a technical reason imposed by the randomization method (see the next Section 3) we have to reformulate our control problems in a product probability space. The main point of this section, see Proposition 2.3, will be to show that this new setting does not affect the value function.

Let be a cylindrical Wiener process with values in , defined on a complete probability space . We define , , as follows: , the -completion of , the extension of to , , , for every .

By , we denote the natural filtration of , augmented with the family of -null sets of . Clearly, satisfies the usual conditions of right-continuity and -completeness. Finally, we denote by the family of -progressively measurable processes with values in . In this section measurability will always be referred to such a filtration.

As before, given and , we consider the controlled stochastic differential equation

(2.2)

Exactly as for equation (2.1), we have the following result.

Proposition 2.2

Assume (A.1)(A.5). Then, for any and , there exists a unique (up to indistinguishability) process that belongs to for all and is a mild solution of (2.2), that is:

Moreover the following estimates hold:

  • for every and there exists a positive constant , independent of and , such that

    (2.3)
  • there exists a positive constant , independent of , , , such that

    (2.4)

Again, for every , and for any , , we define the infinite horizon cost

and the corresponding value function

On the other hand, for every , , and for any , , we define the finite horizon cost:

and the corresponding value function

In the next proposition we give a detailed argument showing that, as expected, the value function of both infinite and finite horizon problems is not affected by the product space formulation. For the definition of and see Section 2.2.

Proposition 2.3

Suppose that Assumptions (A.1)(A.7) hold. Then:

  • For all , , for every .

  • For all and , , for every .

Proof. We prove only the first statement, since the proof of (ii) can be done proceeding along the same lines.

Fix and . We begin noting that the inequality is immediate. As a matter of fact, given (thus is a process defined on ) let , for every . Then, , and . Taking the infimum over , we conclude that .

We now prove the other inequality. To this end, we recall that is the natural filtration on of , augmented with the family of -null sets. In a similar way, we define . Now, let be the filtration defined as , for every . Observe that is right-continuous (as it can be shown proceeding for instance as in the proof of Theorem 1 in [13]), but not necessarily -complete. We also notice that is the augmentation of .

Now, fix . Since is -progressively measurable, by for instance Lemma B.21 in [1] or Theorem 3.7 in [4], we deduce that there exists an -predictable process with values in such that , -a.e., so, in particular, . By Lemma 2.17-b) in [15], it follows that there exists an -predictable process with values in which is indistinguishable from , so that . In addition, since is in particular -progressively measurable, for every we have that the process on , given by , is -progressively measurable. In other words, for every .

Consider now, for every , the process solving the following controlled equation:

On the other hand, we recall that the process solves the controlled equation

So, in particular, there exists a -null set such that the above equality holds, for all and for every . Therefore, there exists a -null set such that, for every ,

which can be rewritten in terms of as

Then, we see that, for every , the two processes and solve the same equation. By pathwise uniqueness, it follows that, for every , and are indistinguishable. An application of Fubini’s Theorem yields

Recalling that , the claim follows taking the infimum over all .

3 Randomized optimal control problems

In the present section we formulate the randomized versions (see [16]) of both the infinite and the finite horizon stochastic optimal control problems introduced in the previous Section 2.

We consider the same probabilistic setting as in subsection 2.3. In particular, we adopt the same notations: , , , , . Progressive measurability of processes will always be intended with respect to the filtration .

By we denote the family of progressively measurable processes with values in such that , -almost surely. Moreover is the set of progressively measurable and essentially bounded processes with values in .

State process.

Given and , we consider the system of controlled stochastic differential equations:

(3.1)

On and we impose the same assumptions as in Section 2, while on we impose the following:

  • is a trace class injective linear operator with dense image.

Proposition 3.1

Assume (A.1)(A.5) and (A.8). Then, for any and , there exists a unique (up to indistinguishability) pair of processes and (the process is independent of ) such that:

  • is given by:

    (3.2)

    thus satisfies the second equation in (3.1) and belongs to for all ;

  • belongs to for all and is a mild solution of the first equation in (3.1), that is:

Moreover for every (with as in Assumption (A.3)), and for any , the following estimate holds:

(3.3)

for some positive constant , depending only on , , , and on the constants introduced in Assumptions (A.1)(A.5), but independent of and .

Proof.This is quite a classical result, see for instance [6] and the randomization framework has nothing special here (we just formulate the result in the case in which we need it). For the sake of completeness, we report the proof of estimate (3.3) . We have

Thus, for every such that , we deduce estimate (3.3).  

Remark 3.1

Notice that , with defined in Proposition 2.2. Indeed, for every and , the process belongs to . Thus the analogue of estimates (2.3) and (2.4) hold for (uniformly with respect to and ).

Once more we define, in this new setting, the finite and infinite horizon costs as well as the corresponding value functions.

Infinite horizon control problem.

For every , and for any , , the infinite horizon cost functional is

with corresponding value function

Finite horizon control problem.

For every , and for any , , the finite horizon cost cost is

with corresponding the value function

Next statement entitles us to study the (asymptotic) behavior of and instead of and (or of and ). Moreover it implies that and do not depend on their last argument.

Proposition 3.2

Suppose that Assumptions (A.1)(A.8) hold. Then, we have (recalling Proposition 2.3):

  • For every , , for all . In particular, the function is independent of its second argument.

  • For every and ,