Solvability conditions for indefinite linear quadratic optimal stochastic control problems and associated stochastic Riccati equations

# Solvability conditions for indefinite linear quadratic optimal stochastic control problems and associated stochastic Riccati equations

Kai Du Institute for Mathematics and Its Applications, School of Mathematics and Applied Statistics, University of Wollongong, Wollongong, NSW 2522, Australia (kaid@uow.edu.au).
###### Abstract

A linear quadratic optimal stochastic control problem with random coefficients and indefinite state/control weight costs is usually linked to an indefinite stochastic Riccati equation (SRE) which is a matrix-valued quadratic backward stochastic differential equation along with an algebraic constraint involving the unknown. Either the optimal control problem or the SRE is solvable only if the given data satisfy a certain structure condition that has yet to be precisely defined. In this paper, by introducing a notion of subsolution for the SRE, we derive several novel sufficient conditions for the existence and uniqueness of the solution to the SRE and for the solvability of the associated optimal stochastic control problem.

\keyphrases

linear quadratic optimal stochastic control, stochastic Riccati equation, backward stochastic differential equation, subsolution, solvability condition \AMclass60H10; 49N10, 93E20

## 1 Introduction

A classical form of a linear quadratic optimal stochastic control (SLQ for short) problem is to minimize the quadratic cost functional

 J(u;ξ)=E{x(T)⊤Hx(T)+∫T0[u⊤(t)R(t)u(t)+x⊤Q(t)x(t)]dt} (1)

with the control being a square-integrable adapted process and the state being the solution to the linear stochastic control system

 dx=(Ax+Bu)dt+∑i(Cix+Diu)dwit,x(0)=ξ∈Rn, (2)

where is a given final time, is a -dimensional Wiener process, and are given coefficients, in particular, and are all symmetric matrix-valued processes, and where we have used a convenient notation

 ∑i:=∑di=1

that will also be used throughout the paper. As in (2), the time variable will be suppressed for simplicity in many circumstances, when no confusion occurs. We assume in this article all the given coefficients to be random.

Under a definiteness assumption that and are positive semi-definite and is positive definite, Bismut [1] made a deep investigation into the above control problem. To characterize the minimal cost and construct the optimal feedback control, he formally derived a backward stochastic differential equation (BSDE) called the stochastic Riccati equation (SRE) as follows:

 dP=∑iΛidwit−[A⊤P+PA+∑i(C⊤iPCi+C⊤iΛi+ΛiCi)+Q]dt (3a) P(T)=H, where the unknown is the matrix-valued process (P,Λ1,…,Λd) adapted to the filtration generated by w. The minimum of J(⋅;ξ) coincides with ξ⊤P(0)ξ once the SRE is solvable “properly”. However, the existence and uniqueness of the solution of (3a) was not completely proved in his work, although he had showed that the original control problem has a unique solution. He left the solvability of the SRE as an open problem which was resolved decades later by Tang [15]111Recently, Tang [16] gave another approach to this problem via the dynamic programming principle.. The systematic study on SLQ problems without the definiteness assumption was initiated by Chen et al. [4] who observed that an SLQ problem where R is possibly indefinite may still be solvable. This finding has triggered an extensive research on the so-called indefinite SLQ problem that has applications in many practical areas, especially in finance (see [18, 11, 10, 17] for example). In [4] they also formulated a related indefinite SRE combining (3a) with the constraint R+∑iD⊤iPDi>0over [0,T], (3b)

and proved that the solvability of this equation yields the well-posedness of the original control problem. This key fact catalyzed quite a few works investigating the existence and uniqueness of the solution to the indefinite SRE. As indicated in the existing literature, the solvability of (3) is by no means unconditional (see [4] for ill-posed examples); in other words, the equation may have no solution if or is “too negative”. The problem is then to specify the conditions that the given data must satisfy to ensure the solvability of indefinite SREs. As far as we know, the existing results are limited to several very special cases (see [8, 13]).

In this paper we derive several novel sufficient conditions that ensure the existence and uniqueness of the solution to the indefinite SRE and also imply the solvability of the associated SLQ problem. According to our understanding, the constraint (3b) seems to be some kind of coercivity condition that plays a similar role to what the positive definiteness of does in the definite case. But it is too implicit to use. Our idea is to reveal the coercivity to some degree by means of a new-defined notion of “subsolution” for SREs (see Definition 2 below). We prove that the existence of subsolutions of (3) implies the well-posedness of the related SLQ problem; moreover, if SRE (3) has a subsolution in a strict sense (see Theorem 3 below), then the equation is solvable and the associated SLQ problem admits a unique optimal feedback control. The original problem is largely converted into finding the new object of the equation. A subsolution is an adapted process that satisfies only an inequality form of (3a) — this relaxing gives us more probabilities to find the target. Further, considering subsolutions of certain particular forms will bring us several practicable criteria of solvable SREs. Consequently, we recover many existing results on the solvability of (3), for instance, obtained in [8, 15, 13]. The proof of Theorem 3 below occupies most of the technical part of our argument, in which we borrow an idea from Tang [15], that is, in a nutshell, as long as an associated forward-backward SDE is solvable, a solution of the SRE can be constructed by using the solution of the former — we succeed to verify the precondition under our setting, and then achieve our aim.

The rest of the paper is organized as follows. Section 2 gives a precise formulation of the problems by introducing several notation and definitions. Section 3 is mainly devoted to the statement of our main results, including some remarks and examples. Section 4 is the most technical part, containing the proofs of several auxiliary lemmas and the main results.

## 2 Preliminaries

Let be a filtered probability space where the filtration is generated by a -dimensional standard Wiener process and satisfies the usual conditions, be the predictable -algebra associated with . Fix a finite terminal time .

Let be the -dimensional Euclidean space, and the set of matrices. We identify and , and use as the norm of . Denote by the set of symmetric matrices. The inequality signs are used to express the usual semi-order of symmetric matrices. For -valued functions (including processes) and , the expression means that is uniformly positive definite almost everywhere (a.e.), i.e., a.e. for some ; the meaning of “” is obvious.

For stopping times and such that , we define

 [[σ,τ))={(t,ω):t∈[σ(ω),τ(ω))};

similarly, we will also use and . For , we write

 Hp(σ,τ;Rn×m):= Lp([[σ,τ)),P,Rn×m), Sp(σ,τ;Rn×m):= Hp(Rn×m)∩Lp(Ω;C([σ,τ];Rn×m)),

and simply

 Hp(Rn×m)=Hp(0,T;Rn×m),Sp(Rn×m)=Sp(0,T;Rn×m).
{definition}

denotes the set of all -valued continuous processes such that

 dV(t)=˚V(t)dt+∑i˘Vi(t)dwitwith  (˚V,˘Vi)∈H1×H2(Sn);

Elements of this set are defined up to indistinguishability. consists of all bounded processes in . We write .

With these preparations, let us restate the main problems. The following assumption is in force throughout the paper.

###### Assumption \thetheorem

The data satisfy that

###### Problem \thetheorem (LQ optimal stochastic control)

Minimize the cost functional (1) over subject to the control system (2). Define the value function

 V(ξ)=infu∈H2(Rk)J(u;ξ).

The problem is said to be well-posed if , to be solvable if for each there is a control depending on such that . We will refer this optimal control problem as SLQ , or simply, SLQ  in some circumstances.

###### Problem \thetheorem (stochastic Riccati equation)

Define the following functions associated with the parameters :

 Δ(P):= R+∑iD⊤iPDi, (4a) Γ(P,Λ):= −Δ(P)−1[B⊤P+∑iD⊤i(PCi+Λi)], Θ(P,Λ):= A⊤P+PA+∑i(C⊤iPCi+C⊤iΛi+ΛiCi)+Q −Γ(P,Λ)⊤Δ(P)Γ(P,Λ). The problem is to find a P∈S such that ˚P+Θ(P,˘P)=0,Δ(P)>0,P(T)=H. (4b)

A solution is said to be bounded if . Here and in what follows, the notation and are understood in the sense of Definition 2. We will refer (4) as SRE , or simply, SRE  in some circumstances.

In order to define our sufficient solvability conditions for SLQs and SREs, we propose an auxiliary notion as follows.

{definition}

is called a subsolution to SRE  if

 ˚F+Θ(F,˘F)≥0,Δ(F)>0,F(T)≤H. (5)

A subsolution is said to be bounded if .

It will be showed that the existence of subsolutions “almost” implies the solvability of problems 2 and 2 (see Theorem 3 below). On the other hand, it is usually much easier to verify whether (4) has a subsolution. These could help us to derive some explicit solvability conditions for SREs. We remark that such a notion can be regarded as a stochastic counterpart of LMI proposed by Rami et al. [14] in their study of deterministic Riccati equations.

## 3 Results

The main results stated as the following two theorems are the basis of our further discussion.

{theorem}

SLQ  is well-posed if SRE  has a bounded subsolution.

{theorem}

Assume that there is a constant such that SRE  has a bounded subsolution . Then

(i) there exists a unique process in the following set

 Sb≥F:={K∈Sb:K(t)≥F(t)~{}almost surely~{}∀t∈[0,T]}

solving SRE ;

(ii) SLQ  is solvable; the value function , and the unique optimal control .

Theorem 3 will be proved in Subsection 4.2. The proof of Theorem 3, deferred to Subsection 4.3, is based on an idea of Tang [15], i.e., to represent a solution of SRE via the solution of a forward-backward stochastic differential equation (FBSDE). The latter is usually called the generalized Hamiltonian system with respect to the associated control problem.

###### Remark \thetheorem

By Definition 2 we can see that, if , then each subsolution to SRE  is also a subsolution to SRE . Therefore, the assumption of Theorem 3 can be stated equivalently as follows: and SRE  has a bounded subsolution.

###### Remark \thetheorem

In Theorem 3 the existence and uniqueness result for the SRE is restricted within a subset of (namely ), which is natural and sensible from the point of view of optimal control. Nevertheless, it is not clear so far whether the SRE admits a solution outside the set .

The result in Theorem 3 is, of course, not optimal. A more satisfactory assertion might be “an SRE is solvable if and only if it has a subsolution”; unfortunately, this is not true, even in the deterministic case. Let us consider the following example.

###### Example \thetheorem

Consider the following ODE over the time interval :

 ˙P=P2(1−t)2χ[0,1)(t)+χ[1,2](t),P(2)=1.

Clearly, is a subsolution to this Riccati equation. However, it is easily verified that it has no continuous solution.

Nevertheless, this assertion would be true under some additional condition. For instance, when the coefficients are all deterministic, the equation (3a) subject to the stronger constraint

 Δ(P)=R+∑iD⊤iPDi≫0 (6)

is solvable if and only if it has a subsolution satisfying (6); we thus conjecture that this may also be available for the stochastic case, but have not found any proof at the moment.

Next we derive from Theorem 3 some explicit sufficient conditions that ensure the existence of solutions to SREs. A basic idea is to consider the subsolutions with certain particular forms. An interesting question is how “negative” the datum could be to maintain the solvability of (4) when and are given. Let us make a first attempt to this question. In the following two results, we provide two “robust” criteria of the “admissible” .

In what follows, we denote

 λ∗(M)=the minimal eigenvalue of a symmetric matrix M. (7)

Note that, for a matrix-valued stochastic process , is a scalar stochastic process.

{proposition}

Let , and be predictable. Assume that the square-integrable predictable processes with satisfy the following BSDE:

 dφ=−[λ∗(Υ(φ,ψ,ζ))φ+λ∗(Q)]dt+ψdwt,φ(T)=λ∗(H), (8)

where

 Υ(φ,ψ,ζ):=A⊤+A+∑iC⊤iCi+∑iψiφ(C⊤i+Ci) −11−ζ(B+∑iC⊤iDi+∑iψiφDi)(∑iD⊤iDi)−1(B+∑iC⊤iDi+∑iψiφDi)⊤.

Then, SRE admits a bounded solution provided .

{proof}

According to Remark 3, it is sufficient to show that is a subsolution to SRE where . Using the notation introduced in Definition 2, we have

 ˘Fi=ψiIn,˚F=−λ∗(Υ(φ,ψ,ζ))φIn−λ∗(Q)In.

Then the expression (recall (4a)) associated to SRE  reads

 ×[(−ζ+1)φ∑iD⊤iDi]−1(φB+φ∑iC⊤iDi+∑iψiDi)⊤+Q =φΥ(φ,ψ,ζ)+Q.

Keeping (7) in mind, since , we have

 φΥ(φ,ψ,ζ)+Q≥φλ∗(Υ(φ,ψ,ζ))In+λ∗(Q)In=−˚F.

This along with the fact that yields that is a subsolution to SRE . The proof is complete.

Equation (8) is actually a one-dimensional quadratic BSDE, of which the existence of the solution was proved by Kobylanski [9]. Nevertheless, due to its high nonlinearity, (8) is often difficult to solve explicitly. Therefore, we formulate a simplified version. First of all, let us introduce another notation: for a matrix-valued random variable , define

 λ#(M)=essinfω∈Ωλ∗(M(ω)).

Note that when is a process, is a deterministic function of time variable.

{theorem}

Let , and . Let satisfy the following ODE:

 ˙φ+λ#(Υ(α))φ+λ#(Q)=0,φ(T)=λ#(H), (9)

where

 Υ(α) := A⊤+A+∑iC⊤iCi−11−α(B+∑iC⊤iDi) ⋅(∑iD⊤iDi)−1(B+∑iC⊤iDi)⊤.

Then, SRE admits a bounded solution provided . {proof} The proof is analogous to that of Proposition 3 — to show that is a subsolution to SRE . This is even simpler here as in this case, so we omit the detail.

An implicit condition of the above two results is that . Although the second criterion is rougher than the previous one, it is significantly more feasible since (9) is a linear ODE that can be resolved explicitly as follows:

 φ(t)=Φ(t,1)λ#(H)+∫1tΦ(t,s)λ#(Q(s))dswith  Φ(t,s)=e∫st[λ#(Υ(α))](r)dr.

Since , an appropriate choice of , even being negative, can also ensure . Therefore, from the above result, one can easily construct various examples of solvable indefinite SREs, including those in which not only but also is indefinite. As far as we know, such a kind of solvability conditions seemed also new for the deterministic case.

Since only one-dimensional condition is concerned, this criterion is still rough, especially for multidimensional equations. Likely some refinement of the analysis will yield a more precise solvability condition, for instance, can be matrix-valued; this is planned as future work. Nevertheless, the above result would be sharp in some one-dimensional cases.

###### Example \thetheorem

Consider the following equation

 dP(t)=(P(t)+Λ(t))2r(t)+P(t)dt+Λ(t)dwt,r(t)+P(t)>0,P(1)=1. (10)

Take a function . By Theorem 3, if

 r(t)>r0(t):=−α(t)φ(t)=−α(t)exp(−∫1t11−α(s)ds), (11)

then (10) admits a solution. Indeed, how to choose the function for different is tricky business. Herein we consider, as an example, a special case that the threshold is a constant, i.e.,

 dr0dt(t)=0 ⟹ dαdt(t)=−α(t)1−α(t).

Thus, the inverse function of is where is a constant. Evidently, is decreasing on , and increasing on , and . To make as low as possible, we choose , i.e., , then is a solution of the equation ; approximately, . Hence, (10) is solvable as long as . This value coincides with that given in [4, Example 3.2] where they considered deterministic equations.

In particular, Theorems 3 and 3 yield directly the following known results.

{corollary}

Let . SRE admits a solution if, i) , or ii) and .

The first case was an open problem proposed by Bismut [2] and Peng [12], respectively, and resolved by Tang [15]. The other was indicated by Kohlmann–Tang [10, 11].

Finally we extend a recent result of Qian–Zhou [13], where certain data of the SRE are not necessarily bounded.

{proposition}

Let and . Take such that

 KB+∑i(C⊤iKDi+˘KiDi)=0. (12)

Assume that , and defined as below are all bounded and positive semi-definite:

 ^Q=¯Q+(˚K+A⊤K+KA)+∑i(C⊤iKCi+C⊤i˘Ki+˘KiCi), ^R=¯R+∑iD⊤iKDi, ^H=¯H−K(T).

Then, SRE  admits a solution if either of the following cases occurs: i) ; ii) and . Moreover, is positive definite and bounded. {proof} According to the assumptions, SRE  admits a solution, denoted by . With (12) in mind, it is easily verified that is a solution of SRE . Moreover, is positive definite and bounded. The proof is complete.

###### Remark \thetheorem

Qian–Zhou [13] introduced a direct approach to deal with a special case that and (that means the Wiener process is one-dimensional). Proposition 3 extends their results into great generality, and thus also recovers those obtained in [8] (see the comments in [13, Section 5]). We also note that, the assumption that and are bounded did not appear in the statement of their main result, i.e., [13, Theorem 2.2], but was involved actually in their proofs, see Lemmas 3.1, 4.2 and 4.4 there; in addition, they assumed the boundedness of , and .

## 4 Proofs

### 4.1 Auxiliary lemmas

Let us first derive a basic a priori estimate for bounded solutions to SREs. An analogous result has been obtained by Tang [15, Theorem 5.1] for the definite case.

{lemma}

Let be a solution to (4). Then, there is a generic constant , depending only on and the bounds of and , such that

 E∫T0(|˚P(t)|+|˘P(t)|2)dt≤κ. (13)
{proof}

Recall (4) that

 −˚P=Θ(P,˘P)= A⊤P+PA+∑i(C⊤iPCi+C⊤i˘Pi+˘PiCi)+Q −Γ(P,˘P)⊤Δ(P)Γ(P,˘P).

Denote . Applying Itô’s formula to , we have

 d|P+P∞|2=|˘P|2dt+2Tr[(P+P∞)˚P]dt+2∑iTr[(P+P∞)˘Pi]dwit.

Taking expectation and by some standard arguments, we gain

 E∫T0|˘P(t)|2dt≤κ+κE∫T0{|˘P|−Tr[(P+P∞)Γ(P,˘P)⊤Δ(P)Γ(P,˘P)]}(t)dt.

Since and ,

 Tr[(P+P∞)Γ(P,˘P)⊤Δ(P)Γ(P,˘P)] =Tr[(P+P∞)1/2Γ(P,˘P)⊤Δ(P)Γ(P,˘P)(P+P∞)1/2]≥0.

Thus, we get

 E∫T0|˘P(t)|2dt≤κ+κE∫T0|˘P(t)|dt≤12E∫T0|˘P(t)|2dt+κ,

which yields the estimate for . Finally, note that

 0≤ E∫T0Γ(P,˘P)⊤Δ(P)Γ(P,˘P)(t)dt ≤ EP(T)−P(0)+E∫T0[A⊤P+PA+∑i(C⊤iPCi+C⊤i˘Pi+˘PiCi)+Q](t)dt ≤ κ+κE∫T0|˘P(t)|2dt≤κ.

This yields the estimate for . The proof is complete.

The following result is a key step toward Theorem 3, which indicates that a solution of the SRE can be constructed from the solution of a forward-backward stochastic differential equation (FBSDE) system, provided that the latter exists and satisfies some appropriate conditions.

{lemma}

Assume that the following FBSDE system

 ⎧⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎩dX=(AX+BU)dt+∑i(CiX+DiU)dwit,dY=−(A⊤Y+∑iC⊤iZi+QX)dt+∑iZidwit,0=RU+B⊤Y+∑iD⊤iZi,X(0)=In,Y(T)=HX(T) (14)

has a solution

 (X,U,Y,Z)∈S2(Rn×n)×H2(Rk×n)×S2(Rn×n)×(H2(Rn×n))d,

moreover, there are a process and a constant such that

 X⊤KX≤X⊤Y≤κX⊤X,R+∑iD⊤iKDi≫0. (15)

Then, has a continuous version, and

 P=YX−1∈Sb (16)

is a solution of SRE  with

 ˘Pi=ZiX−1−YX−1(Ci+DiUX−1), (17)

and a.s. for each .

###### Remark \thetheorem

The process takes values in . Indeed, using Itô’s formula to , we have

 X(t)⊤Y(t)=EFt[∫Tt(U⊤RU+X⊤QX)(r)dr+X(T)⊤HX(T)],∀t∈[0,T].

Clearly, the right-hand side is an -valued random variable.

Before the rigorous proof, let us do some heuristic computations. Suppose exists. Set

 P=YX−1,Λi=ZiX−1−YX−1(Ci+DiUX−1). (18)

By use of the fact that , we derive the equation of as

 d(X−1)= −X−1[A+BUX−1−∑i(Ci+DiUX−1)2]dt −X−1∑i(Ci+DiUX−1)dwit.

Thus, with in mind, we gain that

 dP= d(YX−1)=−∑iZiX−1(Ci+DiUX−1)dt+Yd(X−1)+(dY)X−1 (19) = −∑i[ZiX−1−YX−1(Ci+DiUX−1)](Ci+DiUX−1)dt −[A⊤YX−1+Q+∑iC⊤iZiX−1]dt−YX−1(A+BUX−1)dt +∑i[ZiX−1−YX−1(Ci+DiUX−1)]dwit = −[A⊤P+PA+∑i(C⊤iPCi+C⊤iΛi+ΛiCi)+Q]dt −[PB+∑i(C⊤iPDi+ΛiDi)]UX−1dt+∑iΛidwit.

On the other hand, it follows from (14) that

 0= RUX−1+B⊤YX−1+∑iD⊤iZiX−1 = RUX−1+B⊤P+∑iD⊤i[Λi+P(Ci+DiUX−1)] = (R+∑iD⊤iPDi)UX−1+B⊤P+∑i(D⊤iPCi+D⊤iΛi);

if , then

 UX−1=−(R+∑iD⊤iPDi)−1[B⊤P+∑i(D⊤iPCi+D⊤iΛi)]. (20)

Substituting (20) into (19), one can find that defined in (18) satisfies (3a) formally.

###### Remark \thetheorem

From (20), the equation of can be rewritten as

 dX=[A+BΓ(P,Λ)]Xdt+∑i[Ci+DiΓ(P,Λ)]Xdwit,X(0)=In, (21)

as long as is well-defined, where is defined in (4a).

We are now in a position to prove Lemma 4.1. The key point, suggested by the above heuristic analysis, is to show the existence and continuity of the reciprocal process of .

Proof of Lemma 4.1. First of all, is well-defined. Recall (7) the definition of , and introduce the stopping times:

 τm =inf{t:λ∗(X⊤(t)X(t))≤1m}∧T,m∈N∗, τ =τ∞=inf{t:λ∗(X⊤(t)X(t))≤0}∧T.

Clearly, , and exists on the set , and is bounded on ; is thus well-defined on . Keeping in mind (15), we have

 [X⊤KX](t,ω)≤[X⊤PX](t,ω)≤κ[X⊤X](t,ω)∀(t,ω)∈[[0,τ)), (22)

thus,

 K(t,ω)≤P(t,ω)≤κIn∀(t,ω)∈[[0,τ)), (23)

and furthermore, on . Define

 P(m):=Pχ[[0,τm]]+P(τm)χ((τm,T]],Λ(m):=Λχ[[0,τm]].

Clearly, . According to our heuristic computations, is a bounded solution to

 SRE (A(m),B(m),C(m),D(m);R(m),Q(m),H(m))

with

 A(m):=Aχ[[0,τm]],B(m):=Bχ[[0,τm]],C(m):=Cχ[[0,τm]],D(m):=Dχ[[0,τm]],R(m):=R,Q(m):=Qχ[[0,τm]],H(m):=P(τm).

Define, as in (4a), the corresponding

 Δ(m)(P),Γ(m)(P,Λ),Θ(m)(P,Λ).

By means of Lemma 4.1, there is a constant independent of such that

 E∫T0(|Λ(m)(t)|2+|Θ(m)(P(m)(t),Λ(m)(t))|)dt≤κ1<∞.

Since, as ,

by Fatou’s lemma we have

 E∫T0χ[[0,τ))(t)(|Λ(t)|2+|Θ(P(t),Λ(t))|)dt≤κ1<∞.

Since

 |Λχ[[τm,τ))|≤|Λχ[[0,τ))|,|Θ(P(t),Λ(t))χ[[τm,τ))|≤|Θ(P(t),Λ(t))χ[[0,τ))|,

it follows from Lebesgue’s dominated convergence theorem that

Therefore, we obtain

 ∫T0∑iΛ(m)i(t)dwit→∫T0∑iχ[[0,τ))Λi(t)dwita.s.,

Define

 H(∞):=P