Pointwise second-order necessary conditions for stochastic optimal controls, Part I: The case of convex control constraintThis work is partially supported by the National Basic Research Program of China (973 Program) under grant 2011CB808002, by NSF of China under grant 11231007, and by the PCSIRT (from the Chinese Education Ministry) under grant IRT1273.

# Pointwise second-order necessary conditions for stochastic optimal controls, Part I: The case of convex control constraint††thanks: This work is partially supported by the National Basic Research Program of China (973 Program) under grant 2011CB808002, by NSF of China under grant 11231007, and by the PCSIRT (from the Chinese Education Ministry) under grant IRT1273.

Haisen Zhang School of Mathematics, Sichuan University, Chengdu 610064, Sichuan Province, China. E-mail:haisenzhang@yeah.net.    Xu Zhang School of Mathematics, Sichuan University, Chengdu 610064, Sichuan Province, China. E-mail:zhangxu@scu.edu.cn.
###### Abstract

This paper is the first part of our series work to establish pointwise second-order necessary conditions for stochastic optimal controls. In this part, both drift and diffusion terms may contain the control variable but the control region is assumed to be convex. Under some assumptions in terms of Malliavin calculus, we establish the desired necessary condition for stochastic singular optimal controls in the classical sense.

Key words. Stochastic optimal control, Malliavin calculus, pointwise second-order necessary condition, variational equation, adjoint equation.

AMS subject classifications. Primary 93E20; Secondary 60H07, 60H10.

## 1 Introduction

Let and be a complete filtered probability space (satisfying the usual conditions), on which a -dimensional standard Wiener process is defined such that is the natural filtration generated by (augmented by all of the -null sets).

In this paper, we shall consider the following controlled stochastic differential equation

 {dx(t)=b(t,x(t),u(t))dt+σ(t,x(t),u(t))dW(t),   t∈[0,T],x(0)=x0, \hb@xt@.01(1.1)

with a cost functional

 J(u(⋅))=E[∫T0f(t,x(t),u(t))dt+h(x(T))]. \hb@xt@.01(1.2)

Here is the control variable valued in a set (for some ), is the state variable valued in (for some ), and , and are given functions (satisfying some conditions to be given later). As usual, when the context is clear, we omit the argument in the defined functions.

Denote by the Borel -field of a metric space , and by the set of -measurable and -adapted stochastic processes valued in . Any is called an admissible control. The stochastic optimal control problem considered in this paper is to find a control such that

Any satisfying (LABEL:minimum_J) is called an optimal control. The corresponding state (to (LABEL:controlsys)) is called an optimal state, and is called an optimal pair.

In optimal control theory, one of the central topics is to establish the first-order necessary condition for optimal controls. We refer to [15] for an early study on the first-order necessary condition for stochastic optimal controls. After that, many authors contributed on this topic, see [2, 3, 12] and references cited therein. Compared to the deterministic setting, new phenomenon and difficulties appear when the diffusion term of the stochastic control system contains the control variable and the control region is nonconvex. The corresponding first-order necessary condition for this general case was established in [18].

For some optimal controls, it may happen that the first-order necessary conditions turn out to be trivial. For deterministic control systems, there are two types of such optimal controls. One of them, called the singular optimal control in the classical sense, is the optimal control for which the gradient and the Hessian of the corresponding Hamiltonian with respect to the control variable vanish/degenerate. The other one, called the singular optimal control in the sense of Pontryagin-type maximum principle, is the optimal control for which the corresponding Hamiltonian is equal to a constant in the control region. When an optimal control is singular, the first-order necessary condition cannot provide enough information for the theoretical analysis and numerical computing, and therefore one needs to study the second-order necessary conditions. In the deterministic setting, one can find many references in this direction (See [1, 7, 9, 10, 11, 13, 14, 16] and references cited therein).

Compared to the deterministic control systems, there are only two papers ([4, 19]) addressed to the second-order necessary condition for stochastic optimal controls. In [19], a pointwise second-order maximum principle for stochastic singular optimal controls in the sense of Pontryagin-type maximum principle was established for the case that the diffusion term is independent of the control ; while in [4], an integral-type second-order necessary condition for stochastic optimal controls was derived under the assumption that the control region is convex.

The main purpose of this paper is to establish a pointwise second-order necessary condition for stochastic optimal controls. In this work, both drift and diffusion terms, i.e., and , may contain the control variable , and we assume that the control region is convex. The key difference between [4] and our work is that we consider here the pointwise second-order necessary condition, which is easier to be verified in practical applications. We remark that, quite different from the deterministic setting, there exist some essential difficulties to derive the pointwise second-order necessary condition from an integral-type one when the diffusion term of the control system contains the control variable, even for the case of convex control constraint (See the first 4 paragraphs of Subsection LABEL:sub3.2 for a detailed explanation). We overcome these difficulties by means of some technique from the Malliavin calculus. The method developed in this work can be adopted to establish a pointwise second-order necessary condition for stochastic optimal controls for the general case when the control region is nonconvex but the analysis is much more complicated, and therefore we shall give the details in another paper [21].

The rest of this paper is organized as follows. In Section 2, we list some notations, spaces and preliminary results from Malliavin calculus. In Section 3, we introduce the main results of this paper and give some examples. Finally, in Section 4 we give the proofs of the main results.

## 2 Some preliminaries

In this section, we present some preliminaries.

### 2.1 Some notations and spaces

We introduce some notations and spaces which will be used in the sequel.

Denote by and respectively the inner product and norm in or , which can be identified from the contexts. Let be the space of all -matrices. For any , denote by the transpose of and by the norm of . Also, write .

Let be a given function. For a.e. , we denote by , the first order partial derivatives of with respect to and at , by the Hessian of with respect to at , and by , , the second order partial derivatives of with respect to and at .

For any and , we denote by the space of -valued, measurable random variables such that ; by the space of -valued, -measurable processes such that ; by the space of -valued, -measurable, -adapted processes such that ; by the space of -valued, -measurable, and -adapted continuous processes such that , by the space of -valued, -measurable processes such that ; and by the -valued, measurable functions such that for any , is -adapted and .

### 2.2 Some concepts and results from Malliavin calculus

In this subsection, we recall some concepts and results from Malliavin calculus (See [17] for a detailed discussion on this topic).

Denote by the set of -smooth functions with bounded partial derivatives. For any , write . Define

 S:={ζ=φ(W(h1), W(h2), ⋯, W(hd)) ∣∣ φ∈C∞b(Rd;Rn), d∈N,h1,h2,⋯,hd∈L2(0,T)}. \hb@xt@.01(2.1)

Clearly, is a linear subspace of . For any (in the form of that in (LABEL:mJ)), its Malliavin derivative is defined as follows:

 Dsζ:=d∑i=1hi(s)∂φ∂xi(W(h1), W(h2), ⋯, W(hd)),     s∈[0,T].

Write

 |||ζ|||2:=[E |ζ|2+E∫T0|Dsζ|2ds]12.

Obviously, is a norm on . It is shown in [17] that the operator has a closed extension to the space , the completion of with respect to the norm . When , the following Clark-Ocone representation formula holds:

 ζ=E ζ+∫T0E (Dsζ | Fs)dW(s). \hb@xt@.01(2.2)

Furthermore, if is -measurable, then for any .

Define to be the space of processes such that

1. For , ;

2. The function admits a measurable version; and

Denote by the set of all adapted processes in .

 L1,22+(Rn):={φ(⋅)∈L1,2(Rn)∣∣ ∃ D+φ(⋅)∈L2([0,T]×Ω;Rn) such that fε(⋅) is measurable on  [0,T] for any ε>0, and limε→0+∫T0fε(s)ds=0};
 L1,22−(Rn):={φ(⋅)∈L1,2(Rn)∣∣ ∃ D−φ(⋅)∈L2([0,T]×Ω;Rn) such that gε(s):=sup(s−ε)∨00, and limε→0+∫T0gε(s)ds=0}.

Denote

 L1,22(Rn)=L1,22+(Rn)∩L1,22−(Rn).

For any , denote .

When is adapted, for any . In this case, , and . Denote by the set of all adapted processes in .

Roughly speaking, an element is a stochastic process whose Malliavin derivative has suitable continuity on some neighbourhood of . Examples of such process can be found in [17]. Especially, if is continuous from (for some ) to , then and, .

To end this section, we show the following technical result which will be use in the sequel.

###### Lemma 2.1

Let . Then, there exists a sequence of positive numbers such that as and

 limn→∞1θ2n∫τ+θnτ∫tτE ∣∣Dsφ(t)−∇φ(s)∣∣2dsdt=0,a.e. τ∈[0,T]. \hb@xt@.01(2.3)

Proof. For any , we take the convention that

 supt∈[τ,τ+θ]∩[0,T]E∣∣Dτφ(t)−∇φ(τ)∣∣2=0

whenever . From the definition of , it follows that

 limθ→0+1θ2∫T0∫τ+θτ∫tτE∣∣Dsφ(t)−∇φ(s)∣∣2dsdtdτ = limθ→0+1θ2∫T0∫τ+θτ∫τ+θsE∣∣Dsφ(t)−∇φ(s)∣∣2dtdsdτ ≤ limθ→0+1θ∫T0∫τ+θτ[supt∈[s,s+θ]∩[0,T]E∣∣Dsφ(t)−∇φ(s)∣∣2]dsdτ ≤ limθ→0+1θ∫T0∫θ0[supt∈[s+τ,s+τ+θ]∩[0,T]E∣∣Ds+τφ(t)−∇φ(s+τ)∣∣2]dsdτ ≤ limθ→0+1θ∫θ0∫T0[supt∈[s+τ,s+τ+θ]∩[0,T]E∣∣Ds+τφ(t)−∇φ(s+τ)∣∣2]dτds ≤ limθ→0+1θ∫θ0∫Ts[supt∈[τ,τ+θ]∩[0,T]E∣∣Dτφ(t)−∇φ(τ)∣∣2]dτds ≤ limθ→0+1θ∫θ0∫T0[supt∈[τ,τ+θ]∩[0,T]E∣∣Dτφ(t)−∇φ(τ)∣∣2]dτds ≤ = 0,

which implies (LABEL:equence_converg_for_malliavin_deriv).

## 3 Second-order necessary conditions

In this section, we shall present several second-order necessary conditions for stochastic optimal controls.

To begin with, we assume that

1. The control region is nonempty, bounded, and convex.

2. The functions , , , and satisfy the following:

1. For any , the stochastic processes and are -measurable and -adapted. For a.e. , the functions and are continuously differentiable up to order , and all of their partial derivatives are uniformly bounded (with respect to ). There exists a constant such that for a.e. and for any and ,

 ⎧⎪ ⎪⎨⎪ ⎪⎩|b(t,0,u)|+|σ(t,0,u)|≤L,|b(x,u)2(t,x,u)−b(x,u)2(t,~x,~u)|≤L(|x−~x|+|u−~u|),|σ(x,u)2(t,x,u)−σ(x,u)2(t,~x,~u)|≤L(|x−~x|+|u−~u|).
2. For any , the stochastic process is -measurable and -adapted, and the random variable is -measurable. For a.e. , the functions and are continuously differentiable up to order , and for any and ,

When the condition is satisfied, the state (of (LABEL:controlsys)) is uniquely defined by any given initial datum and admissible control , and the cost functional (LABEL:costfunction) is well-defined on . In what follows, represents a generic constant, depending on and , but independent of any other parameter, which can be different from line to line.

### 3.1 Integral-type second-order conditions

Let be an optimal pair, and be any given admissible control. Let , and write

 v(⋅)=u(⋅)−¯u(⋅),uε(⋅)=¯u(⋅)+εv(⋅). \hb@xt@.01(3.1)

Since is convex, . Denote by the state of (LABEL:controlsys) with respect to the control , and put . For , denote

 φx(t)=φx(t,¯x(t),¯u(t)), φu(t)=φu(t,¯x(t),¯u(t)), φxx(t)=φxx(t,¯x(t),¯u(t)), φxu(t)=φxu(t,¯x(t),¯u(t)), φuu(t)=φuu(t,¯x(t),¯u(t)).

First, similar to [4], we introduce the following two variational equations:

 ⎧⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪⎩dy1(t)=[bx(t)y1(t)+bu(t)v(t)]dt+[σx(t)y1(t)+σu(t)v(t)]dW(t),t∈[0,T],y1(0)=0 \hb@xt@.01(3.2)

and

 ⎧⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪ ⎪⎩dy2(t)=[bx(t)y2(t)+y1(t)⊤bxx(t)y1(t)+2v(t)⊤bxu(t)y1(t)+v(t)⊤buu(t)v(t)]dt+[σx(t)y2(t)+y1(t)⊤σxx(t)y1(t)+2v(t)⊤σxu(t)y1(t)+v(t)⊤σuu(t)v(t)]dW(t),t∈[0,T],y2(0)=0. \hb@xt@.01(3.3)

By (LABEL:firstvariequconvex)–(LABEL:secondvariequconvex) and similar to [4, Lemmas 3.5 and 3.11], one has the following estimates.

###### Proposition 3.1

Let (C2) hold. Then, for any ,

 ∥y1∥κ∞,κ≤C,∥y2∥κ∞,κ≤C,∥δx∥κ∞,κ≤Cεκ, ∥δx−εy1∥κ∞,κ≤Cε2κ,∥δx−εy1−ε22y2∥κ∞,κ≤Cε3κ.

Proof. The proof is very close to that of [4, Lemmas 3.5 and 3.11], and therefore, we omit the details.

Next, define the Hamiltonian

 H(t,x,u,y1,z1):=⟨y1,b(t,x,u)⟩+⟨z1,σ(t,x,u)⟩−f(t,x,u), \hb@xt@.01(3.4)

We introduce respectively the following two adjoint equations for (LABEL:firstvariequconvex)–(LABEL:secondvariequconvex):

 ⎧⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪⎩dP1(t)=−[bx(t)⊤P1(t)+σx(t)⊤Q1(t)−fx(t)]dt+Q1(t)dW(t), t∈[0,T],P1(T)=−hx(¯x(T)) \hb@xt@.01(3.5)

and

 ⎧⎪ ⎪ ⎪ ⎪⎨⎪ ⎪ ⎪ ⎪⎩dP2(t)=−[bx(t)⊤P2(t)+P2(t)bx(t)+σx(t)⊤P2(t)σx(t)+σx(t)⊤Q2(t)+Q2(t)σx(t)+Hxx(t)]dt+Q2(t)dW(t), t∈[0,T],P2(T)=−hxx(¯x(T)), \hb@xt@.01(3.6)

where .

From [8], it is easy to check that, for any , the equation (LABEL:firstajointequconvex) admits a unique strong solution , and (LABEL:secondajointequconvex) admits a unique strong solution .

Also, we define

 S(t,x,u,y1,z1,y2,z2) := Hxu(t,x,u,y1,z1)+bu(t,x,u)⊤y2 +σu(t,x,u)⊤z2+σu(t,x,u)⊤y2σx(t,x,u),

, and denote

 S(t)=S(t,¯x(t),¯u(t),P1(t),Q1(t),P2(t),Q2(t)),t∈[0,T]. \hb@xt@.01(3.8)

We have the following result.

###### Proposition 3.2

Let (C1)–(C2) hold. Then, the following variational equality holds for any :

 J(uε(⋅))−J(¯u(⋅)) = −E∫T0[ε⟨Hu(t),v(t)⟩+ε22⟨Huu(t)v(t),v(t)⟩ +ε22⟨P2(t)σu(t)v(t),σu(t)v(t)⟩+ε2⟨S(t)y1(t),v(t)⟩]dt+o(ε2), (ε→0+),

where , .

Proof. By (LABEL:3z1), using Taylor’s formula and Proposition LABEL:estimateofvariequ, similar to [4, Subsection 3.2], we have

 J(uε)−J(¯u) = E∫T0[⟨fx(t),δx(t)⟩+ε⟨fu(t),v(t)⟩+12⟨fxx(t)δx(t),δx(t)⟩ +ε⟨fxu(t)δx(t),v(t)⟩+ε22⟨fuu(t)v(t),v(t)⟩]dt +E[⟨hx(¯x(T)),δx(T)⟩+12⟨hxx(¯x(T))δx(T),δx(T)⟩]+o(ε2)(ε→0+) = E∫T0[ε⟨fx(t),y1(t)⟩+ε22⟨fx(t),y2(t)⟩+ε⟨fu(t),v(t)⟩ +E[ε⟨hx(¯x(T)),y1(T)⟩+ε22⟨hx(¯x(T)),y2(T)⟩ +ε22⟨hxx(¯x(T))y1(T),y1(T)⟩]+o(ε2),(ε→0+).

By Itô’s formula, we have

 E ⟨hx(¯x(T)),y1(T)⟩=−E ⟨P1(T),y1(T)⟩ =
 E ⟨hx(¯x(T)),y2(T)⟩=−E ⟨P1(T),y2(T)⟩ = −E∫T0[⟨P1(t),y1(t)⊤bxx(t)y1(t)⟩+2⟨P1(t),v(t)⊤bxu(t)y1(t)⟩ +⟨P1(t),v(t)⊤buu(t)v(t)⟩+⟨Q1(t),y1(t)⊤σxx(t)y1(t)⟩ +2⟨Q1(t),v(t)⊤σxu(t)y1(t)⟩+⟨Q1(t),v(t)⊤σuu(t)v(t)⟩ +⟨fx(t),y2(t)⟩]dt,

and (noting that and