Infinite-horizon specifications over Markov processes

# Characterization and computation of infinite-horizon specifications over Markov processes

Ilya Tkachev and Alessandro Abate
###### Abstract.

This work is devoted to the formal verification of specifications over general discrete-time Markov processes, with an emphasis on infinite-horizon properties. These properties, formulated in a modal logic known as PCTL, can be expressed through value functions defined over the state space of the process. The main goal is to understand how structural features of the model (primarily the presence of absorbing sets) influence the uniqueness of the solutions of corresponding Bellman equations. Furthermore, this contribution shows that the investigation of these structural features leads to new computational techniques to calculate the specifications of interest: the emphasis is to derive approximation techniques with associated explicit convergence rates and formal error bounds.
Keywords: discrete-time Markov processes, PCTL model checking, infinite-horizon properties, Bellman equations, absorbing sets.

I. Tkachev is with the Delft Center for Systems & Control, Delft University of Technology, The Netherlands. Email: i.tkachev@tudelft.nl.
A. Abate is with the Department of Computer Sciences, University of Oxford, United Kingdom, and with the Delft Center for Systems & Control, Delft University of Technology, The Netherlands. Email: alessandro.abate@cs.ox.ac.uk.

## 1. Introduction

The use of formal verification notions and methods for dynamical systems has recently become an active inter-disciplinary area of research in systems and control theory [Tab09]. One of the most efficient techniques is model-checking, which aims at determining the satisfaction set of a given specification, i.e. the set of all states that initialize realizations verifying that specification. Probabilistic Computation Tree Logic (PCTL) is a modal logic which is widely used in formal verification to express specifications for discrete-time probabilistic processes [BK08, Chapter 10]. The special case of discrete-time Markov Chains (dt-MC) – models over discrete (countable) spaces – is well-studied in the literature and PCTL specifications can be verified over these models in an automatic manner by employing computationally advantageous probabilistic model checking techniques [HKNP06, KKZ05]. PCTL model checking has also been validated over numerous compelling applications [FKNP11].

The formal extension of PCTL to discrete-time Markov processes (dt-MP) over general (uncountable) state spaces has recently been discussed in [Hut05, RCSL10]. The latter work in particular has expressed the satisfaction set of a given PCTL specification as the level set of an associated state-dependent value function, and has further characterized the computation of such value function via dynamic programming (DP) [BS78]. Within PCTL, there is a clear distinction between finite-horizon specifications (the satisfiability of which depends on finite realizations of the system) and infinite-horizon specifications (those characterized over infinite paths). In the context of dt-MC with a finite state space, DP over a finite horizon is performed by iterative matrix multiplications, whereas DP over an infinite horizon is reduced to solving systems of linear equations. On the other hand, over a general state space the corresponding procedures – namely Bellman iterations and Bellman equations – involve integral operators. Recent work (see e.g. [APLS08]) has shown that explicit analytical solutions over uncountable state-spaces are not to be found in general, and has stressed the need for methods to compute value functions with any given precision.

In the context of dt-MP, the work in [Hut05] has put forward finite abstractions, where measures are approximated by monotone functions of sets. Although these abstractions are sound and upper and lower bounds for the expression of value functions have been derived [Hut05, Theorem 33], no method to tune them has been provided. Also, their tightness, usefulness, or possible triviality (i.e. conditions for the error bounds to be less than ) have not been addressed. The work in [RCSL10], in turn, has characterized PCTL specifications and their associated value functions with an emphasis on the issue of uniqueness of solutions of the related Bellman equations. The following questions have been left open to investigation:

1. how to compute finite-horizon value functions in PCTL with any given precision;

2. since in general value functions are not known exactly and satisfaction sets are expressed as level sets of these functions, how to verify nested PCTL formulae (namely, specifications where the satisfaction set for the first formula appears in the definition of a second one);

3. how to verify infinite-horizon PCTL specifications, particularly if the sufficient conditions for the uniqueness of solutions of Bellman equations in [RCSL10] are not satisfied.

With focus on 1), finite-horizon computations have recently received considerable attention. For discrete-time Stochastic Hybrid Systems (a class of dt-MP), the work in [AKLP10] has put forward finite abstraction techniques to perform DP iterations over corresponding finite state-space dt-MC. These results have been further sharpened in [SA11], where abstractions by state-space partitioning are obtained adaptively, in accordance to a specification-dependent error. In both works the explicit abstraction error grows linearly with the time horizon of the corresponding PCTL specification, which does not allow applying the developed methods directly to the verification of infinite-horizon properties.

The contribution of this work is hence focused on questions 2) and 3) and is twofold: the first goal is to complete the formal discussion on general state-space PCTL verification by dealing with nested formulae; the second goal (and the main task of this work) is to provide both analysis and computational tools for infinite-horizon PCTL specifications under conditions on the model that are as weak as possible and that are easy to verify.

In order to address question 2), we introduce the concepts of sub- and super-satisfaction sets for PCTL specifications, the characterization of which requires only approximate knowledge of the corresponding value functions. Specifically, we show how the sub- and super-satisfaction sets of a nested sub-formula propagate to the corresponding sets of the main formula: this is achieved by using monotonicity properties of corresponding value functions.

In order to tackle question 3), we extend and generalize recent results in [TA11, TA12], showing that the sufficient condition provided in [RCSL10] for the uniqueness of the solution of a Bellman equation is only satisfied if the solution is trivial in some sense. We further show that a weaker version of this condition is both necessary and sufficient if the dt-MP admits certain continuity properties. This result leads to novel techniques to solve Bellman equations whenever their solution is not unique, and provides approximation techniques with associated explicit convergence rates and error bounds. These techniques are based on the reduction of the infinite-horizon problem to a finite-horizon one, for which computational methods available in the literature [AKLP10, SA11] can be directly applied. We furthermore discuss the relationship between the issue of uniqueness of solution and the presence of absorbing sets over the (uncountable) state space: absorbing sets are shown to play a fundamental role for both the characterization and the computation of infinite-horizon PCTL properties.

The contribution is organized as follows. Section 2 introduces discrete-time Markov processes and PCTL specifications, and discusses the verification of nested PCTL formulae. Section 3 dives in depth into infinite-horizon problems. Section 4 provides two case studies to discuss the results and finally Section 5 concludes the work.

Throughout the article we use tools of measure theory and of functional analysis. The following references can be consulted: [Dur04] for probability theory, [Rev84] for Markov processes and [Rud87] for functional analysis and measure theory.

## 2. Markov processes and PCTL

### 2.1. Discrete-time Markov processes

Let be some measurable space and let be a stochastic kernel, so that is a non-negative measurable function for any set and is a probability measure on for any . The space of trajectories is denoted by (here ) and its product -algebra by . It follows from [Rev84, Theorem 2.8] that there exists a unique discrete-time Markov process (dt-MP) with the transition kernel , that is, for any there exists a unique probability measure on such that , and for any measurable set and any time epoch

 (2.1) Px(xn+1∈B|x0,x1,…,xn)=P(xn+1∈B|xn)=P(xn,B).

Equation (2.1) characterizes the Markov property and it indicates that the future of the process is independent of its past history , given its current value . As a result, any dt-MP can be characterized equivalently by the triple .

A familiar class of dt-MP is that of stochastic dynamical systems. If is a sequence of iid random variables and is a measurable map, then

 (2.2) xn+1=f(xn,n),x0=x∈X,

is always a Markov process characterized by a kernel

 Q(x,A):=ν({ξ∈\mdmathbbR:f(x,ξ)∈A})

where is the distribution of . Conversely, under some mild conditions on the structure of the state space, any dt-MP admits a dynamical representation as in (2.2), for an appropriate choice of the function [Kal97, Proposition 8.6]. However, theoretical studies of dt-MP, as well as the current article, usually employ the representation via stochastic kernels.

The reader interested in further discussions about modeling aspects of dt-MP is referred to [Mey08, Appendix A1]. Among other models related to dt-MP, Labeled Markov Processes (LMP) [dgjp04] are of interest as they embed non-determinism and allow for sub-stochastic transition kernels.

### 2.2. Probabilistic Computation Tree Logic (PCTL)

PCTL is a modal logic employed to characterize classes of temporal properties of dt-MC [BK08] and of dt-MP [Hut05, RCSL10]. Properties are expressed as formulae in PCTL and are constructed according to the grammar of this logic. The grammar is based on AP, the set of atomic propositions, which can be thought of as tags or labels associated to the states. Let and ; we write if the atomic proposition is valid at state . Since there is no substantial difference between and its satisfaction set , we define atomic propositions to be measurable subsets of , or equivalently , and require that . The grammar of PCTL is defined as follows. Atomic propositions are basic formulae that are used to build more complex formulae via logical rules. PCTL state formulae are subsets of , whereas path formulae are subsets of . More precisely:

• each atomic proposition is a formula with itself as its satisfaction set;

• if and are formulae, then so are and ;

• if is a path formula and , then is a (state) formula, where can be any symbol from the collection ;

• if and are formulae and , then , , and are path formulae.

The semantics of PCTL state formulae is given as follows:

 x⊨X for allx∈X x⊨A ⇔x∈A x⊨¬A ⇔x∈Ac:=X∖A x⊨A∧B ⇔x∈A∩B x⊨\mdmathbbP⋈p[Phi] ⇔Px(Phi)⋈p

With regards to path formulae, the meaning of (the next operator) is , thus if and only if . The two additional path formulae depend on the bounded until operator and on the unbounded until operator . In order to characterize them through subsets of , let us introduce for any set

 A:=inf{n≥0:xn∈A}

to be the first hitting time of a set over a realization . Clearly, is a random variable with values in . We define , so that whenever , which means that the path formula is satisfied over a trajectory for which holds at least once within the -step horizon, while is persistently valid until that moment. Similarly, for the infinite-horizon case, we define

 AUB:={B≤Ac,B<∞}.

To characterize satisfaction sets for until operators, we introduce the so called reach-avoid111Alternatively known as constrained reachability [BK08]. value functions: for any , let us define

 wn(x;A,B):=Px(AU≤nB),w(x;A,B):=Px(AUB),

which leads to expressing . Functions are measurable, thus all PCTL formulae are well-defined measurable subsets of and all path formulae are elements of [RCSL10].222Although the theory in [RCSL10] has been developed for models with carrying a topological structure, all the results on measurability hold without this requirement and as such they are also valid in the present instance. This work resorts to a topological structure over the state space only in Section 3.2.

Let us provide a few examples: if are PCTL formulae, then is a PCTL formula. Likewise, is a PCTL formula, since and . However, is not a PCTL formula, since the logical operation is defined for state formulae but not over path formulae. Furthermore, PCTL path formula is known as a reachability event for a given set and relates to a wide and important class of problems in systems and control [APLS08]. Its dual, the invariance (or safety) event , cannot be directly expressed in PCTL since the negation of path formulae is not allowed. On the other hand,

 Px(Ac>n)=1−wn(x;X,Ac),

thus one can define , where the symbol stands for , the symbol stands for etc. We denote the invariance value functions by

 (2.3) un(x;A):=1−wn(x;X,Ac),u(x;A):=1−w(x;X,Ac).

The results for reach-avoid and invariance given in this work can thus be directly exported to the reachability property. The latter represents also a crucial property for other types of logics, for instance linear temporal logic (LTL) [BK08, Chapter 5]. In particular, [AKM11] has argued that the verification of a subclass of LTL specifications over a dt-MP can be reduced to a reachability problem [AKM11, Theorem 4].

### 2.3. Nested PCTL properties

As mentioned in the introduction, it is in general not expected that the value functions and can be expressed explicitly. An alternative goal is the following [AKLP10]: given any precision level , find approximate functions and such that and , for all . Consider however the formula : if the value function can only be characterized approximately, what set should be considered to characterize ? And how could this set be used in the parent formula? To resolve this issue we need the following fact.

###### Proposition 1.

Let and be elements of and let . For all :

 wn(x;A,B)≤wn(x;A∗,B∗),w(x;A,B)≤w(x;A∗,B∗).
###### Proof.

Since the proof immediately follows as the probability measure is a monotonic function of sets for any . ∎

For a PCTL formula , we say that () is a subsatisfaction (supersatisfaction) set if (). Clearly, denotes a conservative set, the states of which also satisfy , while denotes a relaxed set: any state in does not satisfy either333The approach here is similar to three-valued approximations in [FLW06], [Hut05, Section 4], and [HPW09]..

As done above, let denote some abstract -approximations of and , respectively. Let us show as an example, how the formula can be verified. Since for all it holds that

 ^w(x;B,C)−δ≤w(x;B,C)≤^w(x,B,C)+δ,

it follows that implies , and that implies . As a result, if we denote , then the sets

 D∗:={x∈X:^w(x;B,C)≤p2−δ},D∗:={x∈X:^w(x;B,C)≤p2+δ}

represent sub- and super-satisfaction sets for . Finally, from Proposition 1 we obtain:

 E∗:={x∈X:^w(x;A,D∗)≥p1+δ},E∗:={x∈X:^w(x;A,D∗)≥p1−δ},

which are sub- and super-satisfaction sets for . The application of this procedure over formulae including the operator is direct, since is a monotonic function of a set-valued argument for any .

A general algorithm for the verification of nested formulae follows: given the ability to approximately compute value functions with a precision , find sub- and super-satisfaction sets for the sub-formulas on the lowest level (leaves) of a given formula tree, then use these sets to find sub- and super-satisfaction sets for higher-level formulae inductively, until the sub- and super-satisfaction sets for the given formula are found (at the root). Note the similarity between our approach and the three-valued approximations of PCTL.

## 3. Verification of infinite-horizon PCTL specifications

The goal of this section is to investigate the verification of infinite-horizon PCTL specifications and to provide methods to compute associated value functions with any given precision. For this purpose Section 3.1 introduces DP techniques to characterize the corresponding value functions via Bellman recursions and fixpoint equations, points out related issues in their evaluation and provides sufficient conditions for the precise reduction of infinite-horizon problems to finite-horizon ones. In Section 3.2, the concept of absorbing set is used to show that for a class of problems the aforementioned conditions are also necessary, and that they relate to the uniqueness of the solution of Bellman fixpoint equations. This result is further applied to derive methods to solve Bellman equations with non-unique solutions, both in the general case (which is done leveraging Lyapunov-like locally excessive functions – cfr. Section 3.3), and in the special case of integral kernels (where such functions are not needed – cfr. Section 3.4). The presented techniques depend on the characterization of absorbing sets, which is discussed in Section 3.5. The obtained results are further compared with approaches in the literature on dt-MP in Section 3.6.

### 3.1. Dynamic programming and Bellman equations

Let denote the space of all real-valued, bounded and measurable functions on . It is a Banach space with a norm given by for . An operator is called linear if

 J(αf+βg)=αJ(f)+βJ(g)

for any constants , and functions , . The quantity

 (3.1) ∥J∥:=sup∥f∥≤1∥Jf∥

is called the norm of the linear operator . We say that a linear operator is a contraction whenever it holds that . An important example of a linear operator associated to a dt-MP is the transition operator , which is induced by the kernel . The action of operator on function is given by the following formula:

 Pf(x):=∫Xf(y)P(x,dy).

Let us furthermore introduce an invariance operator , parameterized by a measurable set , and given by . Clearly, is also a linear operator and . Moreover, is a monotone operator, which means that for all functions and any set it holds that for all whenever it holds that for all .

As an abbreviation, for a function and a constant we further write ; a similar notation is used for any of the other symbols in the collection .

Let us introduce a DP procedure for until-like specifications in PCTL. Let be given sets (equivalently, state formulae in PCTL). From [RCSL10, SL10] it follows:

 (3.2) {wn+1(x;A,B)=1B(x)+IA∖Bwn(x;A,B),w0(x;A,B)=1B(x).

The computation in (3.2) involves iterations of the integral operator . Results in [AKLP10, SA11] allow one to compute a piece-wise constant function approximation , which is such that , where the constant depends on the quality of the state space partitioning (see e.g. [SA11, Theorem 4]). Thus, in the remainder of this work we assume that finite-horizon problems can be solved approximately and with any given precision by any of the techniques given in the literature, and instead focus on the reduction of infinite-horizon problems to finite-horizon ones.

For infinite-horizon problems, it holds that , where the limit is point-wise non-decreasing [RCSL10]. In [RCSL10, Lemma 5] the monotone convergence theorem is applied to , in order to show that the function solves the fixpoint Bellman equation

 (3.3) w(x;A,B)=1B(x)+IA∖Bw(x;A,B).

However the convergence of is not necessarily uniform. Moreover, equation (3.3) may have multiple solutions: since it is an affine equation, if it does not have a unique solution then it admits infinitely many, spanning an affine subspace of . To further look into this issue we leverage value functions for invariance. As discussed above, the until specification can be used to express the invariance over a given set . Using formulae (2.3) and (3.2) we obtain the following DP recursion

 (3.4) {un+1(x;A)=IAun(x;A),u0(x;A)=1A(x).

It easily follows that converges point-wise non-increasingly to function , thus

 (3.5) u(x;A)=IAu(x;A).

Clearly, the verification of the invariance specification inherits issues of non-uniform convergence and of non-uniqueness of the Bellman equation (3.5) from the until specification in (3.3). However, the Bellman equation for the invariance specification has the advantage of being linear and thus always admits the trivial solution . Moreover, the analysis of the affine equation on a linear space can be reduced to the analysis of its homogeneous (linear) version: dealing with (3.5) leads to finding methods for solving (3.3) as well.

###### Remark 1.

There exists a least fixed-point characterization for the infinite-horizon value functions [RCSL10, Lemma 6]: is the least non-negative solution of (3.3), i.e. if is any other non-negative solution of (3.3), then for all . As a result, is the largest solution of (3.5) not exceeding . Although such characterization adds little to the computation of and , it results in the useful fact that whenever is non-trivial, namely whenever is not identically equal to zero.

One sufficient condition for the uniqueness of the solution of (3.5) is given as follows: [RCSL10, Proposition 7], which in turn leads to the contractivity of the operator . While this condition may be easy to check, it can be restrictive: in this case (3.5) admits the unique solution . As a result, any invariance problem with a non-trivial solution will not satisfy this sufficient condition. It follows that the weaker condition , for some , is also sufficient for the uniqueness of the solution of (3.5). Let us introduce the quantities

 m(A):=inf{m≥0:∥um(⋅,A)∥<1},ρ(A):=∥∥um(A)(⋅,A)∥∥,

for any , where we set if . Note that both and are monotone functions on , i.e. if are measurable sets, then and . The quantity is discussed in more detail for the special case of Markov Chains in Section 3.4.

###### Proposition 2.

Let and denote for simplicity and . Then:

• if , then , and for all it holds that ;

• if are disjoint444In the following, for the sake of the simplicity the set-valued arguments of the reach-avoid value functions are assumed to be disjoint. This assumption does not affect the generality of the results, since and hence any reach-avoid problem can be always considered as a problem on disjoint sets. and , then for all

 (3.6) 0≤w(x;A,B)−wn(x;A,B)≤m1−ρ⌊nm⌋.
###### Proof.

For part (i), we have from (3.4) that , for all . Clearly, from the finiteness of and the definition of it follows that , so

 un(⋅;A)≤ρ⋅(IA)n−m1A(⋅)=ρun−m(⋅;A).

for . By induction we obtain that , so that

 u(⋅;A)=limn→∞un(⋅;A)=0.

For part (ii), we define functions . Clearly, it holds that and . Moreover, from the fact that and the monotonicity of the operator , we have that . It further follows that

as desired. ∎

As mentioned before, one goal of this section is to reduce a given infinite-horizon problem to a finite-horizon one, with the ability to tune the error incurred in this reduction. If , and since the right-hand side in (3.6) decreases exponentially fast with respect to , Proposition 2 provides a method to achieve this. In the following, the condition , for an appropriate set-valued argument, indicates that the corresponding infinite-horizon problem can be reduced (and thus solved).

It is worth mentioning that Proposition 2 elucidates the difficulty in the direct extension of the error bounds in [AKLP10, SA11] from finite- to infinite-horizon problems: the developed finite-horizon approximation techniques can be interpreted as providing a perturbation of the original stochastic kernel . Thus, they are tailored at rendering the one-step error (under the operator norm in (3.1)) as small as possible. However, in general a bound on the one-step error cannot be extended over an infinite time horizon, as the following argument shows. Let us consider the case where the solution of the invariance problem on a set for the dt-MP is non-trivial. We denote the corresponding value function as . It follows from Remark 1 that . Let be any probability measure on such that , and define , for . We have

for any function . Hence , so that it can be made arbitrarily small. On the other hand, if we denote by the solution of the invariance problem on for dt-MP , we obtain that . As a result, by Proposition 2, so that , regardless of how small is.

### 3.2. Absorbing and simple sets

From Proposition 2 it follows that the condition in particular implies the uniqueness of the solution of the corresponding Bellman equation. It turns out that under some continuity assumptions on the kernel this condition is also necessary. Before we proceed, we introduce the notion of absorbing set, which is crucial for further discussions.

###### Definition 1.

A set is called absorbing if , for all . If for there is an absorbing subset such that whenever is absorbing, then we say that is the largest absorbing subset of and write . The set is called simple if it does not have non-empty absorbing subsets, i.e. , and non-simple otherwise.

Clearly, the whole state space and the empty set are always absorbing, and if is a countable sequence of non-empty absorbing sets, then their union is absorbing and non-empty. However, it is by no means clear that exists for any given set , since may contain uncountably many absorbing subsets and their union may not be even measurable. Surprisingly, invariance value functions are useful to show that is always well-defined.

###### Lemma 1.

Let and denote for all , so that . Further, let , then for all it holds that and

 (3.7) An+1={x∈A:P(x,An)=1}.

The set admits the representation , i.e. it is the largest absorbing subset of . In particular, if then is simple.

###### Proof.

Let us first prove (3.7): for an arbitrary it holds that

 (3.8) P(x,A)≤1=un+1(x;A)=1A(x)∫Xun(y;A)P(x,dy)=∫Aun(y;A)P(x,dy).

Subtracting the right-hand side of (3.8) from the left-hand side, we obtain that

 ∫A(1−un(y;A))P(x,dy)=0

as it is non-positive from (3.8) and the integrand is non-negative. Due to the latter fact, we obtain that or equivalently .

Conversely, let be an arbitrary state that satisfies . Let us show that . Indeed,

 un+1(x;A)=∫Xun(y;A)P(x,dy)≥∫Anun(y;A)P(x,dy)=P(x,An)=1,

thus . As a result, we have shown that (3.7) holds true.

Since and , we obtain that . Furthermore, by induction it holds that for all . If for some , then and for all , hence . If , then for all , so that .

Suppose now that is a non-simple set and that is an arbitrary absorbing subset of . Clearly, it holds that for all , hence . Furthermore, if , then for any and it holds that , hence . This implies that is absorbing since

 P(x,A∞)=P(x,∞⋂n=0An)=limn→∞P(x,An)=1,

which leads to conclude that is the largest absorbing subset of . ∎

As it has been mentioned above, some continuity assumptions on the kernel are needed in order to sharpen the results. To do so, the state space needs to be endowed with a certain topological structure (see e.g. [HLL96]).

###### Definition 2.

A state space is called topological if is a Borel subset of a Polish (i.e. a metrizable, complete, and separable) space and if is a Borel -algebra of . A kernel on a topological space is called weakly continuous (or Feller) if the function is upper semi-continuous (u.s.c.) whenever is u.s.c. [HLL96, Appendix C].

A dt-MP is said to be weakly continuous whenever is a topological state space and is weakly continuous.

The next theorem shows that for a weakly continuous dt-MP, the infinite-horizon problem over a compact set can be directly reduced to the finite-horizon one (in the sense that ) if and only if the set is simple.

###### Theorem 1.

Let be a topological state space and be a compact set. If is weakly continuous then is a compact set and the following statements are equivalent:

1. ;

2. is a contraction on for some finite (contractivity);

3. equation (3.5) has a unique solution (uniqueness);

4. for all (triviality);

5. the set is simple: (simplicity).

###### Proof.

1) 2) Clearly, for any function it follows that for all states . Thus if , then is a contraction since

 ∥Im(A)+1Af∥≤∥f∥⋅∥Im(A)A1A∥=∥f∥⋅∥um(A)(⋅;A)∥≤ρ(A)∥f∥.

2) 3) If be a solution of (3.5), i.e. . By induction we have , which by contraction mapping theorem [HL89, Proposition A.1] implies the uniqueness of the fixpoint .

3) 4) follows from the linearity of (3.5) and 4) 5) from Lemma 1, so we only have to show that 5) 1). Suppose this is not true, i.e. but is simple. It follows that for all . Since is compact and is metrizable, is closed and hence is u.s.c. Hence is u.s.c. for all by the weak continuity of , which implies that all sets are compact. Moreover, they are not empty and so their intersection is compact and non-empty, which leads to a contradiction. ∎

###### Remark 2.

Within the main goal of reducing infinite-horizon problems over a set to finite-horizon ones, let us remark that numerical methods for finite-horizon problems leading to the computation of PCTL value functions with any given precision have been developed, up to our knowledge, only for compact subsets of finite-dimensional metric spaces [AKLP10, SA11] – this aligns with the assumption raised for Theorem 1. Also, conditions required on the kernel in loc. cit. are stronger than the weak continuity raised above. Taking all of this into account, the assumptions in Theorem 1 are rather mild. Furthermore, some of the relations in the theorem are true under even weaker conditions: we postpone the discussion of these facts to Section A (Appendix).

###### Remark 3.

It follows directly from Theorem 1 that if then is a contraction and furthermore, .

### 3.3. A decomposition technique

Although Theorem 1 is stated in terms of value functions for the invariance problem, its application to the issue of uniqueness of the solution of a reach-avoid problem is direct, since (3.5) is a homogeneous version of (3.3). As a result, if the dt-MP is weakly continuous, sets , are disjoint and is compact and simple, then and the reach-avoid problem can be solved. Thus, the next goal is to study the case of a non-simple set . For this objective the characterization given in Theorem 1 is again useful. We proceed assuming that the of a given set is known, and leave the discussion on the characterization of the of a given set and the verification of the simplicity of a set to Section 3.5.

If is non-simple, the main issue preventing an efficient solution of the problem is the presence of an absorbing subset . This leads to the lack of contractivity of the operator and to the non-uniqueness of the solution of (3.3). Intuitively, if we were to remove some neighborhood , then we would expect that , so that a related problem can be solved on . Moreover, recall that the solution of the original problem on is known: for all , since such states initialize trajectories that never reach the set . The following result relates the solutions of the two problems:

###### Lemma 2 (Decomposition technique).

Let sets be disjoint, and let the set be such that the invariance value function . Then is the unique solution of the corresponding Bellman equation

 (3.9) w(x;A∖C,B)=1B(x)+IA∖Cw(x;A∖C,B),

and for all the following holds:

 (3.10) 0≤w(x;A,B)−w(x;A∖C,B)≤supy∈Cw(y;A,B).
###### Proof.

As an abbreviation, let us denote and and let us partition the event space by the following four disjoint hypotheses555 Alternatively, these hypotheses can be defined using PCTL framework, and are given by path formulae , , and . :

 H1:={2<1,2<∞},H2:={1=∞,2=∞},
 H3:={1<2,1<∞,1=B},H4:={1<2,1<∞,1=C}.

Recall that , thus

 w(x;A,B)=4∑i=1Px({B<2,B<∞}∩Hi).

Note that the first term is zero since clearly . The second term vanishes because . Since the third term equals to , which leaves only the fourth term to be studied. Let be any such that and define a measure on by

 x(D):=Px(xC∈D∣∣H4),D∈B,

so that clearly . For such fixed it holds that

 0≤Px({B<2,B<∞}∩H4) =Px({B<2,B<∞}|H4)Px(H4) =(∫Cw(y;A,B)x(dy))⋅w(x;A,C) ≤supy∈Cw(x;A,B).

The same bounds clearly hold in the alternative case .

Finally, it follows that is the unique solution of the corresponding Bellman equation (3.9) from (see Proposition 4 in Section A). ∎

###### Corollary 1.

[From Lemma 2] Let be a weakly continuous dt-MP and let be disjoint and such that is a compact, non-simple set. Let be an open neighborhood of in the subspace topology of . Then (3.10) holds for all , and .

###### Proof.

Since is open in , the set is a closed subset of a compact set and thus itself compact. From the inclusions it follows that is simple, hence Theorem 1 ensures that all the conditions of Lemma 2 are satisfied. ∎

In order to render the result in Corollary 1 useful for the computation of the infinite-horizon reach-avoid value function, we should provide a method to choose an open neighborhood of , such that , where is a given precision level. We use the theory of excessive functions [SRG08] to achieve this goal.

###### Definition 3.

Given a function , the excessive set of is . If , i.e. if for all , then the function is called excessive.

The relation between excessive functions and infinite-horizon invariance is given via Doob’s inequality [SRG08]: if is an excessive, non-negative function, then

 (3.11) Px{supn≥0g(xn)≥δ}≤g(x)δ.

for all . The inequality (3.11) can be rewritten via the invariance value function:

 (3.12) u(x;{g<δ})≥1−g(x)δ.

Excessive functions for stochastic systems are akin to Lyapunov functions for deterministic systems, since they are characterized by decreasing behavior along the dynamics of the process, as the inequality suggests666From the definition of is follows that , where denotes the expectation with respect to . Thus the condition means that the expected value of the function at the next time step is bounded by its current value, so that the function does not increase on average along realizations of the dt-MP. Thus the function is excessive if and only if the process is a -supermartingale, for all [PS06, p.20].. As is the case with Lyapunov functions for deterministic systems, it is non trivial to find excessive functions. However, it is possible to relax the assumption on global excessivity and to employ a local version of Doob’s inequality.

###### Lemma 3.

[Kus67, Theorem 12] Let be a non-negative function such that for some it holds that . Whenever , it follows that

 (3.13) Px{supn≥0g(xn)≥δ}≤g(x)δ.

The idea behind the proof of this lemma is to consider a set . The related invariance value function does not depend on for , where it is simply equal to zero (recall that all the integrals in the DP recursion (3.4) are equivalently taken over the set ). As a result, exclusively the dynamics within the set are important for the process. Note also that for one trivially has (3.13) as in such case the term in the right-hand side is greater or equal than .

###### Definition 4.

For a topological state space we say that a non-negative continuous function is -locally excessive on the set if and for some real number it holds that and .

###### Theorem 2.

Let be a weakly continuous dt-MP and let be disjoint and such that is a compact, non-simple set. If there exists a -locally excessive function on , then for any it holds that and that

 (3.14) 0≤w(x;A,B)−w(x;A∖{g<εδ},B)≤ε.
###### Proof.

First, we show that for any , if then . Indeed, as , by Lemma 3 we have that for all , so

 u(x;A)≥1−g(x)δ

for all , which follows from . Since , then

 w(x;X,Ac)≤g(x)δ

for all , and since and , from Proposition 1 it follows that . As a result, for any it holds that .

Second, let us fix any and denote . Clearly, and the set is open in since is continuous on . The statement of the theorem then follows from Corollary 1. ∎

### 3.4. Integral kernels and discrete-space Markov Chains

From Theorem 2 it follows that for weakly continuous dt-MPs a reach-avoid problem on a non-simple set can be solved if an appropriate locally excessive function is found. For a known and studied subclass of these processes the problem can be solved even without resorting to such functions. We write if is an integral kernel with a basis and a density , namely when is a -finite non-negative measure, the function is jointly measurable, and for any it holds that

 P(x,A)=∫Ap(x,y)μ(dy).

Furthermore, we raise the following assumption, which generalizes the one used for related studies over the finite horizon [AKLP10, SA11].

###### Assumption 1.

For a subset of a topological state space assume that the function is continuous on for any , .

Let us mention some sufficient conditions for the Assumption 1 to hold true for integral kernels. It follows from [HLL96, Example C.6] that whenever is a continuous function on the set for all , then Assumption 1 is satisfied for the set . It is thus milder than aforementioned assumptions of [AKLP10, SA11] where the stronger Lipschitz continuity is required instead.

Before we prove the main result, we need the following lemma that connects the condition with an important notion of the uniform transitivity [MT93].

###### Lemma 4.

Let be a dt-MP. Suppose that the set is uniformly transient, i.e. there exists such that for all . Then .

###### Proof.

Suppose that , then for any there exists a point such that . Clearly, for any it further holds that

 uk(xn;A)≥un(xn;A)≥12.

Note, that for any non-negative function and for any it holds that for all . As a result:

 ∞∑n=0Pn1A<