Characterization and computation of infinitehorizon specifications over Markov processes
Abstract.
This work is devoted to the formal verification of specifications over general discretetime Markov processes, with an emphasis on infinitehorizon properties.
These properties, formulated in a modal logic known as PCTL, can be expressed through value functions defined over the state space of the process.
The main goal is to understand how structural features of the model (primarily the presence of absorbing sets) influence the uniqueness of the solutions of corresponding Bellman equations. Furthermore, this contribution shows that the investigation of these structural features leads to new computational techniques to calculate the specifications of interest: the emphasis is to derive approximation techniques with associated explicit convergence rates and formal error bounds.
Keywords:
discretetime Markov processes,
PCTL model checking,
infinitehorizon properties,
Bellman equations,
absorbing sets.
A. Abate is with the Department of Computer Sciences, University of Oxford, United Kingdom, and with the Delft Center for Systems & Control, Delft University of Technology, The Netherlands. Email: alessandro.abate@cs.ox.ac.uk.
1. Introduction
The use of formal verification notions and methods for dynamical systems has recently become an active interdisciplinary area of research in systems and control theory [Tab09]. One of the most efficient techniques is modelchecking, which aims at determining the satisfaction set of a given specification, i.e. the set of all states that initialize realizations verifying that specification. Probabilistic Computation Tree Logic (PCTL) is a modal logic which is widely used in formal verification to express specifications for discretetime probabilistic processes [BK08, Chapter 10]. The special case of discretetime Markov Chains (dtMC) – models over discrete (countable) spaces – is wellstudied in the literature and PCTL specifications can be verified over these models in an automatic manner by employing computationally advantageous probabilistic model checking techniques [HKNP06, KKZ05]. PCTL model checking has also been validated over numerous compelling applications [FKNP11].
The formal extension of PCTL to discretetime Markov processes (dtMP) over general (uncountable) state spaces has recently been discussed in [Hut05, RCSL10]. The latter work in particular has expressed the satisfaction set of a given PCTL specification as the level set of an associated statedependent value function, and has further characterized the computation of such value function via dynamic programming (DP) [BS78]. Within PCTL, there is a clear distinction between finitehorizon specifications (the satisfiability of which depends on finite realizations of the system) and infinitehorizon specifications (those characterized over infinite paths). In the context of dtMC with a finite state space, DP over a finite horizon is performed by iterative matrix multiplications, whereas DP over an infinite horizon is reduced to solving systems of linear equations. On the other hand, over a general state space the corresponding procedures – namely Bellman iterations and Bellman equations – involve integral operators. Recent work (see e.g. [APLS08]) has shown that explicit analytical solutions over uncountable statespaces are not to be found in general, and has stressed the need for methods to compute value functions with any given precision.
In the context of dtMP, the work in [Hut05] has put forward finite abstractions, where measures are approximated by monotone functions of sets. Although these abstractions are sound and upper and lower bounds for the expression of value functions have been derived [Hut05, Theorem 33], no method to tune them has been provided. Also, their tightness, usefulness, or possible triviality (i.e. conditions for the error bounds to be less than ) have not been addressed. The work in [RCSL10], in turn, has characterized PCTL specifications and their associated value functions with an emphasis on the issue of uniqueness of solutions of the related Bellman equations. The following questions have been left open to investigation:

how to compute finitehorizon value functions in PCTL with any given precision;

since in general value functions are not known exactly and satisfaction sets are expressed as level sets of these functions, how to verify nested PCTL formulae (namely, specifications where the satisfaction set for the first formula appears in the definition of a second one);

how to verify infinitehorizon PCTL specifications, particularly if the sufficient conditions for the uniqueness of solutions of Bellman equations in [RCSL10] are not satisfied.
With focus on 1), finitehorizon computations have recently received considerable attention. For discretetime Stochastic Hybrid Systems (a class of dtMP), the work in [AKLP10] has put forward finite abstraction techniques to perform DP iterations over corresponding finite statespace dtMC. These results have been further sharpened in [SA11], where abstractions by statespace partitioning are obtained adaptively, in accordance to a specificationdependent error. In both works the explicit abstraction error grows linearly with the time horizon of the corresponding PCTL specification, which does not allow applying the developed methods directly to the verification of infinitehorizon properties.
The contribution of this work is hence focused on questions 2) and 3) and is twofold: the first goal is to complete the formal discussion on general statespace PCTL verification by dealing with nested formulae; the second goal (and the main task of this work) is to provide both analysis and computational tools for infinitehorizon PCTL specifications under conditions on the model that are as weak as possible and that are easy to verify.
In order to address question 2), we introduce the concepts of sub and supersatisfaction sets for PCTL specifications, the characterization of which requires only approximate knowledge of the corresponding value functions. Specifically, we show how the sub and supersatisfaction sets of a nested subformula propagate to the corresponding sets of the main formula: this is achieved by using monotonicity properties of corresponding value functions.
In order to tackle question 3), we extend and generalize recent results in [TA11, TA12], showing that the sufficient condition provided in [RCSL10] for the uniqueness of the solution of a Bellman equation is only satisfied if the solution is trivial in some sense. We further show that a weaker version of this condition is both necessary and sufficient if the dtMP admits certain continuity properties. This result leads to novel techniques to solve Bellman equations whenever their solution is not unique, and provides approximation techniques with associated explicit convergence rates and error bounds. These techniques are based on the reduction of the infinitehorizon problem to a finitehorizon one, for which computational methods available in the literature [AKLP10, SA11] can be directly applied. We furthermore discuss the relationship between the issue of uniqueness of solution and the presence of absorbing sets over the (uncountable) state space: absorbing sets are shown to play a fundamental role for both the characterization and the computation of infinitehorizon PCTL properties.
The contribution is organized as follows. Section 2 introduces discretetime Markov processes and PCTL specifications, and discusses the verification of nested PCTL formulae. Section 3 dives in depth into infinitehorizon problems. Section 4 provides two case studies to discuss the results and finally Section 5 concludes the work.
2. Markov processes and PCTL
2.1. Discretetime Markov processes
Let be some measurable space and let be a stochastic kernel, so that is a nonnegative measurable function for any set and is a probability measure on for any . The space of trajectories is denoted by (here ) and its product algebra by . It follows from [Rev84, Theorem 2.8] that there exists a unique discretetime Markov process (dtMP) with the transition kernel , that is, for any there exists a unique probability measure on such that , and for any measurable set and any time epoch
(2.1) 
Equation (2.1) characterizes the Markov property and it indicates that the future of the process is independent of its past history , given its current value . As a result, any dtMP can be characterized equivalently by the triple .
A familiar class of dtMP is that of stochastic dynamical systems. If is a sequence of iid random variables and is a measurable map, then
(2.2) 
is always a Markov process characterized by a kernel
where is the distribution of . Conversely, under some mild conditions on the structure of the state space, any dtMP admits a dynamical representation as in (2.2), for an appropriate choice of the function [Kal97, Proposition 8.6]. However, theoretical studies of dtMP, as well as the current article, usually employ the representation via stochastic kernels.
The reader interested in further discussions about modeling aspects of dtMP is referred to [Mey08, Appendix A1]. Among other models related to dtMP, Labeled Markov Processes (LMP) [dgjp04] are of interest as they embed nondeterminism and allow for substochastic transition kernels.
2.2. Probabilistic Computation Tree Logic (PCTL)
PCTL is a modal logic employed to characterize classes of temporal properties of dtMC [BK08] and of dtMP [Hut05, RCSL10]. Properties are expressed as formulae in PCTL and are constructed according to the grammar of this logic. The grammar is based on AP, the set of atomic propositions, which can be thought of as tags or labels associated to the states. Let and ; we write if the atomic proposition is valid at state . Since there is no substantial difference between and its satisfaction set , we define atomic propositions to be measurable subsets of , or equivalently , and require that . The grammar of PCTL is defined as follows. Atomic propositions are basic formulae that are used to build more complex formulae via logical rules. PCTL state formulae are subsets of , whereas path formulae are subsets of . More precisely:

each atomic proposition is a formula with itself as its satisfaction set;

if and are formulae, then so are and ;

if is a path formula and , then is a (state) formula, where can be any symbol from the collection ;

if and are formulae and , then , , and are path formulae.
The semantics of PCTL state formulae is given as follows:
With regards to path formulae, the meaning of (the next operator) is , thus if and only if . The two additional path formulae depend on the bounded until operator and on the unbounded until operator . In order to characterize them through subsets of , let us introduce for any set
to be the first hitting time of a set over a realization . Clearly, is a random variable with values in . We define , so that whenever , which means that the path formula is satisfied over a trajectory for which holds at least once within the step horizon, while is persistently valid until that moment. Similarly, for the infinitehorizon case, we define
To characterize satisfaction sets for until operators, we introduce the so called reachavoid^{1}^{1}1Alternatively known as constrained reachability [BK08]. value functions: for any , let us define
which leads to expressing . Functions are measurable, thus all PCTL formulae are welldefined measurable subsets of and all path formulae are elements of [RCSL10].^{2}^{2}2Although the theory in [RCSL10] has been developed for models with carrying a topological structure, all the results on measurability hold without this requirement and as such they are also valid in the present instance. This work resorts to a topological structure over the state space only in Section 3.2.
Let us provide a few examples: if are PCTL formulae, then is a PCTL formula. Likewise, is a PCTL formula, since and . However, is not a PCTL formula, since the logical operation is defined for state formulae but not over path formulae. Furthermore, PCTL path formula is known as a reachability event for a given set and relates to a wide and important class of problems in systems and control [APLS08]. Its dual, the invariance (or safety) event , cannot be directly expressed in PCTL since the negation of path formulae is not allowed. On the other hand,
thus one can define , where the symbol stands for , the symbol stands for etc. We denote the invariance value functions by
(2.3) 
The results for reachavoid and invariance given in this work can thus be directly exported to the reachability property. The latter represents also a crucial property for other types of logics, for instance linear temporal logic (LTL) [BK08, Chapter 5]. In particular, [AKM11] has argued that the verification of a subclass of LTL specifications over a dtMP can be reduced to a reachability problem [AKM11, Theorem 4].
2.3. Nested PCTL properties
As mentioned in the introduction, it is in general not expected that the value functions and can be expressed explicitly. An alternative goal is the following [AKLP10]: given any precision level , find approximate functions and such that and , for all . Consider however the formula : if the value function can only be characterized approximately, what set should be considered to characterize ? And how could this set be used in the parent formula? To resolve this issue we need the following fact.
Proposition 1.
Let and be elements of and let . For all :
Proof.
Since the proof immediately follows as the probability measure is a monotonic function of sets for any . ∎
For a PCTL formula , we say that () is a subsatisfaction (supersatisfaction) set if (). Clearly, denotes a conservative set, the states of which also satisfy , while denotes a relaxed set: any state in does not satisfy either^{3}^{3}3The approach here is similar to threevalued approximations in [FLW06], [Hut05, Section 4], and [HPW09]..
As done above, let denote some abstract approximations of and , respectively. Let us show as an example, how the formula can be verified. Since for all it holds that
it follows that implies , and that implies . As a result, if we denote , then the sets
represent sub and supersatisfaction sets for . Finally, from Proposition 1 we obtain:
which are sub and supersatisfaction sets for . The application of this procedure over formulae including the operator is direct, since is a monotonic function of a setvalued argument for any .
A general algorithm for the verification of nested formulae follows: given the ability to approximately compute value functions with a precision , find sub and supersatisfaction sets for the subformulas on the lowest level (leaves) of a given formula tree, then use these sets to find sub and supersatisfaction sets for higherlevel formulae inductively, until the sub and supersatisfaction sets for the given formula are found (at the root). Note the similarity between our approach and the threevalued approximations of PCTL.
3. Verification of infinitehorizon PCTL specifications
The goal of this section is to investigate the verification of infinitehorizon PCTL specifications and to provide methods to compute associated value functions with any given precision. For this purpose Section 3.1 introduces DP techniques to characterize the corresponding value functions via Bellman recursions and fixpoint equations, points out related issues in their evaluation and provides sufficient conditions for the precise reduction of infinitehorizon problems to finitehorizon ones. In Section 3.2, the concept of absorbing set is used to show that for a class of problems the aforementioned conditions are also necessary, and that they relate to the uniqueness of the solution of Bellman fixpoint equations. This result is further applied to derive methods to solve Bellman equations with nonunique solutions, both in the general case (which is done leveraging Lyapunovlike locally excessive functions – cfr. Section 3.3), and in the special case of integral kernels (where such functions are not needed – cfr. Section 3.4). The presented techniques depend on the characterization of absorbing sets, which is discussed in Section 3.5. The obtained results are further compared with approaches in the literature on dtMP in Section 3.6.
3.1. Dynamic programming and Bellman equations
Let denote the space of all realvalued, bounded and measurable functions on . It is a Banach space with a norm given by for . An operator is called linear if
for any constants , and functions , . The quantity
(3.1) 
is called the norm of the linear operator . We say that a linear operator is a contraction whenever it holds that . An important example of a linear operator associated to a dtMP is the transition operator , which is induced by the kernel . The action of operator on function is given by the following formula:
Let us furthermore introduce an invariance operator , parameterized by a measurable set , and given by . Clearly, is also a linear operator and . Moreover, is a monotone operator, which means that for all functions and any set it holds that for all whenever it holds that for all .
As an abbreviation, for a function and a constant we further write ; a similar notation is used for any of the other symbols in the collection .
Let us introduce a DP procedure for untillike specifications in PCTL. Let be given sets (equivalently, state formulae in PCTL). From [RCSL10, SL10] it follows:
(3.2) 
The computation in (3.2) involves iterations of the integral operator . Results in [AKLP10, SA11] allow one to compute a piecewise constant function approximation , which is such that , where the constant depends on the quality of the state space partitioning (see e.g. [SA11, Theorem 4]). Thus, in the remainder of this work we assume that finitehorizon problems can be solved approximately and with any given precision by any of the techniques given in the literature, and instead focus on the reduction of infinitehorizon problems to finitehorizon ones.
For infinitehorizon problems, it holds that , where the limit is pointwise nondecreasing [RCSL10]. In [RCSL10, Lemma 5] the monotone convergence theorem is applied to , in order to show that the function solves the fixpoint Bellman equation
(3.3) 
However the convergence of is not necessarily uniform. Moreover, equation (3.3) may have multiple solutions: since it is an affine equation, if it does not have a unique solution then it admits infinitely many, spanning an affine subspace of . To further look into this issue we leverage value functions for invariance. As discussed above, the until specification can be used to express the invariance over a given set . Using formulae (2.3) and (3.2) we obtain the following DP recursion
(3.4) 
It easily follows that converges pointwise nonincreasingly to function , thus
(3.5) 
Clearly, the verification of the invariance specification inherits issues of nonuniform convergence and of nonuniqueness of the Bellman equation (3.5) from the until specification in (3.3). However, the Bellman equation for the invariance specification has the advantage of being linear and thus always admits the trivial solution . Moreover, the analysis of the affine equation on a linear space can be reduced to the analysis of its homogeneous (linear) version: dealing with (3.5) leads to finding methods for solving (3.3) as well.
Remark 1.
There exists a least fixedpoint characterization for the infinitehorizon value functions [RCSL10, Lemma 6]: is the least nonnegative solution of (3.3), i.e. if is any other nonnegative solution of (3.3), then for all . As a result, is the largest solution of (3.5) not exceeding . Although such characterization adds little to the computation of and , it results in the useful fact that whenever is nontrivial, namely whenever is not identically equal to zero.
One sufficient condition for the uniqueness of the solution of (3.5) is given as follows: [RCSL10, Proposition 7], which in turn leads to the contractivity of the operator . While this condition may be easy to check, it can be restrictive: in this case (3.5) admits the unique solution . As a result, any invariance problem with a nontrivial solution will not satisfy this sufficient condition. It follows that the weaker condition , for some , is also sufficient for the uniqueness of the solution of (3.5). Let us introduce the quantities
for any , where we set if . Note that both and are monotone functions on , i.e. if are measurable sets, then and . The quantity is discussed in more detail for the special case of Markov Chains in Section 3.4.
Proposition 2.
Let and denote for simplicity and . Then:

if , then , and for all it holds that ;

if are disjoint^{4}^{4}4In the following, for the sake of the simplicity the setvalued arguments of the reachavoid value functions are assumed to be disjoint. This assumption does not affect the generality of the results, since and hence any reachavoid problem can be always considered as a problem on disjoint sets. and , then for all
(3.6)
Proof.
For part (i), we have from (3.4) that , for all . Clearly, from the finiteness of and the definition of it follows that , so
for . By induction we obtain that , so that
For part (ii), we define functions . Clearly, it holds that and . Moreover, from the fact that and the monotonicity of the operator , we have that . It further follows that
as desired. ∎
As mentioned before, one goal of this section is to reduce a given infinitehorizon problem to a finitehorizon one, with the ability to tune the error incurred in this reduction. If , and since the righthand side in (3.6) decreases exponentially fast with respect to , Proposition 2 provides a method to achieve this. In the following, the condition , for an appropriate setvalued argument, indicates that the corresponding infinitehorizon problem can be reduced (and thus solved).
It is worth mentioning that Proposition 2 elucidates the difficulty in the direct extension of the error bounds in [AKLP10, SA11] from finite to infinitehorizon problems: the developed finitehorizon approximation techniques can be interpreted as providing a perturbation of the original stochastic kernel . Thus, they are tailored at rendering the onestep error (under the operator norm in (3.1)) as small as possible. However, in general a bound on the onestep error cannot be extended over an infinite time horizon, as the following argument shows. Let us consider the case where the solution of the invariance problem on a set for the dtMP is nontrivial. We denote the corresponding value function as . It follows from Remark 1 that . Let be any probability measure on such that , and define , for . We have
for any function . Hence , so that it can be made arbitrarily small. On the other hand, if we denote by the solution of the invariance problem on for dtMP , we obtain that . As a result, by Proposition 2, so that , regardless of how small is.
3.2. Absorbing and simple sets
From Proposition 2 it follows that the condition in particular implies the uniqueness of the solution of the corresponding Bellman equation. It turns out that under some continuity assumptions on the kernel this condition is also necessary. Before we proceed, we introduce the notion of absorbing set, which is crucial for further discussions.
Definition 1.
A set is called absorbing if , for all . If for there is an absorbing subset such that whenever is absorbing, then we say that is the largest absorbing subset of and write . The set is called simple if it does not have nonempty absorbing subsets, i.e. , and nonsimple otherwise.
Clearly, the whole state space and the empty set are always absorbing, and if is a countable sequence of nonempty absorbing sets, then their union is absorbing and nonempty. However, it is by no means clear that exists for any given set , since may contain uncountably many absorbing subsets and their union may not be even measurable. Surprisingly, invariance value functions are useful to show that is always welldefined.
Lemma 1.
Let and denote for all , so that . Further, let , then for all it holds that and
(3.7) 
The set admits the representation , i.e. it is the largest absorbing subset of . In particular, if then is simple.
Proof.
Let us first prove (3.7): for an arbitrary it holds that
(3.8) 
Subtracting the righthand side of (3.8) from the lefthand side, we obtain that
as it is nonpositive from (3.8) and the integrand is nonnegative. Due to the latter fact, we obtain that or equivalently .
Conversely, let be an arbitrary state that satisfies . Let us show that . Indeed,
thus . As a result, we have shown that (3.7) holds true.
Since and , we obtain that . Furthermore, by induction it holds that for all . If for some , then and for all , hence . If , then for all , so that .
Suppose now that is a nonsimple set and that is an arbitrary absorbing subset of . Clearly, it holds that for all , hence . Furthermore, if , then for any and it holds that , hence . This implies that is absorbing since
which leads to conclude that is the largest absorbing subset of . ∎
As it has been mentioned above, some continuity assumptions on the kernel are needed in order to sharpen the results. To do so, the state space needs to be endowed with a certain topological structure (see e.g. [HLL96]).
Definition 2.
A state space is called topological if is a Borel subset of a Polish (i.e. a metrizable, complete, and separable) space and if is a Borel algebra of . A kernel on a topological space is called weakly continuous (or Feller) if the function is upper semicontinuous (u.s.c.) whenever is u.s.c. [HLL96, Appendix C].
A dtMP is said to be weakly continuous whenever is a topological state space and is weakly continuous.
The next theorem shows that for a weakly continuous dtMP, the infinitehorizon problem over a compact set can be directly reduced to the finitehorizon one (in the sense that ) if and only if the set is simple.
Theorem 1.
Let be a topological state space and be a compact set. If is weakly continuous then is a compact set and the following statements are equivalent:

;

is a contraction on for some finite (contractivity);

equation (3.5) has a unique solution (uniqueness);

for all (triviality);

the set is simple: (simplicity).
Proof.
1) 2) Clearly, for any function it follows that for all states . Thus if , then is a contraction since
2) 3) If be a solution of (3.5), i.e. . By induction we have , which by contraction mapping theorem [HL89, Proposition A.1] implies the uniqueness of the fixpoint .
3) 4) follows from the linearity of (3.5) and 4) 5) from Lemma 1, so we only have to show that 5) 1). Suppose this is not true, i.e. but is simple. It follows that for all . Since is compact and is metrizable, is closed and hence is u.s.c. Hence is u.s.c. for all by the weak continuity of , which implies that all sets are compact. Moreover, they are not empty and so their intersection is compact and nonempty, which leads to a contradiction. ∎
Remark 2.
Within the main goal of reducing infinitehorizon problems over a set to finitehorizon ones, let us remark that numerical methods for finitehorizon problems leading to the computation of PCTL value functions with any given precision have been developed, up to our knowledge, only for compact subsets of finitedimensional metric spaces [AKLP10, SA11] – this aligns with the assumption raised for Theorem 1. Also, conditions required on the kernel in loc. cit. are stronger than the weak continuity raised above. Taking all of this into account, the assumptions in Theorem 1 are rather mild. Furthermore, some of the relations in the theorem are true under even weaker conditions: we postpone the discussion of these facts to Section A (Appendix).
Remark 3.
It follows directly from Theorem 1 that if then is a contraction and furthermore, .
3.3. A decomposition technique
Although Theorem 1 is stated in terms of value functions for the invariance problem, its application to the issue of uniqueness of the solution of a reachavoid problem is direct, since (3.5) is a homogeneous version of (3.3). As a result, if the dtMP is weakly continuous, sets , are disjoint and is compact and simple, then and the reachavoid problem can be solved. Thus, the next goal is to study the case of a nonsimple set . For this objective the characterization given in Theorem 1 is again useful. We proceed assuming that the of a given set is known, and leave the discussion on the characterization of the of a given set and the verification of the simplicity of a set to Section 3.5.
If is nonsimple, the main issue preventing an efficient solution of the problem is the presence of an absorbing subset . This leads to the lack of contractivity of the operator and to the nonuniqueness of the solution of (3.3). Intuitively, if we were to remove some neighborhood , then we would expect that , so that a related problem can be solved on . Moreover, recall that the solution of the original problem on is known: for all , since such states initialize trajectories that never reach the set . The following result relates the solutions of the two problems:
Lemma 2 (Decomposition technique).
Let sets be disjoint, and let the set be such that the invariance value function . Then is the unique solution of the corresponding Bellman equation
(3.9) 
and for all the following holds:
(3.10) 
Proof.
As an abbreviation, let us denote and and let us partition the event space by the following four disjoint hypotheses^{5}^{5}5 Alternatively, these hypotheses can be defined using PCTL framework, and are given by path formulae , , and . :
Recall that , thus
Note that the first term is zero since clearly . The second term vanishes because . Since the third term equals to , which leaves only the fourth term to be studied. Let be any such that and define a measure on by
so that clearly . For such fixed it holds that
The same bounds clearly hold in the alternative case .
Corollary 1.
Proof.
In order to render the result in Corollary 1 useful for the computation of the infinitehorizon reachavoid value function, we should provide a method to choose an open neighborhood of , such that , where is a given precision level. We use the theory of excessive functions [SRG08] to achieve this goal.
Definition 3.
Given a function , the excessive set of is . If , i.e. if for all , then the function is called excessive.
The relation between excessive functions and infinitehorizon invariance is given via Doob’s inequality [SRG08]: if is an excessive, nonnegative function, then
(3.11) 
for all . The inequality (3.11) can be rewritten via the invariance value function:
(3.12) 
Excessive functions for stochastic systems are akin to Lyapunov functions for deterministic systems, since they are characterized by decreasing behavior along the dynamics of the process, as the inequality suggests^{6}^{6}6From the definition of is follows that , where denotes the expectation with respect to . Thus the condition means that the expected value of the function at the next time step is bounded by its current value, so that the function does not increase on average along realizations of the dtMP. Thus the function is excessive if and only if the process is a supermartingale, for all [PS06, p.20].. As is the case with Lyapunov functions for deterministic systems, it is non trivial to find excessive functions. However, it is possible to relax the assumption on global excessivity and to employ a local version of Doob’s inequality.
Lemma 3.
[Kus67, Theorem 12] Let be a nonnegative function such that for some it holds that . Whenever , it follows that
(3.13) 
The idea behind the proof of this lemma is to consider a set . The related invariance value function does not depend on for , where it is simply equal to zero (recall that all the integrals in the DP recursion (3.4) are equivalently taken over the set ). As a result, exclusively the dynamics within the set are important for the process. Note also that for one trivially has (3.13) as in such case the term in the righthand side is greater or equal than .
Definition 4.
For a topological state space we say that a nonnegative continuous function is locally excessive on the set if and for some real number it holds that and .
Theorem 2.
Let be a weakly continuous dtMP and let be disjoint and such that is a compact, nonsimple set. If there exists a locally excessive function on , then for any it holds that and that
(3.14) 
Proof.
First, we show that for any , if then . Indeed, as , by Lemma 3 we have that for all , so
for all , which follows from . Since , then
for all , and since and , from Proposition 1 it follows that . As a result, for any it holds that .
Second, let us fix any and denote . Clearly, and the set is open in since is continuous on . The statement of the theorem then follows from Corollary 1. ∎
3.4. Integral kernels and discretespace Markov Chains
From Theorem 2 it follows that for weakly continuous dtMPs a reachavoid problem on a nonsimple set can be solved if an appropriate locally excessive function is found. For a known and studied subclass of these processes the problem can be solved even without resorting to such functions. We write if is an integral kernel with a basis and a density , namely when is a finite nonnegative measure, the function is jointly measurable, and for any it holds that
Furthermore, we raise the following assumption, which generalizes the one used for related studies over the finite horizon [AKLP10, SA11].
Assumption 1.
For a subset of a topological state space assume that the function is continuous on for any , .
Let us mention some sufficient conditions for the Assumption 1 to hold true for integral kernels. It follows from [HLL96, Example C.6] that whenever is a continuous function on the set for all , then Assumption 1 is satisfied for the set . It is thus milder than aforementioned assumptions of [AKLP10, SA11] where the stronger Lipschitz continuity is required instead.
Before we prove the main result, we need the following lemma that connects the condition with an important notion of the uniform transitivity [MT93].
Lemma 4.
Let be a dtMP. Suppose that the set is uniformly transient, i.e. there exists such that for all . Then .
Proof.
Suppose that , then for any there exists a point such that . Clearly, for any it further holds that
Note, that for any nonnegative function and for any it holds that for all . As a result: