An Efficient Synthesis Algorithm for Parametric Markov Chains Against Linear Time Properties^{†}^{†}thanks: We have very recently noticed the paper accepted at CAV [2] which treats the nonparametric Markov chains using a similar approach. Our approach has been developed in parallel.
Abstract
In this paper, we propose an efficient algorithm for the parameter synthesis of PLTL formulas with respect to parametric Markov chains. The PLTL formula is translated to an almost fully partitioned Büchi automaton which is then composed with the parametric Markov chain. We then reduce the problem to solving an optimisation problem, allowing to decide the satisfaction of the formula using an SMT solver. The algorithm works also for interval Markov chains. The complexity is linear in the size of the Markov chain, and exponential in the size of the formula. We provide a prototype and show the efficiency of our approach on a number of benchmarks.
1 Introduction
Model checking, an automatic verification technique, has attracted much attention as it can be used to verify the correctness of software and hardware systems [12, 10, 1]. In classical model checking, temporal formulas are often used to express properties that one wants to check.
Probabilistic verification problems have been studied extensively in recent years. Markov chains (MCs) are a prominent probabilistic model used for modelling probabilistic systems. Properties are specified using probabilistic extensions of temporal logics such as probabilistic CTL (PCTL) [22] and probabilistic LTL (PLTL) [5] and their combination PCTL*. In the probabilistic setting, most of the observations about CTL and LTL carry over to their probabilistic counterpart. An exception is the complexity for verifying PLTL: here one could have a double exponential blowup. This is the case, because in general nondeterministic Büchi automata cannot be used directly to verify LTL properties, as they will cause imprecise probabilities in the product. In turn, it is often necessary to construct their deterministic counterparts in terms of other types of automata, for instance Rabin or Parity automata, which adds another exponential blowup. As a result, most of the work in literature focuses on branching time verification problems. Moreover, stateoftheart tools such as PRISM [29] and MRMC [25] can handle large systems with PCTL specifications, but rather small systems –if at all– for PLTL specifications.
In the seminal paper by Courcoubetis and Yannakakis [13], it is shown that for MCs the PLTL model checking problem is in PSPACE. They perform transformations of the Markov chain model recursively according to the LTL formula. At each step, the algorithm replaces a subformula rooted at a temporal operator with a newly introduced proposition; meanwhile, it refines the Markov chain with that proposition, and such refinement preserves the distribution. Then, it is finally boiled down to the probabilistic model checking upon a propositional formula. At the refinement step the state space is doubled, thus resulting in a PSPACE algorithm. Even if it is theoretically a single exponential algorithm for analysing MCs with respect to PLTL, it has not been exploited in the stateoftheart probabilistic model checkers.
In automatabased approaches, one first translates the LTL formula into a Büchi automaton and then analyses the product of the MC and the Büchi automaton. This is sufficient for qualitative properties, i.e., to decide whether the specification is satisfied with probability . For quantitative properties, the Büchi automaton needs to be further transformed into a deterministic variant. Such a determinisation step usually exploits Safra’s determinisation construction [33]. Several improvements have been made in recent years, see for instance [31, 34, 30]. Model checkers such as PRISM [29] and LiQuor [9] handle PLTL formulas by using offtheshelf tools (e.g. (J)Ltl2Dstar [28]) to perform this determinisation step. To avoid the full complexity of the deterministic construction, Chatterjee et al. [7] have proposed an improved algorithm for translating the formulas of the fragment of LTL to an extension of Rabin automata. Recently [18], this algorithm has been extended to the complete LTL. Despite the above improvements, the size of the resulting deterministic automaton is still the bottleneck of the approach for linear temporal properties. In [14], it is first observed that the second blowup can be circumvented by using unambiguous Büchi automata (UBAs) [6]. The resulting algorithm has the same complexity as the one in [13]. Despite the importance of probabilistic model checking, unfortunately, the algorithm in [14] is less recognised. To the best of the authors knowledge, it is not applied in any of the existing model checkers. Recently, in[27], the authors construct the so called limit deterministic Büchi automata that are exponential in the size of LTL\GU formula , which is another fragment of LTL. The approach is only applied to the analysis of qualitative PLTL of the form .
In this paper, we present a further improvement of the solution proposed in [14], adapted directly to solving the parameter synthesis problem for parametric Markov chains. We exploit a simple construction translating the given LTL formula to a reverse deterministic UBA, and then build the product of the parametric Markov chains. We then extract an equation system from the product, then the synthesis problem reduces to the existence of a solution of the equation system. Further, we remark that the related interval Markov chains can be handled by our approach as well. We integrate our approach in the model checker IscasMC [21], and employ SMT solver to solving the obtained equation system. We present detailed experimental results, and observe that our implementation can deal with some realworld probabilistic systems modelled by parametric Markov chains.
Related Work.
In [20], they first use state elimination to compute the reachability probability for parametric Markov models. This has be improved by Dehnert et al.[17]. Another related models is interval Markov chains, which can be interpreted as a family of Markov chains [8, 24] whose transition probabilities lie within the interval ranges. In [8], they also considered model checking regular properties with interval Markov chains. They showed that the synthesis of interval Markov chains problem against PCTL is decidable in PSPACE. In [24], they considered interval Markov chains as abstraction models by using threevalued abstraction for Markov chains. To our best knowledge, it is the first time that one can easily integrate parameter synthesis algorithm that is exponential in the size of LTL formulas over parametric Markov chains.
2 Preliminaries
Given a set , we say that an infinite sequence is an word if . Given a finite word and a finite or infinite word , we denote by the concatenation of and , i.e., the finite or infinite word , respectively. We may just write instead of . We denote by the set of natural numbers .
Probability Theory.
A measure over a measurable space is a function such that and, for each countable family of pairwise disjoint elements of , . If , then we call a subprobability measure and, if , then we call a probability measure. We say that is a discrete measure over if is discrete. In this case, for each , and we drop brackets whenever possible. For a set , we denote by the set of discrete probability measures over , and by the set of discrete subprobability measures over . We call the support of a measure if ; in particular, if is discrete, we denote by the minimum support set . Moreover, we denote by , for , the Dirac measure such that for each , if , otherwise. If is discrete, then it holds that for each , if and if . In case is countable, then the probability measure over the discrete measurable space can be obtained by imposing that ; is also called a probability distribution.
Graph Theory.
A directed graph is a pair where is a finite nonempty set of vertices, also called nodes, and is the set of edges or arcs. Given an arc , we call the vertex the head of , denoted by , and the vertex the tail of , denoted by . In the remainder of the paper we consider only directed graphs and we refer to them just as graphs.
A path is a sequence of edges such that for each , ; we say that is reachable from if there exists a path such that and .
A strongly connected component (SCC) is a set of vertices such that for each pair of vertices , is reachable from and is reachable from ; we say that a graph is strongly connected if is an SCC. We say that an SCC is nonextensible if for each SCC of , implies . Without loss of generality, in the remainder of this paper we consider only nonextensible SCCs.
We define the partial order over the SCCs of the graph as follows: given two SCCs and , if there exist and such that is reachable from . We say that an SCC is maximal with respect to if for each SCC of , implies . We may call the maximal SCCs as bottom SCCs, BSCC for short.
A graph can be enriched with labels as follows: a labelled graph is a triple where is a finite nonempty set of vertices, is a finite set of labels, and is the set of labelled edges. The notations and concepts on graphs trivially extend to labelled graphs.
Generalized Büchi Automata.
A generalized Büchi automaton (GBA) is a tuple where is a finite alphabet, is a finite set of states, is the transition function, is the set of initial states, and is the set of accepting sets.
A run of over an infinite word is an infinite sequence such that and for each it is . Similarly, a run of over a finite word is a finite sequence such that and for each it is . Let be the set of tuples occurring infinitely often in . The run is accepting if for each . The word is accepted by if there is an accepting run of over ; we denote by the language of , i.e., the set of infinite words accepted by .
Given the GBA , for the sake of convenience, we denote by the GBA with initial state and accordingly for we let .
The graph underlying a GBA is the graph whose set of vertices (nodes) is the set of states of and there is an edge labelled with from to if . In this case, we say that is an predecessor of and is an successor of .
Given a GBA , we say that

is deterministic, if and for each and ;

is reverse deterministic if each state has exactly one predecessor for each ;

is unambiguous if for each , , and such that , we have ; and

is separated if for each pair of states , .
We say that a state is reenterable if has some predecessor in . Let be the set of all reenterable states of and consider the GBA where and . Then, we say that is almost unambiguous (respectively almost separated, almost reverse deterministic) if is unambiguous (respectively separated, reverse deterministic).
For an (almost) separated GBA , if for each there exists some state of such that , then we say that is (almost) fully partitioned. Clearly, if an automaton is (almost) fully partitioned, then it is also (almost) separated, (almost) unambiguous and (almost) reverse deterministic.
As an example of GBA that is reversedeterministic and separated but not fully partitioned, consider the automaton in Fig. 1. The fact that is not fully partitioned is clear since no word starting with is accepted by any of the states , , or . One can easily check that is indeed reversedeterministic but checking the separated property can be more involved. The checks involving are trivial, as it is the only state enabling a transition with label or ; for the states and , the separated property implies that given any , it is not possible to find some such that . For instance, suppose the number of the most front ’s in is odd, it must be the case that the most front ’s are directly followed by or . In order to match , we must choose instead of on transition . It follows that the number of the most front ’s in is even. We can get similar result when the number of the most front ’s in is even. Thus and can never be the same.
3 Parametric Markov Chains and Probabilistic LTL
In this section we recall the definitions of parametric Markov chains as presented in [20], interval Markov chain considered in [3, 8, 24] and of the logic PLTL. In addition, we consider the translation of LTL formulas to GBAs which is used later for analysing PLTL properties.
3.1 Parametric Markov Chains
Before introducing the parametric Markov chain model, we briefly present some general notation. Given a finite set with domain in , an evaluation is a partial function . Let denote the domain of ; we say that is total if . A polynomial over is a sum of monomials where each and each . A rational function over is a fraction of two polynomials and over ; we denote by the set of all rational functions over . Given , , and an evaluation , we let denote the rational function obtained by replacing each occurrence of with .
Definition 1
A parametric Markov chain (PMC), is a tuple where is a finite set of states, is a labelling function where is a finite set of state labels, is the initial state, is a finite set of parameters, and is a transition matrix.
We now define the PMC induced with respect to a given evaluation:
Definition 2
Given a PMC and an evaluation , the PMC induced by is the tuple where the transition matrix is given by .
We say that a total evaluation is welldefined for a PMC if and for each . In the remainder of the paper we consider only welldefined evaluations and we require that, for a given PMC and two states , if for some evaluation , then for the considered. We may omit the actual evaluation when we are not interested in the actual value for , such as for the case .
We use to denote the number of states, and for the number of nonzero probabilistic transitions, i.e., .
The underlying graph of a PMC is the graph where and .
A path is a sequence of states satisfying for all . We call a path finite or infinite if the sequence is finite or infinite, respectively. We use to denote the suffix and we denote by and the set of all infinite and finite paths of , respectively. An infinite path defines the word such that for .
For a finite path , we denote by the cylinder set of , i.e., the set of infinite paths starting with prefix . Given an evaluation , we define the measure of the cylinder set by . For a given PMC and an evaluation , we can extend uniquely to a probability measure over the field generated by cylinder sets [26].
We call the bottom SCCs of the underlying graph ergodic sets and for each ergodic set , we call each state ergodic. A nice property of a bottom SCC is the socalled ergodicity property: for each , will be reached again in the future with probability from any state , including itself. Moreover, for each finite path within , will be performed again in the future with probability .
In this paper we are particularly interested in regular properties and the probability for some measurable set . Such properties are known to be measurable in the field generated by cylinders [35]. We write to denote the probability function when assuming that is the initial state of the PMC . To simplify the notation, we omit the superscript whenever is clear from the context and we use as a synonym for .
3.2 Interval Markov chain
In this section we recall the definition of interval Markov chain [8, 24] and show how it can be converted to a parametric Markov chain.
Definition 3
An interval Markov chain (IMC) is a tuple where , and are as for PMCs while are the transition matrices such that for each , .
We show how to convert an IMC to a PMC in the following. Given an IMC , we define the corresponding PMC as follows. For every pair of states, say , we add a new parameter to such that ; then, we define as . For instance, suppose in an IMC, there is a state with two successors, namely and , with , , and . We add two parameters and for the pairs and whose ranges are and respectively. Moreover, in order to get an instance of Markov chain from the resulting PMC, we must make sure that .
3.3 Probabilistic Linear Time Temporal
Throughout the whole paper, we will assume that the state space of any PMC is always equipped with labels that identify distinguishing state properties. For this, we let denote a set of atomic propositions. We assume as state labels, so that specifies the subset of atomic propositions holding in state .
We first recall the linear time temporal logic (LTL). The syntax of LTL is given by:
where . We use standard derived operators, such as: , , , , and . Semantics is standard and is omitted here.
A probabilistic LTL (PLTL) formula has the form where is a nonempty interval with rational bounds and is an LTL formula. In a PMC with evaluation , for a state and a formula , we have:
(1) 
where , or for short, denotes the probability measure of the set of all paths which satisfy . The synthesis problem of this paper is thus to find such a if possible or to prove that the LTL formula is invalid for all valid . From the measurability of regular properties, we can easily show that for any PLTL path formula , the set is measurable in the field generated by the cylinder sets.
3.4 From LTL to Büchi automaton
The following section describes how we can transform a given LTL formula into a GBA which has the required properties for the subsequent model checking procedure.
Definition 4
The set of elementary formulas for a given LTL formula is defined recursively as follows: if ; ; ; ; and .
Given a set and , we inductively define the satisfaction relation for each subformula of as follows:
if in the case of ,  
if it is not the case that ,  
if and ,  
if , and  
if or, and . 
Finally, is the Büchi automaton where:

;

and for each , we have: ; and

where for each subformula of , .
In Definition 4, each formula in is guaranteed to be of the form ; the size of is precisely the number of temporal operators (i.e., and ) occurring in .
Theorem 3.1 (cf. [11])
For the automaton , the following holds:

For each infinite word , we have if and only if .

More generally, for each and we have: if and only if , for every .
It follows directly that
Corollary 1
For each , if then . Moreover, is both almost unambiguous and almost separated.
We observe that for each subset and each , there is exactly one predecessor of , namely the set . Hence, we also have the following conclusion.
Corollary 2
The automaton is almost reverse deterministic and fully partitioned.
Intuitively, for any , we can find a state and we observe that by Theorem 3.1. Since is already almost separated, it follows that it is also almost fully partitioned. Note that because of the nonreenterable initial state, the automaton may not be fully partitioned, but is almost fully partitioned.
4 Parameter Synthesis Algorithm
We consider a parametric Markov chain and an almost fully partitioned automaton obtained from the LTL specification, where if and . To simplify the notation, in the following we assume that for a given PMC we have and for each ; this modification does not change the complexity of probabilistic model checking [13]. Below we define the product graph:
Definition 5
Given the automaton and the PMC , the product graph of and , denoted , is the graph where and (also written ) if and only if and .
Suppose that . We say that an SCC of is accepting if for each , there exist such that and for some .
Given an SCC of , we denote by the corresponding SCC of , where . We denote by a function to get the corresponding path of from the path of , i.e., and we usually call the path the projection of . For convenience, we also write if the (finite) path is a fragment of the (finite) path .
Definition 6 (Complete SCC)
For an SCC of and the corresponding SCC of , we say that is complete if for each finite path in , we can find a finite path in such that .
Consider the product shown in Fig. 2. It has two nontrivial SCCs, namely and . Clearly, and are the corresponding SCCs of and , respectively. We observe that is a complete SCC while is not complete since the finite path of is not a projection of any finite path in . The key observation is that some transitions in the SCCs of may be lost in building the product, so we only consider the complete SCCs in the product to make sure that no transitions of the projection SCCs are missing.
The following lemma characterises the property of a complete SCC of .
Lemma 1
Consider a complete SCC of with and an arbitrary finite path in . Then, there exists some finite path in with the following property: for each finite path in with , contains as a fragment, i.e, .
Proof
Clearly, must be the product of and some component of . Recall that is almost reversedeterministic, then for each path in and each finite path in , there is at most one path in with . In the following, we call the backwardextension of .
Given a finite path in , we define for every finite path in .
Then, for the path , we define a sequence of finite paths , , , … as follows:

. According to the definition, , because . Moreover, we have that is a finite set, let , where is the number of states along the finite path and is the state space of .

For each , we impose the inductive hypothesis that . Then, if , we arbitrarily choose a path . Since is an SCC, then there exists some path connecting the last node of to the first state of .

Let , then it is a finite path in and . It is immediate to see that since the set just consists of the backwardextensions of paths in . As we have mentioned, each path has at most one such extension, and the extension for , namely , does not belong to since it involves the fragment .
Therefore, there must exist such that . Let ; then each finite path in with projection must contain the fragment . ∎
For example, consider the product from Fig. 2. Given the complete SCC and , we find the specific as follows.

. First we get . Since , we choose and for , we have no choices but .

. Similarly, we need to compute . Since and one needs to visit after traversing , which is impossible to bypass to make it happen. Then it gives us . Therefore, we set to be the specific finite path of .
Based on Lemma 1, the following corollary relates the paths in the product and the projected Markov chains:
Corollary 3
Let be a complete SCC of and ; consider two infinite paths in and in such that ; let and be the following properties:

: visits each finite path in infinitely often;

: visits each finite path of infinitely often.
Then holds if and only if holds.
The proof of Corollary 3 can be found in the appendix.
Definition 7
We say that the SCC of is locally positive if:

is accepting and complete.

is maximal with respect to (it is a socalled bottom SCC).
Consider again the example from Fig. 2. Assume the acceptance condition of is ; we observe that the SCC is both accepting and complete but not locally positive since is not a bottom SCC in .
According to Corollary 3, the ergodicity property of Markov chains, and the definition of Büchi acceptance, we have the following result.
Proposition 1
if and only if there exists some locally positive SCC in .
For a given SCC, in order to decide whether it is locally positive, we have to judge whether it is complete. In general, doing so is a nontrivial task. Thanks to [13, Lemma 5.10], completeness can be checked efficiently:
Lemma 2
If is (almost) reverse deterministic, then the following two conditions are equivalent:

is complete, i.e., each finite path of is a projection of some finite path in .

There is no other SCC of with such that .
Intuitively, in the product composed by an almost reverse deterministic automaton and a PMC , the complete SCCs must be the SCCs whose preceded SCCs should not have the same projections. The detailed proof can be found in the appendix.
We now turn to the problem of computing the exact probability.
Theorem 4.1
Given a PMC and a fully partitioned Büchi automaton , let be their product. Let be the set of all locally positive SCCs of and be the set of all BSCCs of which are not locally positive. Further, for an SCC let denote the set of states of occurring in . We define the following equation system:
(2)  
(3)  
and  (4) 
Then, it holds that for any welldefined evaluation .
In general, all locally positive SCCs can be projected to the BSCCs in the induced MC . In the original MC , the reachability probability of every state in the accepting BSCC should be . Thus in a locally positive SCC of , the probability mass distributes on the states in which they share the same second component, i.e, from state . This follows from the fact that the resulting product is almost fully partitioned so that the probability is also partitioned.