Autonomous Agent Behaviour Modelled in PRISM – A Case Study
Abstract
Formal verification of agents representing robot behaviour is a growing area due to the demand that autonomous systems have to be proven safe. In this paper we present an abstract definition of autonomy which can be used to model autonomous scenarios and propose the use of smallscale simulation models representing abstract actions to infer quantitative data. To demonstrate the applicability of the approach we build and verify a model of an unmanned aerial vehicle (UAV) in an exemplary autonomous scenario, utilising this approach.
1 Introduction
Autonomous systems have the ability to decide at runtime what to do and how to do it. A critical question is how this decision making process is implemented.
Increasingly, autonomous systems are being deployed within the public domain (e.g. driverless cars, delivery drones). Naturally, there is concern that these systems are reliable, efficient and  most of all  safe. Although testing is a necessary part of this process, simulation and formal verification are key tools, especially at the early stages of design where experimental testing is both infeasible and dangerous. Simulation allows us to view the continuous dynamics and monitor behaviour of a system. On the other hand, model checking allows us to formally verify properties of a finite representation. Whereas the simulation model is close to an implementation, simulation runs are necessarily incomplete. Verification models, on the other hand, require us to abstract more coarsely.
The decisions made by an autonomous agent depend on the current state of the environment, specifically in terms of data perceived by the agent from its sensors. If model checking is to be used for the verification of autonomous systems we must reflect the uncertainty associated with the state of the environment by using probabilistic model checking.
We propose a framework for analysing autonomous systems, specifically to investigate decisionmaking, using probabilistic model checking of an abstract model where quantitative data for abstract actions is derived from smallscale simulation models. We illustrate our approach for an example system composed of a UAV searching for and collecting objects in an arena. The simulation models for abstract actions are generated using the objectoriented framework Simulink and the abstract models are specified and verified using the probabilistic model checker PRISM. In our example, autonomous decision making involves making a weighted choice between a set of possible actions, and is loosely based on the BeliefDesireIntention architecture [7].
Previous work in which model checking is used to verify autonomy includes [3] in which the decision making process is verified in isolation, while our aim is to integrate this process with the autonomous agent as a whole. Other research includes an investigation of the cooperative behaviour of robots, where each robot is represented by a hybrid automaton [1], and verification of a consensus algorithm using experimental verification and an external observer [2].
2 Autonomy
In order to formally define autonomous behaviour, we introduce finite state machines which abstract the autonomous actions independent of the agent type.
Before we give the formal definitions we require the following notation. For a finite set of variables , a valuation of is a function mapping each variable in to a value in its finite domain. Let be the set of valuations of . For any , and value of , let and be the valuations where for any we have and if and otherwise. For a finite set , a probability distribution over is a function such that . Let be the set of distributions over .
Definition 1
A probabilistic finitestate machine is a tuple where: is a finite set of variables; a set of initial states; a finite set of actions and a (partial) transition function.
The set of states of , denoted , is the set of valuations of . Let denote the actions available from state , i.e. the actions for which is defined. In state an action is chosen nondeterministically from the available actions and, if action is chosen, the transition to the next state is made according to the probability distribution . A probabilistic finitestate machine describes a system without autonomy, we introduce this through a weight function adding decision making to the finitestate machine.
Definition 2
An autonomous probabilistic finitestate machine is a tuple where is a probabilistic finitestate machine and is a weight function such that for any and we have and implies .
In an autonomous machine the nondeterminism in the first step of a transition is removed. More precisely, if a machine is in state , then the action performed is that with the largest weight, that is the action:
Requiring the weights for distinct actions and the same state to be different ensures this action is always well defined. Having removed the nondeterminism through the introduction of a weight function, the semantics of an autonomous finite state machine is a discrete time Markov chain.
3 UAV Example
In this case study we consider a specific example (a simple search and retrieve example, with a UAV in a finite sized arena) to demonstrate our approach.
The UAV first takes off and checks whether the system and sensors are functional. If the UAV detects an issue in the system, then it returns to base. Otherwise it will proceed to search for a given number of objects. When an object is found, the UAV positions itself above the object and descends until the grabber can pick it up. The UAV then ascends to transportation height and transports the object to the deposit site. There is the possibility that the UAV will drop its object along the way and need to retrieve it. Once the UAV is above the deposit site, it releases the object and ascends back to search height. It will then decide whether it continues the search or returns to the base and complete the mission. During operation, the UAV may return to base to recharge if it is low on battery, or conduct an emergency landing, due to an internal system error. If the mission time limit is reached, the UAV abandons the mission and returns to base. Figure 1 represents this scenario, showing the different modes of the UAV and progression between the modes.
We represent this scenario using a autonomous finitestate machine . The variables of are given by:

the number of objects which have not been found;

the position of the UAV in the arena;

the return coordinates when search is interrupted;

the current mode of the UAV;

the mission time;

the battery charge level.
Each state of is a valuation of these variables. The transition and weight functions and are based on Figure 1. We focus on the target approach and search modes of the UAV.
In the target approach mode (), the UAV positions itself above an observed object and we denote this abstract action by . Thus, the weight function is for all states such that , and for any such state we have for any :
where and are the time and battery charge used approaching the object, and is the mode for descending. The abstract action models several different operations of the UAV including the use of its camera and navigational system. A smallscale simulation model was built for this abstract action to provide the required quantitative data.
When the UAV is in search mode () there are two actions that can occur: and with the UAV continuing search if the battery charge level is above a certain threshold, and returning to base otherwise. The weight function for the action from any state such that is given by:
and for the action we have . Concerning the transition function we have for any :
and if and 0 otherwise, where denotes the movement from one discrete square in the arena to the next, and are the time and battery consumption of the UAV while moving one square. The UAV has probability of finding an object in a given position, if an object is found the UAV changes to mode and is set to the current coordinates, as the search has been interrupted. If no object is found, the UAV continues searching.
4 Results
We have modelled our scenario in the probabilistic model checker PRISM [5], building smallscale simulation models to determine individual abstract actions to generate probabilistic and timing values. To encode the timing values in PRISM as integer variables we take the floor and ceiling, introducing nondeterminism into the model and upper and lower bounds for properties [4]. The experiments were performed on a computer with 16GB RAM and a 2.3 GHz Intel Core i5 processor.
The main properties of interest concern the UAV successfully completing the mission (finding and depositing all objects within the time limit) and failing the mission (either due to an emergency landing, missing some objects or running out of time). We have also considered other properties including how often the UAV drops an object and how often it recharges during a mission, more details can be found in the repository [6].
We have analysed two scenarios where there are 3 objects and a time limit of 900s, and 2 objects and a time limit of 500s respectively. For the first scenario the model has states, was built in s and verifying a property varied between s and s. This is far more efficient than running Monte Carlo simulations, as simulating runs of the same scenario takes over two weeks. For the second scenario the model has states and model construction time was s.
For the first scenario, the maximum and minimum probabilities of the UAV completing a mission are and respectively. The maximum and minimum probabilities of running out of time are negligible, searching the arena and missing some objects are and , and and for performing an emergency landing. Figure 2 shows the maximum and minimum probability of a successful mission within a time bound as well as the results obtained when the nondeterminism is replaced by a uniform random choice. The probability increases after a threshold time as the UAV has to search a proportion of the arena before finding all objects.
5 Conclusions
We have proposed using smallscale simulation models to inform probabilistic models used for verification. The simulation models can be used to provide quantitative data for abstract actions. The probabilistic models can be used for fast property specific verification, that is not possible using simulations alone.
Our approach is highly adaptable; once the initial smallscale simulation and probabilistic models have been set up, different decision algorithms can be easily substituted and analysed. Our example illustrates the use of a weight function for decision making. In a more extensive scenario the weight function would be more complex (e.g. involving current values associated with all sensors and guiding systems). Our use of nondeterminism when approximating quantitative data obtained from the smallscale simulation models allows us to provide an range of uncertainty for our results. We aim to formally prove a link between the simulation and the abstract model to allow us to infer results from the abstract model for the actual system. To allow the analysis of more complex scenarios we plan to incorporate abstraction techniques. Acknowledgments. This work was supported by the Engineering and Physical Sciences Research Council [grant number EP/N508792/1].
References
 [1] Chaimowicz, L., Campos, M.F.M., Kumar, V.: Hybrid systems modeling of cooperative robots. In: Proc. Int. Conf. Robotics and Automation (ICRA’03). pp. 4086–4091. IEEE Press (2003)
 [2] Cook, J., Hu, G.: Experimental verification and algorithm of a multirobot cooperative control method. In: Proc. IEEE/ASME Int. Conf. Advanced Intelligent Mechatronics (AIM’10). pp. 109–114. IEEE Press (2010)
 [3] Dennis, L., Fisher, M., Lincoln, N., Lisitsa, A., Veres, S.: Practical verification of decisionmaking in agentbased autonomous systems. Automated Software Engineering pp. 1–55 (2014)
 [4] Kattenbelt, M., Kwiatkowska, M., Norman, G., Parker, D.: A gamebased abstractionrefinement framework for Markov decision processes. Formal Methods in System Design 36(3), 246–280 (2010)
 [5] Kwiatkowska, M., Norman, G., Parker, D.: PRISM 4.0: Verification of probabilistic realtime systems. In: Proc. Int. Conf. Computer Aided Verification (CAV’11). LNCS, vol. 6806, pp. 585–591. Springer (2011)
 [6] PRISM model repository (2016), dx.doi.org/10.5525/gla.researchdata.274
 [7] Veres, S., Molnar, L., Lincoln, N., Morice, C.: Autonomous vehicle control systems  a review of decision making. Proc. Institution of Mechanical Engineers, Part I: Journal of Systems and Control Engineering 225(2), 155195 (2011)