How to Be Both Rich and Happy:
Combining Quantitative and Qualitative Strategic Reasoning about Multi-Player Games (Extended Abstract)
We propose a logical framework combining a game-theoretic study of abilities of agents to achieve quantitative objectives in multi-player games by optimizing payoffs or preferences on outcomes with a logical analysis of the abilities of players for achieving qualitative objectives of players, i.e., reaching or maintaining game states with desired properties. We enrich concurrent game models with payoffs for the normal form games associated with the states of the model and propose a quantitative extension of the logic enabling the combination of quantitative and qualitative reasoning.
F. Mogavero, A. Murano, and M.Y. Vardi (Eds.): 1st Workshop on Strategic Reasoning 2013 (SR’13) EPTCS 112, 2013, pp. How to Be Both Rich and Happy: Combining Quantitative and Qualitative Strategic Reasoning about Multi-Player Games (Extended Abstract)–LABEL:LastPage, doi:10.4204/EPTCS.112.8 © Nils Bulling & Valentin Goranko This work is licensed under the Creative Commons Attribution License.
How to Be Both Rich and Happy:
Combining Quantitative and Qualitative Strategic Reasoning about Multi-Player Games (Extended Abstract)
|Nils Bulling \IfArrayPackageLoaded|
|email@example.com and Valentin Goranko \IfArrayPackageLoaded|
There are two rich traditions in studying strategic abilities of agents in multi-player games:
Game theory has been studying rational behavior of players, relevant for their achievement of quantitative objectives: optimizing payoffs (e.g., maximizing rewards or minimizing cost) or, more generally, preferences on outcomes. Usually, the types of games studied in game theory are one-shot normal form games, their (finitely or infinitely) repeated versions, and extensive form games.
Logic has been mostly dealing with strategic abilities of players for achieving qualitative objectives: reaching or maintaining outcome states with desired properties, e.g., winning states, or safe states, etc.
Among the most studied models in the logic tradition are concurrent game models [6, 22]. On the one hand they are richer than normal form games, as they incorporate a whole family of such games, each associated with a state of a transition system; but on the other hand, they are somewhat poorer because the outcomes of each of these normal form games, associated with a given state, are simply the successor states with their associated games, etc. whereas no payoffs, or even preferences on outcomes, are assigned. Thus, plays in concurrent game models involve a sequence of possibly different one-shot normal form games played in succession, and all that is taken into account in the purely logical framework are the properties – expressed by formulae of a logical language – of the states occurring in the play. Concurrent game models can also be viewed as generalization of (possibly infinite) extensive form games where cycles and simultaneous moves of different players are allowed, but no payoffs are assigned.
Put as a slogan, the game theory tradition is concerned with how a player can become maximally rich, or how to pay as little cost as possible, while the logic tradition – with how a player can achieve a state of ‘happiness’, e.g. winning, or to avoid reaching a state of ‘unhappiness’ (losing) in the game.
The most essential technical difference between qualitative and quantitative players’ objectives is that the former typically refer to (a temporal pattern over) Boolean properties of game states on a given play and can be monitored locally whereas the latter are determined by the entire history of the play (accumulated payoffs) or even the whole play (its value, being a limit of average payoffs, or of discounted accumulated payoffs). It is therefore generally computationally more demanding and costly to design strategies satisfying quantitative objectives or to verify their satisfaction under a given strategy of a player or coalition.
These two traditions have followed rather separate developments, with generally quite different agendas, methods and results, including, inter alia:
extensions of qualitative reasoning (e.g., reachability and Büchi objectives) in multi-player concurrent games with ’semi-quantitative’ aspects by considering a preference preorder on the set of qualitative objectives, see e.g., , , thereby adding payoff-maximizing objectives and thus creating a setting where traditional game-theoretic issues such as game value problems and Nash equlibria become relevant.
on the purely quantitative side, first to mention repeated games, extensively studied in game theory (see e.g., ), which can be naturally treated as simple, one-state concurrent game models with accumulating payoffs paid to each player after every round and no qualitative objectives;
from a more computational perspective, stochastic games with quantitative objectives on discounted, mean or total payoffs, in particular energy objectives, see e.g. .
the conceptually different but technically quite relevant study of counter automata, Petri nets, vector addition systems, etc. – essentially a study of the purely quantitative single-agent case of concurrent game models (see e.g. ), where only accumulated payoffs but no qualitative objectives are taken into account and a typical problem is to decide reachability of payoff configurations satisfying formally specified arithmetic constraints from a given initial payoff configuration.
A number of other relevant references discuss the interaction between qualitative and quantitative reasoning in multi-player games, e.g. , , which we cannot discuss here due to space limitations.
This project purports to combine the two agendas in a common logical framework, by enriching concurrent game models with payoffs for the one-shot normal form games associated with the states, and thus enabling the combination of quantitative game-theoretic reasoning with the qualitative logical reasoning. Again, put as a slogan, our framework allows reasoning about whether/how a player can reach or maintain a state of ‘happiness’ while becoming, or remaining, as rich as (rationally) possible, or paying the least possible price on the way. The purpose of this extended abstract is to introduce and discuss a general framework of models and logics for combined quantitative and qualitative reasoning that would naturally cover each of the topics listed above, and to initiate a long term study on it.
A concurrent game model  (CGM) comprises:
a non-empty, fixed set of players and a set of actions for each .
For any we will denote and will use to denote a tuple from . In particular, is the set of all possible action profiles in .
a non-empty set of game states .
for each a map setting for each state the actions available to at .
a transition function that assigns the (deterministic) successor (outcome) state to every state and action profile such that for every (i.e., every that can be executed by player in state ).
a set of atomic propositions and a labelling function .
Thus, all players in a CGM execute their actions synchronously and the combination of these actions, together with the current state, determines the transition to a (unique) successor state in the CGM.
The logic of strategic abilities (Alternating-Time Temporal Logic), introduced and studied in , is a logical system, suitable for specifying and verifying qualitative objectives of players and coalitions in concurrent game models. The main syntactic construct of is a formula of type , intuitively meaning: “The coalition has a collective strategy to guarantee the satisfaction of the objective on every play enabled by that strategy.” Formally, is a multi-agent extension of the branching time logic CTL*, i.e., multimodal logic extending the linear-time temporal logic – comprising the temporal operators (“at the next state”), (“always from now on”) and (“until”) – with strategic path quantifiers indexed with coalitions of players. There are two types of formulae of , state formulae, which constitute the logic and that are evaluated at game states, and path formulae, that are evaluated on game plays. These are defined by mutual recursion with the following grammars, where , : state formulae are defined by , and path formulae by .
The logic is very expressive and that comes at a high computational price: satisfiability and model checking are -complete. A computationally better behaved fragment is the logic , which is the multi-agent analogue of CTL, only involving state formulae defined by the following grammar, for , : . For this logic satisfiability and model checking are -complete and -complete, respectively. We will, however, build our extended logical formalism on the richer because we will essentially need the path-based semantics for it.
Arithmetic Constraints. We define a simple language of arithmetic constraints to express conditions about the accumulated payoffs of players on a given play. For this purpose, we use a set of special variables to refer to the accumulated payoffs of the players at a given state and denote by the restriction of to any group . The payoffs can be integers, rationals111Note that models with rational payoffs behave essentially like models with integer payoffs, after once-off initial re-scaling., or any reals. We denote the domain of possible values of the payoffs, assumed to be a subset of the reals , by and use a set of constants symbols , with , for names of special real values (see further) to which we want to refer in the logical language.
For fixed sets and we build the set of terms over and from by applying addition, e.g. . An evaluation of a term is a mapping . We write to denote that is satisfied under the evaluation . Moreover, if some order of the elements is clear from context, we also represent an evaluation as a tuple from and often assume that elements from have their canonic interpretation. The set of arithmetic constraints over and consists of all expressions of the form where and . We use to refer to the set of Boolean formulae over ; e.g. for . We note that the language is strictly weaker than Presburger arithmetic, as it involves neither quantifiers nor congruence relations.
We also consider the set of arithmetic path constraints being expressions of the type where , and . The meaning of is to represent the value of the current play for the player . That value can be defined differently, typically by computing the accumulated payoff over the entire play, by using a future discounting factor, or by taking the limit – if it exists – of the mean (average) accumulated payoff (cf. ). We note that the discounted, accumulated, mean or limit payoffs may take real values beyond the original domain of payoffs ; so, we consider the domain for to be a suitable closure of .
3 Concurrent Game Models with Payoffs and Guards
We now extend concurrent game models with utility values for every action profile applied at every state and with guards that determine which actions are available to a player at a given configuration, consisting of a state and a utility vector, in terms of arithmetic constraints on the utility of that player.
A guarded CGM with payoffs (GCGMP) is a tuple where is a CGM and:
is a payoff function assigning at every state and action profile applied at a payoff to every agent. We write for .
, for each player , is a guard function that assigns for each state and action an arithmetic constraint formula that determines whether is available to at the state given the current value of ’s accumulated payoff. The guard must enable at least one action for at . Formally, for each state , the formula must be valid. Moreover, a guard is called state-based if .
is a discount factor, for each , used in order to define realistically values of infinite plays for players or to reason about the asymptotic behavior of players’ accumulated payoffs.
The guard refines the function from the definition of a CGM, which can be regarded as a guard function assigning to every state and action a constant arithmetic constraint true or false. In our definition the guards assigned by only depend on the current state and the current accumulated payoff of . The idea is that when the payoffs are interpreted as costs, penalties or, more generally, consumption of resources the possible actions of a player would depend on her current availability of utility/resources.
Consider the GCGMP shown in Figure 1 with 2 players, I and II, and 3 states, where in every state each player has 2 possible actions, (cooperate) and (defect). The transition function is depicted in the figure.
The normal form games associated with the states are respectively versions of the Prisoners Dilemma at state , Battle of the Sexes at state and Coordination Game at state .
The guards for both players are defined at each state so that the player can apply any action if she has a positive current accumulated payoff, may only apply action if she has accumulated payoff 0; and must play an action maximizing her minimum payoff in the current game if she has a negative accumulated payoff. The discounting factors are 1 and the initial payoffs of both players are 0.
Configurations, plays, and histories. Let be a GCGMP defined as above. A configuration (in ) is a pair consisting of a state and a vector of currently accumulated payoffs, one for each agent, at that state. Hereafter we refer to accumulated payoffs as utility, at a given state. We define the set of possible configurations as . The partial configuration transition function is defined as such that iff:
( is a successor of if is executed).
assigning the value to satisfies the guard for each , i.e. (each agent’s move is enabled at by the respective guard applied to the current accumulated utility value ).
for all (i.e., the utility values change according to the utility function and the discounting rate where denotes the number of steps that took place).
A GCGMP with a designated initial configuration gives rise to a configuration graph on consisting of all configurations in reachable from by the configuration transition function. A play in a GCGMP is an infinite sequence from such that for all . The set of all plays in is denoted by . Given a play we use and to refer to the th element and to the subplay starting in position of , respectively. A history is any finite initial sequence of a play in . The set of all histories is denoted by . For any history we also define as for plays and additionally and to refer to the last state on and to the sub-history between and , respectively. Finally, we introduce functions , , and which denote the projection of a given play or history to the sequence of its configurations, utility vectors, and states, respectively. For illustration, let us consider the play . We have that ; ; ; ; ; ; and where .
Some possible plays starting from in Example 1 are given in the following where we assume that the initial accumulated payoff is for both agents. We note that this implies that the first action taken by any agent is always .
Both players cooperate forever:
After the first round both players defect and the play moves to , where player I chooses to defect whereas II cooperates. Then I must cooperate while II must defect but at the next round can choose any action, so a possible play is:
After the first round player I defects while II cooperates and the play moves to , where they can get stuck indefinitely, until – if ever – they happen to coordinate, so a possible play is:
Note, however, that once player I reaches accumulated payoff 0 he may only apply at that round, so if player II has enough memory or can observe the accumulated payoffs of I he can use the opportunity to coordinate with I at that round by cooperating, thus escaping the trap at and making a sure transition to .
If, however, the guards did not force the players to play C when reaching accumulated payoffs 0, then both players could plunge into an endless misery if the play reaches .
Strategies. A strategy of a player is a function such that if then ; that is, actions prescribed by a strategy must be enabled by the guard. Our definition of strategy is based on histories of configurations and actions, so it extends the notion of strategy from  where it is defined on histories of states, and includes strategies, typically considered e.g. in the study of repeated games, where often strategies prescribe to the player an action dependent on the previous action, or history of actions, of the other player(s). Such are, for instance, Tit-for-tat or Grim-trigger in repeated Prisoners Dillemma; likewise for various card games, etc. Since our notion of strategy is very general, it easily leads to undecidable model checking problems. So, we also consider some natural restrictions, such as: state-based, action-based or configuration-based, memoryless, bounded memory, of perfect recall strategies 222We note that all strategies need to be consistent with the guards, so state-based strategies are only applicable in models where the guards only take into account the current state, but not the accumulated payoffs.. Here we adopt a generic approach and assume that two classes of strategies and are fixed as parameters, with respect to which the proponents and opponents select their strategies, respectively. The proponent coalition selects a -strategy (i.e. one agreeing with the class ) while the opponent coalition selects a -strategy . The outcome play in a given GCGMP determines the play emerging from the execution of the (complete) strategy profile from configuration in .
4 The Logic: Quantitative ATL*
We now extend the logic to the logic with atomic quantitative objectives being state or path arithmetic constraints over the players’ accumulated payoffs. The semantics of naturally extends the semantics of over GCGMPs, but parameterised with the two classes of strategies and .
Definition 2 (The logic )
The language of consists of state formulae , which constitute the logic, and path formulae , generated as follows, where , , , and :
Let be a GCGMP, a configuration, state-formulae, path formulae, and . Further, let and be two classes of strategies as described above. The semantics of the path constraints is specified according to the limit-averaging or discounting mechanism adopted for computing the value of a play for a player. Then the truth of a formula at a position of a configuration in is defined by mutual recursion on state and path formulae as follows:
for iff ; for iff ,
iff there is a collective -strategy for such that for all collective -strategies for we have that .
iff ; iff for .
iff for all ,
iff there is such that and for all .
Ultimately, we define as . Moreover, if not clear from context, we also write for .
The semantics presented above extends the standard semantics for and is amenable to various refinements and restrictions, to be studied further. For instance, if appropriate, an alternative semantics can be adopted, based on irrevocable strategies  or, more generally, on strategy contexts  or other mechanisms for strategy commitment and release . Also, the nested operators as defined here access the accumulated utility values and require plays to be infinite. Similarly to , one can consider variants of these settings which may yield decidable model checking and better complexity results.
As the logic extends , it allows expressing all purely qualitative properties. It can also express purely quantitative properties, e.g.: meaning “Player has a strategy to maintain his accumulated payoff to be always positive”, or meaning “The coalition has a strategy that guarantees the value of the play for player to be at least 3”. Moreover, can naturally express combined qualitative and quantitative properties, e.g. ), etc.
The following state formulae are true at state of the GCGMP in Example 1, where is an atomic proposition true only at state , for each :
5 (Un)Decidability: Related Work and Some Preliminary Results
Generally, the GCGMP models are too rich and the language of is too expressive to expect computational efficiency, or even decidability, of either model checking or satisfiability testing. Some preliminary results and related work show that model checking of in GCGMPs is undecidable under rather weak assumptions, e.g. if the proponents or the opponents can use memory-based strategies. These undecidability results are not surprising as GCGMPs are closely related to Petri nets and vector addition systems and it is known that model checking over them is generally undecidable. In , for example, this is shown for fragments of and (state-based) over Petri nets. Essentially, the reason is that the logics allow to encode a “test for zero”; for Petri nets this means to check whether a place contains a token or not. In our setting undecidability follows for the same reason, and we will sketch some results below.
The logic restricts in the same way as restricts , due to lack of space we skip the formal definition. As a first result we show that model checking is undecidable even if only the proponents are permitted to use perfect recall strategies and the opponents are bound to memoryless strategies. More formally, let denote the class of perfect recall state-based strategies and the class of memoryless state-based strategies. That is, strategies of the former class are functions of type and of the latter class functions of type .
Undecidability can be shown using ideas from e.g. [10, 14]. Here, we make use of the construction of  to illustrate the undecidability by simulating a two-counter machine (TCM). A TCM  can be considered as a transition system equipped with two integer counters that enable/disable transitions. Each step of the machine depends on the current state, symbol on the tape, and the counters, whether they are zero or not. After each step the counters can be incremented (), or decremented () , the latter only if the respective counter is not zero. A TCM is essentially a (nondeterministic) push-down automaton with two stacks and exactly two stack symbols (one of them is the initial stack symbol) and has the same computation power as a Turing machine (cf. ). A configuration is a triple describing the current state (), the value of counter 1 () and of counter 2 (). A computation is a sequence of subsequent configurations effected by transitions.
For the simulation, we associate each counter with a player. The player’s accumulated payoff encodes the counter value; actions model the increment/decrement of the counters; guards ensure that the actions respect the state of the counters. The accepting states of the two-counter machine are encoded by a special proposition . Now, the following lemma stating the soundness of the simulation can be proved:
Lemma 1 (Reduction)
For any two-counter machine we can construct a finite GCGMP with two players and proposition such that the following holds: halts on the empty input iff contains a play with such that there exists with .
The next theorem gives two cases for which the model checking problem is undecidable. By the previous Lemma we have to ensure that the halting state is reached which can be expressed by . We can also use purely state-based guards and encode the consistency checks in the formula as follows: where the proposition is added to the model to indicate that the value of counter is zero. Not that this information is static and obtained from the transition relation of the automaton.
Model checking the logic is undecidable, even for the 2 agent case and no nested cooperation modalities, where and . This does even hold either for formulae not involving arithmetic constraints, or for state-based guards.
There are some natural semantic and syntactic restrictions of where decidability may be restored; these include for instance, the enabling of only memoryless strategies, imposing non-negative payoffs, constraints on the transition graph of the model, bounds on players utilities etc. For instance, the main reason for the undecidability result above is the possibility for negative payoffs that allow for decrementing the accumulated payoffs and thus simulating the TCM operations. Therefore, a natural restriction in the quest for restoring decidability is to consider only GCGMP models with non-negative payoffs. In this case the accumulated payoffs increase monotonically over every play of the game, and therefore the truth values of every arithmetic constraint occurring in the guards and in the formula eventually stabilize in a computable way, which in the long run reduces the model checking of any -formula in an GCGMP to a model checking of an -formula in a CGM. One can thus obtain decidability of the model checking of the logic in finite GCGMP with non-negative payoffs and perfect information. We will discuss these and other decidability results in a future work, where we will also consider restrictions similar to .
6 Concluding Remarks
This paper proposes a long-term research agenda bringing together issues, techniques and results from several research fields. It aims at bridging the two important aspects of reasoning about objectives and abilities of players in multi-player games: quantitative and qualitative, and eventually providing a uniform framework for strategic reasoning in multi-agent systems.
Acknowledgements: We thank the anonymous referees for detailed and helpful comments and additional references.
-  T. Ågotnes, V. Goranko & W. Jamroga (2007): Alternating-time Temporal Logics with Irrevocable Strategies. In D. Samet, editor: Proceedings of TARK XI, pp. 15–24, doi:http://dx.doi.org/10.1145/1324249.1324256.
-  T. Ågotnes, V. Goranko & W. Jamroga (2008): Strategic Commitment and Release in Logics for Multi-Agent Systems (Extended abstract). Technical Report IfI-08-01, Clausthal University of Technology.
-  Natasha Alechina, Brian Logan, Nguyen Hoang Nga & Abdur Rakib (2011): Logic for coalitions with bounded resources. J. Log. Comput. 21(6), pp. 907–937, doi:http://dx.doi.org/10.1093/logcom/exq032.
-  Luca de Alfaro, Thomas A. Henzinger & Orna Kupferman (2007): Concurrent reachability games. Theor. Comput. Sci. 386(3), pp. 188–217, doi:http://dx.doi.org/10.1016/j.tcs.2007.07.008.
-  R. Alur, T. A. Henzinger & O. Kupferman (2002): Alternating-Time Temporal Logic. Journal of the ACM 49, pp. 672–713, doi:http://dx.doi.org/10.1145/585265.585270.
-  Patricia Bouyer, Romain Brenguier, Nicolas Markey & Michael Ummels (2011): Nash Equilibria in Concurrent Games with Büchi Objectives. In S. Chakraborty & A. Kumar, editors: FSTTCS’2011 LIPIcs, Schloss Dagstuhl - Leibniz-Zentrum fuer Informatik, pp. 375–386, doi:http://dx.doi.org/10.4230/LIPIcs.FSTTCS.2011.375.
-  Patricia Bouyer, Romain Brenguier, Nicolas Markey & Michael Ummels (2012): Concurrent Games with Ordered Objectives. In L. Birkedal, editor: Proc. of FoSSaCS’2012, Springer LNCS, vol. 7213, pp. 301–315, doi:http://dx.doi.org/10.1007/978-3-642-28729-9_20.
-  T. Brihaye, A. Da Costa, F. Laroussinie & N. Markey (2008): ATL with Strategy Contexts and Bounded Memory. Technical Report LSV-08-14, ENS Cachan, doi:http://dx.doi.org/10.1007/978-3-540-92687-0_7.
-  Nils Bulling & Berndt Farwer (2010): On the (Un-)Decidability of Model-Checking Resource-Bounded Agents. In H. Coelho & M. Wooldridge, editors: Proc. of ECAI 2010, IOS Press, Amsterdam, pp. 567–572, doi:http://dx.doi.org/10.3233/978-1-60750-606-5-567.
-  Krishnendu Chatterjee, Luca de Alfaro & Thomas A. Henzinger (2011): Qualitative concurrent parity games. ACM Trans. Comput. Log. 12(4), p. 28, doi:http://dx.doi.org/10.1145/1970398.1970404.
-  Krishnendu Chatterjee & Laurent Doyen (2012): Energy parity games. Theor. Comput. Sci. 458, pp. 49–60, doi:http://dx.doi.org/10.1016/j.tcs.2012.07.038.
-  Krishnendu Chatterjee & Thomas A. Henzinger (2012): A survey of stochastic -regular games. J. Comput. Syst. Sci. 78(2), pp. 394–413. Available at http://dx.doi.org/10.1016/j.jcss.2011.05.002.
-  Javier Esparza: Decidability of Model Checking for Infinite-State Concurrent Systems. Acta Informatica 34, pp. 85–107, doi:http://dx.doi.org/10.1007/s002360050074.
-  Javier Esparza (1998): Decidability and complexity of Petri net problems - an Introduction. In: In Lectures on Petri Nets I: Basic Models, Springer-Verlag, pp. 374–428, doi:http://dx.doi.org/10.1007/3-540-65306-6_20.
-  Valentin Goranko & Wojciech Jamroga (2004): Comparing Semantics of Logics for Multi-agent Systems. Synthese 139(2), pp. 241–280, doi:http://dx.doi.org/10.1023/B:SYNT.0000024915.66183.d1.
-  Erich Grädel & Michael Ummels (2008): Solution Concepts and Algorithms for Infinite Multiplayer Games. In Krzysztof Apt & Robert van Rooij, editors: New Perspectives on Games and Interaction, Texts in Logic and Games 4, Amsterdam University Press, pp. 151–178. Available at http://www.logic.rwth-aachen.de/~ummels/knaw07.pdf.
-  JE Hopcroft & JD Ullman (1979): Introduction to Automata Theory, Languages, and Computation. Addison-Wesley, Reading, Massachusetts, doi:http://dx.doi.org/10.1145/568438.568455.
-  W. Jamroga & T. Ågotnes (2007): Constructive Knowledge: What Agents Can Achieve under Incomplete Information. Journal of Applied Non-Classical Logics 17(4), pp. 423–475, doi:http://dx.doi.org/10.3166/jancl.17.423-475.
-  Dario Della Monica, Margherita Napoli & Mimmo Parente (2011): On a Logic for Coalitional Games with Priced-Resource Agents. Electr. Notes Theor. Comput. Sci. 278, pp. 215–228, doi:http://dx.doi.org/10.1016/j.entcs.2011.10.017.
-  M. Osborne & A. Rubinstein (1994): A Course in Game Theory. MIT Press.
-  M. Pauly (2002): A Modal Logic for Coalitional Power in Games. J. of Logic and Computation 12(1), pp. 149–166, doi:http://dx.doi.org/10.1093/logcom/12.1.149.
-  Sophie Pinchinat (2007): A Generic Constructive Solution for Concurrent Games with Expressive Constraints on Strategies. In K. Namjoshi et al, editor: Proc. of ATVA’2007, Springer LNCS, vol. 4762, pp. 253–267, doi:http://dx.doi.org/10.1007/978-3-540-75596-8_19.