# Evolutionary dynamics with stochastic game transitions

###### Abstract.

The environment has a strong influence on a population’s evolutionary dynamics. Driven by both intrinsic and external factors, the environment is subject to continuous change in nature. To model an ever-changing environment, we develop a framework of evolutionary dynamics with stochastic game transitions, where individuals’ behaviors together with the games they play in one time step decide the games to be played next time step. Within this framework, we study the evolution of cooperation in structured populations and find a simple rule: natural selection favors cooperation over defection if the ratio of the benefit provided by an altruistic behavior, , to the corresponding cost, , exceeds , which means , where is the average number of neighbors and captures the effects from game transitions. We show that even if each individual game opposes cooperation, allowing for a transition between them can result in a favorable outcome for cooperation. Even small variations in different games being played can promote cooperation markedly. Our work suggests that interdependence between the environment and the individuals’ behaviors may explain the large-scale cooperation in realistic systems even when cooperation is expensive relative to its benefit.

## 1. Introduction

Cooperation is an integral part of biological systems and is of paramount importance to their prosperity. But cooperation, an altruistic act of bearing a cost to provide another individual with a benefit, reduces the survival advantage for the donor while fostering that of the recipient. Understanding how cooperation can be maintained in a competitive world has long been a focal issue in evolutionary biology and ecology [1]. The spatial distribution of a population makes an individual more likely to interact with neighbors than with those who are more distant, which affects evolutionary dynamics [2]. Past decades have seen an intensive investigation of the evolution of cooperation in structured populations [2, 3, 4, 5, 6, 7, 8]. One of the best-known findings is that weak selection favors cooperation if the ratio of the benefit provided by an altruistic act, , to the cost of expressing such an altruistic trait, , exceeds the average number of neighbors, , i.e. [3, 6]. This simple rule strongly supports the proposition that population structures can promote cooperation, and it is widely accepted as one of the major mechanisms responsible for the evolution of cooperation [1].

Despite these deep insights, empirical studies have produced evidence that many realistic systems are highly-connected, with each individual having many neighbors on average [9]. For example, in the actor-collaboration network, every actor has collaborators on average, meaning [9]. In such cases, the threshold for establishing cooperation, based on the rule , is extremely high: the benefit from an altruistic act must be at least times larger than its cost for cooperators to be favored over defectors. The rule, although providing valuable intuition about when cooperation can evolve in structured populations, is therefore often unattainable in practice [10]. The underlying mechanism for large-scale cooperation in these systems, especially when cooperation is sufficiently costly, is of practical significance but still largely remains an open question in the field. Here, we consider a natural way in which the cost-to-benefit threshold can be relaxed significantly.

In evolutionary game theory, the games played within a population affect individuals’ reproductive success. Most prior studies have relied on an assumption that the environment in which individuals evolve is time-invariant, meaning the individuals play a fixed game throughout the evolutionary process. However, this assumption is not always realistic and can represent an oversimplification of reality [11], as many experimental studies have shown that the environment individuals face changes over time (and often) [12, 13, 14, 15]. Typically, overgrazing leads to the degradation of the common pasture land, leaving herders with fewer resources to utilize in subsequent seasons. By constraining the number of livestock within a reasonable range, herders can achieve a more sustainable use of pasture land [16]. In this type of population, individuals’ actions influence the state of environment, which in turn impacts the actions taken by its members. Apart from endogenous factors like individuals’ actions, exogenous factors like seasonal climate fluctuations and soil conditions can also modify the environment experienced by the individuals. Examples are not limited to human-related activities but also appear in various microbial systems including bacteria and viruses [14, 15].

In this study, we use graphs to model a population’s spatial structure, where nodes represent individuals and edges describe their interactions. We propose a framework of evolutionary dynamics with stochastic game transitions: individuals sharing an edge interact (“play a game”) in each time step, and their strategic actions together with the game played determine the game to be played in the next time step. We find that the game transitions can lower the threshold for establishing cooperation by , which means , where describes how the game transitions affect the evolutionary outcome. We find that even if cooperation is opposed in each individual game, game transitions can favor the evolution of cooperation. In particular, a slight difference between games can dramatically lower the barrier for the success of cooperators. Our work provides a possible explanation for widespread cooperation. It also suggests that transitions between games, if designed properly, give a promising mechanism for overcoming social dilemmas and achieving global cooperation.

## 2. Models

We study a population of players consisting of cooperators and defectors. The population structure is described by a graph. Each player occupies a node on the graph. Edges between nodes describe the events related to interactions and biological reproduction (or behavior imitation). In each time step, each player interacts separately with every neighbor, and the games played in different interactions can be distinct (Fig. 1A). When playing game , mutual cooperation brings each player a “reward”, , whereas mutual defection leads to an outcome of “punishment”, ; unilateral cooperation leads to a “sucker’s payoff”, , for the cooperator and a “temptation”, , for the defector. We assume that each game is a prisoner’s dilemma, meaning . Each player derives an accumulated payoff, , from all interactions, and this payoff is translated into reproductive fitness using the formula , where represents the intensity of selection [17]. We are concerned with the effects of weak selection [18, 19], meaning .

At the end of each time step, one player is selected for death uniformly at random from the population. The neighbors of this player then compete for the empty site, with each neighbor sending an offspring to this location with probability proportional to fitness. Following this “death-birth” update step, the games played in the population also update based on the previous games played and the actions taken in those games (Fig. 1B). For the player occupying the empty site, the games it will play are determined by the interactions of the prior occupant.

The game transition can be deterministic or stochastic (probabilistic). Deterministic transitions are just special cases of stochastic transitions. If the game to be played is independent of the previous game, then the game transition is “state-independent” [11]. When the game that will be played depends entirely on the previous game, the game transition is “behavior-independent”. The simplest case is that games in all interactions are identical initially and remain constant throughout the evolutionary process, which corresponds to the classical setup in most prior studies [3].

## 3. Results

In the absence of mutation, a finite population will eventually reach a monomorphic state in which all players have the same strategy, either all-cooperation or all-defection. We study the competition between cooperation and defection by comparing the fixation probability of a single cooperator, , to that of a single defector, . Concretely, is the probability that a cooperator starting in a random location generates a lineage that takes over the entire population. Analogously, is the probability that a defector in a random position turns a population of cooperators into defectors. We say that selection favors cooperators relative to defectors if [17].

### 3.1. Game transitions between two states

We begin with the case of deterministic game transitions between two states. Here each state corresponds to a donation game (Fig. 2A; see SI Appendix, sections 3 and 4 for a comprehensive investigation of two-state games). In game , a cooperator bears a cost of to bring its opponent a benefit of , and the defector does nothing. Analogously, in game , a cooperator pays a cost of to bring its opponent a benefit of . That is, , , , and in game . Both and are larger than . The preferred choice for each player is defection, but in each game, resulting in the dilemma of cooperation. We say that game is more valuable than game if . Assuming , the game transitions outlined in Fig. 2A imply that mutual cooperation leads to a more valuable game than any other combination of actions since it is the only way to move to and stay in game .

If every player has neighbors (i.e. the graph is “regular”), we find that

(1) |

where and . Note that is positive and independent of payoff values such as , and . We obtain this condition under weak selection and based on the assumption that the population size is much larger than . When , the two games are the same, which leads to the well-known rule of for cooperation evolving on regular graphs [3]. The existence of the term indicates that transitions between different games can reduce the barrier for the success of cooperation. Even when both games oppose cooperation individually, i.e. and , transitions between them such as those described in Fig. 2A can promote cooperation (see Fig. 2B). Our analytical results agree well with numerical simulations.

The beneficial effects of stochastic game transitions on cooperation become more prominent on graphs with a large degree, . We find that a slight difference between games and , , can remarkably lower the barrier for cooperation evolving. For example, when and , the critical benefit-to-cost ratio decreases from to for (Fig. 2C). Therefore, stochastic game transitions may provide a possible explanation for the persistence of cooperation in realistic and highly-connected societies [9]. We find that similar results hold under the closely-related “imitation” update rule (SI Appendix, Fig. S1 and section 3).

Next, we consider “birth-death” [20] and “pairwise-comparison” [21, 22] updating. Under birth-death updating, in each time step a random player is selected for reproduction with probability proportional to its fitness. The offspring replaces a random neighbor. Under pairwise-comparison updating, a player is first selected uniformly-at-random to update his or her strategy. When player is chosen for a strategy updating, it randomly chooses a neighbor and compares payoffs. If and are the payoffs to and , respectively, player adopts ’s strategy with probability and retains its old strategy otherwise. For the game transition pattern shown Fig. 2A, under both birth-death and pairwise-comparison updating, we have the simple rule (SI Appendix, sections 3 and 4)

(2) |

where . When the two games are the same, . Therefore, cooperators are never favored over defectors when the players play a fixed game (Fig. 3AC). Stochastic game transitions provide a chance for cooperation to thrive as long as , which opens an avenue for the evolution of cooperation under birth-death and pairwise-comparison updating. One can attribute this result to the fact that under the transition pattern of Fig. 2A, mutual cooperation results in but when two players use different actions, the cooperator gets and the defector gets . If , then it must be true that , which means that the players are effectively in a coordination game with a preferred outcome of mutual cooperation.

More intriguingly, Eq. 2 shows that the success of cooperators fully relies on the difference between benefits provided by an altruistic behavior in game and game , and it is independent of the exact value in each game (Fig. 3BD). Thus, in a dense population where individuals have many neighbors, even if the benefits provided by an altruistic behavior are low in both game 1 and game 2, transitions between them can still support the evolution of cooperation. In particular, we stress that the difference between the two games required to favor cooperation is surprisingly small. For example, warrants the success of cooperation over defection on graphs of any degree.

We further examine random graphs [23] and scale-free networks [24], where players differ in the number of their neighbors (SI Appendix, Fig. S2). We find that the stochastic game transitions can provide more advantages for the evolution of cooperation than their static counterparts under death-birth and imitation updating, and it gives a way for cooperation to evolve under birth-death and pairwise-comparison updating. In addition, we study evolutionary processes with mutation or behavior exploration (SI Appendix, Fig. S3). The results demonstrate the robustness of the effects of game transitions on the evolution of cooperation.

### 3.2. Game transitions among states

We proceed with the general setup of game transitions among states (i.e. games through ). If two players play game in the current time step and among them there are cooperators, they will play game in the next time step with probability . is for mutual cooperation, for unilateral cooperation/defection, and for mutual defection. In the example of Fig. 2A, we have and since mutual defection () leads to the game transition from game 1 () to game 2 (). This setup can recover deterministic or probabilistic transitions, state-dependent or -independent (SI Appendix, section 4), behavior-dependent or -independent (SI Appendix, section 4), and the traditional framework for playing only a single game [3, 5, 6], as specific cases. We assume that all games are donation games (see SI Appendix, section 3 for any two-player, two-strategy game). In game , a cooperator pays a cost of to bring its opponent a benefit of . Game is the most valuable, meaning for every .

Under death-birth updating, we find that

(3) |

where, for every , , and depends on the game transition pattern but is independent of the benefit in each game, , and cost, (see SI Appendix, sections 3 and 4 for the calculation of ). In fact, the effects of stochastic game transitions on cooperation arise from two sources: the game transition pattern and the variation in games. The former is captured by and the latter by . Eq. 3 shows that the term, , exactly captures how stochastic game transitions influence cooperation. Let denote and let denote . We can interpret Eq. 3 intuitively: weak selection favors cooperation if the ratio of the benefit from an altruistic behavior, , to its cost, , exceeds the average effective number of neighbors, . Analogously, under birth-death or pairwise-comparison updating, we find that

(4) |

We refer the reader to SI Appendix, sections 3 and 4 for the calculation of .

Eq. 3 tells us how different environmental transition patterns affect the evolution of a system. In a two-player interaction, players’ behavior profiles can be mutual cooperation, unilateral cooperation/defection, or mutual defection. We can ask under which behavior profile the game transition has the most prominent impact on evolutionary outcomes. Answering this question is of practical importance since it provides us with insight into finding the right time to intervene in the environment in order to achieve a socially-optimal evolutionary outcome for the population.

We study transitions among three states, namely, the most valuable game (game 1), a moderately valuable game (game 2), and the least valuable game (game 3). We fix the game transition patterns for two behavior profiles and test how varying the transition pattern for the third behavior profile affects the threshold for establishing cooperation (Fig. 4). For example, Fig. 4A shows that when defection (either 1C or 0C) leads to game 1, the critical benefit-to-cost ratio changes notably as the transition pattern responding to mutual cooperation (2C) varies. We find similar results for the behavior profile of unilateral cooperation/defection (Fig. 4B). When the transition pattern responding to mutual defection varies, however, the effects on cooperation are negligible (Fig. 4C). Therefore, game transitions in response to the behavior profile of mutual cooperation or unilateral cooperation/defection can have extremely important effects on evolutionary outcomes. To build a cooperative society, game transitions responding to the two behavior profiles should be carefully designed. A simple and efficient solution is transitioning to the most valuable game once the two players cooperate and to the least valuable game immediately once defection appears.

### 3.3. Pure versus stochastic strategies

So far, in every time step each player is either a cooperator or a defector. But the framework we propose here has a much broader scope than just two pure strategies. For example, we also investigate the competition between stochastic strategies under stochastic game transitions. Let denote a stochastic strategy with which, in each time step, a player chooses cooperation with probability and defects otherwise. thus corresponds to a pure cooperator and a pure defector. We find that the condition for being favored by selection over still follows the format of Eq. 3 under death-birth updating and Eq. 4 under birth-death/pairwise-comparison updating, provided that is modified (SI Appendix, sections 3 and 4). When the game transition patterns follow Fig. 2A under death-birth updating, stochastic game transitions lower the threshold for a cooperative strategy ( with a large ) being favored relative to a less cooperative strategy. We also find that game transitions can favor the evolution of a cooperative stochastic strategy under birth-death/pairwise-comparison updating.

### 3.4. Global versus local game transitions

Our study above assumes that in each time step, games played by any two players are likely to update (“global” transitions). But in some cases, players could present different tendencies in modifying the environment in which they evolve. For example, under death-birth updating, if player is selected for death, then only ’s nearest neighbors compete to reproduce and replace with an offspring. Compared with those not involved in competition to the vacant site, individuals close to the individual to be replaced have stronger incentives to change the environment they face, since the environment indeed affects their success of the replacement. In other words, games played by the nearest neighbors of the dead drive the evolution of a system. Therefore one could impose transitions only to these games, leading to “local” transitions (see Fig. 5A).

Birth-death updating requires competition at the population level, so global and local transitions are identical in this case. For death-birth and pairwise-comparison updating, global and local transitions lead to decidedly different models. We show that, however, the simple rules for cooperation to evolve (Eqs. 3 and 4) still hold provided is modified (SI Appendix, section 1). Specifically, when the game transition pattern follows Fig. 2A, under death-birth updating we have if and only if , where , different from for global transitions (see Eq. 1). Under pairwise-comparison updating, if and only if , where (SI Appendix, section 4).

According to the nature of the critical threshold ( for death-birth updating and for pairwise-comparison updating), global transitions act as a more effective promoter of cooperation than local transitions do (Fig. 5BC). But for both kinds of game transitions, many messages are qualitatively the same: game transitions promote cooperation (Fig. 2, Fig. 3, SI Appendix, Fig. S4); game transitions notably amplify the beneficial effects of game variations on cooperation (Fig. 2, Fig. 3, SI Appendix, Fig. S5); and game transitions responding to mutual cooperation or unilateral cooperation/defection critically affect cooperation (Fig. 4, SI Appendix, Fig. S6). We include a more detailed discussion of global versus local game transitions in SI Appendix.

## 4. Discussion

In this work, we consider evolutionary dynamics with stochastic game transitions, coupling individuals’ actions with the environment. Individuals’ behaviors modify the environment, which in turn affects the viability of future actions in that environment. We find a simple rule for the success of cooperators in an environment that can switch between an arbitrary number of states, namely , where exactly captures how the games and their transitions affect the evolution of cooperation. When all environmental states are identical, we recover the rule [3].

We first study an intuitive scenario: mutual cooperation leads to a valuable game while the existence of defection results in a relatively undesirable game. We find that even if both of the games oppose cooperation individually, transitions between them can allow cooperation to flourish. Moreover, the success of cooperators largely depends on the variation in the two games, and even a slight variation can remarkably lower the threshold for establishing cooperation.

For birth-death and pairwise-comparison updating, weak selection disfavors cooperation in any homogeneous structured population when the environment (game) is constant [3, 25]. Given the practical significance of the two update rules [26], these results are disappointing. However, stochastic game transitions give hope for cooperation evolving under the two rules. Importantly, the variation in the two games, rather than what exactly each one is like individually, determines whether or not cooperation is favored over defection. Thus, even when individuals play two relatively undesirable games, cooperation can still flourish due to game transitions.

Our findings are of great significance to understanding large-scale cooperation in many highly-connected social networks. In these networks, an individual can have hundreds of neighbors [9, 27], and cooperators thus face the risk of being exploited by lots of neighboring defectors. If the environment remains constant, cooperation must be profitable enough to make up for exploitation by defection—the benefit produced by a cooperative behavior should be hundreds of times as large as its cost [3]. This requirement is often unrealistic in real-world social networks. Stochastic game transitions thus provide a possible explanation for the prevailing cooperation in such populations. Our findings hold for various populations structures, from regular and random graphs to scale-free networks.

The main reason that spatial structure can promote cooperation is that local strategy dispersal leads to an assortment of cooperators [2, 3]. Cooperators can resist the invasion of defectors through more interactions with cooperators. But when mutation or random strategy exploration is allowed, a defector is expected to arise within a cluster of cooperators, which then dilutes the spatial assortment. Mutation thus hinders the evolution of cooperation [28]. When the environment changes as a result of individuals’ behaviors, although the defecting mutant indeed exploits its neighboring cooperators temporarily, the environment in which this happens deteriorates rapidly. As a result, the temptation to defect is weakened. In addition, in a constant environment selection favors the establishment of spatial assortment while mutation destroys it continuously. The population state finally reaches a “mutation-selection stationary (MSS)” distribution. But when the environment is subject to transitions, the interacting environment would also be a part of this distribution. We refer to the joint distribution over individuals’ states and games as a “game-mutation-selection stationary (GMSS)” distribution, which is an important avenue for future investigations.

Recent years have seen a growing interest in exploring evolutionary dynamics in a changing environment [29, 30, 31, 32, 33, 34, 35, 36]. Our framework here is somewhat different. First, our study accounts for both exogenous factors and individuals’ behaviors in the change of the environment, modeling general environmental feedback. In addition, the environment that two players face is independent of that for another pair of players. Individuals’ strategic behaviors directly influence the environment in which they evolve, which enables an individual to reciprocate with its opponent in a single interaction through environmental feedback. Therefore, even if cooperators are disfavored in each individual environment, cooperators can still be favored over defectors through environmental reciprocity. Such an effect has never been observed in prior studies where all individuals interact in a homogeneous environment [29, 31]. In those studies, although at different times the environments individuals face are different, at any specific stage the environment is identical for all individuals. When defection is a dominant strategy in each individual environment, defection also dominates cooperation in the context of an ever-changing environment [29, 30, 31]. In a recent seminal work, Hilbe et. al. found that individuals can rely on repeated interactions and continuous strategies to achieve environmental reciprocity [11]. Compared with their model, in our setup individuals play a one-shot game with a pure, unconditional strategy. Our framework shows that without relying on direct reciprocity and any strategic complexity, stochastic game transitions still promote the evolution of cooperation.

The mathematical formulas obtained herein tell us which environmental transition patterns facilitate cooperation and provide possible solutions to relax the dilemma of cooperation. To achieve this, a simple and effective transition pattern is one in which mutual cooperation leads to a valuable game, while defection leads to an immediate transition to a less-desirable game. In this paper, we incorporate interdependence of the interacting environment and individuals’ behaviors into an evolving population. Within this framework, we can further discuss many important issues. For example, by adjusting the transition probability, we can study how different time scales for the evolution of strategies and for the transition of environment affect evolutionary outcomes [31]. In addition, playing edge-dependent games in structured populations has attracted much attention [37, 38]. A prior study has found that when the environment in each interaction is edge-dependent, the evolutionary process can be approximated by that in a transformed and unified environment [37]. Our work provides further, promising results about the dynamics of edge-dependent games.

## Acknowledgments

The authors gratefully acknowledge support from the National Natural Science Foundation of China (grants 61751301, 61533001), the China Scholar Council (grant 201706010277), the Bill & Melinda Gates Foundation (grant OPP1148627), the Army Research Laboratory (grant W911NF-18-2-0265), and the Office of Naval Research (grant N00014-16-1-2914).

## References

- Nowak [2006] M. A. Nowak. Five rules for the evolution of cooperation. Science, 314(5805):1560–1563, 2006. doi: 10.1126/science.1133755.
- Nowak and May [1992] M. A. Nowak and R. M. May. Evolutionary games and spatial chaos. Nature, 359(6398):826–829, 1992. doi: 10.1038/359826a0.
- Ohtsuki et al. [2006] H. Ohtsuki, C. Hauert, E. Lieberman, and M. A. Nowak. A simple rule for the evolution of cooperation on graphs and social networks. Nature, 441(7092):502–505, 2006. doi: 10.1038/nature04605.
- Taylor et al. [2007] P. D. Taylor, T. Day, and G. Wild. Evolution of cooperation in a finite homogeneous graph. Nature, 447(7143):469–472, 2007. doi: 10.1038/nature05784.
- Nowak et al. [2010] M. A. Nowak, C. E. Tarnita, and T. Antal. Evolutionary dynamics in structured populations. Philos. Trans. R. Soc. B-Biol. Sci., 365(1537):19–30, 2010. doi: 10.1098/rstb.2009.0215.
- Rand et al. [2014] D. G. Rand, M. A. Nowak, J. H. Fowler, and N. A. Christakis. Static network structure can stabilize human cooperation. Proc. Natl. Acad. Sci. U.S.A., 111(48):17093, 2014. doi: 10.1073/pnas.1400406111.
- Allen et al. [2017] B. Allen, G. Lippner, Y.-T. Chen, B. Fotouhi, N. Momeni, S.-T. Yau, and M. A. Nowak. Evolutionary dynamics on any population structure. Nature, 544:227, 2017. doi: 10.1038/nature21723.
- Su et al. [2019a] Q. Su, A. Li, L. Wang, and H. E. Stanley. Spatial reciprocity in the evolution of cooperation. Proc. R. Soc. B-Biol. Sci., 286(1900):20190041, 2019a. doi: 10.1098/rspb.2019.0041.
- Watts and Strogatz [1998] D. J. Watts and S. H. Strogatz. Collective dynamics of âsmall-worldâ?networks. Nature, 393(6684):440–442, 1998. doi: 10.1038/30918.
- Sánchez [2018] A. Sánchez. Physics of human cooperation: experimental evidence and theoretical models. J. Stat. Mech: Theory Exp., 2018(2):024001, 2018. doi: 10.1088/1742-5468/aaa388.
- Hilbe et al. [2018] C. Hilbe, Š. Šimsa, K. Chatterjee, and M. A. Nowak. Evolution of cooperation in stochastic games. Nature, 559(7713):246–249, 2018. doi: 10.1038/s41586-018-0277-x.
- Levin [2014] S. A. Levin. Public goods in relation to competition, cooperation, and spite. Proc. Natl. Acad. Sci. U.S.A., 111(Supplement 3):10838–10845, 2014. doi: 10.1073/pnas.1400830111.
- Franzenburg et al. [2013] S. Franzenburg, S. Fraune, P. M. Altrock, S. Künzel, J. F. Baines, A. Traulsen, and T. C. G. Bosch. Bacterial colonization of hydra hatchlings follows a robust temporal pattern. ISME J., 7:781, 2013. doi: 10.1038/ismej.2012.156.
- McFall-Ngai et al. [2013] M. McFall-Ngai, M. G. Hadfield, T. C. G. Bosch, H. V. Carey, T. Domazet-Lošo, A. E. Douglas, N. Dubilier, G. Eberl, T. Fukami, S. F. Gilbert, U. Hentschel, N. King, S. Kjelleberg, A. H. Knoll, N. Kremer, S. K. Mazmanian, J. L. Metcalf, K. Nealson, N. E. Pierce, J. F. Rawls, A. Reid, E. G. Ruby, M. Rumpho, J. G. Sanders, D. Tautz, and J. J. Wernegreen. Animals in a bacterial world, a new imperative for the life sciences. Proc. Natl. Acad. Sci. U.S.A., 110(9):3229–3236, 2013. doi: 10.1073/pnas.1218525110.
- Acar et al. [2008] M. Acar, J. T. Mettetal, and A. van Oudenaarden. Stochastic switching as a survival strategy in fluctuating environments. Nat. Genet., 40:471, 2008. doi: 10.1038/ng.110.
- Rankin et al. [2007] D. J. Rankin, K. Bargum, and H. Kokko. The tragedy of the commons in evolutionary biology. Trends Ecol. Evol, 22(12):643 – 651, 2007. doi: 10.1016/j.tree.2007.07.009.
- Nowak et al. [2004] M. A. Nowak, A. Sasaki, C. Taylor, and D. Fudenberg. Emergence of cooperation and evolutionary stability in finite populations. Nature, 428(6983):646–650, 2004. doi: 10.1038/nature02414.
- Wu et al. [2010] B. Wu, P. M. Altrock, L. Wang, and A. Traulsen. Universality of weak selection. Phys. Rev. E, 82(4):046106, 2010. doi: 10.1103/PhysRevE.82.046106.
- Wu et al. [2013] B. Wu, J. García, C. Hauert, and A. Traulsen. Extrapolating weak selection in evolutionary games. PLoS Comput. Biol., 9(12):e1003381, 2013. doi: 10.1371/journal.pcbi.1003381.
- Moran [1958] P. A. P. Moran. Random processes in genetics. Math. Proc. Cambridge Philos. Soc., 54(01):60, Jan 1958. doi: 10.1017/s0305004100033193.
- Szabó and Tőke [1998] G. Szabó and C. Tőke. Evolutionary prisoner’s dilemma game on a square lattice. Phys. Rev. E, 58(1):69–73, 1998. doi: 10.1103/physreve.58.69.
- Traulsen et al. [2007] A. Traulsen, J. M. Pacheco, and M. A. Nowak. Pairwise comparison and selection temperature in evolutionary game dynamics. J. Theor. Biol., 246(3):522–529, Jun 2007. doi: 10.1016/j.jtbi.2007.01.002.
- Erdös and Rényi [1960a] P. Erdös and A. Rényi. On the evolution of random graphs. Publ. Math. Inst. Hung. Acad. Sci., 5:17–61, 1960a.
- Barabási and Albert [1999a] A.-L. Barabási and R. Albert. Emergence of scaling in random networks. Science, 286(5439):509–512, 1999a. doi: 10.1126/science.286.5439.509.
- Allen and Nowak [2014] B. Allen and M. A. Nowak. Games on graphs. EMS Surv. Math. Sci., 1:113–151, 2014. doi: 10.4171/EMSS/3.
- Traulsen et al. [2010] A. Traulsen, D. Semmann, R. D. Sommerfeld, H.-J. Krambeck, and M. Milinski. Human strategy updating in evolutionary games. Proc. Natl. Acad. Sci. U.S.A., 107(7):2962–2966, 2010. doi: 10.1073/pnas.0912515107.
- Amaral et al. [2000] L. A. N. Amaral, A. Scala, M. Barthélémy, and H. E. Stanley. Classes of small-world networks. Proc. Natl. Acad. Sci. U.S.A., 97(21):11149–11152, 2000. doi: 10.1073/pnas.200327197.
- Allen et al. [2012] B. Allen, A. Traulsen, C. E. Tarnita, and M. A. Nowak. How mutation affects evolutionary games on graphs. J. Theor. Biol., 299:97–105, 2012. doi: 10.1016/j.jtbi.2011.03.034.
- Ashcroft et al. [2019] P. Ashcroft, P. M. Altrock, and T. Galla. Fixation in finite populations evolving in fluctuating environments. J. R. Soc. Interface, 11(100):20140663, May 2019. doi: 10.1098/rsif.2014.0663.
- Assaf et al. [2013] M. Assaf, M. Mobilia, and E. Roberts. Cooperation dilemma in finite populations under fluctuating environments. Phys. Rev. Lett., 111:238101, 2013. doi: 10.1103/PhysRevLett.111.238101.
- Weitz et al. [2016] J. S. Weitz, C. Eksin, K. Paarporn, S. P. Brown, and W. C. Ratcliff. An oscillating tragedy of the commons in replicator dynamics with game-environment feedback. Proc. Natl. Acad. Sci. U.S.A., 113(47):E7518–E7525, 2016. doi: 10.1073/pnas.1604096113.
- Gokhale and Hauert [2016] C. S. Gokhale and C. Hauert. Eco-evolutionary dynamics of social dilemmas. Theor. Popul. Biol., 111:28–42, 2016. doi: https://doi.org/10.1016/j.tpb.2016.05.005.
- Hauert et al. [2006] C. Hauert, M. Holmes, and M. Doebeli. Evolutionary games and population dynamics: maintenance of cooperation in public goods games. Proc. R. Soc. B-Biol. Sci., 273(1600):2565–2571, 2006. doi: 10.1098/rspb.2006.3600.
- Stewart and Plotkin [2014] A. J. Stewart and J. B. Plotkin. Collapse of cooperation in evolving games. Proc. Natl. Acad. Sci. U.S.A., 111(49):17558–17563, 2014. doi: 10.1073/pnas.1408618111.
- Tilman et al. [2019] A. R. Tilman, J. B. Plotkin, and E. Akçay. Evolutionary games with environmental feedbacks. bioRxiv, 2019. doi: https://doi.org/10.1101/493023.
- Hauert et al. [2019] C. Hauert, C. Saade, and A. McAvoy. Asymmetric evolutionary games with environmental feedback. J. Theor. Biol., 462:347–360, 2019. doi: 10.1016/j.jtbi.2018.11.019.
- McAvoy and Hauert [2015] A. McAvoy and C. Hauert. Asymmetric evolutionary games. PLoS Comput. Biol., 11(8):e1004349, 2015. doi: 10.1371/journal.pcbi.1004349.
- Su et al. [2019b] Q. Su, L. Zhou, and L. Wang. Evolutionary multiplayer games on graphs with edge diversity. PLoS Comput. Biol., 15(4):e1006947, 2019b. doi: 10.1371/journal.pcbi.1006947.

SUPPLEMENTARY INFORMATION (SI)

The supplementary information is structured as follows.

In Section 1, we derive a general expression of the critical benefit-to-cost ratio for local game transitions. We study three updating processes, namely death-birth, imitation, and pairwise comparison updating. The birth-death updating process leads to the same result under both local game transitions and global game transitions. We investigate this process in Section 4.

In Section 2, we discuss how the initial conditions (the initial fractions of various games played in the population) affect the evolutionary outcomes. We provide an effective approach to evaluate the sensitivity of evolutionary dynamics to the initial condition.

In Section 3, we derive a general expression of the critical benefit-to-cost ratio for global game transitions. We study four updating processes, namely death-birth, birth-death, imitation, and pairwise comparison updating. We prove that our rules also apply to the case when players use stochastic strategies (cooperate or defect with a probability rather than unconditionally).

In Section 4, we study four representative examples, including state-independent game transitions (the game to be played is independent of the previous games played), strategy-independent game transitions (the game to be played is independent of interactants’ strategic actions in the past), game transitions between two states (the example shown in the main text), and probabilistic game transitions between three states (transitions between different games with a probability). We show that how probabilistic game transitions affect the favorable effects of game transitions on cooperation may depend on the variations in different games.

## section 1. Evolutionary dynamics with local stochastic game transitions

Here we consider stochastic game transitions among states, described by game , game , , game . The payoff structure of game is

(5) |

where each value corresponds to a payoff derived by a player with a strategy in the column against a player with a strategy in the row. The game transition pattern is described by three matrixes, i.e.,

(6) |

where represents the probability that players play game in the next time step conditioned on that they play game in the current time step and there are -players, where and .

On graphs or social networks, each player occupies a node.
If two players (or nodes occupied by players) are connected by an edge or a social tie, they play a one-shot game in each time step.
The main idea about the theoretical analysis is to couple the game played by two connected players and their strategy profiles into edges.
Let denote an edge in which the two connected players take strategy and respectively () and they play game in the current time step ().
For example, in edge , both the two connected players take strategy and they play game .
We then introduce the following variables to describe this evolving system:

: the frequency of -players;

: the frequency of -players;

: the frequency of edge ;

: the probability to find an edge given that one node of this edge is occupied by a player;

: the frequency of edges that connect an -player and a -player;

: the conditional probability to find an player given that the adjacent node is occupied by a player.

Then we have the identities

(7a) | |||

(7b) | |||

(7c) | |||

(7d) | |||

(7e) | |||

(7f) |

Note that players’ strategies and the game they play coevolve throughout the evolutionary process. From the perspective of network dynamics, we need to consider the change in the frequency of nodes occupied by players and the frequency of edge . Based on above identities, we can use and to describe the whole system. In the following, we study a random regular graph, where each node is linked to other nodes.

### .1. Death-birth updating process

In the death-birth updating process, in each time step, a random player is selected to die; then all neighbors compete to reproduce an offspring and this offspring occupies the vacant site with the probability proportional to its fitness. We can also depict it in a social setting: a random player determines to update its strategy; subsequently, it adopts a neighbor’s strategy with a probability proportional to the neighbor’s fitness. The local transitions account for the fact that only the nearest neighbors compete for the vacancy site. Compared with other players not involved in the competition, these neighbors are more incentivized to modify the environment in which they evolve, namely, the games they played. Therefore, under local game transitions, only games played by the nearest neighbors of the dead have the chance to update. We first investigate the change in the frequency of players.

#### .1.1. Change in —updating a B-player

A -player is chosen to die with probability . Let denote the number of neighboring -players with who the focal player plays game . Analogously, denotes the number of neighboring -players with who the focal player plays game . Therefore, . The probability for such a neighborhood configuration is

(8) |

The fitness of a neighboring -player with who the focal player plays game is

(9) |

The fitness of a neighboring -player with who the focal player plays game is

(10) |

The probability that one of neighboring players replaces the vacancy under such a neighborhood configuration is given by

(11) |

The probability that one of neighboring players replaces the vacancy under such a neighborhood configuration is given by

(12) |

Therefore, increases by with probability

(13) |

#### .1.2. Change in —updating a -player

A -player is chosen to die with probability . Let denote the number of neighboring -players with who the focal player plays game . Analogously, denotes the number of neighboring -players with who the focal player plays game . Therefore, . The probability for such a neighborhood configuration is

(14) |

The fitness of a neighboring -player with who the focal player plays game is

(15) |

The fitness of a neighboring -player with who the focal player plays game is

(16) |

The probability that one of neighboring -players replaces the vacancy under such a neighborhood configuration is given by

(17) |

The probability that one of neighboring players replaces the vacancy under such a neighborhood configuration is given by

(18) |

Therefore, decreases by with probability

(19) |

#### .1.3. Change in

Let us now suppose that one replacement event takes place in one unit of time. The time derivative of is given by \linenomath

(20) |

where

(21a) | |||

(21b) | |||

(21c) | |||

(21d) |

#### .1.4. Change in

We proceed with the change in the frequency of each type of edges. Note that when a random player like is chosen to die, edges between and its nearest neighbors (abbreviated ‘neighbors’) and edges between ’s nearest neighbors and the next nearest neighbors are likely to change (see the description of local game transitions). We stress that the change in is different from that in . does not change when neighboring -players replace the focal -player (namely the dead -player) or neighboring -players replace the focal -player (namely the dead -player). But in the same case probably change since games in these edges switch, resulting in the change in edge types.

First we consider the case that a random -player is chosen to die. We take the same neighborhood configuration as we do in Section .1.1, i.e., for . The change in results from two parts: the switching of edges connecting the focal -player and its nearest neighbors, the switching of edges connecting the nearest neighbors and the next nearest neighbors. Under the given neighborhood configuration, the change in due to the former part is \linenomath

(22) |

Equation (22) describes the edge switching of , which occurs when (i) a neighboring -player occupies the vacant site, i.e., ; (ii) neighboring -players who play game with the dead in the current time step play game in the next time step, .

The change in due to edges between the nearest and the next nearest neighbors is \linenomath

(23) |

Equation (.1.4) indicates that regardless of which neighbor (player or player) replaces the focal -player, the change in due to edges between the nearest and next nearest neighbors remains the same.

Then we consider the case that a random -player is chosen to die. We take the same neighborhood configuration as we do in Section .1.2, i.e., for . The change in due to edges between the focal -player and its nearest neighbors is \linenomath

(24) |

and \linenomath

(25) |

Equation (24) captures the case when a neighboring -player successfully occupies the vacant site. Equation (25) describes the case that a neighboring -player successfully occupies the vacant site.

The change in due to edges between the nearest and next nearest neighbors is \linenomath

(26) |

#### .1.5. Change in

When a -player is selected to die and its neighbourhood configuration is the same as that in Section .1.1, the change in due to edges between the nearest and next nearest neighbors is \linenomath

(28) |

and \linenomath

(29) |

Equation (28) captures the case when a neighboring -player successfully occupies the vacant site. Equation (29) describes the case that a neighboring -player successfully occupies the vacant site.

The change in due to edges between the nearest and next nearest neighbors is \linenomath

(30) |

When an -player is selected to die and its neighbourhood configuration is the same as that in Section .1.2, the change in due to edges between the nearest and next nearest neighbors is \linenomath

(31) |

and \linenomath

(32) |

Equation (31) captures the case when a neighboring -player successfully occupies the vacant site. Equation (32) describes the case that a neighboring -player successfully occupies the vacant site.

The change in due to edges between the nearest and next nearest neighbors is \linenomath