# Consequences of fluctuating group size for the evolution of cooperation

## Abstract

Studies of cooperation have traditionally focused on discrete games such as the well-known prisoner’s dilemma, in which players choose between two pure strategies: cooperation and defection. Increasingly, however, cooperation is being studied in continuous games that feature a continuum of strategies determining the level of cooperative investment. For the continuous snowdrift game, it has been shown that a gradually evolving monomorphic population may undergo evolutionary branching, resulting in the emergence of a defector strategy that coexists with a cooperator strategy. This phenomenon has been dubbed the ‘tragedy of the commune’. Here we study the effects of fluctuating group size on the tragedy of the commune and derive analytical conditions for evolutionary branching. Our results show that the effects of fluctuating group size on evolutionary dynamics critically depend on the structure of payoff functions. For games with additively separable benefits and costs, fluctuations in group size make evolutionary branching less likely, and sufficiently large fluctuations in group size can always turn an evolutionary branching point into a locally evolutionarily stable strategy. For games with multiplicatively separable benefits and costs, fluctuations in group size can either prevent or induce the tragedy of the commune. For games with general interactions between benefits and costs, we derive a general classification scheme based on second derivatives of the payoff function, to elucidate when fluctuations in group size help or hinder cooperation.

## I Introduction

Cooperation is ubiquitous in nature, and the cooperative integration of lower-level entities into higher-level units has been vital for the development of life on earth (Marynard Smith and Szathmáry, 1995). While cooperation in the broad sense only implies joint action, the term is often used more strictly to describe situations in which cooperators help others at a cost to themselves. These interactions are typically vulnerable to cheating and exploitation by defectors that benefit without making costly cooperative contributions of their own. Cheating and exploitation are observed in viruses (Turner and Chao, 2003), bacteria (Rainey and Rainey, 2003), yeast (Greig and Travisano, 2004), amoebas (Buss, 1982; Dao et al., 2000; Strassman et al., 2000), fish (Poulin and Vickery, 1995), and humans (Axelrod, 1984).

How cooperation can persist in the presence of cheaters is not obvious. At first glance it often appears as though the fitness of cheaters exceeds that of cooperators. Indeed, the well-known tragedy of the commons (Hardin, 1968) shows that even when cooperation is beneficial for a group, selection acting at the individual level often eliminates cooperation altogether. This has attracted significant scientific interest throughout the last decades, with explanations proposed for the origin and maintenance of cooperation falling into three main categories. First, kin selection (Hamilton, 1963, 1964, 1972) successfully explains many forms of cooperation among genetically related individuals. Second, selection at the level of groups (Wilson, 1980; Wilson and Dugatin, 1997), through which subpopulations with non-cooperative individuals are at a reproductive disadvantage, promotes cooperation under certain conditions. Third, direct and indirect reciprocation have been shown to foster cooperation (Trivers, 1971; Axelrod and Hamilton, 1981; Axelrod, 1984). These alternative mechanisms are further discussed in Nowak (2006).

Most game-theoretical studies of cooperation fall into the third category described above and revolve around a game known as the prisoner’s dilemma (Axelrod and Hamilton, 1981). The classic variant of this game is played by two players choosing between two pure strategies, cooperation or defection, but the game can be generalized to an arbitrary number of players (Kagel and Roth, 1995; Doebeli and Hauert, 2005) and to continuous levels of cooperative contributions (Mar et al., 1994; Killinback et al., 1999). In the latter case, cooperative investments vary continuously and are represented by real numbers, denoted here by or . The payoff of an -strategist facing an -strategist is , where and areq increasing functions that quantify the benefits and costs of cooperative investments. Since cooperative investments do not directly benefit the acting individual, defection is the rational choice when the game is played only once. In many cases, however, it is more reasonable to assume that all players benefit equally from cooperative investments. For example, the digestive enzymes produced by a cell of the yeast Saccharomyces cerevisiae can be used by all nearby cells, including the producing cell itself (Gore et al., 2009). Likewise, while the stalk produced by the amoeba Dictyostelium discoideum can be exploited by cheaters, it also vitally benefits the cooperators. Further examples of processes resulting in shared benefits are cooperative hunting, vigilance behavior, group foraging, and parental care (Kun et al., 2006). Situations in which individuals directly benefit from cooperative acts that they perform can be described by the snowdrift game (Sugden, 1986), synonymously known as the hawk-dove game (Maynard Smith, 1982) or chicken game (Rapoport, 1966).

To better understand the evolution of cooperation and defection when all players are benefiting from cooperative investments, Doebeli et al. (2004) studied the snowdrift game with continuous investments. In this game, the payoff of an -strategist facing an -strategist is , where the functions and are chosen so that cooperation is more successful than defection in groups of defectors, but defection is advantageous in groups of cooperators. Consequently, cooperation in the snowdrift game always develops to some intermediate degree. However, assuming small mutations in continuous cooperative investments, Doebeli et al. (2004) showed that this gradual buildup of cooperation was sometimes followed by the emergence of cheaters with little or no cooperative investments, while the remaining cooperators became even more cooperative. Similar results have been obtained by Brännström and Dieckmann (2005) in the context of the social amoeba Dictyostelium. When starvation is imminent, one or several strains of this amoeba aggregate to form fruiting bodies that enable spore dispersal (Raper, 1984; Fortunato et al., 2003). A strain, however, may forego investing into the stalk of the fruiting body and instead take a free ride on the investments of other strains. Brännström and Dieckmann (2005) modeled this process with spores as players and with the investment of strains into the construction of stalks as strategies. In this model, the payoff of an -strategist facing an -strategist is multiplicative, , with an increasing and a decreasing function of cooperative investments. They showed that fluctuation in player numbers resulted in evolutionary branching and in the subsequent emergence and coexistence of low-investing cheaters and high-investing cooperators.

(a) | (b) |

*Dictyostelium*.

The work of Doebeli et al. (2004) and Brännström and Dieckmann (2005) shows that selection on levels of cooperative investments need not always be stabilizing. Rather, initially monomorphic populations evolving in cooperation games may experience disruptive selection, resulting in evolutionary branching and the emergence of dimorphic evolutionary outcomes in which low-investing and high-investing individuals coexist. Doebeli et al. (2004) investigated this evolutionary phenomenon, which they dubbed the tragedy of the commune, for games played in two-player groups. However, in many situations it is more natural to expect that interactions take place in groups of fluctuating size, for example, as a consequence of abstaining or of local interactions coupled with dispersal or movement. Since environmental fluctuations have been shown to promote the coexistence of competing populations (Hutchinson, 1961) and to facilitate evolutionary branching in some models, such as the site-based model studied by Geritz et al. (1998), one might expect that fluctuating group size would render the tragedy of the commune more likely. This, however, need not be the case. Figure 1 shows a multi-player extension of a game considered by Doebeli et al. (2004), in which players interact in randomly formed groups that change between each interaction. When the size of these groups changes significantly from one interaction to the next, the tragedy of the commune no longer occurs.

The aim of this paper is to explore the evolutionary consequence of fluctuating group size for cooperation games. We first define a large class of games that includes the snowdrift game considered by Doebeli et al. (2004), the Dictyostelium model conceived by Brännström and Dieckmann (2005), the prisoner’s dilemma, the stag-hunt game, and other public-good (joint-effort) games. For this class of games, we explore the evolutionary consequences of fluctuating group size for the establishment of cooperation and the tragedy of the commune.

## Ii Cooperation games with fluctuating group size

In this section we first explain why fluctuations in the size of groups of interacting players are generically expected in nature. We then introduce a class of cooperation games with continuous investments that incorporate fluctuating group size. From the demographic dynamics resulting from games in this class we determine the initial growth rate of a rare mutant strategy. This lays the foundation for our analysis of the evolutionary dynamics of cooperative investments. Finally, we describe the potential outcomes of gradual evolution in a monomorphic population with at most one interior evolutionarily singular strategy.

### ii.1 Fluctuating group size

(a) | (b) |

Figure 2 depicts two situations in which variation in group size naturally occurs. First, movement and infrequent interactions among nearby players leads to a class of games in which groups are formed through the overlap of interaction zones. Although Fig. 2a is most easily interpreted in terms of binary interactions, the interaction strengths among players in general depend on factors such as their distance. Second, Fig. 2b shows how games are formed when players are repeatedly distributed onto patches of different sizes, giving rise to distinct groups of interacting players. Significant variation in patch size leads to a wide distribution of the number of players in a group. This situation arises, for example, when spores or seeds are dispersed over an area with fragmented patches of suitable habitat and subsequently interact within each patch. Our analysis below encompasses both situations depicted in Fig. 2.

### ii.2 Payoffs in cooperation games

When a group of players has been formed, we assume that each player contributes an amount or effort towards the group’s total effort . The contribution is the strategy or trait value of a player and may optionally be constrained to an interval, e.g. . We assume that the payoff for a focal individual playing strategy in a group of players with strategies may depend on the focal player’s own contribution , on the focal player’s share of the total effort, , and on the number of individuals in the group. By choosing appropriately, we recover all traditionally studied cooperation games as special cases. For example, the two-player prisoner’s dilemma is obtained by choosing with increasing functions and , and the -player public-good (joint-effort) game by choosing with a positive factor .

We say that benefits and costs are additively or multiplicatively separable if, respectively, , or . For a fixed number of players, additively and multiplicatively separable benefits and costs give rise to equivalent evolutionary dynamics, as the multiplicatively separable payoff is additively separable on a logarithmic scale. Importantly, however, the evolutionary dynamics of these games is not equivalent in games with fluctuating player numbers, as will become clear in Sect. II.3 below. The distinction between additive and multiplicative payoff structures will play an important role in Sect. III when we examine the effect of fluctuating group size on the tragedy of the commune. When players pay a cost for making a cooperative contribution and benefit from their group’s total effort, it is natural to assume that the payoff decreases with and increases with ,

(1) |

where the subscripts in and denote the partial derivatives of with respect to its first or second argument, respectively. Since this assumption is not needed for most of the arguments below, it will be invoked only when analyzing the sign of mixed derivatives of multiplicative payoff functions.

### ii.3 Demographic dynamics in cooperation games

Based on the general specification of payoffs for players participating in cooperation games provided above, we now introduce the resultant demographic dynamics describing how player abundances change over time. For this we assume that, in successive generations, players are randomly distributed over groups of different size. The probability that an individual joins a game with participants is , where is the fraction of groups with players and is the average number of players in a group. Individuals then interact within the group and produce offspring in proportion to the payoff they received. Survival to the next generation is density-dependent, but independent of trait values. Under these assumptions, the per capita growth rate of an initially rare mutant strategy in an environment dominated by players with resident strategy is

(2) |

In adaptive dynamics theory, this quantity is known as invasion fitness (Metz et al., 1992). Equation (2), the derivation of which is provided in Appendix B, shows that for the cooperation games considered here invasion fitness is given by the excess payoff of a single mutant in groups of residents, relative to the payoff of a resident in groups of residents, averaged according to the probability that the mutant occurs in groups of size . The determination of this invasion fitness allows us to study the long-term evolution of trait values under mutation and selection (Metz et al., 1996; Dieckmann and Law, 1996; Geritz et al., 1998) and thus forms the foundation of our analysis of evolutionary dynamics in Sect. III.

Equation (2) shows that the payoffs players receive in groups of different size are averaged arithmetically. Hence, if two or more group sizes can occur, nonlinear transformations of the scale on which payoffs are measured are not possible, since they would distort the weighted average. Thus, only when the number of players is fixed, does a transformation of payoffs to a logarithmic scale enable a reduction from multiplicatively separable payoff structures to additively separable payoff structures.

### ii.4 Evolutionary dynamics in cooperation games

When mutational steps are small and rare, resident communities will successively be replaced by invading mutants with similar strategies and positive invasion fitness. Driven by directional selection, this process eventually ceases when evolution reaches either a boundary strategy at which constraints prevent further evolution, or an interior strategy at which selection pressures vanish. Strategies of the latter type are called evolutionarily singular; in their vicinity, an initially monomorphic population may experience disruptive selection and thus become dimorphic. Figure 1 illustrates how directional selection leads to a singular strategy at which selection is either disruptive (left panel) or stabilizing (right panel).

Assuming a univariate trait and at most one interior singular strategy, there are only six qualitatively different configurations of selection pressures, as shown in Figure 3. In Fig. 3a and 3c, there is no interior singular strategy and gradual adaptation leads to a monomorphic population of full defectors (tragedy of the commons) or full cooperators, respectively. In Fig. 3d, the interior singular strategy is not convergence stable (i.e., it is a repellor of monomorphic evolution). This results in evolutionary bistability, so that the evolutionary outcome depends on the initial condition (see e.g. Sumpter and Brännström, 2008). In Fig. 3b, the interior singular strategy is convergence stable (i.e., it is an attractor of monomorphic evolution), so that gradual adaptation leads to intermediate cooperative investments. In Fig. 3c and 3d, selection near the interior singular strategy can be either stabilizing or disruptive (i.e., the interior singular strategy is either an attractor or a repellor of dimorphic evolution, respectively). If selection is stabilizing, it results in a locally evolutionarily stable strategy. If selection is disruptive, the population can become dimorphic (tragedy of the commune).

## Iii Evolutionary consequences of fluctuating group size

The initial growth rate of a rare mutant player with strategy in an environment dominated by players with strategy , as given in equation (2), allows us to study the long-term consequences of small mutations and natural selection (Metz et al., 1996; Dieckmann and Law, 1996; Geritz et al., 1998). In what follows, we first study how fluctuations in group size affect the location of the singular strategy. We show that when the payoff function does not have an explicit dependence on group size, the location of the singular strategy is invariant under fluctuations in group size. For two important classes of such payoff functions, with additively or multiplicatively separable benefits and costs, we investigate how fluctuations in group size affect the potential for evolutionary branching and thus for the tragedy of the commune.

### iii.1 Consequences for cooperative investments

The selection gradient is defined as the derivative of the invasion fitness with respect to evaluated at . Accordingly, the function contains information about which nearby strategies can invade a monomorphic population of players with cooperative investment . When the selection gradient is positive (negative) more (less) cooperative strategies can invade. An invading strategy generically replaces the resident strategy, so that the population again becomes monomorphic (Geritz, 2005; Geritz et al., 2002).

From equation (2), we derive the following expression for the selection gradient,

where and , respectively, again denote the partial derivatives of with respect to its first or second argument, is the average number of players in a group, is the probability that a group with players is formed, and is the selection gradient for a fixed group size ,

More details on the derivation of the selection pressure are provided in Appendix B.

We now introduce , so that we can apply Jensen’s inequality, according to which the average of a convex function evaluated at arbitrary arguments is always larger or equal to that function evaluated at the average argument. Thus, if is convex (accelerating) as a function of , then

We can thus see that fluctuations in group size imply a greater selection gradient. This means that cooperation will be established more rapidly and reach higher levels when the group size of interacting players is variable around , than when games are played in groups with a fixed size of players. If is concave (decelerating) as a function of , the opposite is true: fluctuations in player numbers then reduce the speed of evolutionary adjustments in cooperative investments.

Since, as we have now seen, fluctuations in group size typically affect the selection pressures on cooperative investments, such fluctuations can also shift the location of interior singular strategies. If fluctuations in group size shift singular strategies below the natural limit of zero cooperative investments, they prevent the evolution of cooperation altogether. Moreover, if fluctuations in group size shift singular strategies beyond the lowest possible or highest possible cooperative investments, they can prevent evolutionary bistability (if the singular strategy is not convergence stable) or evolutionary branching (if the singular strategy is convergence stable, but not locally evolutionarily stable). However, in the following we show that when payoffs depend only on a player’s own cooperative investment and on its share of the group’s total cooperative investment, selection gradients are independent of the degree of fluctuations in group size around a given average number of players.

### iii.2 Consequences for the tragedy of the commune

We now study the effects of fluctuating group size for games in which payoffs do not explicitly depend on group size,

(3) |

For such games, the selection gradient vanishes for a singular strategy whenever

(4) |

where is the partial derivative of the payoff function with respect to its th argument, evaluated at . Since the only feature of the distribution that appears in equation (4) is the average number of players, fluctuating group size does not have any effect on the location of the singular strategies.

To understand the evolutionary dynamics of a monomorphic populations with a strategy close to the singular strategy , we need to know whether is convergence stable (directional selection drives monomorphic populations toward and whether it is locally evolutionarily stable (selection at is stabilizing). Near a singular strategy that is convergence stable but not locally evolutionarily stable, a monomorphic population experiences disruptive selection and will eventually become dimorphic through evolutionary branching. In Appendix B, we show that the singular strategy is convergence stable if

(5) |

where is the second partial derivative of the payoff function with respect to its th and th argument, evaluated at . The singular strategy is not locally evolutionarily stable if

(6) |

where

The average inverse group size serves as a measure of the strength of fluctuations in group size. It ranges from a minimum of when a group of players always has the same size , to an asymptotic maximum of as group size becomes more and more variable.

(a) | ||

(b) | ||

*Dictyostelium*model considered by Brännström and Dieckmann (2005). The benefits and costs can be separated multiplicatively and evolutionary branching does not occur without fluctuating group size. The regions and the cross (located at and ) correspond to Fig. 4 of Brännström and Dieckmann (2005). Parameters: ; ; (left to right). While the panels in (a) and (b) are produced for specific parameter values, the shown qualitative patterns apply to all games with additively and multiplicatively separable payoff functions, respectively. The parameters used in (b) to illustrate the

*Dictyostelium*model are not identical with those used by Brännström and Dieckmann (2005), but instead define a formally equivalent game.

#### Additively separable benefits and costs

We now analyze the special case in which the effects of the two arguments in the payoff function can be separated additively,

(7) |

In cooperative games, and can be interpreted as the benefit and cost of a cooperative investment, respectively. The conditions in inequalities (1) simply imply that both and are increasing functions.

The separability of arguments implies that and it follows from inequality (5) that combinations of and for which the singular strategy is convergence stable are situated below the line

which is unaffected by variation in the number of players. Likewise, it follows from inequality (6) that the combinations for which the singular strategy is not locally evolutionarily stable are situated above the line

The slope of the latter line ranges from to as fluctuations in group size increase. This is shown in the three panels of Fig. 4a.

The values and depend on the singular strategy . Evolutionarily branching, and hence the tragedy of the commune, eventually occurs when the singular strategy is convergence stable, but not locally evolutionarily stable. In Fig. 4a, this corresponds to the wedge-shaped dark green region. Since this region lies exclusively in the fourth quadrant, where and , it follows immediately that the tragedy of the commune can only occur when both the benefit and the cost are concave around the singular point. Furthermore, we see that for additively separately payoffs fluctuating group size always reduces the parameter range in which evolutionary branching – and hence the tragedy of the commune – occurs. As the fluctuations increase, any point in the plane, including the cross in Fig. 4a that corresponds to the game studied by Doebeli et al. (2004) (see also Fig. 1), eventually falls outside the region in which evolutionary branching occurs. Hence, when payoffs are additively separable, the tragedy of the commune can always be avoided through sufficient fluctuations in group size.

In summary, we have shown that in games with additively separable benefits and costs, or more generally in games with , fluctuations in group size generally reduce the scope for the tragedy of the commune to occur. Moreover, sufficiently large fluctuations in group size can always turn an evolutionary branching point into a locally evolutionarily stable strategy.

#### Multiplicatively separable benefits and costs

For games in which benefits and costs are not additively separable, the situation can be considerably different, as illustrated by the Dictyostelium model studied by Brännström and Dieckmann (2005). In the Dictyostelium model, benefits and costs are multiplicatively separable,

(8) |

Using an exponentially increasing function for the benefit and a linearly decreasing function for the cost, Brännström and Dieckmann (2005) derived an analytical condition demonstrating that evolutionary branching only occurs with fluctuating group size.

Fig. 4b shows why fluctuating group size is required for evolutionary branching. The slopes of the lines are the same as for games with additively separable payoffs, but the line with constant slope now intercepts the -axis at the point , while the remaining line intercepts at . The effect of fluctuating group size thus depends on the sign of . For multiplicatively separable payoffs, is always negative, . Without fluctuations in group size, the region in which evolutionary branching occurs lies entirely in the fourth quadrant. As fluctuations increase, the intercept decreases from to , which is below . Thus, Brännström and Dieckmann (2005) did not find evolutionary branching without fluctuations in group size, because they used a linear cost function, which implies . For sufficiently small values of , evolutionary branching cannot occur without fluctuating group size.

The case is similar to games with additively separable benefits and costs. In particular, sufficiently large fluctuations in group size can always turn an evolutionarily branching point into a locally evolutionarily stable strategy, hence avoiding the tragedy of the commune.

### iii.3 General classification of the consequences of fluctuating group size

The preceding analysis shows that the sign of has a profound impact on the evolutionary dynamics. For , fluctuations in the number of players always reduces the scope for evolutionary branching, whereas for , other outcomes are possible.

To understand the effects of fluctuating group size in general cooperation games with payoffs that are neither additively nor multiplicatively separable, we combine inequalities (5) and (6), resulting in a condition for the singular strategy to be convergence stable but not locally evolutionarily stable,

(9) |

In this double inequality, only the rightmost term changes with the variability in group size. This allows us to obtain a necessary condition for evolutionary branching when the number of players is fixed,

(10) |

Writing and for the left-hand and right-hand side, respectively, we can classify a game according to whether evolutionary branching is possible without variation in group size ( or not (. Analogously, we obtain a necessary condition for evolutionary branching when fluctuations in the number of players are maximal,

which simplifies to . Thus, we may further classify a game according to whether evolutionary branching is possible with maximal fluctuation in group size ( or not (.

Fig. 5 provides a graphical representation of these classifications. In each case, the range of values in which the middle point of inequality (9) must lie for evolutionary branching to occur is shown as a function of the variability in group size. For , and thus for games in which the benefits and costs are multiplicatively separable, the effects of fluctuating group size depend on whether inequality (10) holds. If it does not hold, fluctuating group size always reduces the scope for evolutionary branching. If it does hold, fluctuating group size may prevent or induce evolutionary branching. To distinguish between these two cases, we need to check whether

which applies if . In this case, fluctuations in group size hinder evolutionary branching. If instead , fluctuations in group size promote evolutionary branching.

## Iv Conclusions

Fluctuations are an inherent feature of social and natural systems. We have shown that fluctuating group size can have important consequences for the evolution of cooperation, by impacting both the level and the evolutionary stability of cooperative investments. The former impact can turn stable intermediate cooperative investments into a tragedy of the commons, by shifting the predicted cooperative investments below the natural lower limit of no investment. The latter impact can turn stable intermediate cooperative investments into the recently elucidated tragedy of the commune. Naturally, the consequences of fluctuating group size may also work in the opposite direction, preventing the tragedy of the commons or the tragedy of the commune.

Our results in Sect. III on how convergence stable cooperative investments are expected to shift with fluctuations in group size are based on the assumption that a player’s payoff may be affected by its own investment, the share it receives of the group’s total investment, and the group’s size. We have shown that in games with payoff functions that do not explicitly depend on group size, convergence stable cooperative investments do not shift with fluctuations in group size. For this class of games, the consequences of fluctuations in group size are thus limited to altering the evolutionary stability of cooperative investments. We have also shown that these consequences for potential evolutionary branching, and hence for the tragedy of the commune, can be understood in terms of the mixed second partial derivative of the payoff function. To obtain , the payoff function is differentiated with respect to a player’s investment and with respect to the share it receives of the group’s total investment, with the resultant derivative being evaluated at the evolutionarily singular strategy at which the directional selection pressures on cooperative investments cease. Fluctuations in group size make evolutionary branching less likely if , and we have shown that sufficient fluctuations in group size can always turn disruptive selection into stabilizing selection, hence preventing the tragedy of the commune. If , fluctuations in group size can either turn disruptive selection into stabilizing selection or vice versa, and further information is needed to determine which of these alternative consequences occurs. Most theoretical studies of cooperation are based on games with (which is guaranteed when the payoff function is additively separable; e.g., Doebeli et al., 2004; Maynard Smith, 1982) or on games with (which is guaranteed when the payoff function is multiplicatively separable; e.g., Ross-Gillespie et al., 2009; Brännström and Dieckmann, 2005; Foster, 2004; Brown and Johnstone, 2001; Brown, 1999).

The results obtained in this study demonstrate that fluctuations in group size can significantly affect cooperation evolution in real-world systems by increasing or decreasing, and by stabilizing or destabilizing, cooperative investments. Yet, to date only a handful of studies have considered the role of fluctuating group size for the evolution of cooperation (Parvinen, 2010; Hauert et al., 2008, 2006, 2002), so that many interesting questions as yet remain unexplored. Two extensions of our work here may be of particular relevance. A first promising direction is to consider the consequences of fluctuating group size on the tragedy of the commune for cooperation games with payoff functions that explicitly depend on group size, which includes situations in which a group’s total investment is not divided up equally between players. A second promising direction is to consider processes according to which players adopt new strategies that differ from the gradual-adaptation process we have analyzed here.

Studies of cooperation games with continuous strategies have recently started to add greater realism and new vistas to cooperation research. The future promise of these developments will be further enhanced by accounting for the fluctuations in group size typically arising in realistic multiplayer games.

## Acknowledgements

We thank Duncan Greig, Karl Sigmund, and David Sumpter for valuable discussions and encouragement. This research has been supported by the European Marie Curie Research Training Network FishACE (Fisheries-induced Adaptive Changes in Exploited Stocks), funded through the European Community’s Sixth Framework Programme (Contract MRTN-CT-2004-005578). U.D. gratefully acknowledges additional financial support by the Specific Targeted Research Project FinE, the European Science Foundation, the Austrian Science Fund, the Austrian Ministry of Science and Research, and the Vienna Science and Technology Fund.

## Appendix A: Demographic dynamics

We consider a population of players in which distinct cooperation strategies are represented with densities . The demographic dynamics of these densities is assumed to be given by

where , is the expected number of players with strategy emerging from a game, and the loss rate is included to allow for density regulation. We scale time such that the rate at which a player participates in a game equals 1. Assuming that a player participates in a -player game with probability and that the participants of such games are drawn at random from the population, is given by

where is the payoff function defined in Sect. 2.2. The second sum above reflects the fact that the one focal player with strategy participates in a -player game with other players that follow strategies . With denoting the frequency of strategy in the population, we obtain

(11) |

which is a generalization of the classical replicator equation (Hofbauer and Sigmund, 1998).

## Appendix B: Evolutionary dynamics

From equation (11) we deduce the initial increase in the frequency of a rare mutant strategy in an environment dominated by players with strategy . Writing , , , and when only these two cooperation strategies are present, we have

The invasion fitness of the rare mutant morph is then defined as

which gives

(12) |

We can alternatively express the further calculations in terms of the probability that a game involves players, which is related to the individual’s probability of joining a -player game by

where is the average group size, .

From the invasion fitness in equation (12), we obtain the selection gradient

where denotes the first partial derivative of with respect to its th argument. Of particular interest are the singular strategies at which directional selection ceases, .

A singular strategy is convergence stable, and nearby monomorphic populations will thus evolve toward it, if

(13) |

Here, denotes the second partial derivative of with respect to its th and th arguments, and the asterisks indicate that these derivatives are evaluated at the singular strategy, .

A singular strategy is not locally evolutionarily stable, and selection will thus be disruptive in its vicinity, if

(14) |

We can combine inequalities (13) and (14) into a single criterion for the occurrence of an evolutionary branching point,

The inequality on the left is the condition for convergence stability (implying evolutionary attraction toward , while the inequality on the right is the condition for the lack of local evolutionary stability (implying disruptive selection at .

### References

- Axelrod R (1984) The Evolution of Cooperation. Basic Books, New York, USA
- Axelrod R, Hamilton WD (1981) The evolution of cooperation. Science 211:1390–1396
- Brännström Å, Dieckmann U (2005) Evolutionary dynamics of altruism and cheating among social amoebas. Proceedings of the Royal Society London Series B 272:1609–1616
- Brown SP (1999) Cooperation and conflict in host-manipulating parasites. Proceedings of the Royal Society London Series B 266(1431):1899–1904
- Brown SP, Johnstone RA (2001) Cooperation in the dark: signalling and collective action in quorum-sensing bacteria. Proceedings of the Royal Society London Series B 268:961–965
- Buss LW (1982) Somatic cell parasitism and the evolution of somatic tissue compatibility. Proceedings of the National Academy of Sciences of the USA 79:5337–5341
- Dao DN, Kessin RH, Ennis HL (2000) Developmental cheating and the evolutionary biology of Dictyostelium and Myxococcus. Microbiology 146:1505–1512
- Dieckmann U, Law R (1996) The dynamical theory of coevolution: a derivation from stochastic ecological processes. Journal of Mathematical Biology 34:579–612
- Doebeli M, Hauert C (2005) Models of cooperation based on the Prisoner’s Dilemma and the Snowdrift game. Ecology Letters 8:748–766
- Doebeli M, Hauert C, Killingback T (2004) The evolutionary origin of cooperators and defectors. Science 306:859–863
- Fortunato A, Queller DC, Strassman JE (2003) A linear dominance hierarchy among clones in chimeras of the social amoeba Dictyostelium discoideum. Journal of Evolutionary Biology 16:438–445
- Foster KR (2004) Diminishing returns in social evolution: the not-so-tragic commons. Journal of Evolutionary Biology 17(5):1058–1072
- Geritz SAH (2005) Resident-invader dynamics and the coexistence of similar strategies. Journal of Mathematical Biology 50:67–82
- Geritz SAH, Kisdi E, Meszéna G, Metz JAJ (1998) Evolutionary singular strategies and the adaptive growth and branching of the evolutionary tree. Evolutionary Ecology 12:35–57
- Geritz SAH, Gyllenberg M, Jacobs FJA, Parvinen K (2002) Invasion dynamics and attractor inheritance. Journal of Mathematical Biology 44:548–560
- Gore J, Youk H, van Oudenaarden A (2009) Snowdrift game dynamics and facultative cheating in yeast. Nature 459:253–256
- Greig D, Travisano M (2004) The Prisoner’s Dilemma and polymorphism in yeast SUC genes. Proceedings of the Royal Society London Series B 271:S25–S26
- Hamilton WD (1963) The evolution of altruistic behavior. American Naturalist 97:354–356
- Hamilton WD (1964) The genetical theory of social behaviour I, II. Journal of Theoretical Biology 7:1–52
- Hamilton WD (1972) Altruism and related phenomena, mainly in social insects. Annual Review of Ecology and Systematics 3:193–232
- Hardin G (1968) The tragedy of the commons. Science 162:1243–1248
- Hauert C, Holmes M, Doebeli M (2002) Volunteering as Red Queen mechanism for cooperation in public goods games. Science 296:1129–1132
- Hauert C, Holmes M, Doebeli M (2006) Evolutionary games and population dynamics: maintenance of cooperation in public goods games. Proceedings of the Royal Society London Series B 273:2565–2570
- Hauert C, Wakano JY, Doebeli M (2008) Ecological public goods games: cooperation and bifurcation. Theoretical Population Biology 73:257–263
- Hofbauer J, Sigmund K (1998) Evolutionary Games and Population Dynamics. Cambridge University Press, Cambridge, UK
- Hutchinson GE (1961) The paradox of the plankton. American Naturalist 95:137–145
- Kagel JH, Roth AE (1995) The Handbook of Experimental Economics. Princeton University Press, Princeton, NJ, USA
- Killingback T, Doebeli M, Knowlton N (1999) Variable investment, the Continuous Prisoner’s Dilemma, and the origin of cooperation. Proceedings of the Royal Society London Series B 266:1723–1728
- Kun A, Boza G, Scheuring I (2006) Asynchronous snowdrift game with synergistic effect as a model of cooperation. Behavioral Ecology 17:633–641
- Mar G, Denis PS (1994) Chaos in cooperation – continuous-valued Prisoner’s Dilemmas in infinite-valued logic. International Journal of Bifurcation and Chaos 4:943–958
- Maynard Smith J (1982) Evolution and the Theory of Games. Cambridge University Press, Cambridge, UK
- Maynard Smith J, Szathmáry E (1995) The Major Transitions in Evolution. W. H. Freeman & Co., Oxford, UK
- Metz JAJ, Nisbet RM, Geritz SAH (1992) How should we define “fitness” for general ecological scenarios? Trends in Ecology and Evolution 7:198–202
- Metz JAJ, Geritz SAH, Meszéna G, Jacobs FJA, van Heerwaarden JS (1996) Adaptive dynamics: a geometrical study of the consequences of nearly faithful reproduction. In: van Strien SJ, Lunel SMV (eds) Stochastic and Spatial Structures of Dynamical Systems, North Holland, Amsterdam, The Netherlands, pp. 183–231
- Nowak MA (2006) Five rules for the evolution of cooperation. Science 314:1560–1563
- Parvinen K (2010) Adaptive dynamics of cooperation may prevent the coexistence of defectors and cooperators and even cause extinction. Proceedings of the Royal Society London Series B, in press
- Poulin R, Vickery WL (1995) Cleaning symbiosis as an evolutionary game: to cheat or not to cheat? Journal of Theoretical Biology 175:63–70
- Rainey PB, Rainey K (2003) Evolution of cooperation and conflict in experimental bacterial populations. Nature 425:72–74
- Raper KB (1984) The Dictyostelids. Princeton University Press, Princeton, NJ, USA
- Rapoport A (1966) The game of chicken. American Behavioral Scientist 10:10–28
- Ross-Gillespie A, Gardner A, Buckling A, West SA, Griffin AS (2009) Density dependence and cooperation: theory and a test with bacteria. Evolution 63:2315–2325
- Strassmann JE, Zhu Y, Queller DC (2000) Altruism and social cheating in the social amoeba Dictyostelium discoideum. Nature 408:965–967
- Sugden SR (1986) The Economics of Rights, Cooperation and Welfare. Blackwell Publishing, Oxford, UK
- Sumpter DJT, Brännström Å (2008) Synergy in social communication. In: Hughes D (ed) Social Communication, Oxford University Press, Oxford, UK, pp. 191–209
- Trivers RL (1971) The evolution of reciprocal altruism. Quarterly Review of Biology 46:35–57
- Turner PE, Chao L (2003) Escape from Prisoner’s Dilemma in RNA phage phi 6. American Naturalist 161:497–505
- Wilson DS (1980) The Natural Selection of Populations and Communities. Benjamin-Cummings, Menlo Park, CA, USA
- Wilson DS, Dugatkin LA (1997) Group selection and assortative interactions. American Naturalist 149:336–351