# Public goods games in populations with fluctuating size

## Abstract.

Many mathematical frameworks of evolutionary game dynamics assume that the total population size is constant and that selection affects only the relative frequency of strategies. Here, we consider evolutionary game dynamics in an extended Wright-Fisher process with variable population size. In such a scenario, it is possible that the entire population becomes extinct. Survival of the population may depend on which strategy prevails in the game dynamics. Studying cooperative dilemmas, it is a natural feature of such a model that cooperators enable survival, while defectors drive extinction. Although defectors are favored for any mixed population, random drift could lead to their elimination and the resulting pure-cooperator population could survive. On the other hand, if the defectors remain, then the population will quickly go extinct because the frequency of cooperators steadily declines and defectors alone cannot survive. In a mutation-selection model, we find that (i) a steady supply of cooperators can enable long-term population survival, provided selection is sufficiently strong, and (ii) selection can increase the abundance of cooperators but reduce their relative frequency. Thus, evolutionary game dynamics in populations with variable size generate a multifaceted notion of what constitutes a trait’s long-term success.

## 1. Introduction

The emergence of cooperation is a prominent research topic in evolutionary theory. The problem is usually formulated in such a way that it pays to exploit cooperators, yet the payoff to one cooperator against another is greater than the payoff to one defector against another [1]. In spite of this conflict of interest, cooperation is broadly observed in nature, and various mechanisms have been put forth to explain its evolution [2]. In fact, the question of how cooperators may proliferate in social situations is one of the main concerns of evolutionary game theory, a framework that models cooperation and defection as strategies of a game.

Evolutionary game-theoretic models typically involve a number of assumptions. In this study, we are concerned with two potentially restrictive ones: (i) the population size is infinite or (ii) the population size is finite but fixed and unaffected by evolution. While the classical replicator equation [3, 4, 5] can be used to model large populations that fluctuate in size [6], replicator dynamics capture only the relative frequencies of the strategies. Even stochastic models that account for populations of any finite size, such as the Moran model or the Wright-Fisher model and their game-theoretic extensions, usually require the number of players to remain fixed over time [7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17]. Here, we explore the evolutionary dynamics of cooperation in social dilemmas when the population can fluctuate in size and even go extinct.

Branching processes have a rich history in theoretical biology [see 18] and are a natural way to model populations that vary in size. A number of recent works have considered non-constant population size within evolutionary game theory. Hauert et al. [6] treat ecological dynamics in evolutionary games by modifying the replicator equation to account for population density and show that fluctuating density can lead to coexistence between cooperators and defectors. Melbinger et al. [19] illustrate how the decoupling of stochastic birth and death events can lead to transient increases in cooperation. By allowing a game to influence carrying capacities, Novak et al. [20] demonstrate that variable density regulations can change the stability of equilibria relative to the replicator equation. Furthermore, demographic fluctuations can act as a mechanism to promote cooperation in public goods games [21] and indefinite coexistence (as opposed to fixation) in coexistence games [22]. Fluctuating size in a Lotka-Volterra model also leads to different growth rates for isolated populations of cooperators and defectors [23], and even when the two competing types are neutral at the equilibrium size, fluctuations can still give one type a selective advantage over the other [24]. When traits have the same growth rate, these fluctuations also affect a mutant’s fixation probability [25].

Here, we develop a branching-process model based on the Wright-Fisher model [26, 27] for a population with non-overlapping generations in which trait values of offspring are sampled from the previous generation depending on the success of individuals (parents) in a sequence of interactions [28, 29]. Success is quantified in terms of payoffs, which come from a game and represent competition between the different types, or strategies. Usually, the Wright-Fisher process is defined such that every subsequent generation has exactly the same size as the first generation. We consider a variant of this model for populations that fluctuate in size, in which each individual has a Poisson-distributed number of surviving offspring, with an expected value determined by payoffs from interactions in a game.

Recently, Houchmandzadeh [30] considered a model similar to the one we study here, but under the assumption that the population size in the next generation, , is a deterministic function of the fraction of cooperators in the present generation, . The update rule then has essentially two stages: (i) determine the population size of the next generation, , and (ii) sample offspring from the previous generation using the standard Wright-Fisher rule [30]. In contrast, the model we treat has a stochastic population size that does not need to be prespecified. Moreover, it depends on the numbers of both cooperators and defectors in the current generation, not just on the fraction of cooperators. As mentioned above, we also allow for the possibility that the entire population goes extinct.

We use the public goods game to study the evolution of cooperation in an unstructured population. Cooperators maintain a shared resource or public good, with a cost, , to their fecundity. Defectors neither help maintain the public good nor incur a cost. The resource is distributed evenly among all individuals in the population, but its per-capita effect on fecundity can be greater than the per-capita cost of its production [31]. A multiplication factor, , quantifies this return on the investment made by cooperators toward production of the good. In this model, everyone is better off when the whole population consists of cooperators, but defectors can benefit from cooperation without paying the cost.

We show that when the population size can fluctuate, selection can be essential for the survival of the population as a whole. In our model, population growth and decline are influenced by the public goods game but also by a baseline reproductive capacity, , which is the same for all individuals and which primarily acts to constrain runaway growth. Even when cooperators are less frequent than defectors in the mutation-selection equilibrium, there can be an optimal cost of cooperation, , depending on , at which (i) the population does not immediately go extinct, with the numbers of cooperators and defectors each fluctuating around equilibrium values, and (ii) the frequency of cooperators is maximized subject to (i). In other words, cooperation can be favored by selection at a positive cost of cooperation when there is demographic stochasticity, which marks a departure from the behavior of models with fixed size.

Furthermore, even when the population would survive due to the baseline reproductive capacity alone, selection can increase the number of cooperators while at the same time decreasing their frequency. In models where the population size is assumed to be fixed, cooperators are less frequent than defectors if and only if cooperators are less abundant than defectors. However, this equivalence breaks down when the population size can fluctuate because the frequency of a strategy is determined by both its abundance and the population size. Thus, the evolutionary success of a strategic type depends on more than just the strategy.

## 2. Description of the model

We use the term “reproductive capacity” rather than “fitness” [see 32] to refer to the expected number of offspring of an individual. In a growing population, the average reproductive capacity is greater than one. In a shrinking population, it is less than one. In a population of fixed size or a population at its carrying capacity, the average reproductive capacity is equal to one. If different individuals in the same population have different reproductive capacities, some individuals have a selective advantage over others.

### 2.1. Update rule

We assume that individuals reproduce asexually, so our model corresponds to a model of haploid genetic transmission. In the standard Wright-Fisher process, the population has fixed size, . Thus, in a game with two strategies, (“cooperate”) and (“defect”), the state of the population is determined by number of cooperators, , or by their relative frequency, . If and give the reproductive capacities of cooperators and defectors, respectively, in the state with cooperators, then the probability of transitioning to the state with cooperators (provided ) is {linenomath}

(1) |

In other words, the cooperators in one generation are sampled from the previous generation according to a binomial distribution with mean . One biological interpretation for this transition rule is the following: Each player in one generation produces a large number of gametes from which the surviving offspring in the next generation are selected. These offspring are sampled at random, weighted by the success of the parents in competitive interactions, subject to a constant population size.

In treating populations that fluctuate in size, we drop the assumed dependence that which is implied above, but continue to hold that generations are non-overlapping. Let and give the reproductive capacities of cooperators and defectors, respectively, when the current generation is in state . We assume that the number of offspring per individual follows a Poisson distribution, with parameter for cooperators and parameter for defectors. Then the probability of transitioning from state to state in one generation is {linenomath}

(2) |

Eq. 2 reduces to Eq. 1 when the population size is fixed and equal to (see [33] and also Appendix A).

The transition probabilities of Eqs. 1–2 do not take into account errors in strategy transmission, i.e. mutations. In what follows, we assume that when an individual reproduces, the offspring acquires a random strategy with probability . Thus, with probability , the offspring acquires the strategy of the parent and with probability , becomes either a cooperator or a defector (uniformly at random). For simplicity (and by convention [e.g. see 34]), we assume symmetric mutation, with as likely as .

Using the binomial distribution with parameter and density function , the mutation rate, , is incorporated into the transition rule defined by Eq. 2 as follows: {linenomath}

(3) |

In words, we sum over all transitions defined by Eq. 2 such that, after mutations are accounted for, there are cooperators and defectors. Note that mutations themselves do not affect the population size.

We refer to the process with transitions governed by Eqs. 2–2.1 as a “Wright-Fisher branching process.” Branching processes of this sort have been treated elsewhere [see 33], notably with Poisson-distributed offspring counts but frequency-independent reproductive capacities [35]. Branching processes have also been considered in models with both density-dependent [36] and frequency-dependent [37, 38] reproductive capacities. We consider a Wright-Fisher branching process in which the reproductive capacities of cooperators and defectors in Eqs. 2–2.1 are equal to a baseline reproductive capacity times a factor which depends on the outcome of a public goods game.

### 2.2. Baseline reproductive capacity

The standard Wright-Fisher model, and variants like that in [30], assume that population regulation is very strong or deterministic. This may be a good approximation for large populations and those close to carrying capacity [but see 24]. However, fully stochastic treatments are warranted for populations that fluctuate in size and may go extinct. In the Wright-Fisher branching process we consider, population regulation is achieved through a balance of players’ payoffs in an evolutionary game and a baseline reproductive capacity which represents all other ecological factors. The dynamics depend on the numbers of cooperators and defectors, not just on their relative frequencies. The baseline reproductive capacity is a function of the total population size and is the same for every individual. It captures the ecological constraints which keep populations from growing without limit.

Let be the per-capita reproductive capacity (again, the expected number of offspring) applicable to all individuals when the population size is . We assume that is a non-increasing function of the population size, , so that larger populations lead (in general) to lower per-capita reproductive capacities due to ecological constraints. This baseline reproductive capacity is the fluctuating-size analogue of the “background fitness” that is typically used in models with fixed population size [15].

While the baseline reproductive capacity does not vary from player to player, it can depend on the number of players in the population, . If there are limited resources and reproduction slows as the population grows, then is a decreasing function of . An example we consider is for some , , and . In this case, reflects the growth rate when the population is small, and gives the reproductive capacity when the population has size . To ensure is non-negative, we set whenever . A second example we consider is one in which the baseline reproductive capacity is constant up to a threshold value of and decreasing thereafter, specifically with if and if for some and . For both of these functions, decays to as (see Fig. 1).

### 2.3. Public goods game

Consider a game with two strategies, (“cooperate”) and (“defect”), and suppose that a defector does nothing and a cooperator incurs a cost, , representing a fraction of his or her baseline reproductive capacity, , in order to contribute to the provision of a public good. The public good is distributed evenly among all players in the population [31]. Finally, a multiplication factor, , quantifies the return on investment in this shared resource [39].

The reproductive capacities of cooperators and defectors in state are given by {linenomath}

(4a) | ||||

(4b) |

When , the contribution of this game to reproductive capacity is small. On the other hand, when , cooperators expend their entire baseline reproductive capacity contributing to the public good. Unlike in many evolutionary games in populations of fixed size, where represents selection strength and quantifies relative differences between traits, here the cost of cooperation admits an intuitive biological interpretation: cooperators sacrifice a fraction of their expected number of offspring in hopes of seeing a return.

For a neutral population whose dynamics are governed only by the non-increasing baseline reproductive capacity, , if then the time to extinction will be relatively short. In contrast, if the population will have a positive growth rate until becomes large enough that . Then the population will grow to a stochastic carrying capacity and fluctuate around this size, possibly for considerable time. (For the two classes of baseline reproductive capacities we consider here, this carrying capacity need not be exactly ; see Appendix A). We will refer to situations of this sort as “metastable” because all the populations we consider would eventually go extinct. According to Eq. 4, payoffs from the public goods game can increase reproductive capacities, with the possibility of positive population growth rates even if . Due to our choice of non-increasing functions for (that decay to as grows), this will lead to metastable states but never to unbounded growth of the population.

## 3. Evolutionary dynamics of the Wright-Fisher branching process

Let and denote the expected numbers of cooperator and defectors in the next generation given cooperators and defectors in the current generation. In this section, we are mainly interested in the existence of metastable equilibria, which are defined as states, , such that {linenomath}

(5a) | ||||

(5b) |

Populations will fluctuate around these states for some time, although extinction is inevitable. The time to extinction depends on the population’s carrying capacity (see below and Appendix C). Even when the population eventually goes extinct with probability , it can take extremely long to do so.

We are particularly interested in the case when a population of defectors cannot survive for long on their own but a population of cooperators can. While a population of defectors evolves according to , a population of cooperators evolves according to the reproductive capacity , which can be greater than even when . In polymorphic populations, cooperators and defectors have reproductive capacities given by and in Eq. 4, which are functions of and , but also depend on the baseline reproductive capacity, , the cost of cooperation, , and the multiplication factor for the public good, . We also consider situations in which a population of defectors can reach a metastable carrying capacity, i.e. when . In this case, we are interested in the effects that and can have on the numbers of cooperators and defectors in polymorphic metastable states.

### 3.1. Selection dynamics (without mutation)

When the initial numbers of cooperators and defectors are small, stochastic effects have a profound influence over the long-run composition of the population. We show in Appendix B that any non-zero metastable equilibrium must be monomorphic (all-cooperator or all-defector) for the update rule defined by Eq. 2. Although defectors generally have larger growth rates than cooperators in mixed populations, they can go extinct quickly in small populations, which, in turn, can permit cooperators to prosper. For example, suppose that defectors cannot survive on their own (), which means that any population of defectors shrinks, on average, from one generation to the next. If any population of cooperators grows, due to the multiplication factor , then the only populations that persist beyond a short time horizon are those composed entirely of cooperators. Therefore, cooperators have a type of survivorship bias. Fig. 2 illustrates this phenomenon, showing that defectors often outcompete cooperators (approximately 85% of the time) when both are in the population (A), but once one type goes extinct, the population must consist of just cooperators in order to survive for any considerable length of time (B). These simulations are done with the baseline capacity , where , , and ; cost of cooperation ; and multiplication factor .

### 3.2. Mutation-selection dynamics

A common way to quantify the evolutionary success of cooperators is to introduce strategy mutations and study the frequency of cooperators in the mutation-selection equilibrium [34, 40, 41]. Mutations indicate errors in the transmission (either cultural or genetic) of the two strategies (cooperation and defection) and can be small [42, 43] or large [44] depending on their interpretation. The success of cooperation is quantified by its average frequency in the population over many generations. In a population of cooperators and defectors under neutral drift (i.e. without selective differences between the two types), cooperators are indistinguishable from defectors and are equally frequent in the mutation-selection equilibrium. If selection brings the cooperator frequency above , then selection is said to favor cooperation. By this metric, selection typically disfavors cooperation in unstructured populations [34].

If the population size is static and the update rule is that of the Wright-Fisher process, Eq. 1, then the baseline reproductive capacity appearing in Eq. 4, , cancels out. Only the relative fitnesses of cooperators and defectors matters. The dynamics are then captured in the relative frequencies of cooperators and defectors. Since cooperators are always less frequent than defectors when the intensity of selection, , is positive, selection unambiguously disfavors cooperators relative to defectors. This result can be seen in Fig. 3(D)–(F), in which results are shown for three different values of . These simulations were generated using a multiplication factor of and a mutation rate of . That selection favors defectors is a standard property of many social dilemmas in unstructured populations; additional mechanisms (such as spatial structure) must typically be present in order for cooperators to outperform defectors [15].

When the population size can fluctuate and is the probability that a mutation occurs, then the dynamics are governed by Eq. 2.1. Here, it is still the case that selection decreases the frequency of cooperators relative to defectors. On the other hand, the population can quickly go extinct if selection is not sufficiently strong, which we illustrate in Fig. 3(A)–(C) with , , and with and . Thus, cooperation can be favored in such situations because it protects against extinction.

One key difference from models with fixed population size is that, in a branching process, the population either grows unboundedly or eventually goes extinct [45, 46]. That is, if the population remains bounded in size, then the only true stationary state is extinction. Despite this behavior capturing the long-run dynamics of the process, there can also exist metastable equilibria in which the process persists prior to population extinction. We show in Appendix C that the persistence time in our model grows exponentially in [see also 47, 48, 49], meaning that if is the expected number of generations prior to extinction after starting in the quasi-stationary distribution, then there exists (independent of ) for which .

More informally, if is a metastable equilibrium and denotes standard deviation, then {linenomath}

(6a) | ||||

(6b) |

Therefore, the fluctuations around a metastable equilibrium constitute only small fractions of and when and are sufficiently large (see Appendix B for further details). Since and grow with , and since the fluctuations in and are on the order of and , respectively, the expected amount of time until deviations from the mean destroy the population, i.e. deviates so far as to hit , grows rapidly in . Fig. 4 shows the quasi-stationary distributions of and that result from fluctuations around metastable equilibria, such as those shown in Fig. 3(C).

The dynamics of this public goods game result from the balance among three factors: mutation, selection, and population survival. Although long-term population survival can be achieved by increasing the cost of cooperation, , it can also be destroyed by decreasing the mutation rate, . In Appendix B, we show that for any and any non-zero mutation rate and cost of cooperation, there exists a critical multiplication factor, , such that the population is supported at a metastable equilibrium consisting of at least players whenever . In general, the harmful effect (population extinction) of either low costs of cooperation or low mutation rates can be mitigated by increasing the return on investment in the public good.

Selection can also increase cooperator abundance while decreasing their relative frequency (Fig. 5). This phenomenon is a consequence of the fact that the presence of cooperators can change the carrying capacity of population. That abundance and frequency can move in opposite directions is unique to models with variable population size and presents an interesting question about the definition of cooperator success. We show in Appendix B that the fraction of cooperators present in a metastable equilibrium is independent of and depends on just , , and . Thus, when , , and are fixed, defectors claim a fixed fraction (at least ) of the total population, which means that cooperators are disfavored relative to defectors. However, based on population growth alone, cooperators could be considered to be favored by selection in an absolute sense because their abundance is an increasing function of the cost of cooperation, .

## 4. Discussion

Public goods games have been used to model conflicts of interest ranging from cooperation in microorganisms [50, 51, 52, 53, 54], to alarm calls in monkeys [55, 56], sentinel behavior in meerkats [57], and large-scale human efforts aimed at combating climate change [58, 59] and pollution [60]. Due to its linearity and close relationship to the prisoner’s dilemma, the public goods game we consider is sometimes called the “-person prisoner’s dilemma” [61]. Provided , this game presents a conflict of interest between the group and the individual that can be reduced to a sequence of prisoner’s dilemma interactions [62]. However, the analysis and interpretation of a single public goods game is somewhat more straightforward than that of a series of prisoner’s dilemma interactions when the population size fluctuates over time.

In populations of fixed size, extinction is impossible and defectors can survive without the support of cooperators. This point marks perhaps the most prominent feature of classical models in evolutionary game theory that breaks down when the population size can fluctuate over time. When populations vary in size and defectors cannot sustain themselves on their own, cooperators must be present and selection must be sufficiently strong in order to maintain the existence of the population (Fig. 3). Furthermore, when the population size is assumed to be fixed, selection decreases the frequency of cooperators if and only if it decreases the number, or abundance, of cooperators. In fluctuating populations, selection can decrease the frequency of cooperators while increasing their abundance (Fig. 5).

We have referred to as the “cost of cooperation” because of its interpretation as the expected fraction of offspring that must be sacrificed in order to cooperate. However, we note that because this fraction of an individual’s baseline reproductive capacity is shared across the population, larger also means a greater effect of cooperation (similar to the return, ). Further, if then the population is identical to a population of defectors. In standard Wright-Fisher processes, is irrelevant and it is common to rewrite the terms in Eq. 4 as and , and to refer to the “strength” [40, 63, 64] or “intensity” of selection [8, 65, 66]. When the population size is held constant in our model, corresponds exactly to this well-known notion of selection intensity.

That the frequencies of cooperators relative to defectors in the metastable equilibria of Figs. 3(C), 3(F), and 5(C) are all the same is not a coincidence. The fraction of cooperators, , present at a metastable equilibrium is independent of the baseline reproductive capacity, , and depends on only the mutation rate, ; the cost of cooperation, ; and the multiplication factor for the public goods game, . The population size at a metastable equilibrium, however, does depend on . In Appendix B, we give an explicit formula for and a condition for the existence of a non-zero metastable equilibrium in terms of , , , and .

In the absence of mutation, either cooperators or defectors must be extinct in any metastable equilibrium. Although defectors outperform cooperators in a mixed population, a population of cooperators reaches a higher carrying capacity and persists at this size for a longer time than does a population of defectors. Small populations of cooperators have a distinct advantage over their all-defector counterparts due to larger growth rates. In particular, quick extinction is less likely for all-cooperator populations than it is for all-defector populations, reflecting observations of Huang et al. [23] and Waite et al. [67] for related models.

Unlike in the models of Houchmandzadeh and Vallade [68] and Houchmandzadeh [30], the population size in our model is not a deterministic function of the fraction of cooperators. Rather, it is a random quantity derived from the collective offspring pool of the parental generation. The population size at time depends on both the number of cooperators and the number of defectors at time .

A framework more similar to ours is that of Behar et al. [69], which uses stochastic differential equations to model the numbers of producers and non-producers of a common resource. Both numbers increase when small but eventually non-producers drive the population to extinction. Analogous to the possible role of mutations described here, Behar et al. [69] allow migration to reseed populations with producers. A metastable equilibrium may then occur in the total population even as each local population experiences boom and bust cycles. Our focus here has been on treating baseline reproductive dynamics as an exogenous feature and understanding on how these may be perturbed by a game to allow a variety of different carrying capacities to emerge depending on the parameters of the model.

Since population size can fluctuate in our model, one could also allow the multiplication factor of the public good, represented here by , to change with . If this multiplication factor gets weaker as grows, then one observes dynamics similar to those here even if is independent of . Viewing as a function of presents an alternative way to model populations that cannot have unbounded growth due to environmental constraints. Another extension of our model could involve asymmetric mutation rates with, for example, mutations more likely than . Although the importance of asymmetric mutation in population models is well-established [70, 71, 72], we do not expect this would cause any qualitative changes in the results reported here unless the asymmetry was very extreme.

Incorporating dynamic population size into classical evolutionary models complicates the analysis of their dynamics. Notably, how one measures the evolutionary success of cooperators is not as unambiguous here as it is in models with fixed population size. We have shown that selection can favor cooperator abundance despite disfavoring cooperator frequency, and that even though cooperators are exploited by defectors, they can be crucial to the survival of the population as a whole.

## Appendix A. Wright-Fisher branching process

### a.1. Update rule

Suppose that the population is unstructured but allowed to vary in size. For simplicity, assume that we are dealing with a symmetric game with two strategies, (“cooperate”) and (“defect”). A state of the population is then uniquely defined by a pair, , where and are the number of players using and , respectively. The population size is , which can vary over time.

Suppose that the reproductive capacities of cooperators and defectors in state are given by functions and , respectively. That is, the reproductive capacities are frequency-dependent and determined by the number of each type of player in the population. We define reproductive capacity as the expected number of surviving offspring of an individual over its lifetime. Ours is therefore an “absolute” interpretation of reproductive capacity. We assume a reproductive mechanism in which the number of offspring is Poisson-distributed with mean equal to the parent’s reproductive capacity. Therefore, the probability of transitioning from to over a single generation in this “Wright-Fisher branching process” is {linenomath}

(7) |

As in the standard Wright-Fisher process, we assume that generations are non-overlapping.

###### Remark 1.

### a.2. Reproductive capacities and selection

In the absence of selection, each player in a population of size has a reproductive capacity determined by a baseline reproductive capacity, . We consider the following two functional forms for , examples of which are depicted in Fig. 1 in the main text.

#### Rectified linear

One natural way to model reproductive capacity is as a linear function of the population size, . In this case, we can write for some parameters , , and . We refer to as a “rectified” linear reproductive capacity since it piecewise-linear with the constraint for every . Note that when . Therefore, may be interpreted as the (neutral) carrying capacity of the population since when each individual is replaced by one offspring on average. Note that itself is not necessarily the neutral carrying capacity for this form of .

#### Threshold-constant

If the reproductive capacity is constant, then every player expects to produce offspring that survive into the next generation, where . We assume that this growth is eventually bounded by environmental constraints, so we set for some . We refer to as a “threshold-constant” reproductive capacity since it is constant up to a threshold () and then decreasing to beyond . When , there is no solution to since for each . When , we have , so is the neutral carrying capacity of the population.

#### Selection

In a game with strategies and , let and be the total payoffs to and , respectively, when there are cooperators and defectors. If the population size is fixed, then a payoff of is typically converted to a fitness of by defining , where is the “selection strength” [see 40, 73]. This perturbation approach has even been extended to asymmetric games played between different populations [74]. While our setup is somewhat different, we maintain this convention of using payoffs from a game to perturb reproductive capacities. In particular, if is a parameter representing the intensity of selection, then the reproductive capacities of cooperators and defectors are given by {linenomath}

(9a) | ||||

(9b) |

respectively. In other words, the baseline reproductive capacity, , is perturbed by the game according to the strength of selection, . In order to maintain non-negative reproductive capacities, must be sufficiently small. In the next section, we consider a public goods game in which has a clear biological interpretation.

## Appendix B. Dynamics of the public goods game

In the public goods game, a cooperator sacrifices a fraction, , of his or her baseline reproductive capacity in order to contribute to a public good. This contribution is enhanced by a factor of and then distributed evenly among all of the players in the population. In terms of the payoff function in Eq. 9a, we have and as well as Eq. 4 in the main text.

### b.1. Metastable equilibria

Consider a population evolving according to the update rule of Eq. 2.1. As noted in the main text, such a branching process either grows without bound or eventually goes extinct. Even when the population has an extinction probability of , there can be so-called “metastable” states (or “equilibria”) around which the population fluctuates for many generations. While a quasi-stationary distribution for the process describes the distribution of strategy abundances prior to extinction, a metastable equilibrium describes the mean(s) around which these strategy counts fluctuate. We are interested in when these metastable equilibria exist and how they are influenced by the parameters of the model.

Let (resp. ) be the expected abundance of cooperators (resp. defectors) in the next generation given cooperators and defectors in the current generation. Formally, a metastable equilibrium for this process is a state at which and . That is, cooperator and defector abundances each remain unchanged (on average) at a metastable equilibrium. We use the term “metastable” because the population fluctuates around this state but eventually goes extinct. We discuss extinction time in Appendix C. First, we derive the metastable equilibria for public goods games.

#### Derivation of metastable equilibria

Let be the strategy-mutation rate. With probability , an offspring acquires the strategy of the parent. With probability , the offspring takes on one of and uniformly at random. In state , the expected number of cooperators in the next generation is {linenomath}

(10) |

Similarly, the expected number of defectors in the next generation is . Therefore, the system of equations we need to solve in order to find a metastable equilibrium is {linenomath}

(11a) | ||||

(11b) |

In other words, it must be true that {linenomath}

(12a) | ||||

(12b) |

These equations are trivially satisfied when (population extinction). There can also be solutions to Eq. 12 with or ; we give a condition for the existence of non-zero solutions below.

###### Remark 2.

If , then Eq. 12 reduces to the system {linenomath}

(13a) | ||||

(13b) |

If and satisfy this system and , then {linenomath}

(14) |

Therefore, either and or and . However, for the public goods game, it is always the case that when , so it must be true that and . Thus, if and , then any solution satisfies or . In other words, in the absence of mutation, selection forces the extinction of at least one strategy.

###### Lemma 1.

###### Proof.

If is a solution to Eq. 12 with and , then, with , {linenomath}

(15) |

Since and , Eq. 15 is equivalent to {linenomath}

(16) |

Since is (at most) quadratic, , and , we see that if , then there is a unique solution to Eq. 15 that falls within , and, furthermore, this solution is in . Explicitly, {linenomath}

(17) |

if and , and if or , which completes the proof. ∎

From the proof of Lemma 1, we see that if , then either (i) and every is a solution to Eq. 15 or (ii) and the only solutions to Eq. 15 are and . For any , the fraction of cooperators in a non-zero metastable equilibrium is independent of the baseline reproductive capacity, . However, the existence of a metastable equilibrium and the size of the population at such an equilibrium both depend on the baseline reproductive capacity. Suppose that and satisfy Eq. 12, where is a fraction of cooperators that satisfies Eq. 15. From Eq. 12, {linenomath}