Selection Problems in the Presence of Implicit Bias

Selection Problems in the Presence of Implicit Bias

Jon Kleinberg
Cornell University
   Manish Raghavan
Cornell University

Over the past two decades, the notion of implicit bias has come to serve as an important component in our understanding of discrimination in activities such as hiring, promotion, and school admissions. Research on implicit bias posits that when people evaluate others – for example, in a hiring context – their unconscious biases about membership in particular groups can have an effect on their decision-making, even when they have no deliberate intention to discriminate against members of these groups. A growing body of experimental work has pointed to the effect that implicit bias can have in producing adverse outcomes.

Here we propose a theoretical model for studying the effects of implicit bias on selection decisions, and a way of analyzing possible procedural remedies for implicit bias within this model. A canonical situation represented by our model is a hiring setting: a recruiting committee is trying to choose a set of finalists to interview among the applicants for a job, evaluating these applicants based on their future potential, but their estimates of potential are skewed by implicit bias against members of one group. In this model, we show that measures such as the Rooney Rule, a requirement that at least one of the finalists be chosen from the affected group, can not only improve the representation of this affected group, but also lead to higher payoffs in absolute terms for the organization performing the recruiting. However, identifying the conditions under which such measures can lead to improved payoffs involves subtle trade-offs between the extent of the bias and the underlying distribution of applicant characteristics, leading to novel theoretical questions about order statistics in the presence of probabilistic side information.

1 Introduction

Over the past two decades, the notion of implicit bias [13] has come to provide on important perspective on the nature of discrimination. Research on implicit bias argues that unconscious attitudes toward members of different demographic groups — for example, defined by gender, race, ethnicity, national origin, sexual orientation, and other characteristics — can have a non-trivial impact on the way in which we evaluate members of these groups; and this in turn may affect outcomes in employment [2, 3, 21], education [22], law [14, 15], medicine [12], and other societal institutions.

In the context of a process like hiring, implicit bias thus shifts the question of bias and discrimination to be not just about identifying bad actors who are intentionally discriminating, but also about the tendency of all of us to reach discriminatory conclusions based on the unconscious application of stereotypes. An understanding of these issues also helps inform the design of interventions to mitigate implicit bias — when essentially all of us have a latent tendency toward low-level discrimination, a set of broader practices may be needed to guide the process toward the desired outcome.

A basic mechanism: The Rooney Rule.

One of the most basic and widely adopted mechanisms in practice for addressing implicit bias in hiring and selection is the Rooney Rule [7], which, roughly speaking, requires that in recruiting for a job opening, one of the candidates interviewed must come from an underrepresented group. The Rooney Rule is named for a protocol adopted by the National Football League (NFL) in 2002 in response to widespread concern over the low representation of African-Americans in head coaching positions; it required that when a team is searching for a new head coach, at least one minority candidate must be interviewed for the position. Subsequently the Rooney Rule has become a guideline adopted in many areas of business [5]; for example, in 2015 then-President Obama exhorted leading tech firms to use the Rooney Rule for hiring executives, and in recent years companies including Amazon, Facebook, Microsoft, and Pinterest have adopted a version of the Rooney Rule requiring that at least one candidate interviewed must be a woman or a member of an underrepresented minority group [18]. In 2017, a much-awaited set of recommendations made by Eric Holder and colleagues to address workplace bias at Uber advocated for the use of the Rooney Rule as one of its key points [8, 19].

The Rooney Rule is the subject of ongoing debate, and one crucial aspect of this debate is the following tension. On one side is the argument that implicit (or explicit) bias is preventing deserving candidates from underrepresented groups from being fairly considered, and the Rooney Rule is providing a force that counter-balances and partially offsets the consequences of this underlying bias. On the other side is the concern that if a job search process produces a short-list of top candidates all from the majority group, it may be because these are genuinely the strongest candidates despite the underlying bias — particularly if there is a shortage of available candidates from other groups. In this case, wholesale use of the Rooney Rule may lead firms to consider weaker candidates from underrepresented groups, which works against the elimination of unconscious stereotypes. Of course, there are other reasons to seek diversity in recruiting that may involve broader considerations or longer time horizons than just the specific applicants being evaluated; but even these lines of argument generally incorporate the more local question of the effect on the set of applicants.

Given the widespread consideration of the Rooney Rule from both legal and empirical perspectives [7], it is striking that prior work has not attempted to formalize the inherently mathematical question that forms a crucial ingredient in these debates: given some estimates of the extent of bias and the prevalence of available minority candidates, does the expected quality of the candidates being interviewed by a hiring committee go up or down when the Rooney Rule is implemented? When the bias is large and there are many minority candidates, it is quite possible that a hiring committee’s bias has caused it to choose a weaker candidate over a stronger minority one, and the Rooney Rule may be strengthening the pool of interviewees by reversing this decision and swapping the stronger minority candidate in. But when the bias is small or there are few minority candidates, the Rule might be reversing a decision that in fact chose the stronger applicant.

In this paper, we propose a formalization of this family of questions, via a simplified model of selection with implicit bias, and we give a tight analysis of the consequences of using the Rooney Rule in this setting. In particular, when selecting for a fixed number of slots, we identify a sharp threshold on the effectiveness of the Rooney Rule in our model that depends on three parameters: not just the extent of bias and the the prevalence of available minority candidates, but a third quantity as well — essentially, a parameter governing the distribution of candidates’ expected future job performance. We emphasize that our model is deliberately stylized, to abstract the trade-offs as cleanly as possible. Moreover, in interpreting these results, we emphasize a point noted above, that there are other reasons to consider using the Rooney Rule beyond the issues that motivate this particular formulation; but an understanding of the trade-offs in our model seems informative in any broader debate about such hiring and selection measures.

We now describe the basic ingredients of our model, followed by a summary of the main results.

1.1 A Model of Selection with Implicit Bias

Our model is based on the following scenario. Suppose that a hiring committee is trying to fill an open job position, and it would like to choose the best candidates as finalists to interview from among a large set of applicants. We will think of as a small constant, and indeed most of the subtlety of the question already arises for the case , when just two finalists must be selected.

-candidates and -candidates.

The set of all applicants is partitioned into two groups and , where we think of as the majority group, and as a minority group within the domain that may be subject to bias. For some positive real number and a natural number , there are applicants from group and applicants from group . If a candidate belongs to , we will refer to them as an -candidate, and if belongs to , we will refer to them as a -candidate. (The reader is welcome, for example, to think of the setting of academic hiring, with as candidates from a group that is underrepresented in the field, but the formulation is general.)

Each candidate has a (hidden) numerical value that we call their potential, representing their future performance over the course of their career. For example, in faculty hiring, we might think of the potential of each applicant in terms of some numerical proxy like their future lifetime citation count (with the caveat that any numerical measure will of course be an imperfect representation). Or in hiring executives, the potential of each applicant could be some measure of the revenue they will bring to the firm.

We assume that there is a common distribution that these numerical potentials come from: each potential is an independent draw from . (Thus, the applicants can have widely differing values for their numerical potentials; they just arise as draws from a common distribution.) For notational purposes, when is an -candidate, we write their potential as , and when is a -candidate, we write their potential as . We note an important modeling decision in this formulation: we are assuming that all and all values come from this same distribution . While it is also of interest to consider the case in which the numerical potentials of the two groups and are drawn from different group-specific distributions, we focus on the case of identical distributions for two reasons. First, there are many settings where differences between the underlying distributions for different groups appear to be small compared to the bias-related effects we are seeking to measure; and second, in any formal analysis of bias between groups, the setting in which the groups begin with identical distributions is arguably the first fundamental special case that needs to be understood.

In the domains that we are considering — hiring executives, faculty members, athletes, performers — there is a natural functional form for the distribution of potentials, and this is the family of power laws (also known as Pareto distributions), with and support for a fixed . Extensive empirical work has argued that the distribution of individual output in a wide range of creative professions can be approximated by power law distributions with small positive values of [6]. For example, the distribution of lifetime citation counts is well-approximated by a power law, as are the lifetime downloads, views, or sales by performers, authors, and other artists. In the last part of the paper, we also consider the case in which the potentials are drawn from a distribution with bounded support, but for most of the paper we will focus on power laws.

Selection with Bias.

Given the set of applicants, the hiring committee would like to choose finalists to interview. The utility achieved by the committee is the sum of the potentials of the finalists it chooses; the committee’s goal is to maximize its utility.111Since our goal is to model processes like the Rooney Rule, which apply to the selection of finalists for interviewing, rather than to the hiring decision itself, we treat the choice of finalists as the endpoint rather than modeling the interviews that subsequently ensue.

If the committee could exactly evaluate the potential of each applicant, then it would have a straightforward way to maximize the utility of the set of finalists: simply sort all applicants by potential, and choose the top as finalists. The key feature of the situation we would like to capture, however, is that the committee is biased in its evaluations; we look for a model that incorporates this bias as cleanly as possible.

Empirical work in some of our core motivating settings — such as the evaluation of scientists and faculty candidates — indicates that evaluation committees often systematically downweight female and minority candidates of a given level of achievement, both in head-to-head comparisons and in ranking using numerical scores [23]. It is thus natural to model the hiring committee’s evaluations as follows: they correctly estimate the potential of a -applicant at the true value , but they estimate the potential of an -applicant at a reduced value . They then rank candidates by these values and , and they choose the top according to this biased ranking.

For most of the paper, we focus on the case of multiplicative bias, in which for a bias parameter222When , the ranking has no bias. . This is a reasonable approximation to empirical data from human-subject studies [23]; and moreover, for power law distributions this multiplicative form is in a strong sense the “right” parametrization of the bias, since biases that grow either faster or slower than multiplicatively have a very simple asymptotic behavior in the power law case.

In this aspect of the model, as in others, we seek the cleanest formulation that exposes the key underlying issues; for example, it would be an interesting extension to consider versions in which the estimates for each individual are perturbed by random noise. A line of previous work [4, 10, 11] has analyzed models of ranking under noisy perturbations; while our scenario is quite different in that the entities being ranked are partitioned into a fixed set of groups with potentially different levels of bias and noise, it would be natural to see if these techniques could potentially be extended to handle noise in the context of implicit bias.

1.2 Main Questions and Results

This then is the basic model in which we analyze interventions with the structure of the Rooney Rule: (i) a set of -applicants and -applicants each have an independent future potential drawn from a power law distribution; (ii) a hiring committee ranks the applicants according to a sorted order in which each -applicant’s potential is divided down by , and chooses the top in this ordering as finalists; and (iii) the hiring committee’s utility is the sum of the potentials of the finalists.

Qualitatively, the motivation for the Rooney Rule in such settings is that hiring committees are either unwilling or unable to reasonably correct for their bias in performing such rankings, and therefore cannot be relied on to interview -candidates on their own. The difficulty in removing this skew from such evaluations is a signature aspect of phenomena around implicit bias.

The decision to impose the Rooney Rule is made at the outset, before the actual values of the potentials and are materialized. All that is known at the point of this initial decision to use the Rule or not are the parameters of the domain: the bias , the relative abundance of -candidates , the power law exponent , and the number of finalists to be chosen . The question is: as a function of these parameters, will the use of the Rooney Rule produce a positive or negative expected change in utility, where the expectation is taken over the random draws of applicant values? We note that one could instead ask about the probability that the Rooney Rule produces a positive change in utility as opposed to the expected change; in fact, our techniques naturally extend to characterize not only the expected change, but the probability that this change is positive, as we will show in Section 2.

Figure 1: Fixing , the values for which the Rooney Rule produces a positive expected change for sufficiently large lie above a surface (depicted in the figure) defined by the function .

Our model lets us make precise the trade-off in utility that underpins the use of the Rooney Rule. If the committee selects an -candidate on its own — even using its biased ranking — then their choice already satisfies the conditions of the Rule. But if all finalists are -candidates, then the Rooney Rule requires that the committee replace the lowest-ranked of these finalists with the highest-ranked -candidate . Because was not already a finalist, we know that . But to see whether this yields a positive change in utility, we need to understand which of or has a larger expected value, conditional on the information contained in the committee’s decision, that .

Our main result is an exact characterization of when the Rooney Rule produces a positive expected change in terms of the four underlying parameters, showing that it non-trivially depends on all four. For the following theorem, and for the remainder of the paper, we assume , , and . We begin with the case where .

Theorem 1.1.

For and sufficiently large , the Rooney Rule produces a positive expected change if and only if where


and . Moreover, is increasing in , so for fixed and there exists such that if and only if .

Thus, we have an explicit characterization for when the Rooney Rule produces positive expected change. The following theorem extends this to larger values of .

Theorem 1.2.

There is an explicit function such that the Rooney Rule produces a positive expected change, for sufficiently large and , if and only if .

Interestingly, even for larger values of , there are parts of the parameter space for which the Rooney Rule produces a positive expected change and parts for which the Rooney Rule produces a negative expected change, independent of the number of applicants .

Figure 1 depicts a view of the function , by showing the points in three-dimensional space for which takes the value . The values for which the Rooney Rule produces a positive expected change for sufficiently large lie above this surface.

The surface in Figure 1 is fairly complex, and it displays unexpected non-monotonic behavior. For example, on certain regions of fixed , it is non-monotonic in , a fact which is not a priori obvious: there are choices of and for which the Rooney Rule produces a positive expected change at certain “intermediate” values of , but not at values of that are sufficiently smaller or sufficiently larger. Moreover, there exist pairs above which the surface does not exist. (One example in Figure 1 occurs at and ). Characterizing the function and its level set is challenging, and it is noteworthy that the complexity of this function is arising from our relatively bare-bones formulation of the trade-off in the Rooney Rule; this suggests the function and its properties are capturing something inherent in the process of biased selection.

One monotonocity result we are able to establish for the function is the following, showing that for fixed , increasing the number of positions can’t make the Rooney Rule go from beneficial to harmful.

Theorem 1.3.

For sufficiently large and , if the Rooney Rule produces a positive expected change at a given number of finalists , it also produces a positive expected change when there are finalists (at the same ).

We prove these theorems through an analysis of the order statistics of the underlying power law distribution. Specifically, if we draw samples from the power law and sort them in ascending order from lowest to highest, then the item in the sorted list is a random variable denoted . To analyze the effect of the Rooney Rule, we are comparing with . Crucially, we are concerned with their expected values conditional on the fact that the committee chose the -ranked -candidate over the top-ranked -candidate, implying as noted above that . The crucial comparison is therefore between and . Order statistics conditional on this type of side information turn out to behave in complex ways, and hence the core of the analysis is in dealing with these types of conditional order statistics for power law distributions.

More generally, given the ubiquity of power law distributions [6], we find it surprising how little is known about how their order statistics behave qualitatively. In this respect, the techniques we provide may prove to be independently useful in other applications. For example, we develop a tight asymptotic characterization of the expectations of order statistics from a power law distribution that to our knowledge is novel.

We also note that although our results are expressed for sufficiently large , the convergence to the asymptotic behavior happens very quickly as grows; to handle fixed values of , we need only modify the bounds by correction terms that grow like . In particular, the errors in the asymptotic analysis are small once reaches 50, which is reasonable for settings in which a job opening receives many applications.

Estimating the level of bias .

The analysis techniques we develop for proving Theorem 1.2 can also be used for related problems in this model. A specific question we are able to address is the problem of estimating the amount of bias from a history of hiring decisions.

In particular, suppose that over years the hiring committee makes one offer per year; in of the years this offer goes to an -candidate, and in of the years this offer goes to a -candidate. Which value of the bias parameter maximizes the probability of this sequence of observations?

We provide a tight characterization of the solution to this question, finding again that it depends not only on (in this case, the sequence of values for each year), but also on the power law exponent . The solution has a qualitatively natural structure, and produces (corresponding to no bias) as the estimate when the fraction of -candidates hired over the years is equal to the expected number that would be hired under random selection.

Generalizations to other distributions.

Finally, at the end of the paper we consider how to adapt our approach for classes of distributions other than power laws. A different category of distributions that can be motivated by the considerations discussed here is the set of bounded distributions, which take values only over a finite interval. Just as power laws are characteristic of the performance of employees in certain professions, bounded distributions are appropriate when there are absolute constraints on the maximum effect a single employee can have.

Moreover, bounded distributions are also of interest because they contain the uniform distribution on as a special case. We can think of this special case as describing an instance in which each candidate is associated with their quantile (between and ) in a ranking of the entire population, and the bias then operates on this quantile value, reducing it in the case of -candidates.

For bounded distributions, we can handle much more general forms for the bias — essentially, any function that reduces the values strictly below the maximum of the distribution (for instance, a bias that always prefers a -candidate to an -candidate when they are within some of each other). When and there are equal numbers of -candidates and -candidates, we show that for any bounded distribution and any such bias, the Rooney Rule produces a positive expected change in utility for all sufficiently large .

1.3 An Illustrative Special Case: Infinite Bias

To illustrate some of the basic considerations that go into our analysis and its interpretation, we begin with a simple special case that we can think of as “infinite bias” — the committee deterministically ranks every -candidate above every -candidate. This case already exhibits structurally rich behavior, although the complexity is enormously less than the case of general . We also focus here on . In terms of Figure 1, we can visualize the infinite bias case as if we are looking down at the plot from infinitely high up; thus, reasoning about infinite bias amounts to determining which parts of the plane are covered by the surface .

With infinite bias, the committee is guaranteed to choose the two highest-ranked -candidates in the absence of an intervention; with the Rooney Rule, the committee will choose the highest-ranked -candidate and the highest-ranked -candidate. As we discuss in the next section, for power law distributions with exponent , if is the expected maximum of draws from the distribution, then (i) the expected value of the second-largest of the draws is ; and (ii) the expected maximum of draws from the distribution is asymptotically .

This lets us directly evaluate the utility consequences of the intervention. If there is no intervention, the utility of the committee’s decision will be , and if the Rooney Rule is used, the utility of the committee’s decision will be . Thus, the Rooney Rule produces positive expected change in utility if and only if ; that is, if and only if .

In addition to providing a simple closed-form expression for when to use the Rooney Rule in this setting, the condition itself leads to some counter-intuitive consequences. In particular, the closed-form expression for the condition makes it clear that for every , there exists a sufficiently small so that when the distribution of applicant potentials is a power law with exponent , using the Rooney Rule produces the higher expected utility. In other words, with a power law exponent close to 1, it’s a better strategy to commit one of the two offers to the -candidates, even though they form an extremely small fraction of the population.

This appears to come perilously close to contradicting the following argument. We can arbitrarily divide the -candidates into two sets and of each; and if , each of and is larger than the set of all -candidates. Let be the top candidate in and be the top candidate in . Each of and has at least the expected value of the top -candidate, and moreover, one of them is the top -candidate overall. So how can it be that choosing and fails to improve on the result of using the Rooney Rule?

The resolution is to notice that using the Rooney Rule still involves hiring the top -candidate. So it’s not that the Rooney Rule chooses one of or at random, together with the top -candidate. Rather, it chooses the better of and , along with the top -candidate. The real point is that power law distributions have so much probability in the tail of the distribution that the best person among a set of can easily have a higher expected value than the second-best person among a set of , even when is quite small. This is a key property of power law distributions that helps explain what’s happening both in this example and in our analysis.

1.4 A Non-Monotonicity Effect

As noted above, much of the complexity in the analysis arises from working with expected values of random variables conditioned on the outcomes of certain biased comparisons. One might hope that expected values conditional on these types of comparisons had tractable properties that facilitated the analysis, but this is not the case; in fact, these conditional expectations exhibit some complicated and fairly counter-intuitive behavior. To familiarize the reader with some of these phenomena — both as preparation for the subsequent sections, but also as an interesting end in itself — we offer the following example.

Much of our analysis involves quantities like — the conditional expectation of , given that it exceeds some other random variable multiplied by a bias parameter. (We will also be analyzing the version in which the inequality goes in the other direction, but we’ll focus on the current expression for now.) If we choose and as independent random variables both drawn from a distribution , and then view the conditional expectation as a function just of the bias parameter , what can we say about the properties of this function ?

Intuitively we’d expect to be monotonically increasing in — indeed, as increases, we’re putting a stricter lower bound on , and so this ought to raise the conditional expectation of .

The surprise is that this is not true in general; we can construct independent random variables and for which is not monotonically increasing. In fact, the random variables are very simple: we can have each of and take values independently and uniformly from the finite set . Now, the event consists of four possible pairs of values: (5,1), (9,1), (13,1), and (13,5). Thus, . In contrast, the event consists of three possible pairs of values: (5,1), (9,1), and (13,1). Thus, , which is a smaller value, despite the fact that is required to be a larger multiple of .

The surprising content of this example has a fairly sharp formulation in terms of a story about recruiting. Suppose that two academic departments, Department and Department , both engage in hiring each year. In our stylized setting, each interviews one -candidate and one -candidate each year, and hires one of them. Each candidate comes from the uniform distribution on . Departments and are both biased in their hiring: only hires the -candidate in a given year if they’re more than twice as good as the -candidate, while only hires the -candidate in a given year if they’re more than three times as good as the -candidate.

Clearly this bias hurts the average quality of both departments, more so than . But you might intuitively expect that at least if you looked at the -candidates that has actually hired, they’d be of higher average quality than the -candidates that has hired — simply because they had to pass through a stronger filter to get hired. In fact, however, this isn’t the case: despite the fact that imposes a stronger filter, the calculations performed above for this example show that the average quality of the -candidates hires is , while the average quality of the -candidates hires is .

This non-monotonicity property shows that the conditional expectations we work with in the analysis can be pathologically behaved for arbitrary (even relatively simple) distributions. However, we will see that with power law distributions we are able — with some work — to avoid these difficulties; and part of our analysis will include a set of explicit monotonicity results.

2 Biased Selection with Power Law Distributions

Recall that for a random variable , we use to denote the order statistic in draws from : the value in position when we sort independent draws from from lowest to highest. Recall also that when selecting finalists, the Rooney Rule improves expected utility exactly when

Using linearity of expectation and the fact that , this is equivalent to


We will show an asymptotically tight characterization of the tuples of parameters for which this condition holds, up to an error term on the order of . In order to better understand the terms in (2), we begin with some necessary background.

2.1 Preliminaries

Fact 1.

Let and be, respectively, the probability density function and cumulative distribution function of the order statistic out of draws from the power law distribution with parameter . Using definitions from [9],


Definition 2.

We define

is considered the continuous relaxation of the factorial, and it satisfies

If is a positive integer, . Furthermore, for and for .

2.2 The Case where

For simplicity, we begin with the case where we’re selecting finalists. In this section, we will make several approximations, growing tight with large , that we treat formally in Appendices A and B. This section is intended to demonstrate the techniques needed to understand the condition (2). In the case where , always selecting an -candidate increases expected utility if and only if


Theorems B.1 and B.2 in Appendix B give tight approximations to these quantities; here, we provide an outline for how to find them. For the sake of exposition, we’ll only show this for the denominator in this section, which is slightly simpler to approximate. We begin with

Letting , we can use Lemma D.2 and some manipulation to approximate this by

Conveniently, the function being integrated is (up to a constant factor) , i.e.  times the probability density function of the second-highest order statistic from samples. Since

we have

Then, we can use Lemmas D.10 and D.11 to get , meaning that


For the numerator of (3), a slightly more involved calculation yields


By Lemmas D.10 and D.11, and . Recall that, up to the approximations we made, the Rooney Rule improves utility in expectation if and only if the ratio between (5) and (4) is larger than 1. Therefore, the following theorem holds:

Theorem 2.1.

For sufficiently large , the Rooney Rule with improves utility in expectation if and only if


where .

Note that in the limit as , , and the entire expression goes to , as noted in Section 1.3. Although the full expression in the statement of Theorem 2.1 is complex, it can be directly evaluated, giving a tight characterization of when the Rule yields increased utility in expectation.

With this result, we could ask for a fixed and how to characterize the set of such that the condition in (6) holds. In fact, we can show that this expression is monotonically increasing in .

Theorem 2.2.

The left hand side of (6) is decreasing in and therefore increasing in . Hence for fixed and there exists such that (6) holds if and only if .

Non-monotonicity in .

From Theorem 2.1, we can gain some intuition for the non-monotonicity in shown in Figure 1. For , we can show that even with infinite bias, the Rooney Rule has a negative effect on utility for sufficiently large . Intuitively, this is because the condition for positive change with infinite bias is , which can be written as for . Since this converges to from below, for sufficiently large and , we have . On the other hand, as , the Rooney Rule has a more negative effect on utility. For instance, but . Intuitively, this non-monotonicity arises from the fact that for large and small , the Rooney Rule always has a negative impact on utility, while for very small , samples are very far from each other, meaning that the bias has less effect on the ranking.

2.3 The General Case

We can extend these techniques to handle larger values of . For , we define

We can see that the Rooney Rule improves expected utility when selecting candidates if and only if . While depends on , we will show that it is a very weak dependence: for small , as increases, converges to a function of up to a multiplicative factor. To make this precise, we define the following notion of asymptotic equivalence:

Definition 3.

For nonnegative functions and , define

if and only if there exist and such that

for all . In other words, . When being explicit about and , we’ll write .

Appendix C contains a series of lemmas establishing how to rigorously manipulate equivalences of this form. Now, we formally define a tight approximation to , which serves as an expanded restatement of Theorem 1.2 from the introduction.

Theorem 2.3.

For , define


where . Note that does not depend on . When are fixed, we will simply write this as . For , we have

and therefore the Rooney Rule improves expected utility for sufficiently large if and only if .

This condition tightly characterizes when the Rooney Rule improves expected utility, and its asymptotic nature in becomes accurate even for moderately small : for example, when , the error between and is around for reasonable choices of .

Increasing .

Consider the scenario in which we’re selecting candidates, and for the given parameter values, the Rooney Rule improves our expected utility. If we were to instead select candidates, should we still be reserving a spot for an -candidate? Intuitively, as increases, the Rule is less likely to change our selections, since we’re more likely to have already chosen an -candidate; however, it is not a priori obvious whether increasing should make it better for us to use the Rooney Rule (because we have more slots, so we’re losing less by reserving one) or worse (because as we take more candidates, we stop needing a reserved slot).

In fact, we can apply Theorem 2.3 to understand how changes with . The following theorem, proven in Appendix B, is an expanded restatement of Theorem 1.3, showing that if the Rooney Rule yields an improvement in expected quality when selecting candidates, it will do so when selecting candidates as well.

Theorem 2.4.

For , we have , and therefore for sufficiently large , we have .

Finally, using these techniques, we can provide a tight characterization of the probability that the Rooney Rule produces a positive change. Specifically, we find the probability that the Rooney Rule has a positive effect conditioned on the event that it changes the outcome.

Theorem 2.5.

To determine whether the Rooney Rule is more likely than not to produce a positive effect (conditioned on changing the outcome), we can compare the right-hand side to .

Note that in the case of infinite bias, the right-hand side becomes , and thus, the Rooney Rule produces positive change with probability at least if and only if . It is interesting to observe that this means with infinite bias, the condition is independent of ; in contrast, when considering the effect on the expected value with infinite bias, as we did in Section 1.3, the expected change in utility due to the Rooney Rule did depend on .

2.4 Maximum Likelihood Estimation of

The techniques established thus far make it possible to answer other related questions, including the following type of question that we consider in this section: “Given some historical data on past selections, can we estimate the bias present in the data?” For example, suppose that for the last years, a firm has selected one candidate for each year out of a pool of -candidates and -candidates. If all applicants are assumed to come from the same underlying distribution, then it is easy to see that the expected number of -selections (in the absence of bias) should be

regardless of what distribution the applicants come from. However, if there is bias in the selection procedure, then this quantity now depends on the bias model and parameters of the distribution. In particular, in our model, we can use Theorem B.3 to get

This gives us the following approximation for the likelihood of the data given , where is if an -candidate was selected in year and otherwise:

Taking logarithms, this is

and maximizing this is equivalent to maximizing

where is the number of -candidates selected. Taking the derivative with respect to , we get

Setting this equal to 0 and canceling common terms, we have

Since each is strictly monotonically decreasing in , there is a unique for which equality holds, meaning that the likelihood is uniquely maximized by , up to the approximation we made for . In the special case where for , then the solution is given by

3 Biased Selection with Bounded Distributions

In this section, we consider a model in which applicants come from a distribution with bounded support. Qualitatively, one would expect different results here from those with power law distributions because in a model with bounded distributions, we expect that for large , the top order statistics of any distribution will concentrate around the maximum of that distribution. As a result, when there is even a small amount of bias against one population, for large the probability that any of the samples with the highest perceived quality come from that population goes to 0. This means that the Rooney Rule has an effect with high probability, and the effect is positive if the unconditional expectation of the top -candidate is larger than the unconditional expectation of the -candidate that it replaces.

We focus on the case when , meaning we have equal numbers of applicants from both populations. We use the same order statistic notation as before. While all of our previous results have modeled the bias as a multiplicative factor , we can in fact show that in the bounded distribution setting, for any model of bias such that for , where is strictly less than the maximum of the distribution, the Rooney Rule increases expected utility. Unlike in the previous section the following theorem and analysis are by no means a tight characterization; instead, this is an existence proof that for bounded distributions, there is always a large enough such that the Rooney Rule improves utility in expectation. We prove our results for continuous distributions with support , but a simple scaling argument shows that this extends to any continuous distribution with bounded nonnegative support – specifically, we scale a distribution such that and .

Theorem 3.1.

If is a continuous probability density function on such that and is never more than , then for large enough ,

While we the defer the full proof to Appendix E, the strategy for the proof is as follows:

  1. With high probability, and are both large.

  2. Whenever and are large, is significantly larger than .

  3. The gain from switching from to when and are both large outweighs the loss when at least one of them is not large.

4 Conclusion

In this work we have presented a model for implicit bias in a selection problem motivated by settings including hiring and admissions, and we analyzed the Rooney Rule, which can improve the quality of the resulting choices. For one of the most natural settings of the problem, when candidates are drawn from a power-law distribution, we found a tight characterization of the conditions under which the Rooney Rule improves the quality of the outcome. In the process, we identified a number of counter-intuitive effects at work, which we believe may also help provide insight into how we can reason about implicit bias. Our techniques also provided a natural solution to an inference problem in which we estimate parameters of a biased decision-making process. Finally, we performed a similar type of analysis on general bounded distributions.

There are a number of further directions in which these issues could be investigated. One intriguing direction is to consider the possible connections to the theory of optimal delegation (see e.g. [1]).333We thank Ilya Segal for suggesting this connection to us. In the study of delegation, a principal wants a task carried out, but this task can only be performed by an agent who may have a utility function that is different from the principal’s. In an important family of these models, the principal’s only recourse is to impose a restriction on the set of possible actions taken by the agent, creating a more constrained task for the agent to perform, in a way that can potentially improve the quality of the eventual outcome from the principal’s perspective. Our analysis of the Rooney Rule can be viewed as taking place from the point of view of a principal who is trying to recruit candidates, but where the process must be delegated to an agent whose utilities for -candidates and -candidates are different from the principal’s, and who is the only party able to evaluate these candidates’ potentials. The Rooney Rule, requiring that the agent select at least one -candidate, is an example of a mechanism that the principal could impose to restrict the agent’s set of possible actions, potentially improving the quality of the selected candidates as measured by the principal. More generally, it is interesting to ask whether there are other contexts where such a link between delegation and this type of biased selection provides insight.

Our framework also makes it possible to naturally explore extensions of the basic model. First, the model can be generalized to include noisy observations, potentially with a different level of noise for each group. It would also be interesting to analyze generalizations of the Rooney Rule; for example, if we were to define the -order Rooney Rule to be the requirement that at least of finalists must be from an underrepresented group, we could ask which produces the greatest increase in utility for a given set of parameters. Finally, we could benefit from a deeper undestanding of the function that appears in our main theorems. For example, while we showed in Theorem 1.3 that is monotone in for , Figure 1 shows that is clearly not monotone in . A better understaning of the function may lead to new insights into our model and into the phenomena it seeks to capture.


We thank Eric Parsonnet for his invaluable technical insights. This work was supported in part by a Simons Investigator Grant and an NSF Graduate Fellowship.


  • [1] Ricardo Alonso and Niko Matouschek. Optimal delegation. Review of Economic Studies, 75(1):259–293, January 2008.
  • [2] Marianne Bertrand and Sendhil Mullainathan. Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination. American Economic Review, 94(4):991–1013, September 2004.
  • [3] Iris Bohnet, Alexandra van Geen, and Max Bazerman. When performance trumps gender bias: Joint vs. separate evaluation. Management Science, 62(5):1225–1234, 2016.
  • [4] Mark Braverman and Elchanan Mossel. Sorting from noisy information. arXiv preprint arXiv:0910.1191, 2009.
  • [5] Marilyn Cavicchia. How to fight implicit bias? With conscious thought, diversity expert tells NABE. American Bar Association: Bar Leader, 40(1), 2015.
  • [6] Aaron Clauset, Cosma R. Shalizi, and Mark E. J. Newman. Power-law distributions in empirical data. SIAM Review, 51(4):661–703, 2009.
  • [7] Brian W. Collins. Tackling unconscious bias in hiring practices: The plight of the Rooney Rule. NYU Law Review, 82(3), 2007.
  • [8] Covington and Burling. Recommendations to Uber, 13 June 2017.
  • [9] H. A. David and H. N. Nagaraja. Basic Distribution Theory, pages 9–32. John Wiley & Sons, Inc., 2005.
  • [10] Uriel Feige, Prabhakar Raghavan, David Peleg, and Eli Upfal. Computing with noisy information. SIAM Journal on Computing, 23(5):1001–1018, 1994.
  • [11] Qiang Fu and Jingfeng Lu. Micro foundations of multi-prize lottery contests: a perspective of noisy performance ranking. Social Choice and Welfare, 38(3):497–517, 2012.
  • [12] Alexander R. Green, Dana R. Carney, Daniel J. Pallin, Long H. Ngo, Kristal L. Raymond, Lisa I. Iezzoni, and Mahzarin R. Banaji. Implicit bias among physicians and its prediction of thrombolysis decisions for black and white patients. Journal of General Internal Medicine, 22(9):1231–1238, 2007.
  • [13] Anthony G. Greenwald and Mahzarin R. Banaji. Implicit social cognition: attitudes, self-esteem, and stereotypes. Psychological Review, 102(1):4–27, 1995.
  • [14] Anthony G. Greenwald and Linda Hamilton Krieger. Implicit bias: Scientific foundations. California Law Review, 94:945–967, 2006.
  • [15] Christine Jolls and Cass R. Sunstein. The law of implicit bias. California Law Review, 94:969–996, 2006.
  • [16] Manuel Lopez and James Marengo. An upper bound for the expected difference between order statistics. Mathematics Magazine, 84(5):365–369, 2011.
  • [17] Henrick John Malik. Exact moments of order statistics from the pareto distribution. Scandinavian Actuarial Journal, 1966(3-4):144–157, 1966.
  • [18] Christina Passariello. Tech firms borrow football play to increase hiring of women. Wall Street Journal, 27 September 2016.
  • [19] Hamza Shaban. What is the “Rooney Rule” that Uber just adopted? Washington Post, 13 June 2017.
  • [20] Francesco Giacomo Tricomi and Arthur Erdélyi. The asymptotic expansion of a ratio of gamma functions. Pacific J. Math, 1(1):133–142, 1951.
  • [21] Eric Luis Uhlmann and Geoffrey L. Cohen. Constructed criteria: Redefining merit to justify discrimination. Psychological Science, 16(6):474–480, 2005.
  • [22] Linda van den Bergh, Eddie Denessen, Lisette Hornstra, Marinus Voeten, and Rob W. Holland. The implicit prejudiced attitudes of teachers: Relations to teacher expectations and the ethnic achievement gap. American Education Research Journal, 47(2):497–527, 2010.
  • [23] Christine Wenneras and Agnes Wold. Nepotism and sexism in peer-review. Nature, 387:341–343, 1997.

Appendix A Missing Proofs for Section 2

Proof of Theorems 2.1 and 2.3.

We can expand the statement in Theorem B.1 to

(By Lemma D.10)

This gives us a ratio

(Using Theorem B.2)

Proof of Theorem 2.2.

Since the only influence of is through and is decreasing in , it is sufficient to show that

is decreasing in . Ignoring constants, this is

This has derivative

which is negative if and only if

This is true by Lemma D.9, which proves the theorem. ∎

Proof of Theorem 2.4.

By Theorem 2.3,

We use the fact that for and

If the summation went to , it would be