Removing Algorithmic Discrimination
(With Minimal Individual Error)
Abstract
We address the problem of correcting group discriminations within a score function, while minimizing the individual error. Each group is described by a probability density function on the set of profiles. We first solve the problem analytically in the case of two populations, with a uniform bonusmalus on the zones where each population is a majority. We then address the general case of populations, where the entanglement of populations does not allow a similar analytical solution. We show that an approximate solution with an arbitrarily high level of precision can be computed with linear programming. Finally, we address the inverse problem where the error should not go beyond a certain value and we seek to minimize the discrimination.
1 Introduction
As machine learning is being deployed, a growing number of cases of discriminatory behaviors is being highlighted. In 2016, a study by ProPublica^{1}^{1}1See https://tinyurl.com/machinebiassentencing showed that some algorithmic assessment of recidivism risks was significantly racially biased against black criminals. Indeed, 45% of supposedly highrisk black criminals did not reoffend, as opposed to 22% of supposedly highrisk white criminals. Conversely, 28% of supposedly lowrisk black criminals reoffended, as opposed to 48% of supposedly lowrisk white criminals. Such concerns for algorithmic discrimination have fostered a lot of work.
A major difficulty posed by new machine learning techniques is that algorithms may have learned their biases from highdimensional data, which ironically seems hard to handle without machine learning. Racial inequalities in facial recognition have for instance been showed in [buolamwini2018]. More disturbingly, it was discovered that the popular word2vec package [mikolov2013] yields gender discriminative relations between word representations, e.g., . In other words, word2vec seems to infer from natural language processing that a man is to a woman what a doctor is to a nurse. Evidently, this is only one example out of many. Such examples illustrate the difficulty of mitigating algorithmic discrimination.
Many solutions have been proposed. Some consist in preprocessing data used for machine learning [dxp2, dxp4, dxp5, dxp6, dxp15] or making it unbiased [dxp7]. Some try to prevent discrimination during the learning phase [dxp13, dxp14, dxp3], by using causal reasoning [dxp1], or with graphical dependency models [dxp16]. Other approaches try to achieve independence from specific sensitive attributes [dxp8, dxp18, dxp17]. Dwork et al. [dxp19] introduced the concept of “fair affirmative action”, to improve the treatment of specific groups while treating similar individuals similarly. Algorithmic discrimination was also considered in problems of subsampling [dxp9], voting [dxp10], personalization [dxp11] or ranking [dxp12].
All previous works highlighted the fundamental tradeoff between group discrimination (i.e., some groups being globally penalized compared to other groups) and individual accuracy (i.e., individuals being judged with a high level of precision). In this paper, we propose a postprocessing approach to remove group discrimination while minimizing the individual error^{2}^{2}2The social impact of removing group discrimination and the extent to which such an enterprise is desirable are out of the scope of this paper. We “simply” address the problem of doing it with a minimal error., as well as an approach to minimize group discrimination given an individual error constraint.
More specifically, we assume that we are given a score function that computes a score for each individual . Here, the individual’s profile can be any sort of description of the individual. In simple settings, it may be a collection of realvalued features, i.e. , and the scoring function may be interpretable. However, as machine learning improves, rawer data are being used to score individuals, e.g. they may be textual biographies of undetermined length. In such cases, the scoring function is usually constructed via machine learning, and it often has to be regarded as some “black box”. To remove group discrimination, rather than preprocessing raw data or modifying the learning phase, it may thus be simpler to perform some postprocessing of the score function, i.e. deriving a nondiscriminative score function from the possibly discriminative function .
An additional difficulty is that the individual’s profile may not clearly determine its sensitive features, e.g. gender or race. Nevertheless, evidently, even biographic texts may provide strong indications of the individual’s likely sensitive features. A natural approach to analyze the dependency of the score function on sensitive features is to test its scoring on profiles that are representative of a certain gender or race. Interestingly, this approach can now be simulated using socalled generative models [goodfellow2014, karras2017]. These models allow to draw representative samples of subpopulations of individuals.
Thus, we assume that any population (women, men, black, white, …) can be described by some generative model. Formally, this corresponds to saying that the population is represented by a probability density function on . Given , we can determine the average score of population (i.e., ), which can be well approximated by sampling the generative model associated to population . A toy example of average score is given in Figure 1.
Contributions. We first study in this paper the simple case of two populations with a different average score. The goal here is to determine a new score function where (a) the two populations have the same average score and (b) the individual error is minimized. We define the individual error as the maximal difference between and , i.e., (also written ). We call the problem of determining the best function the 2ODR (2Optimal Discrimination Removal) problem (“2” standing for “two populations”). We present an exact solution to the 2ODR problem. Roughly speaking, we consider the subsets of where and , and apply a uniform bonus (or penalty) on these subsets. We show that our solution is indeed optimal for it minimizes the individual error.
Then we turn to the more general case of populations, which is arguably the most relevant setting in practice. Indeed, it is for instance often considered important that a score function be both nonracist and nonsexist. Similarly, it may be relevant to compare the scores of several races, e.g. Black, White, Asian and Arabic. In fact, we may even demand greater granularity by also comparing black female and white female, in addition to already comparing black and white. We address this population setting by considering some desired average score for each population . This more general goal enables the modelers to describe more subtly what they consider desirable. We call this problem the Optimal Discrimination Removal (ODR) problem.
This problem is significantly more difficult with . In fact, we conjecture that it is computationally intractable for and combinatorially large profile sets . Indeed, intuitively, in the case , the general problem of removing discrimination could be fixed locally for each , by determining whether is more likely to be of population 1 or 2. Unfortunately, this no longer seems to be the case when . To solve the ODR problem, it seems that a global solution first needs to be derived. But this global solution seems to require at least computation steps in general.
Interestingly though, we show that an approximate solution (with an arbitrarily high level of precision) can be obtained with linear programming [zlp0]. Linear programming problems are expressed in terms of a set of inequalities involving linear combinations of variables. These problems have been extensively studied, and a lot of algorithms have been proposed to solve them [zlp1, zlp2, zlp5, zlp6]. Here, we show that this abundant literature of algorithms can also be leveraged to solve discrimination problems.
We proceed incrementally through 6 steps. We first show that the ODR problem is reducible to the simpler (to express) Optimal BonusMalus (OBM) problem, where each desired average score is . We then define an approximate version of OBM, which we denote AOBM. We consider an arbitrary partition of , as well as a set of functions which are “flat” on each subset . The AOBM problem consists in approximating a solution to the OBM problem with a function . The larger , the more precise the solution. We show that the AOBM problem is equivalent to a linear programming problem with variables and inequalities^{3}^{3}3Excluding the inequalities requiring each variable to be positive (which are included in the canonical form of a linear programming problem).. We use the fact that the functions of can only take a finite number of values, to transform the continuous OBM problem into a discrete problem.
We finally also address the inverse problem, where the individual error is not allowed to be greater than . Here, the goal is to be as close as possible to the desired score of each population. We proceed in an analogous way through 6 steps.
2 The Case of Two Populations
Let be a set of profiles. Let be a function from to associating a score to each profile. Let and be any two probability density functions on , representing two populations and .
Let be the set of functions from to such that (i.e. population and have the same average score).
For any function from to , let .
The 2ODR (2Optimal Discrimination Removal) problem consists in finding a function , i.e., a function minimizing the individual error.
Solution.
For , let if and otherwise.
Let , and .
We define by .
Theorem 1.
Function above solves the 2ODR problem.
Proof.
By construction, . If , indeed minimizes . We now suppose that .
The proof is by contradiction. Suppose the opposite of the claim: there exists a function such that . Then, , with . Let . Then, .
By definition, , with and . Thus, , and .
Let (resp. ) be the subset of such that (resp. ). Then, , where . Let if and otherwise.
If (resp. ), (resp. ). Then, as , we have (resp. ). Thus, , and .
Therefore, , and : contradiction. Hence, our result. ∎
3 The General Case
We now consider the case of populations. This problem when is significantly harder than the problem above due to the entanglement of several probability density functions. We show that an approximate solution of this problem can be obtained with linear programming. We proceed incrementally through 6 steps.

We define the general Optimal Discrimination Removal (ODR) problem, corresponding to the case .

We define a simpler (to express) problem, the Optimal BonusMalus (OBM) problem.

We show that solving OBM provides an immediate solution to ODR.

We define an approximate version of the OBM problem (AOBM), where we restrict ourselves to functions which are “flat” on an arbitrarily large number of subsets of .

We define a Linear Programming problem, that we simply call LP for convenience.

We show that LP also solves AOBM.
Step 1: The Optimal Discrimination Removal (ODR) Problem
Let be probability density functions on , each one representing a population. Let be arbitrary values.
Let be the set of functions from to such that, , (i.e. the mean score of population is ).
If , the ODR problem consists in finding a function .
Step 2: The Optimal BonusMalus (OBM) Problem
, let .
Let be the set of functions from to such that, , .
If , the OBM problem consists in finding a function .
Step 3: Reducing ODR to OBM
Theorem 2 below says that a solution to the OBM problem provides an immediate solution to the ODR problem.
Theorem 2.
If solves the OBM problem, then solves the ODR problem.
Proof.
As solves the OBM problem, we have the following: , .
Note that, if , then . Indeed, if , then , . Thus, , . Thus, .
Therefore, , . As , we have: , . Thus, . Thus, the result. ∎
Step 4: The Approximate OBM (AOBM) Problem
Let be a partition of : , and , . Let be the set of functions from to such that, , and , (i.e. is “flat” on each subset ).
If , the AOBM problem consists in finding a function .
Step 5: The Linear Programming (LP) Problem
Let and be two integers. Let be variables. Let and be linear combinations of the variables . Let be constant terms.
A linear programming problem consists in finding values of maximizing L while verifying the following inequalities:

,

,
In the following, we define a specific linear programming problem, that we simply call LP problem for convenience.
and , let .
Let , and be variables.
Consider the following inequalities:

, and , and .

, and .

,

,
The LP problem consists in finding values of , and maximizing while satisfying the aforementioned inequalities.
Step 6: Reducing AOBM to LP
Let , and be a solution to the LP problem. , let be the integer such that . Let be the function from to such that, , .
Theorem 3 below says that solves the AOBM problem. We first prove some lemmas.
Lemma 1.
, where and .
Proof.
Suppose the opposite: . According to inequalities 2, . Thus, .
, we define and as follows:

If , and .

Otherwise, and .
Let .
, and . Thus, and .
We now show that , and satisfy the inequalities of the LP problem.
Inequalities 1 are satisfied by definition. , and . Thus, and , and inequalities 2 are satisfied.
Inequalities 3 and 4 are equivalent to: , . :

If , .

Otherwise, .
Thus, , . Thus, . Thus, inequalities 3 and 4 are satisfied.
Thus, there exists , and satisfying the inequalities of the LP problem with . Thus, , and do not solve the LP problem: contradiction. Thus, the result. ∎
Lemma 2.
.
Proof.
. , and . Thus, , and .
We now show that . Suppose the opposite: . As the LP problem consists in maximizing (and thus, minimizing ), this implies that the inequalities of the LP problem are not compatible with . Variable only appears in inequalities 1 and 2, and these inequalities impose to have , and . Thus, , where and . Thus, according to Lemma 1, .
Therefore, . ∎
Theorem 3.
Function solves the AOBM problem.
Proof.
By definition, .
Inequalities 3 and 4 of the LP problem are equivalent to: , . Thus, , . Thus, .
Therefore, . Now, suppose the opposite of the claim: . Let . Thus, .
Let be such that, , . Let . , we define and as follows:

If , and .

Otherwise, and .
Thus, inequalities 1 are satisfied.
As , , . Thus, inequalities 2 are satisfied.
As , , . Thus, , . Thus, inequalities 3 and 4 are satisfied.
According to Lemma 2, . Thus, as , . Therefore, there exists , and satisfying the inequalities of the LP problem with . Thus, , and do not solve the LP problem: contradiction. Thus, the result. ∎
4 The Inverse Case
In the previous section, we showed how to reach the desired scores for each population with a minimal individual error. However, even when minimized, the individual error may still be very high, and sometimes not acceptable.
In this section, we consider the inverse problem: assuming that we can accept an individual error which is at most , how can we reach a score which is as close as possible from the desired scores of each population? We call this problem the inverse ODR (IODR) problem.
We again proceed in 6 steps, following the same outline as the 6 steps of Section 3.
Step 1: The Inverse ODR (IODR) Problem
Let . Let be the set of functions from to such that (i.e., functions for which the individual error remains acceptable).
Let be a function from to . , let (i.e., the distance between the average score of population and its desired average score ). Let (i.e., the upper bound of these distances).
The IODR problem consists in finding a function .
Step 2: The Inverse OBM (IOBM) Problem
Let be the set of functions from to such that .
, let . Let .
The IOBM problem consists in finding a function .
Step 3: Reducing IODR to IOBM
In Theorem 4, we show that a solution to the IOBM problem provides an immediate solution to the IODR problem.
Theorem 4.
If solves the IOBM problem, then solves the IODR problem.
Proof.
Let be a function from to . , . Thus, , and .
Therefore, if , then . Thus, the result ∎
Step 4: The Inverse AOBM (IAOBM) Problem
The IAOBM problem consists in finding a function .
Step 5: The Inverse LP (ILP) Problem
Let , and be variables.
Consider the following inequalities:

, and , and .

, and .

,

,
The ILP problem consists in finding values of , and maximizing while satisfying the aforementioned inequalities.
Step 6: Reducing IAOBM to ILP
Let , and be a solution to the ILP problem. Let be the function from to such that, , .
In Theorem 5, we show that solves the IAOBM problem.
Lemma 3.
.
Proof.
, , according to inequalities 3 and 4. Thus, .∎
Lemma 4.
.
Proof.
Suppose the opposite: . Let . , . Thus, as , inequalities 3 and 4 are still satisfied if we replace by . Thus, as , , and do not solve the ILP problem: contradiction. Thus, the result. ∎
Theorem 5.
The function solves the IAOBM problem.
Proof.
By definition, . According to inequalities 2, . Thus, .
Now, suppose the opposite of the claim: . Let .
Let be such that, , . Let . , we define and as follows:

If , and .

Otherwise, and .
By construction, inequalities 1 are satisfied.
As , , . Thus, , and . Therefore, inequalities 2 are satisfied.
As , , . Thus, inequalities 3 and 4 are satisfied.
5 Conclusion
We consider the problem of removing algorithmic discrimination between several populations with a minimal individual error. We first describe an analytical solution to this problem in the case of two populations. We then show that the general case (with populations) can be solved approximately with linear programming. We also consider the inverse problem where an upper bound on the error is fixed and we seek to minimize the discrimination.
A major challenge would be to either find an analytical solution to the general case with populations or prove that it is indeed intractable. We conjecture the latter. Another interesting question would be to determine how to optimally choose the subsets used for the approximate solution.