An exploration of algorithmic discrimination in data and classification

An exploration of algorithmic discrimination in data and classification

Jixue Liu, Jiuyong Li, Feiyue Ye, Lin Liu, Thuc Duy Le, Ping Xiong
University of South Australia           Jiangsu University of Technology, China

Algorithmic discrimination is an important aspect when data is used for predictive purposes. This paper analyzes the relationships between discrimination and classification, data set partitioning, and decision models, as well as correlation. The paper uses real world data sets to demonstrate the existence of discrimination and the independence between the discrimination of data sets and the discrimination of classification models.

11footnotetext: supported by Grant 61472166 of National Natural Science Foundation China

Keywords: algorithmic discrimination; fairness; association; classification

1 introduction

Discrimination means “treating a person or particular group of people differently, especially in a worse way from the way in which you treat other people, because of their skin colour, sex, sexuality, etc” ( It can happen in law enforcement applications where people may be unfairly treated and sentenced because of their races and religions, in bank loan applications where people may not get a loan because they live in a suburb with a lower economic status.

The law does not allow discrimination to happen. That is, decisions should not be made based on people’s sex, skin colour, religion etc., called protected attributes of individuals. However discrimination is still a concern in the real world. In car insurance, the insurance company required people in a specific suburb to pay higher premium than those in other suburbs with the reason that the suburb has a higher claim rate. If most dwellers of the suburb are of a certain race, the higher premium forms actually discrimination to the people of this race. Boatwright (2017) reported three real world lending discrimination cases [2].

Discrimination does exist in the data collected from real world applications. We will demonstrate this fact in the experiment section of this paper.

The level of discrimination is context-based. Contexts are defined by explanatory attributes. Assume that an investigation is on whether female employees are less paid. Then, the explanatory attributes like profession and position held by the employees matter. A fair comparison between the payments of female employees and male employees must assume that the employees hold same profession and same position. Without a context, the comparison may between the income of a group of male CEOs with the income of a group of male kitchen hands. We use an example to show how discrimination levels are affected by different contexts.

Example 1 Table 1 shows the tuple frequencies of a data set where standing for (1=high, 0=low) is the outcome/class/target attribute, G standing for Gender and is a protected attribute, and a number in the middle of the tables is the number of tuples in having the same value. The discrimination score for females in in Part (a) is 0 where the score is defined as the percentage of high income females taking away the percentage of high income males [3].

For the same application, if a context is defined by an explanatory variable (standing for employment sector, S=private or public), the data set in (a) is divided into two subsets and , and the tuple frequencies for each subset are shown in Parts b.1 and b.2 respectively. The discrimination scores for the corresponding employment sectors are 0.22 and -0.24 respectively. The scores in (a) and (b) conclude that contexts change discrimination levels.

D Female Male
1 10 15
0 40 60
(b.1) for Sector=1
D Female Male
1 9 3
0 20 30
(b.2) for Sector=0
D Female Male
1 1 12
0 20 30
Table 1: Data split

Classification models built from a data set may become discriminatory too. The discrimination of a model is the discrimination of the predictions that the model produces. Consider data in Table 2(a) where each record is for an individual. The column (1=high,0=low) is the performance measurement of individuals. Without an assumed context, this data set has no discrimination. If a predictive model is build on , the model would be . We note that gender is protected and cannot be used in the model. When the model is applied to predict the outcome for the individuals, the predicted data set is in Part (b). It is easy to see that the discrimination score for is 0.5. This confirms that classification models can be discriminatory even if the training data is fair. This type of models are said to be algorithmic discriminatory.

(a) Data
1 F 1
0 F 0
1 M 0
0 M 0
(b) Prediction
1 F 1
0 F 0
0 M 0
0 M 0

Model from training data is .


Table 2: Model changes discrimination.

Algorithmic discrimination becomes more a concern in automated decision making systems as machine learning and data analytics are used more and more in real applications. In loan approval cases, it is a normal practice that an automated system automatically scores and makes decisions on customer’s applications. Fairness of automated decision systems is important to service receivers and to a harmonic society.

Research work on algorithmic discrimination are from two areas: detection and removal. This paper, instead of continuing on these two directions, goes to the fundamental side of these two areas. It aims to explore deeper understanding of discrimination related issues. It analyzes some properties of algorithmic discrimination and explores the level of discrimination in real world data sets and how classification models affect the discrimination levels. More specifically, the paper will show results on the following points.

  • sources of discrimination in classification models and the relationship between fair data and the discrimination of classifiers,

  • discrimination when data sets are merged and split,

  • the relationship between discrimination and explanatory variables in a decision tree,

  • the interaction between discrimination and correlation,

  • experimental exploration of the existence of discrimination in real world data sets and classification models.

The organization of the paper is the following. Section 2 presents a discrimination score and group discrimination. Section 3 presents our analysis of properties of discrimination. In Section 4, we show the results from experimental exploration of real world data sets. Section 5 presents related work and the final section concludes the paper.

2 Definitions and Problem

In this section, we define basic notation, and present a measurement for algorithmic discrimination.

Let be a data set on a schema of binary attributes/variables. The attributes in are of four types: an outcome/target/class attribute , some protected attributes , some explanatory attributes , and other attributes : . For the outcome attribute , means a favorite outcome like or that an individual prefers to receive. For a protected attribute , (e.g. SexFemale or RaceBlack) means a group of individuals who are protected by the law not to to be discriminated. The explanatory attributes explain why some people receive favorite outcomes more or less frequently than others or identify such people. For example, profession and education are often taken as explanatory attributes. Surgeon is a profession and people who are surgeons are mostly high income earners, while kitchen hand is another profession and people who are kitchen hands are low income earners. Explanatory attributes are often used in selection criteria of employment. The ‘other’ attributes are that attributes not in . Some of these attributes may be correlated with the outcome variable and are used in classifiers for predictions, and some, called red-line attributes, may be correlated to some protected attributes which makes discrimination analysis challenging.

An E-group (or stratum of ) is a subset of all the tuples having the same attribute-value pairs on all explanatory attributes in the data set . , or equivalently , is called the signature of the group and is denoted by . The concept of an E-group is fundamental in our discrimination definition. All groups of in are denoted by . In the case where is empty, the whole data set is an E-group.

Similar to an E-group, we also use group to mean all the tuples with in .

Discrimination score

We employ the well cited discrimination score defined in [3] and score is . Other terms for this score are risk difference [25] and selection lift [22]. In the case where is income and is gender, the score reflects the probability difference of high income earners caused by gender difference. Considering E-groups, the score for each E-group is:


The discrimination score of in data set is the E-group size weighted average [30, 17]:


where is each stratum and is also overloaded to represent the signature of in Formula (1). Obviously the following Lemma is true because of the average.

Lemma 1

. That is, for a given protected attribute , the score of the data set is less than or equal to the maximal score of E-groups.

A data set has multiple protected variables. The discrimination score of a data set is:

Definition 1 (Discrimination)

Given a data set and a user-defined discrimination score threshold ,

  • a group of a protected attribute is group-discriminated in the explanatory group if  ;

  • a group of a protected attribute is globally discriminated if  , and

  • an E-group is discriminatory if .

  • the data set is discriminatory if . is discrimination-safe if . is discrimination-free if .

Consider a classification model and a data set on schema . When is applied to , a new outcome is predicted for each tuple . We use to replace the value in column of , add hat to the attribute , as , to reflect such a change, and denote the updated data set by which is now on the schema . is called the predicted data set of by .

Definition 2 (Discrimination of a model)

Given a data set , a classification model , its predicted data set , and a user-defined discrimination score threshold , is discriminatory if is discriminatory with regard to the outcome (instead of ).

3 Properties of discrimination

In this section, after introducing the notation to calculate Formula (1), we present our results of discrimination properties related to the building of classifiers. Our results are on the basis of an E-group, but they are also true for the whole data set as a data set is a special case of an E-group.

To calculate the probabilities in Formula (1), we partition an E-group into divisions, called DP-divisions, based on the values of the outcome variable and a protected attribute . Each division is a subset of all tuples with the same and value in . The concept of a division is the same as a stratum or a group, but it is used here to make the terminology distinct. thus has four DP-divisions because and are all binary. The tuple count of each division is denoted by where the first subscript means the value and the second subscript represents the value. For example, means the tuple count in the division . The symbols denoting the counts are defined in Table 3, called the counts table.

(a) tuple counts of training data
=1 =0
(b) tuple counts of predictions
pred. =1 =0
=1 =1
=0 =0
Table 3: Tuple counts of DP-divisions

With the notation of the tuple counts of DP-divisions, Formula (1) can be represented in Formula (4). Obviously each fraction in the formula is bounded by 1 and as a result, is bounded to .


In the special cases where there is no tuple in the contrast divisions for the protected attributes, i.e., or , the discussion of discrimination in this case is not meaningful and no discrimination is possible. Then, is defined to be 0.

When a classification model is applied to , the model draws a decision boundary through the space defined by . This boundary then splits the counts in Table 3(a) into the counts in Table 3(b) with the following constraints: () where is the count of correct predictions and is the count of wrong predictions. We note that the decision boundary does not change the value of the values of the protected attributes, so correct predictions and wrong predictions are within the same P-group. The discrimination score of the model is calculated based on the predicted values as the following.


3.1 Where is discrimination from?

Discrimination of the (training) data is from the uneven distribution of the preferable outcome (=1) in a P-group in contrast to the non-P-group. As shown in Figure 1(a) where the protected attribute is gender and the outcome values are ‘+’ are ‘’, the fraction of ‘+’ among females is much less than the fact of ‘+’ among males. In terms of Formula (4), and so discrimination exists on the gender attribute.

Figure 1: Where is discrimination from?

In the case where a classification model is learnt from the training data, the discrimination of the model is the discrimination of predictions. Following Formula (5), we have the following results in Lemma 2.

Lemma 2

Given a data set and a classification model trained from the data,

  • if the model is perfect (does not make any error in predictions), the discrimination of the model is the same as the discrimination of the training data. If the model is perfect and the training data is discrimination-free, the model is discrimination-free.

  • If the model is not perfect, the discrimination of the model and the discrimination of the training data is independent.


Item (1) is correct because when the model is perfect , , and , so Formula (5) becomes Formula (4).

Item (2) is correct because if the model is not perfect, some . Consider a case where , , , and . In this case, and this is irrelevant to the values of , , , , and consequently irrelevant to .  

Figure 1(b-d) illustrate the lemma. In Part (b), one ‘+’ in the female group was wrongly predicted, leading the score of the predicted data to be , an increase in absolute value compared to that of Part (a).

In Part (c), the predicted outcomes are identical to the original outcomes, and the discrimination score of the predictions are the same as that of the original data in Part (a). In Part (d), the decision boundary X=s leads to a score of in the predictions.

This analysis shows that when a data set is discriminatory, the models learnt from it may or may not be discriminatory and this is irrelevant to the error rate levels. A small error rate can lead to large discrimination while a large error rate may lead to 0 discrimination.

Next we show that the classifiers trained from non-discriminatory data may be discriminatory. This seems against intuition, but it is correct. The reason for a classifier trained on non-discriminatory data not to be safe is because the input variables used in the classifier for prediction may be independent to the protected variables. When the values of the outcome variable are fair to the protected and the non-protected groups, the predicted outcomes may break this fairness, making the model unfair.

Figure 2: Discrimination of model predictions

Consider Figure 2(a) where the outcomes are fair with regard to the Gender values and 0.5 fraction of points are getting the ‘+’ outcome in both M and F groups, and the discrimination score is 0.

Assume that a learning process learns the decision boundary and this boundary maximizes the accuracy of the model. The predictions of the model is shown in Figure 2(a.1) and the predictions contain an error indicated by the circle. The predictions are now discriminatory as Females are getting less number of ‘+’ label.

Finally, redline attributes [11] in a model affect the discrimination of the model but this effect relates to the discrimination of the training data and the error distribution along decision boundary. Redline attributes are those correlated with a protected attribute. If this correlation is high and the distribution of the preferable outcome in the protected groups are even in the training data, redline attributes would not be good predictors and their participation in the model is less possible. If the training data itself is discriminatory, using redline attributes in the model has a high chance to make better predictions, generate discrimination in the predictions, and make the model discriminatory. Interestingly as shown in Lemma 2, the model discrimination is also affected by the error distribution along the decision boundary. In some cases, the decision boundary may reduce the impact of redline attributes on the model discrimination.

3.2 Discrimination in subsets of an E-group

In decision tree learning algorithms, a data set is split into subsets and the subsets are moved to the child nodes. We like to know how discrimination changes as a data set is split and as subsets are combined into one set.

Consider an E-group and its sub sets and where . Discrimination of , and follows the Simpson Paradox [26]. The lemma below shows this paradox. It indicates that even if the data set is not discriminatory, its subsets may still be discriminatory, and if every subset is non-discriminatory, the combined data set from the subsets may still be discriminatory.

Lemma 3

Assume that data sets , and belong to the same E-group such that . Then, the discrimination in with regard to a protected attribute is not guaranteed by the discrimination of and or vice versa. That is, for a user specified discrimination score threshold
(1) and , and
(2) and .

We use the following proof to show a method to construct clusters of cases for the lemma.

  1. Proof of (1). The following example shows that has a discrimination score of 0, but after partition, its subsets and both have maximal discrimination scores. In the example, is a non-zero positive integer and the tuple counts meet the requirements of .

  2. Proof of (2). We show that there exist cases where the discrimination scores of subsets and are less than the threshold , but after merging , the discrimination score of is more than the threshold . Let be non-zero positive integers, and be a number so that . We choose the tuple counts to be:

    Then, . If , then .

    Item (2) is approved.

The importance of Lemma 3 is that the consideration of discrimination in subsets of an E-group does not lead to correct discrimination guarantee. Discrimination has to be analyzed against the whole of an E-group.

3.3 Discrimination of a decision tree

Following Lemma 3, we know that discrimination cannot be analyzed at subset level of an E-group. On the other hand, the discrimination of a data set can be averaged over the scores from the scores of multiple E-groups of the data set (Formula (2)). By Lemma 1, if every E-group is non-discriminatory, the whole data set is non-discriminatory.

Now we use this to analyze the discrimination score of the predictions of a decision tree.

A leaf node of a decision tree makes predictions of a fixed label decided by majority voting during the training phase. Any tuple direct to this node by the decision path of the tree will get this label. So all the predictions out of the leaf node has only one outcome: either or . As a result, the discrimination score of the predictions out of this leaf node is 0 (see paragraph following Formula (4)).

Consider Figure 2(b) with two leaf nodes of a parent node where the decision/splitting attribute is explanatory. The discrimination score of the predictions of every leaf node is 0 as explained above. As the splitting is by an explanatory attribute, the discrimination score of the predictions out of the parent node is the average of the scores on the leaf nodes and is 0. From this, we draw the conclusion in Lemma 4.

Lemma 4

In a decision tree, if the splitting attribute of every internal node is explanatory, the predictions out of the whole decision tree are discrimination-free.

We note that in general, the result in the above lemma may not be right. However, because all decision attributes are explanatory, the uneven distribution of the favorite outcome among different leaf nodes are ‘explained’ by the explanatory variables in the internal nodes. The explanation leads to 0 score in the whole tree. This result extends to the general case.

Lemma 5

If a classification model uses explanatory variables as input variables, is non-discriminatory.

Lemma 5 is correct because . All tuples of an E-group will have the same prediction and consequently the E-group is discrimination-free. The discrimination score averaged over all E-groups is 0. So the lemma is correct.

3.4 Discrimination and correlation between and

Correlation between a protected attribute and the outcome attribute , denoted by , is critical to discrimination. If is high, i.e., =1 implies =1, any classification model that has high accuracy will produce high level of discrimination. In this section, we analyze how discrimination is related to . We show that although there is a link between discrimination and correlation of and , the relationship is not monotone.

The correlation between the outcome and a protected attribute can be measured by odds-ratio . means that there is no correlation between and . values further away from 1 indicate strong correlation. For example, if all females are high income earners () and all males are low income earners (), and are extremely correlated and the value is infinite.

If , we define to mean not correlated. If , the =1 group does not have any people, correlation does not make sense. If If , no tuple has =1 outcome and the case does not make sense either. The case where is symmetric.

Odds-ratio and discrimination are related. As an example, assume that if all counts, except for , are fixed in . In the discrimination, assume that . Then as increases, both and increases. This trend is NOT true if other counts are allowed to change.

Interestingly, the following lemma shows that the interaction of odds-ratio and discrimination is not simple.

Lemma 6

Given an E-group and its DV-division tuple counts,

  1. is discrimination-free () if and only if and are independent.

  2. Less correlation does not mean less discrimination.

  1. We transform Formula (4) into . When , then and vice versa. means .

    The extreme cases for and the extreme cases for are corresponding. So the item is proved.

  2. We assume that we have two E-groups and . Their tuple counts are denoted by and respectively. Their odds-ratios and discrimination scores are respectively. We want to prove that when and are less correlated in than in (i.e., ), the discrimination in can be larger than that in (i.e., ).

    We consider only the case of . Let , and . When want to show that when , there are cases satisfying .

    We present a tuple counts table below. Assume non-zero positive integers and the following counts tables.

    . if plus reflecting our assumption . On the other hand, . if . Combining the conditions for and , we have . For example, if we let and , and . We proved that a data set having lower correlation (odds-ratio closer to 1) may have higher discrimination compared to another data set.

    The item is approved.

This result is very important to understanding the complexity between redline attributes and the discrimination of predictions. Let be a redline attribute. It is an intuition that higher correlation between and and higher correlation between and would lead to higher model discrimination. In a previous section we showed that the discrimination caused by redline attributes is modified by the error distribution of the model. Now we show that higher does not mean higher discrimination. Consequently the relationship between model discrimination and redline attributes is not intuitive.

4 Exploration of discrimination levels in real data sets

In this section, we present the results of our exploration of discrimination in four real world data sets. We present the following results. (1) Data from real application is often discriminatory and how the number of explanatory attributes affect the score. (2) Classification models may change the level of discrimination in the predictions compared with the one in the train data. (3) Classification models built from non-discriminatory data may still be discriminatory.

4.1 data sets

We use four real world data sets as shown in the following list. Name, size, source, and attributes of the data sets are shown. All data sets are processed to have binary (0,1) values. The values of ordinal attributes are binary-zed using median. Categorical attributes are binary-zed by taking majority and the rest. The labels (P), (E), and (D) against some attributes indicate the types protected, explanatory, and outcome respectively. The attributes without a label are O-attributes.

  • US Census 1994. numb(rows)=48842; minority class rate=.25 sets/adult
    Attributes: age45(P), natCountryUS(P), raceBlack(P), sexM(P), workPrivate(E), occuProf(E), workhour30(E), eduUni(E), relaNoFamily, married, income50K(D)

  • Canada Census 2011 [20] 111The author wishes to acknowledge the statistical office that provided the underlying data making this research possible: Statistics Canada. numb(rows)=691788; minority class rate=.34
    Attributes: weight100(P), age50(P), sexM(P), edUni(E), occProf(E){}, occSkilled(E), occOther(E), hoursfull(E), govJob, classSalary, income45K(D)

  • German Credit. numb(rows)=1000; minority class rate=.3 sets/statlog+(german+credit+data)
    Attributes: age35(P), single(P), foreign(P), chkAccBal(E), duration20m(E), creditHistGood(E), purposeCar(E), credit2320(E), savings500(E), emp4y(E), installPct3(E), sexM(E), guarantor(E), resid3y(E), propertyYes(E), instPlanNon(E), houseOwn(E), creditAcc(E), jobSkilled(E), people2(E), hasTel(E), approved(D)

  • Recidivate-violent [16]; numb(rows)=4744; minority class rate=.14
    Attributes: sexM(P), age30(P), raceAfrica(P), raceWhite(P), raceOther(P), juvFelonyCnt1(E), juvMisdCnt1(E), juvOthcnt1(E), priorsCnt3(E), cjail1Month(E), cChargeisdemM(E), isRecid(E), score8(D)

The Recidivism data set follows [16]. The score8 column stores predictions from a system called COMPAS. The isRecid column stores whether the person re-committed a crime. We want to see if score8 values can be accurately re-predicted.

We now present the results of experimental evaluation of our method.

4.2 Discrimination in original data sets

In the experiments, each (original) data set is stratified into E-groups by using the explanatory variables specified in the data set descriptions above. A discrimination score is calculated for each protected attribute in each E-group following Formula (4). The discrimination scores of different E-groups for the same protected attribute are averaged with the weights of the group sizes. The global discrimination score of the data set is the maximum of the averaged scores over different protected attributes.

The results are shown in Table 4 where is the global discrimination score. The table also lists the worst (maximal) E-group discrimination score () and the percentage () of tuples in this worst group out of all the tuples in the data set. In the data set, the discrimination score of some groups is over the threshold and these groups are called the over-limit groups. The discrimination scores of these groups are averaged to get the average score and the percentage of tuples in these groups is . The top three protected attributes ranked by absolute values of the their score are listed in the right-most column.

Table 4: Discrimination of original data sets

For the Adult data, all E-groups are discriminatory (100%) with the score 2 times more than the threshold, and the over-limit groups are 4 times more than the threshold. The worst discrimination happened to the attributes of Gender and Age.

In the Canada data, the discrimination scores are larger than those of the Adult data set. The worst discrimination happened to Age and Sex male. People who takes Salary (instead of Wage) were slightly discriminated.

German Credit data’s discrimination level is lower although some E-groups (with 52% of tuples) have a discrimination score of 0.28, and the worst E-group (with 5 tuples or 0.5%) has a score of 1. After some investigation, we found that this extreme score is caused by a small group size. When the size of an E-group is small, the discrimination score can be dramatic. The tuple counts of this worst group is (3,0,0,2) and the score calculation is .

The Recidivate data has an overall discrimination score of 0.098, lowest among all data sets. The worst discrimination (0.778) happened to the protected attribute RaceAfrica in an E-group with 11 tuples (0.2% of the data set). The tuple counts of the group is (7,2,0,2) and the score calculation is . Among 9 Africans, 7 were predicted to recidivate, but among the 2 non-Africans, 0 were predicted to recidivate.

We note that the above observations are conditional. They are dependent on the way in which data is discretized and on what and how many variables are specified as explanatory ones.

Figure 3 describes the relationship between discrimination and the number of explanatory attributes. The experiments are done by using a specified number of explanatory attributes among all the described explanatory variables in turn to calculate an average discrimination score. For example, with the Adult data set, there are 4 possible explanatory attributes in the description. In the case of using 1 explanatory attribute, we run 4 experiments, each with a different explanatory attribute and the scores from these 4 experiments are averaged to get the final score for the one explanatory variable case. From Figure 3, we observe that as the number of explanatory attributes increases, the discrimination score becomes lower. This trend is reasonable because when more explanatory attributes are used, more discrimination can be justified and consequently the discrimination level reduces.

Figure 3: Number of explanatory attributes on discrimination

We note that as more explanatory attributes are used, the data becomes more fragmented and the sizes of strata of explanatory attributes reduce to small numbers or 0, which can make discrimination scores change dramatically to large values.

4.3 Discrimination of classifiers

In this exploration, we use commercially available modeling algorithms to build classifiers on the training data sets (the original data sets used above). The models are then used to predict a new outcome for each tuple in the training data sets. The predicted outcomes replace the original observed outcomes to form new data sets called predicted data sets. We calculate the discrimination scores on the predicted data sets.

For the same training data set, we get different predicted data sets when the classifiers are different. We choose five well-used classification algorithms, namely decision tree (DT), Bayes network (BN), neural network (NN), logistic regression (LR), and support vector machine (SVM) from SAS Enterprise miner and use these algorithms with the default parameters to generate predicted data sets. Discrimination scores are calculated for the predicted data sets and the results are shown in Table 5. BCR stands for balance classification rate which is the average of the true positive rate and the true negative rate. Err stands for misclassification rate. These two measures indicate the quality of the classification models. Better models have larger BCR and smaller Err.

Table 5: Effect of models on discrimination of Adult data set

First we look at the results for the Adult data set in the top-left corner. Compared to the discrimination scores of the original data (line Orig), the predicted data set has (1) a slightly lower global score (glbds), (2) much higher over-limit score (ogds) with a lower percentage (og%), and (3) a much higher worst group score (wgds) with smaller percentage (wg%). Points (2) and (3) indicate that the classification errors made by the models are quite high (19% ), some of these errors made some E-groups less discriminatory, others made other groups worse.

The Canada, German, and Recidivate data sets have similar properties. We note that the last line for SVM of these data sets have a global score of 0. The reason is not that the classifier is the best, but that the classifier made so many errors such that the error rate equals to the minor class rate in the data set. For example, in German credit data, the percentage of non-approved cases is 30% while the method’s misclassification rate is also 30%. A close check found that the classifier predicted all negative class as positive class. In this case, all non-approved cases are predicted as approved cases. When all predictions have only one class, there is no discrimination.

From this we see that when examining the discrimination of a classification model, we must consider the balanced accuracy and error rate. Otherwise, the conclusion may not be right.

4.4 Non-discriminatory data does not mean fair classifiers

In this section, we use experiments to show the independence between discrimination and classifiers trained from the non-discriminatory data.

In the experiments, our non-discriminatory data was generated from the CV method [3]. More specifically, we use the original data as training data to run the CV implementation in [7] by specifying Sex as the only protected attribute, rejecting other protected attributes described in Section 4.1. The CV method produces a predicted data set which is close to non-discriminatory. We call the predicted data set the data set. Further, we train a predictive classifier from the CV data set. The classifier again produces a predicted data set . The discrimination in is what we want to analyze. That is, we want to see if the classifier from the non-discriminatory data set is discriminatory.

The results of this experiment are in Table 6. The grey line labelled by ‘CV’ is the data set. The data set for each classifier is labelled by the classifier name. For example, is the data set from the Bayes network classifier. Because the CV method does not support explanatory attributes, the results do not reflect group level discrimination. That is, the whole data set is seen as a large E-group for the only protected attribute Sex.

Table 6: Discrimination of classifiers trained on non-discriminatory data

The results show that the discrimination of training data and the discrimination of the classifiers from the data are independent. Among the four data sets, Adult CV has the smallest discrimination score, but the classifiers from this data set have highest discrimination scores. Although different algorithms among BN, DT, LR, NN, and SVM have different balanced accuracy and different error rates, all of them are discriminatory and the scores are 13-33 times larger than the score on the CV data set.

The results also show that high accuracy does not lead to lower discrimination. For example, DT for Adult CV data has highest BCR, but its discrimination score is not the least.

The numbers for the German credit data set indicate the same independence but in an opposite way. With this data set, the discrimination of the CV is over the threshold, but the discrimination of the classifiers is less than the threshold (NN is only slightly over).

The numbers for the Canada and the Recidivate data sets show that the classifiers did not change the discrimination levels.

From all these, we observe clearly that discrimination of classifiers are independent to the discrimination of training data. It is also independent of accuracy and error rate. These two conclusions are reasonable because with the same error rate, the error distribution in the protected attributes can be very different, leading different discrimination scores.

5 Related Work

Algorithmic discrimination has attracted a lot of research effort. The work focused in two areas: discrimination detection and discrimination removal from data and from models. The work on removal are in three categories: pre-processing, manipulation of model learning algorithms and post processing. We now review the work done in these directions.

Discrimination detection in data The core problem of detection is to define and choose metrics to measure discrimination. Zliobaite [32] has a good summary of previous metrics [21, 22, 33, 8, 24, 25, 5, 6]. Some recently proposed causality-based metrics are [6, 14, 29, 30, 17, 31] and the metrics for individual discrimination are [18, 4, 19, 30].

Removal of discrimination from training data (pre-processing) Feldman et al. [5] proposed to transform data in a data set so that the red-line attributes, those that correlated to the target, become independent to the target variable. Friedler et al. [7] summarized some of the work in this direction.

Modification of model training algorithms Kamiran et al. [10] proposed to combine information gain and discrimination gain in decision tree learning and to use a post-relabelling process to remove discrimination from predictions. Calders and Verwer [3] adjusts the probability in naive Bayes methods so that the predictions are discrimination free. Kamishima et al. [12] proposed a regularization method for logistic regression method. Zafar et al. [28] represented discrimination constraints via a convex relaxation and optimized the accuracy and discrimination in the SVM learning algorithm. Woodworth et al. [27] proposes a two step method to build a non-discriminatory classifier. The data set is divided into two subsets S1 and S2. In the first step, a classifier is built to minimize error rate under the constraints of discrimination in data set S1. In the second step, a post-processing model is built on data set S2. Raff et al. [23] proposed a discrimination-aware measure for decision tree induction for continuous data. Landeiro and Culotta [15] proposes to use a weight under-training method by strengthening confounder features to build a model. Kearns et al. [13] used an optimization method in model learning. Kamishima et al. [12] has done a deep analysis of CV2NB, ROC and proposed a method called universal ROC.

Removal of discrimination from predictions of a model (post-processing) The work of Kamiran et al. [10] relabels the predictions of leaf nodes of a decision tree to achieve discrimination goal. The post-processing model of Step 2 in [27] minimizes the discrimination using the target, the predicted target and the protected variables on the second half of the training data. Hardt et al. [9] uses equalized odds to build a model for post-prediction manipulation. Kamishima et al. [12] proposed a method called universal ROC.

6 Conclusion

In this paper, we analyzed some properties of algorithmic discrimination. The properties reveal relationships between discrimination and correlation, data partition, and classification models. These properties have important implications in building predictive models, assessing discrimination in data sets and models, and evaluating discrimination-aware models. The paper also explored discrimination of real world data sets. The results of exploration show that discrimination does exist in real world data sets, and the discrimination of models is independent to the discrimination of training data. This implies that the fairness of models cannot be achieved by manipulating training data. Our future work is to develop a general method to achieve fairness independent of learning algorithms.


  • [1]
  • Boatwright et al. [2017] Kimberly Boatwright, CRCM CAMS, and Kinsey Sullivan. 2017. 3 Real-World Examples of Fair Lending Discrimination How They Can Damage Your Reputation. (as of 2018Jul) (2017).
  • Calders and Verwer [2010] Toon Calders and Sicco Verwer. 2010. Three naive Bayes approaches for discrimination-free classification. Data Min. Knowl. Discov. 21, 2 (2010), 277–292.
  • Dwork et al. [2012] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. Innovations in Theoretical Computer Science Conf. (2012), 214–226.
  • Feldman et al. [2015] Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and Removing Disparate Impact. ACM SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining (2015), 259–268.
  • Fish et al. [2016] Benjamin Fish, Jeremy Kun, and Adam D. Lelkes. 2016. A Confidence-Based Approach for Balancing Fairness and Accuracy. (2016).
  • Friedler et al. [2018] Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan P. Hamilton, and Derek Roth. 2018. A comparative study of fairness-enhancing interventions in machine learning. (2018).
  • Fukuchi et al. [2013] Kazuto Fukuchi, Jun Sakuma, and Toshihiro Kamishima. 2013. Prediction with Model-Based Neutrality. Euro. Conf. on Machine Learning and Knowledge Discovery in Databases - Volume 8189 (2013), 499–514.
  • Hardt et al. [2016] Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of Opportunity in Supervised Learning. Advances in neural info. proc. systems (2016).
  • Kamiran et al. [2010] Faisal Kamiran, Toon Calders, and Mykola Pechenizkiy. 2010. Discrimination Aware Decision Tree Learning. ITEE Intl. Conf. on Data Mining (2010), 869–874.
  • Kamiran et al. [2012] Faisal Kamiran, Indre Zliobaite, and Toon Calders. 2012. Quantifying explainable discrimination and removing illegal discrimination in automated decision making. Knowl. and Info. Sys. 35, 3 (2012), 613–644.
  • Kamishima et al. [2018] Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. 2018. Model-based and actual independence for fairness-aware classification. Data Min Knowl Disc 32 (2018), 258–286.
  • Kearns et al. [2018] Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. Proc. of Machine Learning Research V80 (2018), 2564–2572.
  • Kleinberg et al. [2016] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016. Inherent Trade-Offs in the Fair Determination of Risk Scores. (2016).
  • Landeiro and Culotta [2016] Virgile Landeiro and Aron Culotta. 2016. Robust Text Classification in the Presence of Confounding Bias. AAAI (2016).
  • Larson et al. [2016] Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How We Analyzed the COMPAS Recidivism Algorithm. (2016).
  • Li et al. [2017] Jiuyong Li, Jixue Liu, Lin Liu, Thuc Le, Saisai Ma, and Yizhao Han. 2017. Discrimination detection by causal effect estimation. BigData (2017), 1087–1094.
  • Luong et al. [2011] Binh Thanh Luong, Salvatore Ruggieri, and Franco Turini. 2011. k-NN as an Implementation of Situation Testing for Discrimination Discovery and Prevention. ACM SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining (2011).
  • Mancuhan and Clifton [2014] Koray Mancuhan and Chris Clifton. 2014. Combating discrimination using Bayesian networks. Artificial Intelligence and Law 22, 2 (2014), 211–238.
  • Minnesota et al. [2018] Minnesota, Population, and Center. 2018. Integrated Public Use Microdata Series, International: Version 7.0 [dataset]. Minneapolis, MN: IPUMS, 2018. IPUMS (2018).
  • Pedreschi et al. [2008] Dino Pedreschi, Salvatore Ruggieri, and Franco Turini. 2008. Discrimination-aware Data Mining. ACM SIGKDD Intl. Conf. on Knowl. Disc. and Data Mining (KDD) (2008).
  • Pedreschi et al. [2009] Dino Pedreschi, Salvatore Ruggieri, and Franco Turini. 2009. Measuring Discrimination in Socially-Sensitive Decision Records. SIAM Intl. Conf. on Data Mining (SDM) (2009).
  • Raff et al. [2018] Edward Raff, Jared Sylvester, and Steven Mills. 2018. Fair Forests: Regularized Tree Induction to Minimize Model Bias. AAAI (2018).
  • Ristanoski et al. [2013] Goce Ristanoski, Wei Liu, and James Bailey. 2013. Discrimination aware classification for imbalanced datasets. Conf. on Information and Knowledge Management (2013), 1529–1532.
  • Ruggieri et al. [2014] Salvatore Ruggieri, Sara Hajian, Faisal Kamiran, and Xiangliang Zhang. 2014. Anti-discrimination Analysis Using Privacy Attack Strategies. Euro. Conf. Machine Learning and Knowledge Discovery in Databases ECML PKDD, Part II (2014), 694–710.
  • Simpson [1951] Edward H. Simpson. 1951. The interpretation of Interaction in Contingency Tables. Journal of the Royal Statistical Society 13 (1951), 238–241.
  • Woodworth et al. [2017] Blake Woodworth, Suriya Gunasekar, Mesrob I. Ohannessian, and Nathan Sreb. 2017. Learning non-discriminatory predictors. arXiv:1702.06081v3 (2017).
  • Zafar et al. [2015] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P. Gummadi. 2015. Fairness Constraints: A Mechanism for Fair Classification. arXiv preprint arXiv:1507.05259 (2015).
  • Zhang et al. [2016a] Lu Zhang, Yongkai Wu, and Xintao Wu. 2016a. On Discrimination Discovery Using Causal Networks. Intl. Conf. on Social Computing, Behavioral-Cultural Modeling, Prediction and Behavior Representation in Modeling and Simulation (2016).
  • Zhang et al. [2016b] Lu Zhang, Yongkai Wu, and Xintao Wu. 2016b. Situation Testing-Based Discrimination Discovery: A Causal Inference Approach. Intl. Joint Conf. on Artificial Intelligence (2016), 2718.
  • Zhang et al. [2017] Lu Zhang, Yongkai Wu, and Xintao Wu. 2017. Achieving Non-Discrimination in Data Release. KDD (2017).
  • Zliobaite [2017] Indre Zliobaite. 2017. Measuring discrimination in algorithmic decision making. Data Mining and Knowledge Discovery 31, 4 (2017), 1060–1089.
  • Zliobaite et al. [2011] Indre Zliobaite, Faisal Kamiran, and Toon Calders. 2011. Handling Conditional Discrimination. ITEE Intl. Conf. on Data Mining (2011), 992–1001.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description