An exploration of algorithmic discrimination in data and classification
Abstract
Algorithmic discrimination is an important aspect when data is used for predictive purposes. This paper analyzes the relationships between discrimination and classification, data set partitioning, and decision models, as well as correlation. The paper uses real world data sets to demonstrate the existence of discrimination and the independence between the discrimination of data sets and the discrimination of classification models.
Keywords: algorithmic discrimination; fairness; association; classification
1 introduction
Discrimination means “treating a person or particular group of people differently, especially in a worse way from the way in which you treat other people, because of their skin colour, sex, sexuality, etc” (dictionary.cambridge.org). It can happen in law enforcement applications where people may be unfairly treated and sentenced because of their races and religions, in bank loan applications where people may not get a loan because they live in a suburb with a lower economic status.
The law does not allow discrimination to happen. That is, decisions should not be made based on people’s sex, skin colour, religion etc., called protected attributes of individuals. However discrimination is still a concern in the real world. In car insurance, the insurance company required people in a specific suburb to pay higher premium than those in other suburbs with the reason that the suburb has a higher claim rate. If most dwellers of the suburb are of a certain race, the higher premium forms actually discrimination to the people of this race. Boatwright (2017) reported three real world lending discrimination cases [2].
Discrimination does exist in the data collected from real world applications. We will demonstrate this fact in the experiment section of this paper.
The level of discrimination is contextbased. Contexts are defined by explanatory attributes. Assume that an investigation is on whether female employees are less paid. Then, the explanatory attributes like profession and position held by the employees matter. A fair comparison between the payments of female employees and male employees must assume that the employees hold same profession and same position. Without a context, the comparison may between the income of a group of male CEOs with the income of a group of male kitchen hands. We use an example to show how discrimination levels are affected by different contexts.
Example 1 Table 1 shows the tuple frequencies of a data set where standing for (1=high, 0=low) is the outcome/class/target attribute, G standing for Gender and is a protected attribute, and a number in the middle of the tables is the number of tuples in having the same value. The discrimination score for females in in Part (a) is 0 where the score is defined as the percentage of high income females taking away the percentage of high income males [3].
For the same application, if a context is defined by an explanatory variable (standing for employment sector, S=private or public), the data set in (a) is divided into two subsets and , and the tuple frequencies for each subset are shown in Parts b.1 and b.2 respectively. The discrimination scores for the corresponding employment sectors are 0.22 and 0.24 respectively. The scores in (a) and (b) conclude that contexts change discrimination levels.
(a)  
D  Female  Male 
1  10  15 
0  40  60 
(b.1) for Sector=1  
D  Female  Male 
1  9  3 
0  20  30 
(b.2) for Sector=0  
D  Female  Male 
1  1  12 
0  20  30 
Classification models built from a data set may become discriminatory too. The discrimination of a model is the discrimination of the predictions that the model produces. Consider data in Table 2(a) where each record is for an individual. The column (1=high,0=low) is the performance measurement of individuals. Without an assumed context, this data set has no discrimination. If a predictive model is build on , the model would be . We note that gender is protected and cannot be used in the model. When the model is applied to predict the outcome for the individuals, the predicted data set is in Part (b). It is easy to see that the discrimination score for is 0.5. This confirms that classification models can be discriminatory even if the training data is fair. This type of models are said to be algorithmic discriminatory.
(a) Data  
D  G  M 
1  F  1 
0  F  0 
1  M  0 
0  M  0 
(b) Prediction  
G  M  
1  F  1 
0  F  0 
0  M  0 
0  M  0 
Model from training data is .
but
Algorithmic discrimination becomes more a concern in automated decision making systems as machine learning and data analytics are used more and more in real applications. In loan approval cases, it is a normal practice that an automated system automatically scores and makes decisions on customer’s applications. Fairness of automated decision systems is important to service receivers and to a harmonic society.
Research work on algorithmic discrimination are from two areas: detection and removal. This paper, instead of continuing on these two directions, goes to the fundamental side of these two areas. It aims to explore deeper understanding of discrimination related issues. It analyzes some properties of algorithmic discrimination and explores the level of discrimination in real world data sets and how classification models affect the discrimination levels. More specifically, the paper will show results on the following points.

sources of discrimination in classification models and the relationship between fair data and the discrimination of classifiers,

discrimination when data sets are merged and split,

the relationship between discrimination and explanatory variables in a decision tree,

the interaction between discrimination and correlation,

experimental exploration of the existence of discrimination in real world data sets and classification models.
The organization of the paper is the following. Section 2 presents a discrimination score and group discrimination. Section 3 presents our analysis of properties of discrimination. In Section 4, we show the results from experimental exploration of real world data sets. Section 5 presents related work and the final section concludes the paper.
2 Definitions and Problem
In this section, we define basic notation, and present a measurement for algorithmic discrimination.
Let be a data set on a schema of binary attributes/variables. The attributes in are of four types: an outcome/target/class attribute , some protected attributes , some explanatory attributes , and other attributes : . For the outcome attribute , means a favorite outcome like or that an individual prefers to receive. For a protected attribute , (e.g. SexFemale or RaceBlack) means a group of individuals who are protected by the law not to to be discriminated. The explanatory attributes explain why some people receive favorite outcomes more or less frequently than others or identify such people. For example, profession and education are often taken as explanatory attributes. Surgeon is a profession and people who are surgeons are mostly high income earners, while kitchen hand is another profession and people who are kitchen hands are low income earners. Explanatory attributes are often used in selection criteria of employment. The ‘other’ attributes are that attributes not in . Some of these attributes may be correlated with the outcome variable and are used in classifiers for predictions, and some, called redline attributes, may be correlated to some protected attributes which makes discrimination analysis challenging.
An Egroup (or stratum of ) is a subset of all the tuples having the same attributevalue pairs on all explanatory attributes in the data set . , or equivalently , is called the signature of the group and is denoted by . The concept of an Egroup is fundamental in our discrimination definition. All groups of in are denoted by . In the case where is empty, the whole data set is an Egroup.
Similar to an Egroup, we also use group to mean all the tuples with in .
Discrimination score
We employ the well cited discrimination score defined in [3] and score is . Other terms for this score are risk difference [25] and selection lift [22]. In the case where is income and is gender, the score reflects the probability difference of high income earners caused by gender difference. Considering Egroups, the score for each Egroup is:
(1) 
The discrimination score of in data set is the Egroup size weighted average [30, 17]:
(2) 
where is each stratum and is also overloaded to represent the signature of in Formula (1). Obviously the following Lemma is true because of the average.
Lemma 1
. That is, for a given protected attribute , the score of the data set is less than or equal to the maximal score of Egroups.
A data set has multiple protected variables. The discrimination score of a data set is:
(3) 
Definition 1 (Discrimination)
Given a data set and a userdefined discrimination score threshold ,

a group of a protected attribute is groupdiscriminated in the explanatory group if ;

a group of a protected attribute is globally discriminated if , and

an Egroup is discriminatory if .

the data set is discriminatory if . is discriminationsafe if . is discriminationfree if .
Consider a classification model and a data set on schema . When is applied to , a new outcome is predicted for each tuple . We use to replace the value in column of , add hat to the attribute , as , to reflect such a change, and denote the updated data set by which is now on the schema . is called the predicted data set of by .
Definition 2 (Discrimination of a model)
Given a data set , a classification model , its predicted data set , and a userdefined discrimination score threshold , is discriminatory if is discriminatory with regard to the outcome (instead of ).
3 Properties of discrimination
In this section, after introducing the notation to calculate Formula (1), we present our results of discrimination properties related to the building of classifiers. Our results are on the basis of an Egroup, but they are also true for the whole data set as a data set is a special case of an Egroup.
To calculate the probabilities in Formula (1), we partition an Egroup into divisions, called DPdivisions, based on the values of the outcome variable and a protected attribute . Each division is a subset of all tuples with the same and value in . The concept of a division is the same as a stratum or a group, but it is used here to make the terminology distinct. thus has four DPdivisions because and are all binary. The tuple count of each division is denoted by where the first subscript means the value and the second subscript represents the value. For example, means the tuple count in the division . The symbols denoting the counts are defined in Table 3, called the counts table.
(a) tuple counts of training data  

=1  =0  
=1  
=0 
(b) tuple counts of predictions  

pred.  =1  =0  
=1  =1  
=0  
=0  =0  
=1 
With the notation of the tuple counts of DPdivisions, Formula (1) can be represented in Formula (4). Obviously each fraction in the formula is bounded by 1 and as a result, is bounded to .
(4) 
In the special cases where there is no tuple in the contrast divisions for the protected attributes, i.e., or , the discussion of discrimination in this case is not meaningful and no discrimination is possible. Then, is defined to be 0.
When a classification model is applied to , the model draws a decision boundary through the space defined by . This boundary then splits the counts in Table 3(a) into the counts in Table 3(b) with the following constraints: () where is the count of correct predictions and is the count of wrong predictions. We note that the decision boundary does not change the value of the values of the protected attributes, so correct predictions and wrong predictions are within the same Pgroup. The discrimination score of the model is calculated based on the predicted values as the following.
(5) 
3.1 Where is discrimination from?
Discrimination of the (training) data is from the uneven distribution of the preferable outcome (=1) in a Pgroup in contrast to the nonPgroup. As shown in Figure 1(a) where the protected attribute is gender and the outcome values are ‘+’ are ‘’, the fraction of ‘+’ among females is much less than the fact of ‘+’ among males. In terms of Formula (4), and so discrimination exists on the gender attribute.
In the case where a classification model is learnt from the training data, the discrimination of the model is the discrimination of predictions. Following Formula (5), we have the following results in Lemma 2.
Lemma 2
Given a data set and a classification model trained from the data,

if the model is perfect (does not make any error in predictions), the discrimination of the model is the same as the discrimination of the training data. If the model is perfect and the training data is discriminationfree, the model is discriminationfree.

If the model is not perfect, the discrimination of the model and the discrimination of the training data is independent.
Proof
Item (2) is correct because if the model is not perfect, some . Consider a case where , , , and . In this case, and this is irrelevant to the values of , , , , and consequently irrelevant to .
Figure 1(bd) illustrate the lemma. In Part (b), one ‘+’ in the female group was wrongly predicted, leading the score of the predicted data to be , an increase in absolute value compared to that of Part (a).
In Part (c), the predicted outcomes are identical to the original outcomes, and the discrimination score of the predictions are the same as that of the original data in Part (a). In Part (d), the decision boundary X=s leads to a score of in the predictions.
This analysis shows that when a data set is discriminatory, the models learnt from it may or may not be discriminatory and this is irrelevant to the error rate levels. A small error rate can lead to large discrimination while a large error rate may lead to 0 discrimination.
Next we show that the classifiers trained from nondiscriminatory data may be discriminatory. This seems against intuition, but it is correct. The reason for a classifier trained on nondiscriminatory data not to be safe is because the input variables used in the classifier for prediction may be independent to the protected variables. When the values of the outcome variable are fair to the protected and the nonprotected groups, the predicted outcomes may break this fairness, making the model unfair.
Consider Figure 2(a) where the outcomes are fair with regard to the Gender values and 0.5 fraction of points are getting the ‘+’ outcome in both M and F groups, and the discrimination score is 0.
Assume that a learning process learns the decision boundary and this boundary maximizes the accuracy of the model. The predictions of the model is shown in Figure 2(a.1) and the predictions contain an error indicated by the circle. The predictions are now discriminatory as Females are getting less number of ‘+’ label.
Finally, redline attributes [11] in a model affect the discrimination of the model but this effect relates to the discrimination of the training data and the error distribution along decision boundary. Redline attributes are those correlated with a protected attribute. If this correlation is high and the distribution of the preferable outcome in the protected groups are even in the training data, redline attributes would not be good predictors and their participation in the model is less possible. If the training data itself is discriminatory, using redline attributes in the model has a high chance to make better predictions, generate discrimination in the predictions, and make the model discriminatory. Interestingly as shown in Lemma 2, the model discrimination is also affected by the error distribution along the decision boundary. In some cases, the decision boundary may reduce the impact of redline attributes on the model discrimination.
3.2 Discrimination in subsets of an Egroup
In decision tree learning algorithms, a data set is split into subsets and the subsets are moved to the child nodes. We like to know how discrimination changes as a data set is split and as subsets are combined into one set.
Consider an Egroup and its sub sets and where . Discrimination of , and follows the Simpson Paradox [26]. The lemma below shows this paradox. It indicates that even if the data set is not discriminatory, its subsets may still be discriminatory, and if every subset is nondiscriminatory, the combined data set from the subsets may still be discriminatory.
Lemma 3
Assume that data sets , and belong to the same Egroup such that . Then, the discrimination in with regard to a protected attribute is not guaranteed by the discrimination of and or vice versa. That is, for a user specified discrimination score threshold
(1) and , and
(2) and .
We use the following proof to show a method to construct clusters of cases for the lemma.
Proof

Proof of (1). The following example shows that has a discrimination score of 0, but after partition, its subsets and both have maximal discrimination scores. In the example, is a nonzero positive integer and the tuple counts meet the requirements of .

Proof of (2). We show that there exist cases where the discrimination scores of subsets and are less than the threshold , but after merging , the discrimination score of is more than the threshold . Let be nonzero positive integers, and be a number so that . We choose the tuple counts to be:
Then, . If , then .
Item (2) is approved.
The importance of Lemma 3 is that the consideration of discrimination in subsets of an Egroup does not lead to correct discrimination guarantee. Discrimination has to be analyzed against the whole of an Egroup.
3.3 Discrimination of a decision tree
Following Lemma 3, we know that discrimination cannot be analyzed at subset level of an Egroup. On the other hand, the discrimination of a data set can be averaged over the scores from the scores of multiple Egroups of the data set (Formula (2)). By Lemma 1, if every Egroup is nondiscriminatory, the whole data set is nondiscriminatory.
Now we use this to analyze the discrimination score of the predictions of a decision tree.
A leaf node of a decision tree makes predictions of a fixed label decided by majority voting during the training phase. Any tuple direct to this node by the decision path of the tree will get this label. So all the predictions out of the leaf node has only one outcome: either or . As a result, the discrimination score of the predictions out of this leaf node is 0 (see paragraph following Formula (4)).
Consider Figure 2(b) with two leaf nodes of a parent node where the decision/splitting attribute is explanatory. The discrimination score of the predictions of every leaf node is 0 as explained above. As the splitting is by an explanatory attribute, the discrimination score of the predictions out of the parent node is the average of the scores on the leaf nodes and is 0. From this, we draw the conclusion in Lemma 4.
Lemma 4
In a decision tree, if the splitting attribute of every internal node is explanatory, the predictions out of the whole decision tree are discriminationfree.
We note that in general, the result in the above lemma may not be right. However, because all decision attributes are explanatory, the uneven distribution of the favorite outcome among different leaf nodes are ‘explained’ by the explanatory variables in the internal nodes. The explanation leads to 0 score in the whole tree. This result extends to the general case.
Lemma 5
If a classification model uses explanatory variables as input variables, is nondiscriminatory.
Lemma 5 is correct because . All tuples of an Egroup will have the same prediction and consequently the Egroup is discriminationfree. The discrimination score averaged over all Egroups is 0. So the lemma is correct.
3.4 Discrimination and correlation between and
Correlation between a protected attribute and the outcome attribute , denoted by , is critical to discrimination. If is high, i.e., =1 implies =1, any classification model that has high accuracy will produce high level of discrimination. In this section, we analyze how discrimination is related to . We show that although there is a link between discrimination and correlation of and , the relationship is not monotone.
The correlation between the outcome and a protected attribute can be measured by oddsratio . means that there is no correlation between and . values further away from 1 indicate strong correlation. For example, if all females are high income earners () and all males are low income earners (), and are extremely correlated and the value is infinite.
If , we define to mean not correlated. If , the =1 group does not have any people, correlation does not make sense. If If , no tuple has =1 outcome and the case does not make sense either. The case where is symmetric.
Oddsratio and discrimination are related. As an example, assume that if all counts, except for , are fixed in . In the discrimination, assume that . Then as increases, both and increases. This trend is NOT true if other counts are allowed to change.
Interestingly, the following lemma shows that the interaction of oddsratio and discrimination is not simple.
Lemma 6
Given an Egroup and its DVdivision tuple counts,

is discriminationfree () if and only if and are independent.

Less correlation does not mean less discrimination.
Proof

We transform Formula (4) into . When , then and vice versa. means .
The extreme cases for and the extreme cases for are corresponding. So the item is proved.

We assume that we have two Egroups and . Their tuple counts are denoted by and respectively. Their oddsratios and discrimination scores are respectively. We want to prove that when and are less correlated in than in (i.e., ), the discrimination in can be larger than that in (i.e., ).
We consider only the case of . Let , and . When want to show that when , there are cases satisfying .
We present a tuple counts table below. Assume nonzero positive integers and the following counts tables.
. if plus reflecting our assumption . On the other hand, . if . Combining the conditions for and , we have . For example, if we let and , and . We proved that a data set having lower correlation (oddsratio closer to 1) may have higher discrimination compared to another data set.
The item is approved.
This result is very important to understanding the complexity between redline attributes and the discrimination of predictions. Let be a redline attribute. It is an intuition that higher correlation between and and higher correlation between and would lead to higher model discrimination. In a previous section we showed that the discrimination caused by redline attributes is modified by the error distribution of the model. Now we show that higher does not mean higher discrimination. Consequently the relationship between model discrimination and redline attributes is not intuitive.
4 Exploration of discrimination levels in real data sets
In this section, we present the results of our exploration of discrimination in four real world data sets. We present the following results. (1) Data from real application is often discriminatory and how the number of explanatory attributes affect the score. (2) Classification models may change the level of discrimination in the predictions compared with the one in the train data. (3) Classification models built from nondiscriminatory data may still be discriminatory.
4.1 data sets
We use four real world data sets as shown in the following list. Name, size, source, and attributes of the data sets are shown. All data sets are processed to have binary (0,1) values. The values of ordinal attributes are binaryzed using median. Categorical attributes are binaryzed by taking majority and the rest. The labels (P), (E), and (D) against some attributes indicate the types protected, explanatory, and outcome respectively. The attributes without a label are Oattributes.

US Census 1994. numb(rows)=48842; minority class rate=.25
https://archive.ics.uci.edu/ml/data sets/adult
Attributes: age45(P), natCountryUS(P), raceBlack(P), sexM(P), workPrivate(E), occuProf(E), workhour30(E), eduUni(E), relaNoFamily, married, income50K(D) 
Canada Census 2011 [20] ^{1}^{1}1The author wishes to acknowledge the statistical office that provided the underlying data making this research possible: Statistics Canada. numb(rows)=691788; minority class rate=.34
https://international.ipums.org
Attributes: weight100(P), age50(P), sexM(P), edUni(E), occProf(E){}, occSkilled(E), occOther(E), hoursfull(E), govJob, classSalary, income45K(D) 
German Credit. numb(rows)=1000; minority class rate=.3
https://archive.ics.uci.edu/ml/data sets/statlog+(german+credit+data)
Attributes: age35(P), single(P), foreign(P), chkAccBal(E), duration20m(E), creditHistGood(E), purposeCar(E), credit2320(E), savings500(E), emp4y(E), installPct3(E), sexM(E), guarantor(E), resid3y(E), propertyYes(E), instPlanNon(E), houseOwn(E), creditAcc(E), jobSkilled(E), people2(E), hasTel(E), approved(D) 
Recidivateviolent [16]; numb(rows)=4744; minority class rate=.14
https://github.com/propublica/compasanalysis
Attributes: sexM(P), age30(P), raceAfrica(P), raceWhite(P), raceOther(P), juvFelonyCnt1(E), juvMisdCnt1(E), juvOthcnt1(E), priorsCnt3(E), cjail1Month(E), cChargeisdemM(E), isRecid(E), score8(D)
The Recidivism data set follows [16]. The score8 column stores predictions from a system called COMPAS. The isRecid column stores whether the person recommitted a crime. We want to see if score8 values can be accurately repredicted.
We now present the results of experimental evaluation of our method.
4.2 Discrimination in original data sets
In the experiments, each (original) data set is stratified into Egroups by using the explanatory variables specified in the data set descriptions above. A discrimination score is calculated for each protected attribute in each Egroup following Formula (4). The discrimination scores of different Egroups for the same protected attribute are averaged with the weights of the group sizes. The global discrimination score of the data set is the maximum of the averaged scores over different protected attributes.
The results are shown in Table 4 where is the global discrimination score. The table also lists the worst (maximal) Egroup discrimination score () and the percentage () of tuples in this worst group out of all the tuples in the data set. In the data set, the discrimination score of some groups is over the threshold and these groups are called the overlimit groups. The discrimination scores of these groups are averaged to get the average score and the percentage of tuples in these groups is . The top three protected attributes ranked by absolute values of the their score are listed in the rightmost column.
For the Adult data, all Egroups are discriminatory (100%) with the score 2 times more than the threshold, and the overlimit groups are 4 times more than the threshold. The worst discrimination happened to the attributes of Gender and Age.
In the Canada data, the discrimination scores are larger than those of the Adult data set. The worst discrimination happened to Age and Sex male. People who takes Salary (instead of Wage) were slightly discriminated.
German Credit data’s discrimination level is lower although some Egroups (with 52% of tuples) have a discrimination score of 0.28, and the worst Egroup (with 5 tuples or 0.5%) has a score of 1. After some investigation, we found that this extreme score is caused by a small group size. When the size of an Egroup is small, the discrimination score can be dramatic. The tuple counts of this worst group is (3,0,0,2) and the score calculation is .
The Recidivate data has an overall discrimination score of 0.098, lowest among all data sets. The worst discrimination (0.778) happened to the protected attribute RaceAfrica in an Egroup with 11 tuples (0.2% of the data set). The tuple counts of the group is (7,2,0,2) and the score calculation is . Among 9 Africans, 7 were predicted to recidivate, but among the 2 nonAfricans, 0 were predicted to recidivate.
We note that the above observations are conditional. They are dependent on the way in which data is discretized and on what and how many variables are specified as explanatory ones.
Figure 3 describes the relationship between discrimination and the number of explanatory attributes. The experiments are done by using a specified number of explanatory attributes among all the described explanatory variables in turn to calculate an average discrimination score. For example, with the Adult data set, there are 4 possible explanatory attributes in the description. In the case of using 1 explanatory attribute, we run 4 experiments, each with a different explanatory attribute and the scores from these 4 experiments are averaged to get the final score for the one explanatory variable case. From Figure 3, we observe that as the number of explanatory attributes increases, the discrimination score becomes lower. This trend is reasonable because when more explanatory attributes are used, more discrimination can be justified and consequently the discrimination level reduces.
We note that as more explanatory attributes are used, the data becomes more fragmented and the sizes of strata of explanatory attributes reduce to small numbers or 0, which can make discrimination scores change dramatically to large values.
4.3 Discrimination of classifiers
In this exploration, we use commercially available modeling algorithms to build classifiers on the training data sets (the original data sets used above). The models are then used to predict a new outcome for each tuple in the training data sets. The predicted outcomes replace the original observed outcomes to form new data sets called predicted data sets. We calculate the discrimination scores on the predicted data sets.
For the same training data set, we get different predicted data sets when the classifiers are different. We choose five wellused classification algorithms, namely decision tree (DT), Bayes network (BN), neural network (NN), logistic regression (LR), and support vector machine (SVM) from SAS Enterprise miner and use these algorithms with the default parameters to generate predicted data sets. Discrimination scores are calculated for the predicted data sets and the results are shown in Table 5. BCR stands for balance classification rate which is the average of the true positive rate and the true negative rate. Err stands for misclassification rate. These two measures indicate the quality of the classification models. Better models have larger BCR and smaller Err.
First we look at the results for the Adult data set in the topleft corner. Compared to the discrimination scores of the original data (line Orig), the predicted data set has (1) a slightly lower global score (glbds), (2) much higher overlimit score (ogds) with a lower percentage (og%), and (3) a much higher worst group score (wgds) with smaller percentage (wg%). Points (2) and (3) indicate that the classification errors made by the models are quite high (19% ), some of these errors made some Egroups less discriminatory, others made other groups worse.
The Canada, German, and Recidivate data sets have similar properties. We note that the last line for SVM of these data sets have a global score of 0. The reason is not that the classifier is the best, but that the classifier made so many errors such that the error rate equals to the minor class rate in the data set. For example, in German credit data, the percentage of nonapproved cases is 30% while the method’s misclassification rate is also 30%. A close check found that the classifier predicted all negative class as positive class. In this case, all nonapproved cases are predicted as approved cases. When all predictions have only one class, there is no discrimination.
From this we see that when examining the discrimination of a classification model, we must consider the balanced accuracy and error rate. Otherwise, the conclusion may not be right.
4.4 Nondiscriminatory data does not mean fair classifiers
In this section, we use experiments to show the independence between discrimination and classifiers trained from the nondiscriminatory data.
In the experiments, our nondiscriminatory data was generated from the CV method [3]. More specifically, we use the original data as training data to run the CV implementation in [7] by specifying Sex as the only protected attribute, rejecting other protected attributes described in Section 4.1. The CV method produces a predicted data set which is close to nondiscriminatory. We call the predicted data set the data set. Further, we train a predictive classifier from the CV data set. The classifier again produces a predicted data set . The discrimination in is what we want to analyze. That is, we want to see if the classifier from the nondiscriminatory data set is discriminatory.
The results of this experiment are in Table 6. The grey line labelled by ‘CV’ is the data set. The data set for each classifier is labelled by the classifier name. For example, is the data set from the Bayes network classifier. Because the CV method does not support explanatory attributes, the results do not reflect group level discrimination. That is, the whole data set is seen as a large Egroup for the only protected attribute Sex.
The results show that the discrimination of training data and the discrimination of the classifiers from the data are independent. Among the four data sets, Adult CV has the smallest discrimination score, but the classifiers from this data set have highest discrimination scores. Although different algorithms among BN, DT, LR, NN, and SVM have different balanced accuracy and different error rates, all of them are discriminatory and the scores are 1333 times larger than the score on the CV data set.
The results also show that high accuracy does not lead to lower discrimination. For example, DT for Adult CV data has highest BCR, but its discrimination score is not the least.
The numbers for the German credit data set indicate the same independence but in an opposite way. With this data set, the discrimination of the CV is over the threshold, but the discrimination of the classifiers is less than the threshold (NN is only slightly over).
The numbers for the Canada and the Recidivate data sets show that the classifiers did not change the discrimination levels.
From all these, we observe clearly that discrimination of classifiers are independent to the discrimination of training data. It is also independent of accuracy and error rate. These two conclusions are reasonable because with the same error rate, the error distribution in the protected attributes can be very different, leading different discrimination scores.
5 Related Work
Algorithmic discrimination has attracted a lot of research effort. The work focused in two areas: discrimination detection and discrimination removal from data and from models. The work on removal are in three categories: preprocessing, manipulation of model learning algorithms and post processing. We now review the work done in these directions.
Discrimination detection in data The core problem of detection is to define and choose metrics to measure discrimination. Zliobaite [32] has a good summary of previous metrics [21, 22, 33, 8, 24, 25, 5, 6]. Some recently proposed causalitybased metrics are [6, 14, 29, 30, 17, 31] and the metrics for individual discrimination are [18, 4, 19, 30].
Removal of discrimination from training data (preprocessing) Feldman et al. [5] proposed to transform data in a data set so that the redline attributes, those that correlated to the target, become independent to the target variable. Friedler et al. [7] summarized some of the work in this direction.
Modification of model training algorithms Kamiran et al. [10] proposed to combine information gain and discrimination gain in decision tree learning and to use a postrelabelling process to remove discrimination from predictions. Calders and Verwer [3] adjusts the probability in naive Bayes methods so that the predictions are discrimination free. Kamishima et al. [12] proposed a regularization method for logistic regression method. Zafar et al. [28] represented discrimination constraints via a convex relaxation and optimized the accuracy and discrimination in the SVM learning algorithm. Woodworth et al. [27] proposes a two step method to build a nondiscriminatory classifier. The data set is divided into two subsets S1 and S2. In the first step, a classifier is built to minimize error rate under the constraints of discrimination in data set S1. In the second step, a postprocessing model is built on data set S2. Raff et al. [23] proposed a discriminationaware measure for decision tree induction for continuous data. Landeiro and Culotta [15] proposes to use a weight undertraining method by strengthening confounder features to build a model. Kearns et al. [13] used an optimization method in model learning. Kamishima et al. [12] has done a deep analysis of CV2NB, ROC and proposed a method called universal ROC.
Removal of discrimination from predictions of a model (postprocessing) The work of Kamiran et al. [10] relabels the predictions of leaf nodes of a decision tree to achieve discrimination goal. The postprocessing model of Step 2 in [27] minimizes the discrimination using the target, the predicted target and the protected variables on the second half of the training data. Hardt et al. [9] uses equalized odds to build a model for postprediction manipulation. Kamishima et al. [12] proposed a method called universal ROC.
6 Conclusion
In this paper, we analyzed some properties of algorithmic discrimination. The properties reveal relationships between discrimination and correlation, data partition, and classification models. These properties have important implications in building predictive models, assessing discrimination in data sets and models, and evaluating discriminationaware models. The paper also explored discrimination of real world data sets. The results of exploration show that discrimination does exist in real world data sets, and the discrimination of models is independent to the discrimination of training data. This implies that the fairness of models cannot be achieved by manipulating training data. Our future work is to develop a general method to achieve fairness independent of learning algorithms.
References
 [1]
 Boatwright et al. [2017] Kimberly Boatwright, CRCM CAMS, and Kinsey Sullivan. 2017. 3 RealWorld Examples of Fair Lending Discrimination How They Can Damage Your Reputation. https://www.trupointpartners.com/blog/author/kimberlyboatwrightcrcmcamsandkinseysullivan (as of 2018Jul) (2017).
 Calders and Verwer [2010] Toon Calders and Sicco Verwer. 2010. Three naive Bayes approaches for discriminationfree classification. Data Min. Knowl. Discov. 21, 2 (2010), 277–292.
 Dwork et al. [2012] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness through awareness. Innovations in Theoretical Computer Science Conf. (2012), 214–226.
 Feldman et al. [2015] Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and Removing Disparate Impact. ACM SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining (2015), 259–268.
 Fish et al. [2016] Benjamin Fish, Jeremy Kun, and Adam D. Lelkes. 2016. A ConfidenceBased Approach for Balancing Fairness and Accuracy. https://arxiv.org/abs/1601.05764 (2016).
 Friedler et al. [2018] Sorelle A. Friedler, Carlos Scheidegger, Suresh Venkatasubramanian, Sonam Choudhary, Evan P. Hamilton, and Derek Roth. 2018. A comparative study of fairnessenhancing interventions in machine learning. arxiv.org/pdf/1802.04422.pdf (2018).
 Fukuchi et al. [2013] Kazuto Fukuchi, Jun Sakuma, and Toshihiro Kamishima. 2013. Prediction with ModelBased Neutrality. Euro. Conf. on Machine Learning and Knowledge Discovery in Databases  Volume 8189 (2013), 499–514.
 Hardt et al. [2016] Moritz Hardt, Eric Price, and Nathan Srebro. 2016. Equality of Opportunity in Supervised Learning. Advances in neural info. proc. systems (2016).
 Kamiran et al. [2010] Faisal Kamiran, Toon Calders, and Mykola Pechenizkiy. 2010. Discrimination Aware Decision Tree Learning. ITEE Intl. Conf. on Data Mining (2010), 869–874.
 Kamiran et al. [2012] Faisal Kamiran, Indre Zliobaite, and Toon Calders. 2012. Quantifying explainable discrimination and removing illegal discrimination in automated decision making. Knowl. and Info. Sys. 35, 3 (2012), 613–644.
 Kamishima et al. [2018] Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. 2018. Modelbased and actual independence for fairnessaware classification. Data Min Knowl Disc 32 (2018), 258–286.
 Kearns et al. [2018] Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. Proc. of Machine Learning Research V80 (2018), 2564–2572.
 Kleinberg et al. [2016] Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016. Inherent TradeOffs in the Fair Determination of Risk Scores. arxiv.org/abs/1609.05807 (2016).
 Landeiro and Culotta [2016] Virgile Landeiro and Aron Culotta. 2016. Robust Text Classification in the Presence of Confounding Bias. AAAI (2016).
 Larson et al. [2016] Jeff Larson, Surya Mattu, Lauren Kirchner, and Julia Angwin. 2016. How We Analyzed the COMPAS Recidivism Algorithm. https://www.propublica.org/ (2016).
 Li et al. [2017] Jiuyong Li, Jixue Liu, Lin Liu, Thuc Le, Saisai Ma, and Yizhao Han. 2017. Discrimination detection by causal effect estimation. BigData (2017), 1087–1094.
 Luong et al. [2011] Binh Thanh Luong, Salvatore Ruggieri, and Franco Turini. 2011. kNN as an Implementation of Situation Testing for Discrimination Discovery and Prevention. ACM SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining (2011).
 Mancuhan and Clifton [2014] Koray Mancuhan and Chris Clifton. 2014. Combating discrimination using Bayesian networks. Artificial Intelligence and Law 22, 2 (2014), 211–238.
 Minnesota et al. [2018] Minnesota, Population, and Center. 2018. Integrated Public Use Microdata Series, International: Version 7.0 [dataset]. Minneapolis, MN: IPUMS, 2018. IPUMS https://doi.org/10.18128/D020.V7.0 (2018).
 Pedreschi et al. [2008] Dino Pedreschi, Salvatore Ruggieri, and Franco Turini. 2008. Discriminationaware Data Mining. ACM SIGKDD Intl. Conf. on Knowl. Disc. and Data Mining (KDD) (2008).
 Pedreschi et al. [2009] Dino Pedreschi, Salvatore Ruggieri, and Franco Turini. 2009. Measuring Discrimination in SociallySensitive Decision Records. SIAM Intl. Conf. on Data Mining (SDM) (2009).
 Raff et al. [2018] Edward Raff, Jared Sylvester, and Steven Mills. 2018. Fair Forests: Regularized Tree Induction to Minimize Model Bias. AAAI (2018).
 Ristanoski et al. [2013] Goce Ristanoski, Wei Liu, and James Bailey. 2013. Discrimination aware classification for imbalanced datasets. Conf. on Information and Knowledge Management (2013), 1529–1532.
 Ruggieri et al. [2014] Salvatore Ruggieri, Sara Hajian, Faisal Kamiran, and Xiangliang Zhang. 2014. Antidiscrimination Analysis Using Privacy Attack Strategies. Euro. Conf. Machine Learning and Knowledge Discovery in Databases ECML PKDD, Part II (2014), 694–710.
 Simpson [1951] Edward H. Simpson. 1951. The interpretation of Interaction in Contingency Tables. Journal of the Royal Statistical Society 13 (1951), 238–241.
 Woodworth et al. [2017] Blake Woodworth, Suriya Gunasekar, Mesrob I. Ohannessian, and Nathan Sreb. 2017. Learning nondiscriminatory predictors. arXiv:1702.06081v3 (2017).
 Zafar et al. [2015] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P. Gummadi. 2015. Fairness Constraints: A Mechanism for Fair Classification. arXiv preprint arXiv:1507.05259 (2015).
 Zhang et al. [2016a] Lu Zhang, Yongkai Wu, and Xintao Wu. 2016a. On Discrimination Discovery Using Causal Networks. Intl. Conf. on Social Computing, BehavioralCultural Modeling, Prediction and Behavior Representation in Modeling and Simulation (2016).
 Zhang et al. [2016b] Lu Zhang, Yongkai Wu, and Xintao Wu. 2016b. Situation TestingBased Discrimination Discovery: A Causal Inference Approach. Intl. Joint Conf. on Artificial Intelligence (2016), 2718.
 Zhang et al. [2017] Lu Zhang, Yongkai Wu, and Xintao Wu. 2017. Achieving NonDiscrimination in Data Release. KDD (2017).
 Zliobaite [2017] Indre Zliobaite. 2017. Measuring discrimination in algorithmic decision making. Data Mining and Knowledge Discovery 31, 4 (2017), 1060–1089.
 Zliobaite et al. [2011] Indre Zliobaite, Faisal Kamiran, and Toon Calders. 2011. Handling Conditional Discrimination. ITEE Intl. Conf. on Data Mining (2011), 992–1001.