POTs: Protective Optimization Technologies

POTs: Protective Optimization Technologies


Algorithmic fairness aims to address the economic, moral, social, and political impact that digital systems have on populations through solutions that can be applied by service providers. Fairness frameworks do so, in part, by mapping these problems to a narrow definition and assuming the service providers can be trusted to deploy countermeasures. Not surprisingly, these decisions limit fairness frameworks’ ability to capture a variety of harms caused by systems.

We characterize fairness limitations using concepts from requirements engineering and from social sciences. We show that the focus on algorithms’ inputs and outputs misses harms that arise from systems interacting with the world; that the focus on bias and discrimination omits broader harms on populations and their environments; and that relying on service providers excludes scenarios where they are not cooperative or intentionally adversarial.

We propose Protective Optimization Technologies (POTs). POTs, provide means for affected parties to address the negative impacts of systems in the environment, expanding avenues for political contestation. POTs intervene from outside the system, do not require service providers to cooperate, and can serve to correct, shift, or expose harms that systems impose on populations and their environments. We illustrate the potential and limitations of POTs in two case studies: countering road congestion caused by traffic-beating applications, and recalibrating credit scoring for loan applicants.


1. Introduction

Advances in computational power, software engineering, and machine learning algorithms have been instrumental in the rise of digital systems. Their ubiquity in our everyday activities raises concerns regarding the centralization of decisional power (Yeung, 2018). These concerns are amplified by the opaque and complex nature of these systems which results in hard-to-explain outputs (Mittelstadt et al., 2019) and unjust outcomes for historically marginalized populations (Browne, 2015; Gandy, 2016; Angwin et al., 2016; Ensign et al., 2018).

Computer scientists counter these inequities through frameworks studied under the rubric of fairness, using a variety of formal fairness notions (Barocas et al., 2018). Their results have been instrumental in our understanding of the discriminatory effects of algorithmic decisions. The frameworks, however, rely on narrowing the inequity problem to primarily consider the discriminatory impact of algorithms, and assuming trustworthy providers. The narrow view has enabled valuable breakthroughs centered around the service providers’ ability to address some inequities, but fails to capture broader harms or explore other ways to contest service-provider power (Overdorf et al., 2018).

In this work we investigate digital systems from a new perspective in order to understand how to address the broader class of harms that they cause. To achieve this, we characterize the type of systems in which algorithms are integrated and deployed. These systems typically build on distributed service architectures and incorporate real-time feedback from both users and third-party service providers (Gurses and van Hoboken, 2018; Kaldrack and Leeker, 2015). This feedback is leveraged for a variety of novel forms of optimization that are geared towards extraction of value through the system. Typically, optimization is used for technical performance and minimizing costs, e.g., by optimizing cloud usage orchestration or allocation of hardware resources. It has also become part and parcel of “continuous development” strategies based on experimentation that allow developers to define dynamic objective functions and build adaptive systems. Businesses can now design for “ideal” interactions and environments by optimizing feature selection, behavioral outcomes, and planning that is in line with a business growth strategy. We argue that optimization-based systems are developed to capture and manipulate behavior and environments for the extraction of value. As such, they introduce broader risks and harms for users and environments beyond the outcome of a single algorithm within that system. These impacts go beyond the bias and discrimination stemming from algorithmic outputs fairness frameworks consider.

Borrowing concepts and techniques from software and requirements engineering, and from economics and social sciences, we characterize the limitations of current fairness approaches in capturing and mitigating harms and risks arising from optimization. Among others, we show that focusing on the algorithms and their outputs overlooks the many ‘externalities’ caused by optimizing every aspect of a system; that discrimination is only one of the injustices that can arise when systems are designed to maximize gain; and that ignoring service providers’ incentives and capabilities to enforce proposed solutions limits our understanding of their operation and further consolidates the power providers have regarding decisions and behaviors that have profound effects in society.

Finally, we propose Protective Optimization Technologies (POTs), which aim at addressing risks and harms that cannot be captured from the fairness perspective and cannot be addressed without a cooperative service provider. The ultimate goal of POTs is to eliminate the harms, or at least mitigate them. When these are not possible, POTs can shift harms away from affected users or expose abusive or non-social practices by service providers. We illustrate the potential of POTs to address externalities of optimization-based systems in two case studies: traffic-beating routing applications and credit scoring. We also identify numerous techniques developed by researchers in the fields of security and privacy, by artists, and by others, that, though not necessarily designed to counter externalities, can be framed as POTs.

2. Fairness from a system’s perspective

We introduce Michael A. Jackson’s theory of requirements engineering (Jackson, 1995b) to discuss the focus, goals, and assumptions behind fairness framework from a systems’ perspective. This theory argues that computer scientists and engineers “are concerned both with the world, in which the machine serves a useful purpose, and with the machine itself […] The purpose of the machine […] is located in the world in which the machine is to be installed and used.” In other words, our objective is to design systems that fulfill requirements in the world where the system is to be deployed.

More precisely, a portion of the world becomes the environment, or the application domain,9 of a machine. In this world there are phenomena in the application domain (e.g., events, behavior of people, activities) and in the machine (e.g., data, algorithms, state). The machine’s inputs and outputs, i.e., the things the machine can sense or affect, are shared phenomena: they are observable both to the application domain and the machine.

We introduce machines into existing application domains to effect change in these domains. To achieve the desired change, we need descriptions of the phenomena before the machine is introduced, known as domain assumptions ; and statements about the desired situation once the machine is introduced to the domain, known as requirements . A specification is a set of requirements providing enough information for engineers to implement the machine. typically describes phenomena shared between the machine and application domain. A program derived from the specification is a description of a machine. If implemented correctly, programs satisfy the specification. If the specification is derived correctly, programs generate phenomena that attain the desired effects in the application domain, i.e., they fulfill the requirements.

Jackson’s explicit treatment of the application domain and its interaction with the machine helps us to project known problems with algorithms to a systems view. It enables us to distinguish problems due to badly derived requirements (description of the problem) from those due to badly derived specifications (description of the solution) and those due to badly implemented programs (how solutions are implemented) (Jackson, 1995a).

2.1. The focus on algorithms is insufficient for addressing inequitable outcomes of systems

In the light of Jackson’s theory, we claim that the focus on algorithms leaves out the systems’ view. Algorithms are often a part of a larger technical system, which is deployed in an environment. Fairness proposals rarely evaluate the systems’ environmental conditions, thereby leaving out the possibility that, even when a fairness metric is satisfied by the algorithm, the system (environment plus the machine) could still be unfair or have other negative side effects. Such focus on the specification of the machine also promotes the idea “that the difficulty in addressing [unfairness] lies in devising a solution” (Jackson, 1995b) and not necessarily in rethinking the world.

Most fairness frameworks focus on describing independent of and and then guarantee that states of the machine, and its input and outputs, have certain properties. However, a machine that has fair inputs and outputs and fulfills a specification does not guarantee the fulfillment of requirements of fairness in the application domain. This would require evaluating , establishing what phenomena the machine is expected to change in the application domain, and articulating requirements with fairness as a goal. Only then could we evaluate whether an fulfills .

Focusing on has a number of repercussions. First, it does not reflect how harms manifest themselves in the environment. Without an understanding of and , a specification may simply not lead to a fair outcome. Imagine a hypothetical “fair predictive policing algorithm” that can fairly distribute police officers to different neighborhoods. If the algorithm does not consider that the policing institution is already configured to control minorities (Lavigne et al., 2017) and that interactions with police pose greater risk of harms for minorities, a “fair allocation” can still disparately impact those minorities (Ensign et al., 2018).

Second, focusing on achieving fairness for users might leave out the impact of the system on phenomena in the application domain that is not shared with the machine. Unjust outcomes could arise due to the optimization of certain behavior in the application domain, and not because was unfair. For instance, self-regulated housing markets such as Airbnb (AirBnB, ) may not actively discriminate against their users, but reports have shown that they can disrupt neighborhoods by changing rent dynamics and neighborhood composition (Hinsliff, 2018).

Third, the focus on algorithms abstracts away potential harms of phenomena in the machine. Much of the fairness literature focuses on ways in which algorithms can be biased or on harms caused by decision-making algorithms. This overlooks that, when the system hosting the algorithms optimizes its operation, it may gather more inputs and outputs than those of the algorithm. Therefore, focusing on the algorithm may miss effects on the world that can go beyond those generated by the outputs of the algorithm actions.

When phenomena in the machine domain are subject to optimization, unfairness can arise from optimization programs fulfilling the specification but not the requirements in the application domain. For instance, prediction techniques to optimize targeted advertising can create discriminatory effects (Ali et al., 2019); and exploration strategies to optimize may gather inputs from the application domain that put some users unfairly at risk (Bird et al., 2016).

2.2. Discriminatory effects are not the only concern for building just systems

In Jackson’s terms, considering only discriminatory effects constrains the requirements to a particular class of harms. This approach risks missing other harms caused by the system when evaluating the performance of the specification in the environment.

We assume the introduction of a machine in an environment aims to improve specific phenomena. The fact that this machine follows a specification that does not discriminate according to does not guarantee that this machine will not induce other harms to the environment in any applications domain. Take as examples unsafe housing or bad working conditions. If we use a machine to distribute these resources more efficiently, even if it does so fairly, all users are harmed. Bad housing conditions and lack of labor protection are problems in and of themselves so users will be badly served regardless of fairness conditions. When the system is by design unjust, or when the phenomena are structurally unjust or harmful, claims on are meaningless. In Jackson’s terms: the requirements are incomplete with respect to just outcomes.

Sometimes, however, injustices are tightly tied to the machine: is not the solution but the source of problems. The way requirements are optimized might lead to externalities for (a subpopulation of) users. When a specification optimizes an asocial outcome, such as excessive user engagement in social networks (Tufekci, 2018), it can expose users to harms like addiction. Solving fairness in this system will not resolve the underlying problem: the system is harmful.

Striving to fulfill the requirements itself might bring new harms to the application domain. Consider a fairness solution that alleviates distributional shift based on increasing diversity in the training set. If its specification requires collecting data from more individuals or collects new attributes to implement the fairness measure, it will exacerbate privacy issues. These issues might result in many other harms in the application domain.

This ontology assumes that is built to “solve problems” and “improve phenomena” in the world. Thus, it does not provide the conceptual tools to address adverse situations. This positive valence hinders the consideration of cases in which a machine amplifies existing injustices or introduce new ones. Neither this ontology nor fairness frameworks account for power imbalances or economic incentives, and how they impact how machine requirements are considered and prioritized. We consider these matters in the next section by augmenting our systems view to consider socio-economic aspects of the application domain.

3. Fairness, incentives, and power

In this section, we extend our analysis to problems related to the political economy of systems. To model harms, we borrow the term negative externalities from the economics literature. A system causes negative externalities when its consumption, production, and investment decisions cause significant repercussions to users, non-users, or the environment (Starrett, 2011). The introduction of a machine might cause externalities in the application domain, independent of the completeness or correctness of its requirements and specification. For example, the heavy use of traffic-beating apps such as Waze can worsen congestion for all drivers in the application domain (Cabannes et al., 2018). We argue that validating the specification against the requirements is not enough. To build just systems one must consider externalities of the machine in the application domain.

Congruent with models in fair optimization and economics, to express externalities and incentives we introduce two utility functions that capture the machine’s impact on the application domain: the service provider’s utility, which measures how much value the provider extracts from introducing the machine, and the social utility, which measures the machine’s utility for the environment and people. We define two versions of social utility: one with a “god’s view” of the application domain and one from the specification’s perspective. These utility functions enable us to capture injustices due to the introduction of the machine into the environment.

3.1. Idealized Fair-by-Design Service Provider

We first consider an idealized fair-by-design service provider that is willing to address the externalities of the machine. That is, this provider aims to maximize both their own utility and the social utility. Using this setting, we show the ways in which fairness models fail to address a broad class of systems’ externalities.

We consider that a system is parametrized by a vector of internal parameters for some convex set of possible parameters . Let be a population, a set of individuals or other environmental entities that might be affected by the system. Let be the utility function of the provider when they use parameters . Let denote a hypothetical social-utility function, or benefit, defined in the requirements . Let be the social utility in the provider’s specification. The provider optimizes its operation by solving the following multi-objective optimization problem:


This problem is considered in fair learning literature in its scalarized form or constraint-form (Kamishima et al., 2011; Zafar et al., 2017a, b; Celis et al., 2017; Donini et al., 2018), with the social utility modeling a notion of fairness. We assume an ideal situation in which the chosen parameter vector is a Pareto-optimal solution (Miettinen, 2012), that is, it cannot improve any of the objectives without hurting at least one of them. Pareto-optimality, however, is not sufficient to guarantee fairness or equity as different trade-offs between the objectives are possible (Le Grand, 1990). We assume that out of the possible Pareto solutions, the provider chooses one that maximizes social utility: where is the set of all Pareto-optimal solutions.

Limitations in the Face of Externalities

Consider , the system obtained when using the “god’s view” values of social utility: with being the set of the corresponding Pareto-optimal solutions when using in Eq. 1.

Let . We say that a system with parameters induces externalities on its environment when the social utility of is not equal to that of the system : .

This definition is analogous to the neoclassical economic interpretation of externalities: due to an “inefficiency,” a partial optimum () is achieved, that is different from the optimum if no inefficiencies were present ((Dahlman, 1979). One can also parallel the divergence between social-utility values in to the Pigouvian “divergence of social and private costs” (Pigou, 1920). Because we are analyzing harms, we focus on negative externalities: .

Clearly, when . If does not precisely match , however, there will likely be externalities present, as illustrated in Fig. 1. In general, it is not known how exactly is impacted by deviations of from . In Appendix A, for a class of strictly concave utility functions we show that the sensitivity of to infinitisemal error of is approximately quadratic in the magnitude of the error. Intuitively, grows quadratically fast as diverges.

Incomplete Information of Social Utilities. We first study the case in which the provider’s view of the social utility, , does not fully reflect the requirements and the context of the application domain.

Social benefit is often modeled as a function incorporating individual models of social utility. A common assumption in neoclassical economics is that social utility is the sum of individual utilities (Yew-Kwang, 1983): where is a utility of individual . In practice, however, the fair-by-design provider has incomplete information: they are aware of the users, yet lack the full knowledge of their needs (similar to “imperfect knowledge” in economics and game theory (Phlips, 1988)). That is, they could misspecify for some user , inadvertently ignoring the needs and well-being of this individual. Consider a hypothetical fair-by-design predictive policing (Lum and Isaac, 2016) application that assumes that is maximized when there is equality of false positives in patrol dispatches across regions. The resulting dispatching rates might be fair for this definition, but the overall social benefit might be unchanged as minorities could still be over-policed in terms of the number of dispatches.

Figure 1. Pareto frontiers with real , and provider’s version . Misspecification of results in externalities , difference in values of the benefit function between provider’s system and .

Omitting Impact on Non-Users and Environmental Impact. Another case is when the fair-by-design provider has structural lack of knowledge of the application domain, e.g., knows the utility of their users, but not the utility of anything else (i.e., of phenomena not shared with the machine). Thus, a fair solution for the users could harm non-users as their benefits were never specified in the model . This is exemplified in the case of self-regulating housing markets, which could hypothetically be optimized for fairness in acceptance rates for guests (Benner, 2016), but disrupt neighborhoods impacting housing conditions, especially for people with low incomes.

Limitations Under Complete Knowledge

Even in the presence of complete information regarding and , there can exist externalities that the provider cannot mitigate.

First, the provider can maximize and yet cause externalities if the system’s goal itself is harmful (Keyes et al., 2019). For example, a facial recognition surveillance system might be completely fair with respect to skin color, but it still causes risks and harms to the population as a whole in terms of privacy loss.

Second, the fair-by-design model has as premise that there exists a solution that, if not maximizes, then at least satisfies minimum standards of everyone’s benefit. However, this may not be the case. Recall the examples about unsafe housing or bad jobs in Sec. 2.2. In those cases, the fair outcome might still be harmful for all users.

3.2. Limitations on Ideals

Mitigating externalities becomes a greater issue when incentives, capacities, and power structures are not aligned. One way this manifests itself is in the modeling of social benefit as a function. Indeed, a function cannot encode all the nuance regarding human needs. Moreover, neither the provider nor the users’ utility functions model the political context or the power asymmetries in the environment of the machine (Power, 2004). These can heavily influence or skew what we assume to be the ground truth . Thus, power and politics may come to render the system specification unfair even when it was designed considering the perfect benefit function for users.

Furthermore, the fairness-by-design approach inherently assumes that the provider always has enough resources to implement the fair solution that maximizes social utility. However, it is unreasonable to believe that such an assumption will hold in practice. For instance, even if Waze was cognizant of the needs of all individuals and all the peculiarities in their streets, it is unlikely that they could afford an operations in which all of those constraints are taken into account or be capable of mitigating the impact of externalities due to the interaction of multiple traffic beating applications.

Finally, not all service providers have the incentives to implement fair-by-design solutions. The provider might not be incentivized to care about social utility or could benefit from externalizing certain costs. In such cases, the magnitude of the externalities is likely to be more pronounced than in the case of incomplete knowledge.

4. Protective Optimization Technologies

The systems view on algorithms in previous sections enables us to systematically explore a problem space that had not been formalized before. A major source of problems is the use of optimization techniques that help to capture and manipulate phenomena in the application domains for the extraction of value. This practice causes intentional or unforeseen changes to the environment which result in (often neglected) harms to the environment.

We showed that existing fairness frameworks produce solutions that can only address a subset of these harms, at the discretion of a service provider. These frameworks have limited capability to mitigate harms arising from inequities inherent to the application domain and from harmful impacts introduced by the machine. By focusing solely on actions that can be taken by service providers, who have opposite incentives, fairness frameworks narrow both politics and contestation to the re-design of the algorithm which may not always be the site of either the problem or the solution.

In this section, we discuss means to address these issues. Although these means could be of socio-legal nature, we focus on technological approaches that we call Protective Optimization Technologies (POTs). Their goal is to shape the application domain in order to reconfigure the phenomena shared with the machine to address harms in the environment. POTs are designed to be deployed by actors affected by the optimization system. As these actors directly experience the externalities, a) they have intimate knowledge of the system’s negative effects, b) they are in position to have a better view of their social utility than a system provider can model—because it is their own utility (Dolan, 1998). Lastly, c) POTs do not rely on the incentives of the provider.

POTs seek to equip individuals and collectives with tools to counter or contest the externalities of the system. Their goal is not to maximize benefits for both users and service providers, nor to find the best strategy to enable optimization systems to extract value at minimum damage. POTs are intended to eliminate the harms induced by the optimization system, or at least mitigate them. In other cases, POTs may shift the harms to another part of the problem space where the harm can be less damaging for the environment or can be easier to deal with. Finally, when service providers react to reduce the effectiveness of POTs, this very action exposes the service providers’ need to maintain the power relationship and their capability to manipulate the environment to their own benefit.

POTs and Optimization Systems. To extract value through optimization, service providers obtain inputs from their environments that help them make decisions. We consider three kinds of inputs: i) inputs that users generate when interacting with the system, ii) inputs about individuals and environments received from third parties (Barreneche and Wilken, 2015), and iii) inputs from regulations and markets that define the political and economic context in which the system operates. Note that third parties can be public or private, and the data they provide can be gathered online or offline. We use Uber ride-hailing service (uber, ) as an example to showcase this complexity (Fig. 2).

Figure 2. Uber: a complex optimization system. Inputs to Uber optimization (black), effects on the environment (blue), and alternatives to deploy POTs (red).

In order to maximize its profit, Uber optimizes the prices offered to riders and the wages offered to drivers. Uber uses the following inputs (black arrows). First, it uses the direct inputs it receives from both riders and drivers. Second, it uses data from other sources such as online service providers, e.g., Google for maps and data that they might collect using cookies or pixels on other sites their users visit (uber, ). Uber also receives offline data from parties like municipalities interested in promoting the use of Uber to reduce costs of public transport (Uber, 2019). Lastly, Uber uses inputs from the market and regulators to evaluate the economic context in order to adjust wages and ride prices.

Uber ultimately uses these inputs and all the political and economic context in a combination of managerial and mathematical optimization to deliver outcomes to the environment: match riders to drivers and set ride prices. Reports and studies demonstrate that these outcomes cause externalities: Uber’s activity increases congestion and misuse of bike lanes (Kunkle, 2017), increases pollution, and decreases public support for transit (Graehler et al., 2019; Schmitt, 2019).

In the previous sections’ vocabulary, Uber and its optimization techniques are the machine; and the application domain comprises Uber drivers and riders, non-users, the online and offline environments, and the market and regulatory frameworks in which Uber operates. Roughly speaking, Uber’s requirements are to match drivers to riders with associated dynamic pricing. The specification defines what Uber applications should do to fulfill .

Uber is known to have unfairness problems. For example, Rosenblat et al. show that customer-based reviews are biased against minority drivers. As getting blocked from the system depends on these reviews, even if the Uber’s algorithms do not discriminate drivers on their attributes per se, the rules of the system result in disparate impact on blocked drivers (Rosenblat et al., 2017). Further, even when Uber algorithms are fair, e.g., they reward all drivers the same irrespective of their protected attributes, the optimization processes underlying Uber’s operation result in unjust outcomes: low wages (Sainato, 2019). The former is an externality stemming from structural biases in the application domain (thus a hypothetical does not result on ); whereas the latter is a problem of incentives misalignment.

In the Uber scenario, POTs can be deployed by users and non-users with the goal of changing the phenomena captured by Uber. These can come in three forms (red lines in  Fig. 2): by changing the inputs of the users to the system (e.g. the surge-induction POT (Sweeney, 2019) as we describe shortly in Sec. 4.1), by changing the online or offline signals gathered by Uber (e.g., mayors changing the city urban planning), or by affecting the market (e.g., by changing regulations or mandating salary increases (Rosenblat et al., 2017)).

4.1. Examples of POTs

Origin Optimization System Externality POT Desired Outcome Deployment Technique
Academic Face Recognition Privacy, discrimination Wear printed eyeglasses (Sharif et al., 2016) Evade face detection Individual Optimization
Academic Copyright Infringement Detection Fair use takedowns Adversarial examples (Saadatpanah et al., 2019) Avoid a fair use takedown Individual Optimization
Academic Psychometric Profiling Privacy, manipulations Text style transfer (Fu et al., 2018; dos Santos et al., 2018) Prevent from attribute inference Individual Optimization
Academic YouTube Recommendations Manipulation Poisoning (Fang et al., 2018) Breaking out of content bubbles Individual Optimization
Academic Waze Routing Local traffic congestion Sybil devices simulate traffic (Wang et al., 2018) Prevent routing into towns Individual Heuristic
Academic GRE Scorer Biased grading system Generate essay to pass GRE (Perelman, ) Higher test score Individual Heuristic
Deployed Ad Network Privacy, manipulations Click on all ads (Howe and Nissenbaum, ) Ad Network destroyed Collective Heuristic
Deployed Uber Pricing System Low wages Shut off app, turn it back on (Sweeney, 2019) Induce surge Collective Heuristic
Deployed Instacart Pricing Low wages Tip 22¢ in app, cash at door (Eidelson, 2019) Fair pay for jobs Collective Heuristic
Deployed Automated Hiring Bias, discrimination Edit resume (Jobscan, ) Flip automated hiring decision Individual Optimization
Deployed Pokemon Go Resource Spawn Unfairness Edit Open Street Maps (Akhtar, 2016) Encourage resources to spawn Individual Heuristic
Deployed FitBit for Insurance Premium Privacy, surveillance Spoof device location (Brain and Mattu, ) Get insurance benefits Individual Heuristic
Deployed Pharma Optimizing Patents End of humanity Find potential drugs using ML (Tseng and Raspet, ) Get drugs in the public domain Individual Mixed
Deployed Insurance Coverage Optimization High costs of treatment Doctors changing claim codes (Wynia et al., 2000) Get higher reimbursements Individual Heuristic
Artistic Face Recognition Privacy, surveillance Scarf that is classified as a face (Harvey, ) Evade face detection Individual Optimization
Artistic Face Recognition Privacy, surveillance Camouflage to cover features (Harvey, ) Evade face detection Individual Heuristic
Artistic Autonomous Cars Exploration risks Ground markings (Bridle, ) Trap autonomous cars Individual Heuristic
Table 1. Repurposing technologies as POTs. We detail the origin of the technology; the externality it addresses, the optimization system causing the externality and the desired outcome of the POT; deployment requirements (individual or cooperative action); and the underlying design techniques (counter-optimization or heuristics to decide how to shape the environment).

Our vision for POTs systematizes the use of technologies as tools to both explore the effects that algorithms and optimization systems have on our society, and the design of countermeasures to contest their negative effects in the absence of regulatory or other accountability measures.

We now revisit recent academic technologies, artistic interventions, and deployed tools, that can be reframed as POTs (Table 1 provides a summary). POTs formalize such technologies and interventions, enabling the systematic study and design of such solutions. We illustrate this in Sec. 5, where we design two POTs from scratch.

These technologies have different origins. First, we observe that there are technologies proposed in the academia that can be repurposed as POTs. For instance, in the field of computer security, research that aims to protect against attackers gaming the YouTube algorithm (Fang et al., 2018) can be repurposed by users to fight against filter bubbles; in the field of adversarial machine learning, tools developed to evade copyright detection (Saadatpanah et al., 2019), originally developed to strengthen DRM, can be reframed as a way to prevent fair-use takedowns (Foundation, ). Second, we draw from works produced by artists looking into the impact of technology on society. For instance, counter-surveillance fashion that tricks facial recognition technologies (Harvey, ) can be repurposed to evade discriminatory facial recognition algorithms or discriminatory uses of facial recognition. Finally, we look to deployed technologies that are already countering optimization, either intentionally or as a side effect. For instance, Jobscan (Jobscan, ) assists job applicants in getting past the automated sorting implemented by large companies and job-posting sites. This tool could be repurposed to reduce the gender or racial bias reported for these tools (Dastin, 2018).

These examples can also be categorized based on the means used to design the POT: those based relying on adversarial machine learning (Sharif et al., 2016; Fu et al., 2018; Fang et al., 2018; dos Santos et al., 2018; Harvey, ; Jobscan, ), and those that use heuristics to exploit the target optimization process. For instance, as a response to low wages, Uber drivers have developed heuristics for inducing surges (Sweeney, 2019).

Finally, some technologies can be deployed individually and others require collective action. AdNauseam (Howe and Nissenbaum, ), for example, reduces the utility of ad networks by clicking all ads served to the user, flooding the network with false information. Without a critical mass of users, however, AdNauseam’s effect would not be observable.

5. Designing Protective Optimization Technologies: Case studies

In this section, we show how, given an optimization system and a negative externality, one can design new POTs from scratch.

Like many of the technologies and interventions in Table 1, we make use of optimization techniques to design the operation of POTs. Starting from the model introduced in  Sec. 3, we model the optimization system’s objective as a function where denotes the space of environment inputs to the system (i.e., phenomena that are sensed by the machine), and the vector is a set of inputs coming to the system. Each concrete input can come from a different source (users, non-users, or other actors in the environment). For simplicity, we model the time dimension through discrete time steps, and we assume that there are two possible time steps: and .

To maximize its gain, the optimization system strives to solve a mathematical optimization problem for a set of inputs at each point in time . Given this system, the goal of a POT is finding actionable—feasible and inexpensive—modifications to the inputs of the optimization system so as to maximize social utility that we denote as . Note that this definition of social utility needs not to correspond to the social utility considered by the optimization system ( in Section 3). We consider that each input has an associated modification cost , that represents how hard it is to modify it. The cost of changing a set of inputs can be any function of the individual costs.

In this model, we define the POT design as a bi-level multi-objective optimization problem:


where is the vector of inputs if no intervention happened, and are possible vectors of modified inputs. represents the system’s state in the next step, after the POT has been deployed through the modified inputs .

We now instantiate this problem for two different use cases, one where the POT can be deployed by an individual, and one where effective deployment requires a collective.

5.1. Thwarting Traffic from Routing Apps

In our first case study, we look at Waze, a crowdsourced mobile routing application that optimizes the routes of its users based on information about traffic, road closures, etc. (Waze, ). Waze causes negative externalities for residents of towns and neighbourhoods that are adjacent to busy routes. For example, take the town of Leonia, New Jersey, USA, which lies just outside one of the bridges into New York City. As Waze rose in popularity and directed an increasing amount of users through the town when the highway was busy, the town became crowded during rush hours. To prevent Waze traffic, the town was briefly closed off to non-local traffic, which was determined illegal (Shkolnikova, 2018). In this section we propose a solution for discouraging Waze from selecting routes through the town while minimizing the impact on its inhabitants.

Problem Setup. We set up this problem as a planning problem in which the town’s traffic network is modeled as a weighted directed graph, and the goal is to increase the cost of paths between the highway ramps. We define Waze’s utility as the capability to provide fastest routes for its users. Routing through town can increase this utility when it takes less time than traveling via the highway. The POT is designed to increase the minimum time through town so that Waze users are not routed through it. We define the social utility as a binary variable that takes value 1 when no vehicle is routed through the town (i.e., the cost in time of traversing the town is greater than traversing the highway) and 0 otherwise.

Let be a weighted directed graph representing the traffic network within the town. Each edge represents a road segment, and each vertex represents a junction of segments. The edges have associated time to pass the segment given by the function . We define the time cost of traversing a path in the graph as the total time to pass its edges: .

Let be the source and sink vertices that represent the entry to the town from the highway , and the exit to return to the highway . Let the time to travel from point to via the highway be . While we do not know the routing algorithm used by Waze, we assume that Waze will send users through the town when the path through town is quicker than the highway. That is, if there is a path from to in with cost , Waze routes users through the town.

Avoiding Routing via Planning

We aim to transform the graph into a graph such that the time cost of any path from to in is . We focus on what the town can control: time to traverse a road segment. We express these changes as the increase in time that it takes to traverse a segment, . We abstract the exact method for adding time to the segment, which could be changes to speed limits, traffic lights, stop signs, speed bumps etc. We construct by modifying the time values in the original graph : where is a function representing the edge time in .

We acknowledge that some roads are more costly to change than others by associating a modification cost to every road. In practice, this cost will be specified by the town. We use length of a road segment as such a cost, capturing that changing longer roads will likely have more impact on the town. Let be binary interdiction variables that represent whether we are modifying the edge in graph . To express the cost of modifying a graph into we use the following modification-cost function: where represents the cost of modifying the edge .

We now formalize the POT through posing the multi-objective optimization problem in Eq. 2 in a constraint form: minimize subject to the constraint, given that Waze is maximizing :


This is equivalent to the problem known as shortest-path network interdiction (Golden, 1978; Fulkerson and Harding, 1977; Israeli and Wood, 2002; Wei et al., 2018). In the form equivalent to ours it can be solved as a mixed-integer linear program (MILP) (Israeli and Wood, 2002; Wei et al., 2018). We refer to Appendix B for the exact formulation of the MILP in our context. We use ortools (ortools, ) for specifying this MILP in Python, and the CBC open-source solver (Forrest et al., 2019) for solving.

Empirical Evaluation

We apply this POT to three towns, all of which reported issues with Waze: Leonia, NJ, USA; Lieusaint, France; and Fremont, CA, USA. We retrieve the map data for each via the Open Street Maps API (Map, ).

To assign to each segment , we estimate the time it takes to traverse the segment using its length and type (e.g., we consider residential streets to have a speed of 25mph) and add extra time for traversing the intersection. We infer the intersection time using the travel time from Google Maps for accuracy. For each city, we run the solver for different values, corresponding to different travel times on the highway. Larger values of require more changes to roads in the town.

Figure 3. Solutions for Leonia, NJ (left) and Lieusaint, France (right), when the time of road segments is allowed to be increased by 75%. Town streets are marked in blue, highways in orange, and surroundings in grey. The red dots signify and , the closest points to the highway in the town. The roads marked by thick, black lines are the optimal set of segments in which the time should be increased.

For simplicity of implementation, we choose two points at the edge of the town (red dots in Fig. 3) and not the sink/source points, which would be on the highway, for the solver. We then add the time that it takes to travel from the actual source point on highway to the first point and from the second point back to the sink point in order the calculate the results. For Leonia, for instance, we approximated this as 30 seconds on each side of the town.

We consider scenarios in which the town is able to increase the time that it takes to traverse each road segment by 25%, 50%, and 75%. For each town we found the value of for which Waze begins sending cars through the town and the value of in which no further changes to the road can prevent Waze from sending its users through the town.

The graph for Leonia, NJ contains 251 vertices and 661 edges. Given the parameters we choose, without any changes to the town, Waze begins to send its users through Leonia when . That is, it normally takes 4 minutes to travel through Leonia, so when it takes longer than 4 minutes to traverse the highway, Waze routes users through town. This corresponds to traffic traveling at about 31mph on the highway. If we limit the amount of time that can be added to traverse a road segment to 25% of the original time, we can prevent cars from being routed through Leonia until the average highway speed has reached 26mph (). That is, we can find a solution that prevents routing through town as long as the highway speed is greater than 26mph. For 75% time increase, we can change the town such that traffic will not be routed through Leonia until the average speed on the highway has fallen to 19mph. This solution for Leonia is shown in Figure 3.

Lieusaint is larger than Leonia, with 645 vertices and 1,248 edges. Given the parameters we choose, Waze routes its users through Lieusaint when , which corresponds to the speed on the highway dropping to 16mph. Allowing the road segments to be lowered by 75%, we can prevent traffic from being routed through Lieusaint until the highway speed has dropped below 10mph () (See Lieusaint in Figure 3). We report the solution for Fremont, CA, a significantly larger town, in Appendix B.

Finally, we measure the cost of implementing these solutions (Fig. 4). For each town, we consider the impact to the town to be how much longer, on average, it takes to travel between any two points in town. We compute the shortest path between every pair of points in the town and average these times before and after the POT solution. We then compute the percentage increase from the initial average time to the post-POT average time. The higher impact solutions are those which will tolerate a lower highway speed. That is, they will prevent cars from being routed through the town at lower highway speeds. We see that even though allowing road segments to take 75% longer to traverse can prevent cars from entering the town at a lower highway speed, the impact to the town is much higher. The inhabitants of Lieusaint also suffer more from the changes than the residents of the much smaller Leonia.

Figure 4. Effect of Changes on In-Town Travel.

POT Impact and Limitations

The intervention we propose in this section focuses on alleviating the problems routing applications cause in one town. The POT would support this town’s government to decide which changes should be implemented in the city layout and traffic rules so that the cost of traversing the city becomes undesirable for passthrough drivers. While this intervention indeed mitigates the effect of external traffic on the target town, it is likely that then vehicles are routed elsewhere, i.e., the POT shifts the harms from this particular town to other regions. Moreover, we acknowledge that our POT can only help in cases when vehicles can still circulate on the highway. The moment the congestion forces vehicles to halt, the town becomes the better option regardless of any modification on its road network.

5.2. Reducing False Negatives in Credit Scoring

In this case study, we explore solutions for countering harmful effects of a machine-learning credit scoring system: A system that a bank uses to decide how to price a loan application according to the predicted risk. The underlying algorithms that support such decisions are designed to maximize banks’ profits and minimize their risks. These algorithms can be discriminatory (Citron and Pasquale, 2014), or cause feedback loops for populations disadvantaged by the financial system (Poon, 2016). These harms are often caused by inputs that represent unjust realities which are propagated to the model’s decisions.

Problem Setup. We model the credit scoring system as a classifier that takes as input the information about the loan applicant and the loan details, , and outputs a confidence score for whether the applicant would repay the loan or not. This function optimizes the bank’s utility that we model here as the negative empirical loss over historical data: where is the parameters of the classifier, is the loss function, and is the bank’s dataset of historical loans () and their repayment or default outcomes (). We assume that the classifier is retrained as new data points arrive.

In our case study, we model the harms of the system as the rate of false negatives (wrong default predictions) for economically disadvantaged populations. In this POT, we counter this problem using adversarial machine learning techniques deployed by a collective of individuals with the means to take and repay loans, as explained shortly. This POT aims to increase the social utility defined as the loss for a target group : where is the disadvantaged subset of applicants (in this case study, we define disadvantaged as having little funds in the bank account) who were wrongfully denied a loan. This POT can be thought as promoting equality of false-negative rates between the target group and everyone else (Hardt et al., 2016; Chouldechova, 2017), with the difference that we do not limit our view of externalities to a commonly protected subgroup, i.e., economical disadvantage is not commonly considered as protected.

Reducing False Negatives with Adversarial Machine Learning

We first identify what inputs ( in the abstract model) can be modified by the collective deploying the POT. First, the deployers can only add new inputs to the dataset by taking and repaying loans. Second, the demographic attributes of these added loan applications have to be similar to those of individuals in . Thus, the POT must inform the collective about who and for which loans they should apply for and repay, in such a way that they reduce the false-negative rate on the target group after retraining. This POT is idealistic in that it assumes that this collective will include applicants of diverse backgrounds to be able to provide different inputs to the classifier. However, in the absence of other means of feedback, communication, and accountability, it represents the only means to influence an unjust system. It is also consistent with the existing practices people resort to in this setting (Friedman, 2017; O’Shea, 2018).

Finding which inputs to inject into a training dataset to modify its outputs is known as poisoning in adversarial machine learning (Papernot et al., 2016). Typically, poisoning attacks aim to increase the average error of the classifier or increase the error on specific inputs (Biggio and Roli, 2018). We note that our use of poisoning is different. First, we poison to decrease the error for a given target group. Second, our use of poisoning is not adversarial. On the contrary, we use it to protect users against harmful effects of the model.

With this in mind, we design the POT using the following bi-level optimization problem:


where is the current parameters of the classifier, is the set of poisoned applications, is the maximum number of poisoned applications, and is a set of feasible loan applications. That is, we minimize the classifier’s loss for the target group, where the classifier is trained using the poisoned examples . In our evaluation below, we additionally make use of a regularizer to minimize the effect of poisoning on any other applications:

This formulation makes two assumptions. First, we assume that when designing the POT we have access to the training dataset of the provider , and we can obtain retrained loan-approval models. We consider this assumption reasonable as poisoning attacks tend to transfer even if the dataset and model do not match the real ones, especially when the models have low complexity (Demontis et al., 2019). Second, the added loan applications must be feasible: there has to exist a person in the with demographics required for this loan application.

We solve this problem by scoring each example in according to the value of our optimization objective , retraining for each example, and then employing a greedy algorithm to assemble that satisfies the constraints. We refer to the Appendix C for the details of the algorithm.

Empirical Evaluation

We create a simulated loan approval system using the German credit risk dataset from the UCI Machine Learning Repository (Dheeru and Karra Taniskidou, 2017). This dataset contains 1,000 feature vectors representing loan applications, including applicants’ job type, bank account details, gender, etc., and loan details: amount, duration, and purpose. Each has a binary label encoding whether the loan was repaid (70%) or not (30%). We implement the loan approval system as a logistic regression classifier. For our simulation, we split the dataset into the bank’s training dataset (800 examples), and the test set (200). The classifier achieves 75.5% test accuracy. We obtained the best results with the use of as the trade-off parameter for the regularization term (out of 0.25, 0.5, 0.1) in .

We simulate the set of feasible applications using a subset of successfully repaid applications from users that are not in the target group by generating all possible changes of modifiable attributes.

We evaluate our POT in two settings: a “clean” setting in which the only applications received by the bank are those from the collective (); and a “noisy” setting in which other people take loans between the time when the POT is deployed and the time when the classifier is retrained. We report the results in the noisy setting for 10 and 50 additional loans (1.25% and 6.25% of the original dataset, respectively). We repeat the noisy experiments 10 times in which we draw the “noise” loan applications at random.

We present the effect of poisoning on the target group and everyone else in Fig. 5, top. Unsurprisingly, the more poisoning applications there are, the more significant the effect. With 10 poisoning applications (1.25% of the training dataset), our algorithm reduces the number of false negatives in the target group by 9; in the presence of noise this decreases to 7. The false negatives in the rest of the dataset are on average not impacted by our POT.

Figure 5. Effects of the POT on the error rates of the classifier for the target group. Top: decrease in the number of false negatives (wrongly denied loans). Bottom: increase in the number of false positives (wrongly granted loans).

POT Impact and Limitations

Unfortunately, the POT increases the false positives (Fig. 5, bottom). That is, it shifts the harm to the bank, which would give more loans to people who cannot repay, increasing its risk. This effect sharply increases with the seventh application (out of 10). This is due to the poisoning inputs starting to change the model parameter corresponding to the loan purpose. In turn, this increase in false positives could lead to adverse social impact over time if banks try to counter the POT effect (Liu et al., 2019). The POT deployers could adjust the trade-off parameter to control for the side-effects. Yet, such a reaction would expose the incentives and motivations behind the bank’s choice of parameters.

To be consistent with the fairness literature (Hardt et al., 2016; Yang and Stoyanovich, 2017), we assume that the target group is interested in getting higher credit ratings (through decrease in false-negatives). This model that postulates the access to credit as beneficial, however, is naïve. Even though the loan-making process that relies on risk-based pricing has economic backing, by definition it implies that the less finanically advantaged people will get more expensive loans. Hence, an intervention that aims at increasing inclusion of disadvantaged populations in the lending process can be seen as an instance of predatory inclusion (Poon, 2016; Seamster and Charron-Chénier, 2017; Taylor, 2019). Even if it results in lower loan prices in the short term, it can lead to dispossession in the long run. When harms are viewed through this lens, It is not clear if any technological intervention is capable of counteracting such systems.

6. Discussion

Our inspiration to provide a more holistic view on optimization systems and their harms comes from works that point to the logic and potential impact of optimization systems. In particular, Poon has drawn attention to the ways in which optimization systems are driven by outcomes, as exemplified in our utility functions in Sec. 3. This allows for techniques like operational control and statistical management to be the primary mode with which machines interact with phenomena in the world (Poon, 2016). As a result, these techniques function both as a means for engineering and as “a mathematical state that poses as a solution to political contention” (McKelvey, 2018). Optimization is a technique long established in, for example, resource allocation. However, its increasing supremacy in systems that reconfigure every aspect of life (education, care, health, love, work etc.) for extraction of value is new and refashions all social, political, and governance questions into economic ones. This shift allows companies to commodify aspects of life in a way that conflates hard questions around resource allocation with maximization of profit and management of risk (Gandy, 2010). The impact is subtle but fundamental, as evident in the way even we start framing complex and historical questions of justice in terms of utility functions.

To ground ourselves in the world, we used the requirements engineering model of Michael A. Jackson, aligning it with calls for decentering technology and centering communities in the design of systems (Peña Gangadharan and Niklas, 2019). We extended the model to ensure that we are not suggesting that problems and solutions are neatly separable and void of power and political economy (Powles and Nissenbaum, 2018). However, many elements that might be crucial for conceptualizing optimization systems are still missing. The ontology offers few concepts to capture matters around data, machine learning, or services, and it does not provide deeper insights into addressing issues like fairness or justice. It is, however, a nod to the importance of ontological work for systematizing reflections on our frameworks (Corbett-Davies and Goel, 2018; Green and Hu, 2018).

To capture the political economy of optimization systems, we turned to utilitarian models and calculations of externalities. Such models are commonly used both in mathematical and managerial forms of optimization and are the cornerstone of neoclassical economics. However, utilitarian models have been thoroughly critiqued for, among others, perpetuating inequalities (Power, 2004). Most prominently, Sen has highlighted the limitations of assessing value through consequences, assessing value through subjective utility, maximizing welfare without regard for its distribution, and fetishizing resources over relations between resources and people (Sen, 1979). Overall, utilitarian approaches are weak in capturing collective interests, social well-being, forms of power, and subjugation. Given these critiques, the central role that these models play in designing large-scale optimization systems is a problem in and of itself.

One possible way forward is to consider alternative economic models for the design and evaluation of systems, e.g., (Sen, 1984; Diaz, 2015; Power, 2004; Fleurbaey et al., 2018). POTs depend and build on the existence of such alternative economic models and the availability of collectivity, altruism and reciprocity. They assume there is something to be gained both individually and collectively, dismissing the selfish agents presumed in utilitarian approaches. In fact, we struggled to express POTs in the utilitarian logic: if we optimize for the utility of the service provider it is hard to justify any POT that may reduce the utility of the service provider.

Beyond economic gains, POTs strategically support people’s agency. Optimization systems offer little agency to effectively contest their value proposition (Thomas, 2019; Myanmar Civil Society Groups (Phandeeyar, Mido, Burma Monitor, Center for Social Integrity, Equality Myanmar, Myanmar Human Rights Education Network), ; Miracle, 2018; Citron and Pasquale, 2014) and offer more optimization as solutions for externalities (Vincent, 2018). POTs can be used to exercise some agency towards an unaccountable (Pasquale, 2018) and “authoritative system” (Ziewitz, 2019). Nevertheless, service providers may argue that POTs are gaming the system. Our focus is on social-justice contexts, in which POTs can be cast as “weapons of the geek” for the least equipped to deal with the consequences of optimization (Coleman, 2017). POTs can also serve to expose systems’ injustices, achieving transparency and accountability goals. In that sense, they also can come to act like rhetorical software “that turn the logic of a system against itself […] to demonstrate its pathology” (Tseng, 2019). This includes making apparent the damaging results of utilitarian forms of governance prominent in optimization systems.

Despite their positive potential, designing and deploying POTs is not trivial. By virtue of modifying, subverting, or sabotaging an optimization system, POTs may elicit transitions in the system state that result in externalities. If several POTs are deployed and enter in an arms race, those agents with the most knowledge and resources are likely to deploy the most aggressive and effective POTs and have the most leverage. This undermines the ability of less powerful populations, who may need POTs the most, to have any effect on the system. This signals that well-thought POTs must be built to provide less powerful actors with the means to respond to the potential abuse of power by those that have more capabilities.

Just as much, the multi-input form of most optimization systems poses a serious challenge: when optimization is based on continuous tracking across many channels, POTs cannot be built short of creating “optimized doubles” of entities in the environments (Ruckenstein, 2014). The fact that a whole infrastructure for optimizing populations and environments is built in a way that minimizes their ability to object to or contest the use of their inputs for optimization is of great concern to the authors—and should be also to anyone who believes in choice, subjectivity, and democratic forms of governance.


We are indebted to Martha Poon for her original framing of the optimization problem. We also thank Ero Balsa for his collaboration on the previous versions of this work, Jakub Tarnawski for his invaluable help with finding the right tool for the traffic-routing case study, and Muhammad Bilal Zafar for his helpful comments.

This work was supported in part by the Research Council KULeuven: C16/15/058, and the European Commission through KULeuven BOF OT/13/070, H2020-DS-2014-653497 PANORAMIX, and H2020-ICT-2015-688722 NEXTLEAP. Seda Gürses is supported by a Research Foundation–Flanders (FWO) Fellowship. Rebekah Overdorf is supported by the Open Technology Fund–Information Controls Fellowship.

Appendix A Sensitivity to Misspecifications

We theoretically estimate the impact of misspecifications on the severity of externalities. For that, we use influence functions from the toolkit of robust statistics [48].

We assume that the utility functions , , and are strictly concave and twice-differentiable; and we strengthen the ideal property of the fair-by-design provider. Besides picking the Pareto-optimal solution that maximizes their model of social utility, we now assume the near-optimality of the social-utility objective for this solution: . That is, the system is close to the ideal solution for the social-utility objective  [65].

Consider a partially corrected optimization objective:

where we move a pointmass away from the provider’s model of the social utility to its “god’s view” value. Let us take its a Pareto-optimal solution that has highest benefit. We define the influence function of as follows:

This models how fast the magnitude of the externality grows as more weight is given to the corrected in the optimization problem.

Let us restate a known property of Pareto-optimal solutions due to Kuhn and Tucker [57]:

Theorem A.1 ([57]).

Let be a Pareto-optimal solution to the optimization problem of the form:

Then there exists , such that the following holds:

Let us denote by the Hessian matrix of at , and for convenience set . Additionally, denote by the difference in system parameters coming from the corrected and the original optimization problems. We can now present our estimate for the influence function.

Statement A.1 ().

Using linearization techniques, we can obtain the following approximation for the influence function for some :

Derivation Using Theorem A.1, we can say there exists such that:


We now rewrite Eq. A.1 in terms of the original system parameters using a first-order Taylor approximation:

We can rearrange and further approximate following Koh and Liang [56], keeping in mind that is small:

We now approximate the influence function using its first-order Taylor expansion and the obtained expression for :

Statement A.2 ().

Given the assumptions on and , our linear approximation for the influence function of is asymptotically lower bounded as follows:


As is negative-definite by the concavity assumption and the fact that convex combinations preserve concavity, so is its inverse. Hence, is positive-definite. By a lower bound of a symmetric positive-definite quadratic form we have:

For concave functions, can serve as a measure of error of the solution  [10], which confirms our intuition.

Appendix B Details for the Traffic Thwarting Case Study

b.1. MILP Formulation

With a simple reparameterization, it is possible to formulate the optimization problem as follows:


In this form, the optimization problem is the shortest-path interdiction problem [39, 36], and can be solved as an MILP [49, 104]:


where are additional vertex-potential variables that represent the smallest time cost for getting from to in graph .

Assume that each edge is associated with a length defined by , and a speed limit . In the case of changing the speed limits through , can be obtained from , and as follows:


b.2. Evaluation Details for Fremont, California

Figure 6. Solution for Fremont, California (in black)

The graph for Fremont, CA, USA, is much larger than for the other towns considered in our evaluation, with a total of 9,215 nodes and 19,313 edges. The normal time from to in the town is 8.5 minutes. Figure 6 shows the optimal set of roads to lower the speed limit on for , using a 75% decrease in time for the allowed road changes.

Appendix C Details for the Credit-Scoring Case Study

We detail the heuristic algorithm we use to solve the optimization problem of the POT in Algorithm 1. To compute the scores, we retrain a classifier for each example . In our case of the logistic regression as the bank’s model, retraining is inexpensive. For more complex models, approximation techniques can be used [22].

  1. for

  2. continue if

  3. simulate

  4. compute the score

  5. add to along with the computed score

  6. while :

  7. remove with the lowest score from

  8. add to R.

Algorithm 1 Algorithm for selecting poisoning loan applications in order to reduce the false-negative rate on the target group.


  1. journalyear: 2020
  2. copyright: acmcopyright
  3. conference: Conference on Fairness, Accountability, and Transparency; January 27–30, 2020; Barcelona, Spain
  4. booktitle: Conference on Fairness, Accountability, and Transparency (FAT* ’20), January 27–30, 2020, Barcelona, Spain
  5. price: 15.00
  6. doi: 10.1145/3351095.3372853
  7. isbn: 978-1-4503-6936-7/20/01
  8. Bogdan Kulynych and Rebekah Overdorf contributed equally to this work.
  9. Application domain does not refer to a class of applications, like health or banking domain, but to actors and things in the environment where the machine is introduced.


  1. AirBnB Note: \urlhttps://airbnb.com, visited 2019-12-05 Cited by: §2.1.
  2. A. Akhtar (2016) Is Pokémon Go racist? how the app may be redlining communities of color. USA Today. Note: \urlhttps://usatoday.com/story/tech/news/2016/08/09/pokemon-go-racist-app-redlining-communities-color-racist-pokestops-gyms/87732734/, visited 2019-12-05 Cited by: Table 1.
  3. M. Ali, P. Sapiezynski, M. Bogen, A. Korolova, A. Mislove and A. Rieke (2019) Discrimination through optimization: how Facebook’s ad delivery can lead to skewed outcomes. CoRR. Cited by: §2.1.
  4. J. Angwin, J. Larson, S. Mattu and L. Kirchner (2016) Machine bias: there’s software used across the country to predict future criminals. and it’s biased against blacks.. ProPublica. Note: \urlhttps://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing, visited 2019-12-05 Cited by: §1.
  5. S. Barocas, M. Hardt and A. Narayanan (2018) Fairness and machine learning. fairmlbook.org. Note: \urlhttp://www.fairmlbook.org, visited 2019-12-05 Cited by: §1.
  6. C. Barreneche and R. Wilken (2015) Platform specificity and the politics of location data extraction. European Journal of Cultural Studies. Cited by: §4.
  7. K. Benner (2016) Airbnb vows to fight racism, but its users can’t sue to prompt fairness. The New York Times. Note: \urlhttps://www.nytimes.com/2016/06/20/technology/airbnb-vows-to-fight-racism-but-its-users-cant-sue-to-prompt-fairness.html, visited 2019-12-05 Cited by: §3.1.1.
  8. B. Biggio and F. Roli (2018) Wild patterns: ten years after the rise of adversarial machine learning. Pattern Recognition. Cited by: §5.2.1.
  9. S. Bird, S. Barocas, K. Crawford and H. Wallach (2016) Exploring or exploiting? social and ethical implications of autonomous experimentation in AI. In Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT-ML), Cited by: §2.1.
  10. S. Boyd and L. Vandenberghe (2004) Convex optimization. Cambridge university press. Cited by: Appendix A.
  11. T. Brain and S. Mattu(Website) Note: \urlhttps://www.unfitbits.com, visited 2019-12-05 Cited by: Table 1.
  12. J. Bridle(Website) Note: \urlhttps://jamesbridle.com/works/autonomous-trap-001, visited 2019-12-05 Cited by: Table 1.
  13. S. Browne (2015) Dark matters: on the surveillance of blackness. Duke University Press. Cited by: §1.
  14. T. Cabannes, F. Shyu, E. Porter, S. Yao, Y. Wang, M. A. S. Vincentelli, S. Hinardi, M. Zhao and A. M. Bayen (2018) Measuring regret in routing: assessing the impact of increased app usage. In International Conference on Intelligent Transportation Systems, ITSC, Cited by: §3.
  15. L. E. Celis, D. Straszak and N. K. Vishnoi (2017) Ranking with fairness constraints. In ICALP, Cited by: §3.1.
  16. A. Chouldechova (2017) Fair prediction with disparate impact: a study of bias in recidivism prediction instruments. Big data. Cited by: §5.2.
  17. D. K. Citron and F. Pasquale (2014) The scored society: due process for automated predictions. Washington Law Review. Cited by: §5.2, §6.
  18. G. Coleman (2017) From internet farming to weapons of the geek. Current Anthropology. Cited by: §6.
  19. S. Corbett-Davies and S. Goel (2018) The measure and mismeasure of fairness: a critical review of fair machine learning. CoRR. Cited by: §6.
  20. C. J. Dahlman (1979) The problem of externality. The journal of law and economics. Cited by: §3.1.1.
  21. J. Dastin (2018) Amazon scraps secret AI recruiting tool that showed bias against women. Reuters. Cited by: §4.1.
  22. A. Demontis, M. Melis, M. Pintor, M. Jagielski, B. Biggio, A. Oprea, C. Nita-Rotaru and F. Roli (2019) Why do adversarial attacks transfer? Explaining transferability of evasion and poisoning attacks. In USENIX Security Symposium, Cited by: Appendix C, §5.2.1.
  23. D. Dheeru and E. Karra Taniskidou (2017) UCI machine learning repository. University of California, Irvine, School of Information and Computer Sciences. Note: \url= http://archive.ics.uci.edu/ml, visited 2019-12-05 Cited by: §5.2.2.
  24. N. Q. Diaz (2015-10) Decolonial feminist economics: a necessary view for strengthening social and popular economy. ViewPoint Magazine. Note: \urlhttps://doi.org/10.6084/m9.figshare.11336048 Cited by: §6.
  25. P. Dolan (1998) The measurement of individual utility and social welfare. Journal of Health Economics. Cited by: §4.
  26. M. Donini, L. Oneto, S. Ben-David, J. S. Shawe-Taylor and M. Pontil (2018) Empirical risk minimization under fairness constraints. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §3.1.
  27. C. N. dos Santos, I. Melnyk and I. Padhi (2018) Fighting offensive language on social media with unsupervised text style transfer. CoRR. Cited by: §4.1, Table 1.
  28. J. Eidelson (2019) These instacart workers want you to leave them a 22-cent tip. Bloomberg. Note: \urlhttps://www.bloomberg.com/news/articles/2019-01-17/these-instacart-workers-want-you-to-leave-them-a-22-cent-tip, visited 2019-12-05 Cited by: Table 1.
  29. D. Ensign, S. A. Friedler, S. Neville, C. Scheidegger and S. Venkatasubramanian (2018) Runaway feedback loops in predictive policing. In Conference on Fairness, Accountability and Transparency, FAT*, Cited by: §1, §2.1.
  30. M. Fang, G. Yang, N. Z. Gong and J. Liu (2018) Poisoning attacks to graph-based recommender systems. Cited by: §4.1, §4.1, Table 1.
  31. M. Fleurbaey, M. Ferranna, M. Budolfson, F. Dennig, K. Mintz-Woo, R. Socolow, D. Spears and S. Zuber (2018) The social cost of carbon: valuing inequality, risk, and population for climate policy. The Monist. Cited by: §6.
  32. J. Forrest, S. Vigerske, T. Ralphs, H. G. Santos, L. Hafer, B. Kristjansson, J. P. Fasano, E. Straver, M. Lubin, rlougee, jpgoncal1, h-i-gassmann and M. Saltzman (2019) Coin-or/cbc: version 2.10.3. Cited by: §5.1.1.
  33. E. F. Foundation A guide to youtube removals. Note: \urlhttps://www.eff.org/issues/intellectual-property/guide-to-youtube-removals, visited 2019-12-05 Cited by: §4.1.
  34. Z. Friedman (2017-05) How personal loans can boost your credit score.. Forbes. Note: \urlhttps://www.forbes.com/sites/zackfriedman/2017/05/25/personal-loans-credit-score/#79bce94175ba, visited 2019-12-05 Cited by: §5.2.1.
  35. Z. Fu, X. Tan, N. Peng, D. Zhao and R. Yan (2018) Style Transfer in Text: Exploration and Evaluation. In AAAI Conference on Artificial Intelligence, Cited by: §4.1, Table 1.
  36. D. R. Fulkerson and G. C. Harding (1977) Maximizing the minimum source-sink path subject to a budget constraint. Mathematical Programming. Cited by: §B.1, §5.1.1.
  37. O. H. Gandy (2010) Engaging rational discrimination: exploring reasons for placing regulatory constraints on decision support systems. Ethics and Information Technology 12 (1), pp. 29–42. Cited by: §6.
  38. O. H. Gandy (2016) Coming to terms with chance: engaging rational discrimination and cumulative disadvantage. Routledge. Cited by: §1.
  39. B. Golden (1978) A problem in network interdiction. Naval Research Logistics Quarterly. Cited by: §B.1, §5.1.1.
  40. M. Graehler, A. Mucci and G. Erhardt (2019) Understanding the recent transit ridership decline in major us cities: service cuts or emerging modes?. In Transportation Research Board 98th Annual Meeting, pp. . Cited by: §4.
  41. B. Green and L. Hu (2018) The myth in the methodology: towards a recontextualization of fairness in machine learning. In the ICML 2018 Debates Workshop, Cited by: §6.
  42. S. Gurses and J. van Hoboken (2018) Privacy after the agile turn. In The Cambridge Handbook of Consumer Privacy, E. Selinger, J. Polonetsky and O. Tene (Eds.), pp. 579–601. Cited by: §1.
  43. M. Hardt, E. Price and N. Srebro (2016) Equality of opportunity in supervised learning. In Advances in Neural Information Processing Systems (NeurIPS), Cited by: §5.2.3, §5.2.
  44. A. Harvey(Website) Note: \urlhttps://cvdazzle.com/, visited 2019-12-05 Cited by: §4.1, Table 1.
  45. A. Harvey(Website) Note: \urlhttps://ahprojects.com/hyperface/, visited 2019-12-05 Cited by: §4.1, Table 1.
  46. G. Hinsliff (2018) Airbnb and the so-called sharing economy is hollowing out our cities. The Guardian. Note: \urlhttps://www.theguardian.com/commentisfree/2018/aug/31/airbnb-sharing-economy-cities-barcelona-inequality-locals, visited 2019-12-05 Cited by: §2.1.
  47. D. C. Howe and H. Nissenbaum(Website) Note: \urlhttps://adnauseam.io, visited 2019-12-05 Cited by: §4.1, Table 1.
  48. P. J. Huber (2011) Robust statistics. Springer. Cited by: Appendix A.
  49. E. Israeli and R. K. Wood (2002) Shortest-path network interdiction. Networks: An International Journal. Cited by: §B.1, §5.1.1.
  50. M. Jackson (1995) Software requirements and specifications: a lexicon of practice, principles and prejudices. Addison-Wesley. Cited by: §2.
  51. M. Jackson (1995) The world and the machine. In International Conference on Software Engineering (ICSE), Cited by: §2.1, §2.
  52. Jobscan(Website) Note: \urlhttps://jobscan.co, visited 2019-12-05 Cited by: §4.1, §4.1, Table 1.
  53. I. Kaldrack and M. Leeker (2015) There is no software, there are just services. pp. 9–20. Cited by: §1.
  54. T. Kamishima, S. Akaho and J. Sakuma (2011) Fairness-aware learning through regularization approach. In IEEE International Conference on Data Mining Workshops, Cited by: §3.1.
  55. O. Keyes, J. Hutson and M. Durbin (2019) A mulching proposal: analysing and improving an algorithmic system for turning the elderly into high-nutrient slurry. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, Cited by: §3.1.2.
  56. P. W. Koh and P. Liang (2017) Understanding black-box predictions via influence functions. In International Conference on Machine Learning (ICML), Cited by: Appendix A.
  57. H. Kuhn and A. W. Tucker (2014) Nonlinear programming. In Traces and emergence of nonlinear programming, pp. 247–258. Cited by: Theorem A.1, Appendix A.
  58. F. Kunkle (2017) San Francisco police say most traffic tickets go to Uber and Lyft drivers. Washington Post. Cited by: §4.
  59. S. Lavigne, B. Clifton and F. Tseng (2017) Predicting financial crime: augmenting the predictive policing arsenal. CoRR. Cited by: §2.1.
  60. J. Le Grand (1990) Equity versus efficiency: the elusive trade-off. Ethics. Cited by: §3.1.
  61. L. T. Liu, S. Dean, E. Rolf, M. Simchowitz and M. Hardt (2019) Delayed impact of fair machine learning. In International Joint Conference on Artificial Intelligence, IJCAI, Cited by: §5.2.3.
  62. K. Lum and W. Isaac (2016) To predict and serve?. Significance. Cited by: §3.1.1.
  63. O. S. Map Note: \urlhttps://openstreetmap.org, visited 2019-12-05 Cited by: §5.1.2.
  64. F. McKelvey (2018) Internet daemons: digital communications possessed. U of Minnesota Press. Cited by: §6.
  65. K. Miettinen (2012) Nonlinear multiobjective optimization. Springer Science & Business Media. Cited by: Appendix A, §3.1.
  66. V. Miracle (2018-04) Los Angeles councilman tries to work with map apps to alleviate traffic in neighborhoods. ABC7. Note: \urlhttps://abc7.com/traffic/la-councilman-tries-to-work-with-map-apps/3329737/ , visited 2019-12-05 Cited by: §6.
  67. B. Mittelstadt, C. Russell and S. Wachter (2019) Explaining explanations in AI. In Conference on Fairness, Accountability, and Transparency (FAT*), Cited by: §1.
  68. Myanmar Civil Society Groups (Phandeeyar, Mido, Burma Monitor, Center for Social Integrity, Equality Myanmar, Myanmar Human Rights Education Network)(Website) Note: \urlhttps://drive.google.com/file/d/1Rs02G96Y9w5dpX0Vf1LjWp6B9mp32VY-/view, visited 2019-12-05 Cited by: §6.
  69. B. O’Shea (2018-04) What is a credit-builder loan?. Forbes. Note: \urlhttps://www.nerdwallet.com/blog/finance/what-is-credit-builder-loan/, visited 2019-12-05 Cited by: §5.2.1.
  70. ortools Note: \urlhttps://developers.google.com/optimization, visited 2019-12-05 Cited by: §5.1.1.
  71. R. Overdorf, B. Kulynych, E. Balsa, C. Troncoso and S. F. Gürses (2018) Questioning the assumptions behind fairness solutions. In Critiquing and Correcting Trends in Machine Learning, Cited by: §1.
  72. N. Papernot, P. D. McDaniel, A. Sinha and M. P. Wellman (2016) Towards the science of security and privacy in machine learning. CoRR. Cited by: §5.2.1.
  73. F. Pasquale (2018-08) Odd numbers. algorithms alone can’t meaningfully hold other algorithms accountable.. Real Life. Note: \urlhttp://reallifemag.com/odd-numbers/, visited 2019-12-05 Cited by: §6.
  74. S. Peña Gangadharan and J. Niklas (2019) Decentering technology in discourse on discrimination. Information, Communication & Society. Cited by: §6.
  75. L. Perelman(Website) Note: \urlhttp://lesperelman.com/writing-assessment-robo-grading/babel-generator/, visited 2019-12-05 Cited by: Table 1.
  76. L. Phlips (1988) The economics of imperfect information. Cambridge University Press. Cited by: §3.1.1.
  77. A. Pigou (1920) The economics of welfare. London:McMillan and Co.. Cited by: §3.1.1.
  78. M. Poon (2016) Corporate capitalism and the growing power of big data: review essay. Science, Technology, & Human Values 41 (6), pp. 1088–1108. Cited by: §5.2.3, §5.2, §6.
  79. M. Power (2004) Social provisioning as a starting point for feminist economics. Feminist economics 10 (3), pp. 3–19. Cited by: §3.2, §6, §6.
  80. J. Powles and H. Nissenbaum (2018)(Website) Note: \urlhttps://medium.com/s/story/the-seductive-diversion-of-solving-bias-in-artificial-intelligence-890df5e5ef53, visited 2019-12-05 Cited by: §6.
  81. A. Rosenblat, K. E. Levy, S. Barocas and T. Hwang (2017) Discriminating tastes: uber’s customer ratings as vehicles for workplace discrimination. Policy & Internet. Cited by: §4, §4.
  82. M. Ruckenstein (2014) Visualized and interacted life: personal analytics and engagements with data doubles. Societies. Cited by: §6.
  83. P. Saadatpanah, A. Shafahi and T. Goldstein (2019) Adversarial attacks on copyright detection systems. CoRR. Cited by: §4.1, Table 1.
  84. M. Sainato (2019) I made $3.75 an hour: Lyft and Uber drivers push to unionize for better pay. The Guardian. Note: \urlhttps://www.theguardian.com/us-news/2019/mar/22/uber-lyft-ipo-drivers-unionize-low-pay-expenses, visited 2019-12-05 Cited by: §4.
  85. A. Schmitt (2019) All the bad things about Uber and Lyft in one simple list. Note: \urlhttps://usa.streetsblog.org/2019/02/04/all-the-bad-things-about-uber-and-lyft-in-one-simple-list/, visited 2019-12-05 Cited by: §4.
  86. L. Seamster and R. Charron-Chénier (2017) Predatory inclusion and education debt: rethinking the racial wealth gap. Social Currents. Cited by: §5.2.3.
  87. A. Sen (1979) Utilitarianism and welfarism. The Journal of Philosophy. Cited by: §6.
  88. A. Sen (1984) Resources, values and development. Harvard University Press. Cited by: §6.
  89. M. Sharif, S. Bhagavatula, L. Bauer and M. K. Reiter (2016) Accessorize to a crime: real and stealthy attacks on state-of-the-art face recognition. In Conference on Computer and Communications Security (CCS), Cited by: §4.1, Table 1.
  90. S. Shkolnikova (2018) Leonia’s road closure ordinance ruled invalid. North Jersey Record. Note: \urlhttps://northjersey.com/story/news/bergen/leonia/2018/08/30/leonia-nj-road-closure-ordinance-ruled-invalid/1145418002/, visited 2019-12-05 Cited by: §5.1.
  91. D. A. Starrett (2011) Economic externalities. In Fundamental Economics Vol I., Cited by: §3.
  92. S. Sweeney (2019) Uber, Lyft drivers manipulate fares at reagan national causing artificial price surges. WLJA. Note: \urlhttps://wjla.com/news/local/uber-and-lyft-drivers-fares-at-reagan-national, visited 2019-12-05 Cited by: §4.1, Table 1, §4.
  93. K. Taylor (2019) Race for profit: how banks and the real estate industry undermined black homeownership. UNC Press Books. Cited by: §5.2.3.
  94. E. Thomas (2019-21. June) Facebook keeps failing in Myanmar. Foreign Policy. Note: \urlhttps://foreignpolicy.com/2019/06/21/facebook-keeps-failing-in-myanmar-zuckerberg-arakan-army-rakhine/, visited 2019-12-05 Cited by: §6.
  95. F. Tseng and S. Raspet(Website) Note: \urlhttps://matter.farm/about/, visited 2019-12-05 Cited by: Table 1.
  96. F. Tseng (2019-06) Sabotage, heresy and traps. Note: Conference Talk Cited by: §6.
  97. Z. Tufekci (2018) YouTube, the great radicalizer. The New York Times 10. Cited by: §2.2.
  98. uber Note: \urlhttps://uber.com, visited 2019-12-05 Cited by: §4.
  99. uber Uber: Cookie notice. Note: \urlhttps://www.uber.com/legal/privacy/cookies/en/, visited 2019-12-05 Cited by: §4.
  100. Uber (2019) The story of Innisfil. Note: \urlhttps://www.uber.com/en-CA/blog/the-story-of-innisfil/, visited 2019-12-05 Cited by: §4.
  101. J. Vincent (2018-04) Why AI isn’t going to solve Facebook’s fake news problem. The Verge. Note: \urlhttps://www.theverge.com/2018/4/5/17202886/facebook-fake-news-moderation-ai-challenges, visited 2019-12-05 Cited by: §6.
  102. G. Wang, B. Wang, T. Wang, A. Nika, H. Zheng and B. Y. Zhao (2018) Ghost riders: sybil attacks on crowdsourced mobile mapping services. IEEE/ACM Trans. Netw.. Cited by: Table 1.
  103. Waze Note: \urlhttps://www.waze.com, visited 2019-12-05 Cited by: §5.1.
  104. X. Wei, C. Zhu, K. Xiao, Q. Yin and Y. Zha (2018) Shortest path network interdiction with goal threshold. IEEE Access. Cited by: §B.1, §5.1.1.
  105. M. K. Wynia, D. S. Cummins, J. B. VanGeest and I. B. Wilson (2000) Physician manipulation of reimbursement rules for patients: between a rock and a hard place. Jama 283 (14), pp. 1858–1865. Cited by: Table 1.
  106. K. Yang and J. Stoyanovich (2017) Measuring fairness in ranked outputs. In Proceedings of the 29th International Conference on Scientific and Statistical Database Management, Cited by: §5.2.3.
  107. K. Yeung (2018) A study of the implications of advanced digital technologies (including AI systems) for the concept of responsibility within a human rights framework. MSI-AUT. Cited by: §1.
  108. N. Yew-Kwang (1983) Welfare economics: introduction and development of basic concepts. Springer. Cited by: §3.1.1.
  109. M. B. Zafar, I. Valera, M. Gomez Rodriguez and K. P. Gummadi (2017) Fairness beyond disparate treatment & disparate impact: learning classification without disparate mistreatment. In International Conference on World Wide Web (WWW), Cited by: §3.1.
  110. M. B. Zafar, I. Valera, M. Gomez-Rodriguez and K. P. Gummadi (2017) Fairness constraints: mechanisms for fair classification. In International Conference on Artificial Intelligence and Statistics, AISTATS, Cited by: §3.1.
  111. M. Ziewitz (2019) Rethinking gaming: the ethical work of optimization in web search engines. Social studies of science. Cited by: §6.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description