An Unethical Optimization Principle
Abstract
If an artificial intelligence aims to maximise riskadjusted return, then under mild conditions it is disproportionately likely to pick an unethical strategy unless the objective function allows sufficiently for this risk. Even if the proportion of available unethical strategies is small, the probability of picking an unethical strategy can become large; indeed unless returns are fattailed tends to unity as the strategy space becomes large. We define an Unethical Odds Ratio Upsilon () that allows us to calculate from , and we derive a simple formula for the limit of as the strategy space becomes large. We give an algorithm for estimating and in finite cases and discuss how to deal with infinite strategy spaces. We show how this principle can be used to help detect unethical strategies and to estimate . Finally we sketch some policy implications of this work.
pnasresearcharticle \leadauthorBeale \significancestatementThis paper formulates the Unethical Optimization Principle for AI and analytically quantifies the risk amplification involved. Under mild assumptions we show that an AI is almost certain to adopt an unethical strategy when the returns are Gaussian or have a similar thintailed distribution, and that although the probability that such a strategy is adopted decreases as the returns become heaviertailed, it is still appreciably higher than the incidence of unethical strategies in the strategy space as a whole. The implications for owners and regulators are that special care must be taken, but the Principle can also be used to help root out ethically problematic strategies \authorcontributionsNB had the initial idea, formulated the Principle, cowrote the paper, derived equation [4] from the analysis by AD and equation [5]. HB indicated that the extremal types theorem could be used to quantify the risk in wide generality. RM did the initial analysis, leading to formulating the problem in terms of the Odds Ratio. AD provided most of the analysis and cowrote the paper. All authors contributed importantly to the review and editing of the paper, and did extensive background analysis. \authordeclarationThe authors declare no conflict of interest. \correspondingauthor^{1}To whom correspondence should be addressed. Email: nicholas.beale@sciteb.com \datesThis manuscript was compiled on 12 November 2019 \doiwww.pnas.org/cgi/doi/10.1073/pnas.XXXXXXXXXX
Artificial intelligence (AI) is increasingly deployed in commercial situations. Consider for example using AI to set prices of insurance products to be sold to a particular customer. There are legitimate reasons for setting different prices for different people, but it may also be profitable to “game” their psychology or willingness to shop around. The AI has a vast number of potential strategies to choose from, but some are unethical — by which we mean, from an economic point of view, that there is a risk that stakeholders will apply some penalty, such as fines or boycotts, if they subsequently understand that such a strategy has been used. Such penalties can be huge: although these happened too early for an AI to be involved, the penalties levied on Banks for misconduct are currently estimated to be over $276 billion (see SI). In an environment in which decisions are increasingly made without human intervention, there is therefore a strong incentive to know under what circumstances AI systems might adopt unethical strategies. Society and governments are closely engaged in such issues. Principles for ethical use of AI have been adopted at national (1) and international (2) levels and the whole area of AI Ethics is one of very considerable activity (3, 4).
Ideally there would be no unethical strategies in the AI’s strategy space. But the best that can be achieved may be to have only a small fraction of such strategies being unethical. Unfortunately this runs up against the Unethical Optimization Principle, which we formulate as follows.
If an AI aims to maximise riskadjusted return, then under mild conditions it is disproportionately likely to pick an unethical strategy unless the objective function allows sufficiently for this risk.
Problem formulation
The following is a deliberately oversimplified representation that emphasises certain aspects and ignores others. Consider an AI that is searching a strategy space for a strategy that maximises the riskadjusted return for its owners. It does this by attempting to maximise its estimate of apparent riskadjusted return function , which we treat as random because it is based on potentially noisy data — for example data from existing clients who are themselves taken from a much larger number of potential clients. However, unknown to the AI, certain strategies in would be considered unethical by stakeholders, who in the future may impose a penalty for adopting them. Such penalties may be fines, reparations/compensation or boycotts: what they have in common from our point of view is that they have a nonzero riskadjusted expected cost which we denote by . We will call the subset of for which “unethical” or , and the complementary subset, for which , “ethical” or . Hence the true riskadjusted return may be expressed as
(1) 
where the ‘error’ accounts for other differences between and even when , due to imperfections in the algorithm’s ability to predict the future accurately.
Let denote the probability that the chosen strategy is unethical, and assume there is some measure on so that one could in principle compute the proportion of that is Red. The strategies comprise the remaining proportion of . Then we can define an Unethical Odds Ratio, Upsilon, as:
(2) 
which represents the increase in odds of choosing an unethical strategy by using the AI, relative to choosing a strategy at random. If is small, then will not represent a significant increase in risk due to use of the AI, whereas if then the AI acts as a significant unethical amplifier. If regulation reduces to 0.05 (or 0.01), for example, having would mean (or 0.09).
Unless there is a difference in the distribution of between the Red and Green regions or the mean returns are infinite, the expected riskadjusted returns in and satisfy
where means the average value of for .
Moreover if varies within , the corresponding standard deviations will satisfy . Thus returns for strategies in will have higher means and variabilities than those in . Suppose that the mean estimated return in is larger than that in , and that the estimated standard deviation in is a factor larger than that in . The tradeoff between returns from ethical and unethical strategies will depend on , and and on the tail of the distribution of returns.
Asymptotic strategy space
Suppose that the strategy space contains strategies, of which are unethical and are ethical. Let and respectively denote the maximum returns for strategies in and . In many cases the maximum of a random sample of size from a distribution can be renormalized using sequences and in order that converges as to a limiting random variable having a generalized extremevalue distribution. This distribution has a tail index parameter that controls the weight of its righthand tail, with increasing corresponding to fatter tails; it includes the Gumbel distribution as a special case for . Following the discussion above, we can write and , where and are respectively the maxima of and mutually independent variables from , and we suppose that and converge to variables and , which are independent and have the same generalized extremevalue distribution. In the Supporting Information we obtain general expressions for the limiting probability under mild conditions, and compute and for some special cases:

if is Gaussian, then the limiting variables and are Gumbel, and if , or both are positive;

if is lognormal or exponential, then the limiting variables and are Gumbel and if ;

if is Pareto, i.e., for and , then and have Fréchet distributions with tail indexes , and
(3) which yields
(4) and

if is Student with degrees of freedom, then the Pareto limit applies.
The significance of these results is that if the strategy space is large, then unless the distribution of the returns is fattailed, as in the cases of the Pareto or distributions, a responsible regulator or owner should be extremely cautious about allowing AI systems to operate unsupervised in situations with real consequences. If the returns are fattailed, then (4) gives some idea of the risk of using an unethical strategy.
Results for finite strategy space
For large but finite there is a simple and widelyapplicable algorithm to estimate . Numerical experiments show that its limiting value is reached quite rapidly for fattailed distributions, whereas grows roughly as for Gaussian returns.
Figure 2 shows how the finitesample unethical odds ratio depends on for some special cases. In the Gaussian case the probabilities approach unity most rapidly when the volatility is inflated, i.e., , and the Unethical Odds Ratio appears to be ultimately loglinear in . In the case of Student returns with degrees of freedom, the probabilities overshoot their asymptotic values when , and the asymptote (4) is approached rather slowly.
Infinite strategy spaces and correlated returns
So far we have discussed finite strategy spaces in which the returns for each strategy are independent. For many purposes this may be enough: if the asymptotic values of and are known it may be irrelevant whether is or . However there may be an effective upper bound on even when is infinite, if is viewed as a stochastic process with state space . For example, if there is a metric on and there are correlations between neighbouring points. Understanding the best approach in particular cases will depend on knowing the structure of and of , but these are the one part of the system that are essentially under the control of the AI and therefore the least imponderable. The more that is known about and the better one can estimate the effective value of . We discuss this further in the SI.
Estimating the parameters
The Unethical Optimisation Principle can help risk managers and regulators to detect unethical strategies. Consider a reasonably large sample . Manually examining for potential unethical elements may be prohibitively expensive if this requires human judgement. Suppose however that we rank the elements of by their values of and focus our attention on the subset with the largest values of , where . We assume that careful manual inspection can divide this set into and elements and write . By (2) we then have an estimator
(5) 
which allows a rough estimate of given and . Perhaps more importantly, focusing on to find examples of unethical strategies that might be adopted not only weeds out those most likely to be used, but will help develop intuition on where problems might be found. Observing the bulk distribution of gives an idea of overall shape of and an idea of . To generate reasonably robust estimates of and it will generally be necessary to do some more manual inspection of another subset of to determine and elements but this can be relatively small if well targeted. Details are discussed in the SI.
Implications
Practical advice to the regulators and owners of AI is to sample the strategy space and observe whether the returns have a fattailed distribution. If not, then the “optimal” strategies are likely to be unethical regardless of the value of . If, however, the observed return distribution is fattailed, then the tail index can be estimated using standard techniques (5, 6) and can be estimated as discussed above. However, it would be unwise to place much faith in the precision of such estimates: there are so many imponderable factors that the main point is to avoid sailing close to the wind. In addition the Principle can be used to help regulators, compliance staff and others to find problematic strategies that might be hidden in a large strategy space.
The Principle also suggests that it may be necessary to rethink the way AI operates in very large strategy spaces, so that unethical outcomes are explicitly rejected in the optimisation/learning process.
This work was supported by the Swiss National Science Foundation, the UK Engineering and Physical Sciences Research Council and Capital International. We thank Prof He Ping, Deputy Governor Pan Gonsheng and Alex Brazier for organising seminars at Tsinghua School of Economics and Management, the PBOC/SAFE and the Bank of England in March and April 2019 where NB presented the initial ideas that led to this paper. RM is grateful to the Alan Turing Institute for a Fellowship TU/B/000101 that helped enable this collaboration. We thank Andrew Bailey, Karen Croxson, and Wolfram Peters for helpful discussions.
References
 (1) UK Government Data Ethics Framework (2019).
 (2) OECD Principles on AI (2019).
 (3) N Bostrom, E Yudkowsky, Ai ethics in Cambridge Handbook of Artificial Intelligence, eds. W Ramsey, K Frankish. (Cambridge University Press), (2011).
 (4) V Dignum, Ethics in artificial intelligence: Introduction to the special issue. \JournalTitleEthics and Information Technology 20, 1–3 (2019).
 (5) P Embrechts, C Klüppelberg, T Mikosch, Modelling Extremal Events for Insurance and Finance. (Springer, Berlin), (1997).
 (6) SG Coles, An Introduction to Statistical Modeling of Extreme Values. (Springer, New York), (2001).
 (7) N Megaw, The UK’s slowburn £50bn banking scandal. \JournalTitleFinancial Times (2019).
 (8) MR Leadbetter, G Lindgren, H Rootzén, Extremes and Related Properties of Random Sequences and Processes. (Springer, New York), (1983).
 (9) RA Fisher, LHC Tippett, Limiting forms of the frequency distributions of the largest or smallest member of a sample. \JournalTitleProceedings of the Cambridge Philosophical Society 24, 180–190 (1928).
 (10) AC Davison, RL Smith, Models for exceedances over high thresholds (with Discussion). \JournalTitleJournal of the Royal Statistical Society, series B 52, 393–442 (1990).
 (11) AC Davison, Statistical Models. (Cambridge University Press, Cambridge), (2003).
 (12) R Core Team, R: A Language and Environment for Statistical Computing (R Foundation for Statistical Computing, Vienna, Austria), (2018).
 (13) SG Coles, J Heffernan, JA Tawn, Dependence measures for extreme value analyses. \JournalTitleExtremes 2, 339–365 (1999).
 (14) RA Davis, T Mikosch, The extremogram: A correlogram for extreme events. \JournalTitleBernoulli 15, 977–1009 (2009).
 (15) MR Leadbetter, On a basis for ‘Peaks over Threshold’ modeling. \JournalTitleStatistics & Probability Letters 12, 357–362 (1991).
Supporting Information Appendix (SI)
Recent penalties in Financial Services
The Financial Times listed (7) the major sets of fines and penalties levied on Western Banks for various forms of misconduct. There were 11 types of misconduct and the fines and penalties totaled $276Bn. Penalties (including compensation) for Payment Protection Insurance totaled $62Bn and was the second largest category.
Derivation of limiting
The extremal types theorem (8, Theorem 1.4.2) implies that in wide generality, the maximum of a random sample with cumulative distribution function may be renormalized using sequences and in order that converges as to a limiting random variable having a generalized extremevalue distribution. A simple sufficient condition for this is that is twice continuously differentiable with density and that the reciprocal hazard function is such that converges to a constant as approaches the upper support point of . Then we can take , and the distribution of is
(6) 
where ; setting gives the Gumbel distribution . The quantity , sometimes called the tail index, typically satisfies , with smaller values corresponding to lighter tails. If , then the limiting density has an upper support point at , whereas if then the limiting density has no finite upper support point, so the limiting random variable has no upper bound.
This implies that we can write for sufficiently large , where the quality of the approximation depends on ; it has long been known that the convergence is extremely slow for Gaussian variables (9). A result of Khintchine (8, Theorem 1.2.3) implies that if and for some fixed , then as ,
with when .
To apply these results, let denote the maximum of independent random variables with common distribution function , which represent the returns of ethical, Green, strategies, and suppose that converges in distribution to a random variable as . Let denote the maximum of independent random variables representing the returns of unethical, red, strategies. We suppose that is a random sample from and that and quantify the increase in mean return and in volatility for unethical returns. We briefly discuss the case where the and have different distributions below. Then
where means ‘has the same distribution as’, and as , will converge in distribution to a random variable with the same distribution as .
If is large enough, then we can write , and so the probability that the best return from an unethical strategy exceeds the best return from a ethical one satisfies
as , where depends on , , and the normalising sequence for .
We now discuss the behaviour for large of
(7) 

If , then and , so . In this case the distributions of and become more and more concentrated for large , and any advantage for red leads to it beating green with probability one, in the limit, because red returns have a higher upper limit than green ones.

If , then as , so , which is infinite if . The behaviour of depends on the limit of as . For example, if is exponential, then converges to a constant, whereas if is Gaussian, then . For exponential maxima, therefore, is infinite if , but is finite if , for any . For Gaussian maxima, and , so if either of or is positive, i.e., if there is any systematic advantage for red strategies.
Other limits might appear when and depend on , but one would need to consider whether this is realistic; for example, this might apply if , i.e., red strategies are a vanishingly small fraction of all possible ones. This does not seem very realistic, since presumably any ethical strategy could be tweaked slightly to make it more profitable but unethical.
Here are the details for the special cases in the main text.

If is Gaussian, then we can take and , giving , so and . The limiting variables and are Gumbel, and red will beat green if either or is positive.

If is logGaussian, then we can take and , so , and . The limiting variables and are Gumbel. Here and , so red always beats green, owing to its higher volatility.

If is exponential, then , and , so and are Gumbel, , and
tends to infinity unless : red beats green in the limit owing to its higher volatility.
The discussion above presupposes that the red and green returns only differ by a location and/or scale shift. If the limiting variables have the same support but different tail indexes, then the variable with the higher asymptotically dominates the other: if has a higher tail index than , then red returns will beat green returns with probability one for large .
Estimation
To estimate the distributions for the ethical and unethical strategies, we suppose that the sampled strategies with the highest riskadjusted returns have been divided into unethical and ethical strategies, with respective returns and , and we denote by the largest sampled return that is not among these . In our asymptotic framework the generalized Pareto distribution (GPD) (10) provides a suitable probability model for and , i.e., the ‘excess’ returns over . The probability density functions for the red and green excesses are
for and . The shape parameter is the same as in (6), and are scale parameters. The effect of changes in both and appears in the ratio , which will be larger than unity if there is an advantage for red returns, whereas should be the same for red and green subsets. This last property is helpful: can be hard to estimate from small samples, but inference for it will be based on all of the largest returns. The adequacy of the GPD is readily checked using standard techniques (6, Ch. 4), and the parameters can be estimated, and models compared, using standard likelihood methods (11, Ch. 4).
Having obtained estimates , and , we estimate by Monte Carlo simulation as follows. We generate standard uniform variables and Poisson variables with mean , all mutually independent. We then compute , for , and estimate by
where denotes the fitted cumulative distribution function for the green exceedances over , which is generalized Pareto with parameters and . In the simulations described below we took , which reduces variation in to the third decimal place.
We performed a small simulation experiment to check these ideas. For different settings with normal and returns, we simulated 10,000 samples, each with and . We constructed each sample by generating , and then made red returns , with the green returns being . We took the largest returns for each sample, ascertained whether they were red or green, and obtained , and . We then fitted the GPD to the entire sample of excesses, and to the red and green excesses separately, using a common value of ; this enabled us to compute the likelihood ratio statistic for testing whether , based on the largest returns; the proportion of times this is rejected is the statistical power for testing the hypothesis at a nominal 5% significance level. If the return distributions differ greatly, then this power should be high. We also computed the empirical value of , based on whether the largest return in each sample was red or green, which would not be useful in practice, as it would equal either 0 or 1, based on the single sample available. As estimates of we computed the empirical proportion and the estimate described above, both of which would be available in practice.
Power  

Normal  0  0  10.2  10.0  13.4  5.9 
0.5  0  41.4  25.7  47.7  19.3  
0  0.2  54.0  20.0  57.5  46.4  
0.5  0.2  86.8  38.6  90.3  90.4  
0  0  9.8  10.0  12.8  5.2  
0.5  0  20.4  21.3  25.4  5.4  
0  0.2  33.7  18.3  37.6  20.1  
0.5  0.2  50.1  32.1  58.4  33.0 
Table 1 summarises the results of this experiment. The rows with show that and are both close to the expected value of 10% when there is no difference between red and green returns, and the power is close to the anticipated value, 5%. Although increases when either of or is positive, it generally has a downward bias, and appears to provide a better estimate of . On the other hand computations not shown indicate that can be highly variable, though taking reduces its variance. The power increases when or is positive, as predicted by the asymptotic theory; the power shows that when and , for example, a difference between red and green returns can be detected in around 91% of samples. For the returns, and its estimates again increase, but more modestly, and more for increased volatility, , than for increased mean, . Again, this corresponds to the asymptotic theory.
Computation of
Let and . It is straightforward to check that
which can be estimated by Monte Carlo simulation as follows:

generate , then set for ;

compute an estimate
of ;

repeat the steps above, with replaced by to give an estimate ;

return as an estimate of .
The first step uses inversion to generate maxima directly from , the second step averages the exact probabilities , and the third and fourth steps use antithetic sampling to reduce the variance of . With this gives probabilities accurate to three decimal places almost instantaneously. The R (12) code below embodies this.
prob.sim < function(S, eta, delta, gamma, R=10^5) { # F is distribution function and Finv its inverse n < (1eta)*S m < eta*S u < runif(R) x < Finv( u^(1/m) ) m1 < mean( F(delta+(1+gamma)*x)^n ) x < Finv( (1u)^(1/m) ) m2 < mean( F(delta+(1+gamma)*x)^n ) (m1+m2)/2 }
Highprecision arithmetic may help in computing more accurately for very large , though its precise value is rarely crucial.
Infinite strategy spaces and correlated returns
As one example of the kind of approach discussed in the paper, consider the following:
Let denote the copula that determines the dependence of random variables and having uniform marginal distributions. One standard measure of extremal dependence is (13)
where is of most interest in the present context. If , then and are said to be asymptotically dependent, with corresponding to total dependence and to socalled asymptotic independence. The quantity can be roughly interpreted as the equivalent number of independent extremes at high levels of , so yields one ‘equivalent independent’ variable, and yields two ‘equivalent independent’ variables. Rankbased estimators for from independent data pairs are available for high values of , e.g., . As these are based on the ranks, the marginal distributions of and are irrelevant.
To apply these ideas, suppose that can be treated as a stationary process, that there is a measure of distance on , and evaluate on an equispaced grid, at , say. Thus we can observe the joint properties of at distances and so forth, taking and for each in the grid. If we take all such distinct pairs a distance apart and estimate as described above, then we can assess the dependence of the extremes of the process at lag , for example by plotting the estimate against . This extremogram (14) will equal unity for , and should drop to zero as increases, and thus can be used to assess the approximate number of equivalent independent values in .
To illustrate this, we took , created a function by linear interpolation between independent Gaussian variables at , and evaluated on a grid with random initial value and . Figure 3 shows these plots for four simulated functions. The sampling properties of for large mimic those for the usual time series correlogram in the presence of strong dependence and are not good, but the sharp decline near the origin shows precisely the behaviour we expect; it appears that extreme values of would be independent of those for or perhaps , as we would anticipate from its construction. Thus if we sampled at sites no closer than two units apart, the corresponding values of could be taken as independent at extreme levels.
Although further refinement is certainly feasible, the discussion above strongly suggests that it should be possible to identify an approximate number of ‘independent’ extrema in an infinite strategy space, under assumptions similar to those above, perhaps using a development of the ideas in Leadbetter (15).