Uncertainty in Cyber Security Investments
Abstract
When undertaking cyber security risk assessments, we must assign numeric values to metrics to compute the final expected loss that represents the risk that an organization is exposed to due to cyber threats. Even if risk assessment is motivated from realworld observations and data, there is always a high chance of assigning inaccurate values due to different uncertainties involved (e.g., evolving threat landscape, human errors) and the natural difficulty of quantifying risk per se. Our previous work Fielder2016 () has proposed a model and a software tool that empowers organizations to compute optimal cyber security strategies given their financial constraints, i.e., available cyber security budget. We have also introduced a general gametheoretic model Rass2015 () with uncertain payoffs (probabilitydistributionvalued payoffs) showing that such uncertainty can be incorporated in the gametheoretic model by allowing payoffs to be random. In this paper, we combine our aforesaid works and we conclude that although uncertainties in cyber security risk assessment lead, on average, to different cyber security strategies, they do not play significant role into the final expected loss of the organization when using our model and methodology to derive this strategies. We show that our tool is capable of providing effective decision support. To the best of our knowledge this is the first paper that investigates how uncertainties on various parameters affect cyber security investments.
keywords:
Cyber security investments, uncertainty, game theory.1 Introduction
Many organizations do not have a solid foundation for an effective information security risk management. As a result, the increasingly evolving threat landscape in combination with the lack of appropriate cyber security defenses poses several and important risks. On the other hand, the implementation of an optimal cyber security strategy (i.e., formal information security processes; technical mechanisms; and organizational measures) is not a straightforward process. In particular Small and Medium Enterprises (SMEs) are a priority focus sector for governments’ economic policy. Given that the majority of SMEs are restricted by limited budgets for investing in cyber security, the situation becomes cumbersome, as without cyber security mechanisms in place, they may be significantly impacted by inadvertent attacks on their information systems and networks leading, in most cases, to undesirable business effects.
Yet, it is not only the limited budgets. Even if these are available to some extent, investing in cyber security is challenging due to the evolving nature of cyber threats that introduces serious uncertainties when undertaking cyber security risk assessments. This asymmetry can highlight an investment decision from optimal to inefficient due to: (i) exploitation of newly found vulnerabilities that were not patched by the latest investment; and/or (ii) the mistaken values to risk assessment parameters, which lead to erroneous optimal cyber security strategies. The purpose of this paper is exactly that; “to investigate how uncertainties in conducting cyber security risk assessment affect cyber security investments”.
1.1 Cyber security investments
According to a 2017 IBM report ibm2017report (), despite the decline (10% percent) in the overall cost of a data breach over previous years to $3.62 million, companies in this year’s study are having larger breaches. A study conducted by the Ponemon Institute ponemon2015cost (), in 2015, on behalf of the security firm Damballa shows that although businesses spend an average of $1.27 million annually and 395 peoplehours each week responding to false alerts, thanks to faulty intelligence and alerts, breaches have actually gone up dramatically in the past three years.
The main challenges faced by organizations when it comes to investing in cyber security can be summarized as follows:

lack of methods of determining accurate values for risk assessment parameters;

complexity of developing a holistic methodology that models an organization’s environments, performs risks assessment and finally derives an optimal investment solution; and

new threats emerge changing the level of risk derived prior to their appearance and therefore making the most recent investment nonoptimal.
The literature of economics of security is quite rich and it comes to methodologies for investing in cyber security Lee2011 (); chronopoulos2017options (); benaroch2017real (); gordon2015increasing (); moore2016identifying (). In our previous works Fielder2016 (), Panaousis2014 () we compared different decision support methodologies for security managers to tackle the challenge of investing in security for SMEs. To undertake the risk assessment of the proposed model, we used fixed values for the payoffs of the players (i.e., defender and attacker). These values were set by using a mapping from the SANS Critical Security Controls web:SANS () combined with the Common Weakness Enumeration (CWE) Top 25 Software Vulnerabilities web:CWE (). The data for this paper was published here Casestudy2015 (). Although the use of data from wellknown sources made our risk assessment valid and important, this approach ignored the fact that in realworld scenarios there is a very high amount of uncertainty when setting the payoff values. And in fact, even the data used in Fielder2016 (), is just as accurate as the activities taken by experts when defining these values. But such activities are prone to error due to: (i) being subjective to the human experience each time; (ii) the evolving threat landscape that unavoidably dictates new risk assessment values; and (iii) new assets being added to an organization’s environment (i.e., infrastructure) therefore altering the current security posture of the organization.
1.2 Decision under uncertainty
Decision problems often involve uncertainty about the consequences of the potential actions. Currently existing decision support methods use to either ignore this uncertainty or reduce existing information (e.g., by aggregating several values into a single number) to simplify the process. However, such approaches lose a lot of information. In Rass2015 (), we introduce a game theoretic model where the consequences of actions and the payoffs are indeed random and, consequently, they are described as probability distributions. Even though the full space of probability distributions cannot be ordered, a subset of suitable loss distributions that satisfy a few mild conditions can be totally ordered in a way that agrees with a general intuition of risk minimization. We show that existing algorithms from the case of scalarvalued payoffs can be adapted to the situation of distributionvalued payoffs. In particular, an adaption of the fictitious play algorithm allows computation of a Nash equilibrium for a zerosum game. This equilibrium then represents the optimal way to decide among several options.The model is described in more depth and illustrated with an example in Rass2016 ().
An area where such a framework is particularly useful is risk management. Risk is often assessed by experts and thus depends on many factors, including the risk appetite of the person doing the assessment. Additionally, the effects of actions are rarely deterministic but rather depend on external influences. Therefore, it is recommended by the German Federal Office of Information Security to do a qualitative risk assessment which is consistent with our approach. We have applied the framework to model security risks in critical utilities such as a water distribution system in Busby2016 (). In this situation, consequences are difficult to predict as consumers are not homogeneous and thus do not act like a single (reasonable) person. Another situation that can be modeled with this generalized gametheoretic approach is that of an advanced persistent threat (APT) Rass2017 (). Recently, this type of attack has gained a lot of attention due to major incidents such as Stuxnet Karnouskos2011 () or the attack on the Ukrainian power grid eisac_analysis_2016 ().
2 Proposed Methodology
Our work is inspired from two previous papers Fielder2016 () and Rass2015 () to investigate how uncertainties regarding cyber security risk assessment values affect the efficiency of cyber security investments that have been built upon gametheoretic and combinatorial optimization techniques (a multiobjective multiple choice Knapsack based strategy). These uncertainties are reflected on the payoffs of the organization (henceforth refered to as the Defender). Although Fielder2016 () was proven interesting and validated the UK’s government aforesaid advice, it certainly did not account for uncertainties in the payoffs of the Defender. In real world scenarios, defenders almost always operate with incomplete information, and often a rough estimate on the relative magnitude of known cyber threats is the only information available to the cyber security managers. Furthermore, practical security engineers will argue that it is already difficult to obtain detailed information on risk assessment parameters. We envisage that by merging these two approaches, we will be able to offer a decision support tool for cyber security investments with increased resiliency against threats facing SMEs. More importantly, our work addresses a wider class of cyber threats than commodity cyber threats, which were investigated in Fielder2016 (). Although this assumption does not negate the possibility of zeroday vulnerabilities, it removes the expectation that it is in the best interest of either player to invest heavily in order to either find a new vulnerability or be able to protect against these unknown vulnerabilities. Therefore, in the present paper, we address even cyber attacks that target an organization with all means (i.e., advanced persistent threats).
2.1 Cyber security Control Games with Uncertainty
The Cyber security Control Games (CSCGs) developed so far Fielder2016 () do not yet capture a problem that often arises in real life and especially in cyber security: a crisp prediction of the efficacy of cyber security controls as well as the values of the various other risk assessment parameters is often not possible. Rather, some intuitive information is available that describe some values as more likely than others. In this paper, we enrich the model recently presented in Fielder2016 () by considering uncertainty in payoffs of the Defender (and of the Attacker since we play a zero sum game) in CSCG. This is a twostage cyber security investments model that supports security managers with decisions regarding the optimal allocation of their financial resources in presence of uncertainty regarding the different risk assessment values.
For a specific set of targets of the Attacker and security controls to be implemented by the Defender, our approach to cyber security risk assessment consists of two main steps. First, a zerosum CSCG is solved to derive the optimal level at which the control should be implemented to minimize the expected damage if a target is attacked. This game accounts for uncertainty about the effectiveness of a control using probabilitydistribution as payoffs instead of crisp numbers. In previous work Rass2015 (), we show that imposing some mild restrictions on these distributions admits the construction of a total ordering on a (useful) subset of probability distributions which allows to transfer solution concepts like the Nash equilibrium to this new setting.
The most critical part in estimating the damage caused by a cyber security attack is predicting the efficacy of a control to protect a target . Let us assume that we decide to implement the control at some level ; then we denote the efficacy of the control to protect target as . Typically, it is difficult to estimate this value, even if and are known. Thus, we replace the exact value of by a Gaussian distribution centered around the most likely value with a fixed variance . For simplicity, we assume that the uncertainty is equal for each cyber security control and implementation level. This assumption can be relaxed if we have obtained an accurate value about the efficacy of a cyber security process (i.e., a control implemented at some level). In order to avoid negative efficacy, we truncate the Gaussian distributions to get a proper probability distribution on . Allowing the efficacy of an implementation of a control at level on target to be random yields a random cyber security loss . This is the expected damage (e.g., losing some data asset) that the Defender suffers when is attacked and a control has implemented at level . This definition of loss is in line with the wellknown formula, risk = expected damage probability of occurrence Oppliger2015 (). We assume that this loss will take values in a compact subset of . The losses in our games are thus random variables, so at this point, we explicitly deviate from the classical route of game theory. In particular, we do not reduce the random payoffs to expected values or similar realvalued representatives. Instead, we will define our games to reward us in terms of a complete probability distribution, which is convenient for several reasons:

working with the entire probability distribution preserves all information available to the modeler when the games are defined. In other words, if empirical data or expertise on losses or utilities is available, then condensing it into a humble average sacrifices unnecessarily large amounts of information;

it equips the modeler with the whole armory of statistics to define the payoff distribution, instead of forcing the modeler to restrict oneself to a “representative value”. The latter is often a practical obstacle, since losses are not always easily quantifiable nor expressible on numeric scales (for example, if the game is about critical infrastructures and if human lives are at stake, a quantification in terms of “payoff” simply appears inappropriate).
Note that uncertainty in our case is essentially different to the kind of uncertainty that Bayesian or signaling games capture. While the latter is about uncertainty in the opponent the uncertainty in our case is about the payoff itself. The crucial difference is that Bayesian games nonetheless require a precise modeling of payoffs for all players of all types. This is only practically feasible for a finite number of types (though theoretically not limited to this). In contrast, our games embody an infinitude of different possible outcomes (types of opponents) in a single payoff, thus simplifying the structure of the game back into a standard matrix game, while offering an increased level of generality over Bayesian or signaling games.
In CSCG (a matrix game), Defender and Attacker have finite pure strategy spaces (where ) and a payoff structure of the Defender, denoted by , which in light of the uncertainties intrinsic to cyber security risk assessment, is a matrix of random variables. During the gameplay, each player takes its actions at random, which determines a row and column for the payoff distribution . Repeating the game, each round delivers a different random payoff whose distribution is conditional on the chosen scenario . Thus, we obtain the function . By playing mixed strategies, the distribution of the overall expected random payoff is obtained from the law of total probability by
(1) 
when are the mixed strategies supported on and the player’s moves are stochastically independent (e.g., no signaling).
Unlike classical repeated games, where a mixed strategy is chosen to optimize a longrun average revenue, equation (1) optimizes the distribution , which is the same (identical) for every repetition of the game. The game is in that sense static, but (unlike its conventional counterpart) does not induce repetitions in practice, since the payoffs are random (in each round), but all having the same distribution. Thus, the “distributionvalued payoff” is always the same (whether there are repetitions of the game or not).
2.2 Investment Optimization Problem with Uncertainty
When having cyber security controls, our plan for cyber investment is to solve CSCGs by splitting each of them up to a set of control subgames with targets and up to implementation levels for each control, where (we set to indicate that the control is not implemented at all). For a CSCG the Control Subgame equilibria constitute the CSCG solution Fielder2016 (). Given the Control Subgame equilibria we then use a Knapsack algorithm to provide the general investment solution. The equilibria provide us with information regarding the way in which each security control is best implemented, so as to maximize the benefit of the control with regard to both the ’s strategy, and the indirect costs of the organization. For convenience, we denote the Control Subgame solution by the maximum level of implementation available. For instance, for control the solution of Control Subgame is denoted by . Let us assume that for control the equilibria of all Control Subgames are given by the set . For each control there exists a unique Control Subgame solution , which dictates that control should not be used.
We define an optimal solution to the Knapsack problem as
.
A solution takes exactly one solution (i.e., equilibrium or cyber security plan) for each control as a policy for implementation. To represent the cyber security investment problem, we need to expand the definitions for both expected damage and effectiveness to incorporate the Control Subgame solutions. Hence, we expand such that , which is the expected damage on target given the implementation of . Likewise, we expand the definition of the effectiveness of the implemented solution on a given target as . Additionally, we consider as the direct cost of implementing . If we represent the solution by the bitvector , we can then represent the 01 Multiple Choice, MultiObjective Knapsack Problem as presented in (2.2).
(2) 
where is the available cyber security budget, and when . In addition, we consider a tiebreak condition in which if multiple solutions are viable, in terms of maximizing the minimum, according to the above function we will select the solution with the lowest cost. This ensures that an organization is not advised to spend more on security than would produce the same net effect. In Fig. 1, we have illustrated the overview of the methodology followed to provide an optimal cyber security advice supporting decision makers with deciding about optimal cyber security investments.
3 Experiments
The results presented here represent the outcomes of experiments run using a test case comprised of a sample of 10 controls and 13 vulnerabilities from Casestudy2015 () with different levels of uncertainty at each budget level. All the reported results are collected in Fig. 2 and the expected damage is defined as a normalised value between 0 and 100. In the following paragraphs, we will discuss the characteristics of each budget level.
The tables presented in this section present the best strategies seen at each budget level when tested with different levels of uncertainty. The number represents the optimal level that a control should be implemented at, where 1 dictates the simplest possible configuration, 5 dictates the best but most restrictive possible configuration, and 0 represents no implementation of the control.
Budget 5: The expected damage is distributed primarily between 35 and 45. With lower budgets, there are fewer viable solutions. There are few solutions that provide both good coverage and fall within the budget range, making the discovery of optimal solutions more difficult. The closer the direct cost of a solution tends towards the budget, the more likely the solution under uncertainty will exceed the budget and incur the penalty, this is prominent at the 5% level of uncertainty.
With a very limited budget, the number of viable solutions are limited. With low uncertainty we see in Table 1, all optimal solutions tend towards implementing only two controls.
With uncertainty greater than 0.2, we see a different solution, where the first control is implemented at a lower level, with the third control implemented at a higher level.
Budget 10: Unlike the lower budget level, we see that the average expected damage falls in the range of 26 to 29, which is half the range seen at budget 5. With more controls available, the expected damage should go down, however at the same time we see that the solutions become more consistent. The standard deviation is less than 2.5, with a difference in means that never exceeds 2.
Table 2 shows that the optimal results for budget 10 build on the basic pattern from those at budget 5, suggesting implementations for both controls 1 and 3 regardless of the level of uncertainty. With low uncertainty, control 9 is considered optimal, but at higher levels of uncertainty, controls 7 and 10 are considered optimal.
Budget 15: For a budget of 15, we see that the mean expected damage is between 19 and 22. At this budget and higher, we see that the difference in means between the certain and uncertain solutions never exceeds 1. With the increased budget over the previous results, the optimal solution in Table 3, now always considers a combination of the first three controls, where the rest of the budget is used to sporadically patch the worst remaining vulnerabilities as dictated by uncertainty. This means that at lower levels of uncertainty control 4 is preferred, while at higher levels of uncertainty, we see that control 10 becomes the favoured addition to the base set of controls, with control 9 preferred at 10% uncertainty.
Uncertainty  1  2  3  4  5  6  7  8  9  10 

0%  4  0  1  0  0  0  0  0  0  0 
5%  4  0  1  0  0  0  0  0  0  0 
10%  4  0  1  0  0  0  0  0  0  0 
15%  4  0  1  0  0  0  0  0  0  0 
20%  3  0  2  0  0  0  0  0  0  0 
25%  3  0  2  0  0  0  0  0  0  0 
Uncertainty  1  2  3  4  5  6  7  8  9  10 

0%  3  0  1  0  0  0  0  0  4  0 
5%  3  0  1  0  0  0  0  0  4  0 
10%  3  0  1  0  0  0  0  0  4  0 
15%  4  0  3  0  0  0  1  0  0  0 
20%  4  0  3  0  0  0  1  0  0  0 
25%  4  0  2  0  0  0  0  0  0  2 
Uncertainty  1  2  3  4  5  6  7  8  9  10 

0%  4  2  3  1  0  0  0  0  0  0 
5%  4  2  3  1  0  0  0  0  0  0 
10%  4  2  3  0  0  0  0  0  1  0 
15%  4  3  2  0  0  0  0  0  0  1 
20%  4  3  2  0  0  0  0  0  0  1 
25%  4  3  3  0  0  0  0  0  0  1 
Uncertainty  1  2  3  4  5  6  7  8  9  10 

0%  5  3  2  1  0  0  0  0  0  1 
5%  5  3  2  1  0  0  0  0  0  1 
10%  5  3  2  1  0  0  0  0  0  1 
15%  5  4  2  0  0  0  0  1  2  0 
20%  5  4  2  0  0  0  0  1  2  0 
25%  5  4  3  0  0  0  0  0  3  0 
Budget 20: The range of average expected damage is limited to less than 1, with the biggest discrepancy between certain and uncertain solution at the 20% uncertainty level.
The optimal solutions from Table 4 add little to the general pattern of solutions that precede it, implementing the first 3 controls at varying levels. This is the only time that we see the optimal solution suggest the highest level of implementation for control 1. Here, control 10 is preferred at lower levels of uncertainty. At higher levels, this and control 4 are replace by a combination of controls 7 and 8.
Uncertainty  1  2  3  4  5  6  7  8  9  10 

0%  4  4  3  3  0  0  0  3  0  0 
5%  4  4  3  3  0  0  0  3  0  0 
10%  4  4  3  3  0  0  0  3  0  0 
15%  4  4  1  3  0  0  0  2  3  0 
20%  4  4  1  3  0  0  0  2  3  0 
25%  3  2  3  3  0  0  1  0  4  0 
Budget  1  2  3  4  5  6  7  8  9  10 

5  3  0  1  0  0  0  0  0  0  0 
10  3  0  1  0  0  0  0  0  0  0 
15  4  2  2  0  0  0  0  0  0  0 
20  5  3  2  0  0  0  0  0  0  0 
25  3  2  1  4  0  0  0  0  0  0 
Budget 25: Considering the highest budget tested, we see that the average expected damage has a range of 1, between 13.2 and 14.2. This results in a difference in means of at most 0.4 and a minimum of 0.025. This is combined with standard deviations of no greater than 1.2 to provide consistent results between certain and uncertain solutions.
From Table 6, the main difference in solutions is that control 4 becomes a permanent suggestion for implementation in addition to the other 3 core controls. Up to 20% uncertainty, we see some variation of 6 controls, with consistent solutions up to 10% uncertainty and a common solution at 15% and 20% uncertainty.
At 25% uncertainty we see that the optimal solution deviates away from those solutions below. As with all of the results, despite a different solution, we still see a similar expected damage with the solution created in certain space. With uncertainty and a wide range of available configurations, it is reasonable to consider that there will be a number of solutions that offer similar results. Given that it still shares common factors, we can consider that most of the mitigation is handled by those four controls. The mitigation of the additional controls covers the change in values caused by uncertainty, this is similar to the case seen at 15% uncertainty.
The following section highlights a number of common themes across the results, considering the expected results as well as themes consistent with the optimal solutions.
4 Discussion
Across all of the results in Fig. 2, we see only a small difference in mean expected damage between the optimal results with certain and uncertain parameters. This is represented by a difference in the mean values of comparable results not exceeding one standard deviation. While some of the consistency is due to multiple evaluations of solutions, the nature of the designs of the solutions similarly reduces the impact. The hybrid optimisation approach requires multiple different negative perturbations on values to be offset by positive perturbations on other controls before the impact will be seen. The value suggested by the expected damage captures these differences in the deviation of the results from the mean.
The optimal results demonstrate a number of changes to the investment strategy as the uncertainty increases. This change can be explained as a combination of the factors that are uncertain. In general, this will be as a result of some controls becoming more effective than others at similar tasks. Less common results will have optimal solutions that might not be considered valid under a certain set of parameters, but based on uncertainty in the costs, would appear to be genuine.
It is with this last point that we find one of the sources for deviation in the average expected damage seen in the previous section. Above, we discuss having potentially invalid solutions seen to optimal, but we also need to consider the case, where the most optimal solution was eliminated due to potentially having a cost that would exceed the budget.
Uncertainty in the cost is represented most prominently in the results at low budgets. This is due to the number of viable solutions that can be tested, since most solutions will exceed the budget. With this, the search space for solutions features more local optima, with less coherent strategies for traversal.
The consistency in the results can be explained by the coverage of certain controls and their effectiveness at completing that task. Across all the results displayed in Tables 1, 2, 3 and 4, we see that control 1 is always selected, and with some limited exceptions, so is control 3. This gives us an impact on multiple vulnerabilities tested, causing a reduction in the expected damage. It is only at higher budgets, we see that the impact of multiple controls better filling the role of control 3 causes it to be replaced in the optimal solutions.
In addition to the idea that we see consistent results across low levels of uncertainty, we also see that the results identify that although there are a number of differences in the precise optimal solution, there is commonality among all of the optimal solutions present.
The trial was performed with a small set of attacks and controls. Increasing the number of controls and vulnerabilities could increase the potential for less consistent solutions, due to more overlap of controls. Regardless of the composition, good coverage of attack vectors is achieved as the optimal set of controls will always aim to mitigate the most expected damage across all targets.
A desired outcome of the experimental work was to see the extent of the commonality of optimal solutions for each of the levels of uncertainty. As has been explained above, we see that there are a number of commonalities, especially at the same budget levels. Table 8 shows the minimal set of controls and levels that are implemented regardless of the uncertainty.
In comparison to the optimal results for each of the budget levels, we see that these share common features on the first three controls, and later control 4. It is these controls that provide a base coverage of the attack vectors, as described previously. The worst performing base is that of budget 10, which reflects that of the budget 5, this is due to the deviation between low uncertainty and high uncertainty solutions.
From the cyber security perspective, we consider that there are sets of advice such as the UK’s Cyber Essentials, that promote a number of controls. These pieces of advice suggest a set of controls that are reasonable to implement regardless of the degree of complexity or available budget. The base solutions shown here offer the same approach, demonstrating, what a solution should contain based on a constrained budget and uncertainty. These base solutions should be taken as a reference point for building secure systems, with decisions made regarding company specific requirements.
5 Conclusions and future work
This work extended previous work published in the field of decision support for cyber security. It has demonstrated an approach to cyber security investments under uncertainty, where a previous risk assessment based model was extended for this purpose. To explore this, a series of experiments looking at optimal cyber security investments under uncertainty were performed. Uncertainty is naturally a challenge that all cyber security managers face when they have to take decisions. The derivation of exact values for various risk assessment parameters seems like an impossible task. Our work here highlights, that even with some uncertainty in factors that impact payoffs and viable strategies, there is consistency in the outcomes, where the majority of damage was being mitigated by only a few cyber security controls. Although we have concluded to a set of numerical results that clearly demonstrate the benefit of our model and methodology, the expected extension of this work, would be to apply the proposed tools to a full realistic case study, allowing for a comparison to expert judgments, capturing where and how the uncertainty arises.
References
 (1) A. Fielder, E. Panaousis, P. Malacaria, C. Hankin, and F. Smeraldi, “Decision support approaches for cyber security investment,” Decision Support Systems, vol. 86, pp. 13–23, 2016.
 (2) S. Rass, S. König, and S. Schauer, “Uncertainty in games: Using probabilitydistributions as payoffs,” in International Conference on Decision and Game Theory for Security. Springer, 2015, pp. 346–357.
 (3) Ponemon Institute LLC, “Cost of data breach study,” Tech. Rep., 2017. [Online]. Available: https://www.ibm.com/security/databreach/
 (4) ——, “The cost of malware containment,” Tech. Rep., 2015. [Online]. Available: http://landing.damballa.com/rs/damballa/images/Damballa_Ponemon_Malware_Containment.pdf
 (5) Y. J. Lee, R. J. Kauffman, and R. Sougstad, “Profitmaximizing firm investments in customer information security,” Decision Support Systems, vol. 51, no. 4, pp. 904–920, 2011.
 (6) M. Chronopoulos, E. Panaousis, and J. Grossklags, “An options approach to cybersecurity investment,” IEEE Access, 2017.
 (7) M. Benaroch, “Real options models for proactive uncertaintyreducing mitigations and applications in cybersecurity investment decisionmaking,” 2017.
 (8) L. A. Gordon, M. P. Loeb, W. Lucyshyn, and L. Zhou, “Increasing cybersecurity investments in private sector firms,” Journal of Cybersecurity, vol. 1, no. 1, pp. 3–17, 2015.
 (9) T. Moore, S. Dynes, and F. R. Chang, “Identifying how firms manage cybersecurity investment,” University of California, Berkeley, 2016.
 (10) E. Panaousis, A. Fielder, P. Malacaria, C. Hankin, and F. Smeraldi, “Cybersecurity games and investments: A decision support approach,” in Decision and Game Theory for Security. Springer, 2014, pp. 266–286.
 (11) SANS, “The critical security controls for effective cyber defense (version 5.0),” http://www.counciloncybersecurity.org/attachments/article/12/CSCMASTERVER502272014.pdf, 2014, accessed: 20151219.
 (12) CWE, “Cwe top 25 most dangerous software errors (2011),” http://cwe.mitre.org/top25/, accessed: 20151219.
 (13) A. Fielder and E. Panaousis, “Decision support approaches for cyber security investment: data for cyber essentials case study,” http://www.panaousis.com/papers/casestudy.pdf, 2015, accessed: 20151219.
 (14) S. Rass, S. König, and S. Schauer, “Decisions with uncertain consequences  a total ordering on lossdistributions,” PLOS ONE, vol. 11, 2016.
 (15) J. S. Busby, A. Gouglidis, S. Rass, and S. König, “Modelling security risk in critical utilities: The system at risk as a three player game and agent society,” in 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 2016, pp. 001 758–001 763.
 (16) S. Rass, S. König, and S. Schauer, “Defending against advanced persistent threats using gametheory,” PLOS ONE, vol. 12, 2017.
 (17) S. Karnouskos, “Stuxnet worm impact on industrial cyberphysical system security,” in IECON 2011  37th Annual Conference of the IEEE Industrial Electronics Society, Nov 2011, pp. 4490–4494.
 (18) EISAC, “Analysis of the Cyber Attack on the Ukrainian Power Grid,” Washington, USA, Tech. Rep., 2016. [Online]. Available: https://ics.sans.org/media/EISAC_SANS_Ukraine_DUC_5.pdf
 (19) R. Oppliger, “Quantitative risk analysis in information security management: A modern fairy tale,” IEEE Security Privacy, vol. 13, no. 6, pp. 18–21, Nov 2015.