The Cloudlet Bazaar Dynamic Markets for the Small Cloud

The Cloudlet Bazaar Dynamic Markets for the Small Cloud

Ranjan Pal1, Sung-Han Lin1, Aditya Ahuja2, Leana Golubchik1 1 University of Southern California {rpal, sunghan, leana}@usc.edu 2 Indian Institute of Technology Delhi aditya.ahuja@cse.iitd.ac.in
Abstract

The recent emergence of the small cloud (SC), both in concept and in practice, has been driven mainly by issues related to service cost and complexity of commercial cloud providers (e.g., Amazon) employing massive data centers. However, the resource inelasticity problem faced by the SCs due to their relatively scarce resources (e.g., virtual machines) might lead to a potential degradation of customer QoS and loss of revenue. A proposed solution to this problem recommends the sharing of resources between competing SCs to alleviate the resource inelasticity issues that might arise [1]. Based on this idea, a recent effort ([2]) proposed SC-Share, a performance-driven static market model for competitive small cloud environments that results in an efficient market equilibrium jointly optimizing customer QoS satisfaction and SC revenue generation. However, an important non-obvious question still remains to be answered, without which SC sharing markets may not be guaranteed to sustain in the long-run - is it still possible to achieve a stable market efficient state when the supply of SC resources is dynamic in nature and there is a variation of customer demand over time?. In this paper, we address the problem of efficient market design for SC resource sharing in dynamic environments. We answer our previous question in the affirmative through the use of Arrow and Hurwicz’s disequilibrium process [3, 4] in economics, and the gradient play technique in game theory that allows us to iteratively converge upon efficient and stable market equilibria.

small cloud; dynamic market; stability
\setstretch

1

I Introduction

Cloud computing is becoming increasingly popular and pervasive in the information technology (IT) marketplace due to its on-demand resource provisioning, high availability, and elasticity. These features allow cloud end users (e.g., individuals, small-scale companies, world-wide enterprises) to access resources in a pay-as-you-go manner and to meet varying demands sans upfront resource commitments [5]. Cloud service providers (Amazon AWS [6], Google Compute Engine [7], and Microsoft Azure [8]) allow customers to quickly deploy their services without a large initial infrastructure investment.

I-a The Rise of Small-Scale Data Centers

There are some non-trivial concerns in obtaining service from large-scale public clouds, including cost and complexity. Massive cloud environments can be costly and inefficient for some customers (e.g., Blippex [9]), thus resulting in more and more customers building their own smaller data centers [10] for better control of resource usage; for example, it is hard to guarantee network performance in large-scale public clouds due to their multi-tenant environments [11]. Moreover, smaller data center providers exhibit greater flexibility in customizing services for their users, while large-scale public providers minimize their management overhead by simplifying their services; e.g., Linode [12] distinguishes itself by providing clients with easier and more flexible service customization. The use of small-scale clouds (SCs) is one approach to solve cost and complexity issues.

Despite the potential emergence of small-scale clouds, the latter due to their moderate sizes, are likely to suffer from resource under-provisioning, thus failing to meet peak demand at times. This leads to a resource provisioning dilemma where the SCs have to make the tradeoff between request loss and the cost of over-provisioning. One way out of this dilemma is for such small clouds to cooperate with each other to help meet each others’ user demand via resource sharing at low costs, thereby increasing their individual resources when in need without having to significantly invest in more. Such cooperation is analogous to Business Clusters described in mainstream economics which emerge due to, among other factors, shared interests and geographical proximity [13].

I-B Research Motivation

In this section, we briefly describe the problem setting followed by the challenges that motivate us to alleviate them.

Problem Setting. The effective sharing or borrowing resources by an SC from its peers involves mutually satisfying the interests of the stakeholders in context. In this paper, we consider three different stakeholders: (i) the SC customers, (ii) profit maximizing autonomous SCs, and (iii) a regulatory agency overseeing certain functioning aspects of the autonomous SCs (e.g., ensuring customer data privacy). The SC customers are interested in achieving certain performance measures for their jobs (e.g., low job response time, cheap storage); the SCs are interested in maximizing revenues obtained from serving customers; and a regulatory agency (e.g., the local government, a federated agency [14][15]) is interested in (i) ensuring a proper manner by which the autonomous SCs conduct their business of lending resources to peer SCs (e.g., preserving data privacy, designing policies for (a) customer cost effectiveness by disallowing SCs to charge high customer prices, and (b) maintaining a certain level SC and customer welfare), and (ii) recommending resource exchanges between autonomous SCs in a way (without interfering in the important resource (allocation, scheduling) decisions of the SCs) that encompasses the necessary means for the proper configuration of the resources (e.g., using the OpenNebula manager and its external resource lease manager Haizea [16]). We term the above setting as an SC market.

The Challenges. Ideally, an SC would want to service all its customers solely using its own resources. However, the primary barrier to this goal is its individual resource capacity which might not be enough to service peak customer demand. In such a case, the SC can either resort to peer SCs for additional resources, thereby incurring borrowing costs, and/or buy the services of a big public cloud (e.g., Amazon). The latter option is generally more expensive than the former and also likely to be more privacy threatening. Thus, from an SC’s viewpoint, its challenge is to satisfy two conflicting objectives: (i) to generate as much revenue by serving its customer demands, and (ii) to incur as low as possible, borrowing and/or buying costs from other clouds. For simplicity purposes, we assume that buying resources from big clouds (e.g., Google, Amazon) is the last resort for an SC in events of low resource availability, and in such events it would try its best to get resources from peer SCs. Another challenge is to ensure that at market equilibrium (see below), the SCs and their customers ideally operate on parameters (see Section II) that allow the market to be efficient, a condition commonly characterized in microeconomics by certain popular functions (see Section II.C) of market stakeholder utilities, and one that entails optimal social welfare allocation amongst the SCs and their customers. This is a non-trivial and challenging task as the existence of a market equilibrium does not necessarily imply market efficiency [17]. In this regard, the authors in [2] show the existence of SC market equilibrium through numerical simulations, and do not provide a general theory for equilibrium existence. In addition to the above mentioned challenges, the SC market is dynamic in nature due to the non-static nature of the supply of SC resources, as well as due to the variations in customer demand over time. This dynamic nature of the SC market is likely to lead to frequent market equilibrium perturbations and potentially a state of market disequilibrium. Conditioned on the achievability of a market efficient equilibrium, a state of eventual disequilibrium will threaten the long-term sustainability of SC markets. Here, the term ‘market equilibrium’ refers to a situation in which all market stakeholders mutually satisfy their interests, in which case an important challenge is to design a stable market that is robust to perturbations and always returns to its equilibrium point(s) when perturbations do occur.

Our Goal. In this paper, our goal is to formulate the joint ‘stakeholder satisfaction problem’ in dynamic SC environments as an efficient, stable, and sustainable dynamic market/ecosystem design task, and propose an effective solution for it.

I-C Research Contributions

We make the following research contributions in this paper.

  • We propose a utility theory based small cloud competitive market model comprising of SC customers, profit maximizing autonomous SCs, and a regulatory agency overseeing some functionality aspects of the SCs, as the market stakeholders. The model mathematically expresses the stakeholder interests in terms of utility functions and paves the path for analyzing SC markets for market equilibrium properties (see Section II).

  • We analyze our proposed market model via a convex optimization framework for the existence and uniqueness of a static market equilibrium at which (i) the utilitarian social welfare function (see Section III for a definition ) is maximized, i.e., the market equilibrium is socially efficient, (ii) the market equilibrium is Pareto efficient (see Section III for a definition), (iii) the market is cleared, i.e., the SC supply balances customer demand, and (iv) no stakeholder has any incentive to deviate from the equilibrium. We show that there exists a unique static competitive market equilibrium jointly satisfying (i), (ii), (iii), and (iv), however there are several static market equilibria jointly satisfying (ii), (iii), and (iv). (see Section III).

  • Using notion of a disequilibrium process proposed by Arrow and Hurwicz [3, 4], we apply the gradient play technique in game theory [18] that is based on the theory of differential equations, to investigate the dynamic market setting where a static market equilibrium (conditioned on their existence) is potentially subject to perturbations that might lead to market disequilibrium. In this regard, we show (in theory) that static market equilibrium achieved in small cloud markets is asymptotically stable in dynamic market settings. Our use of the gradient play technique is motivated by the fact that in many practical market environments stakeholders (i) find it behaviorally difficult or computationally expensive to play their best responses [19], (ii) have zero or incomplete knowledge of the utilities of other stakeholders in the market, and (iii) cannot even observe the actions of other stakeholders in the worst case. In such environments, gradient play is a suitable technique to achieve static market equilibrium stability iteratively [20], from a state of disequilibrium. More specifically, for our market setting the occurrence of (i)-(iii) is quite likely. Gradient play also works when issues (i)-(iii) do not arise (see Section IV).

  • We validate our proposed theory through extensive numerical experiments to illustrate the stability of SC markets and the high speed with which such markets converge to stable equilibria despite variations in market supply-demand. Through numerical experiments, we also investigate market equilibria performance of SC markets with respect to three Bergson-Samuelson social welfare functions, viz., the utilitarian function, the egalitarian function, and the Rawl’s function, standard in the economics literature [17]. Here, we study and compare (i) the fairness amongst market players of a welfare allocation achieved through a given social welfare function and (ii) the amount of social welfare of SC markets under the different social welfare functions, when social welfare optimality is achieved. Via our numerical evaluation, we infer that SC markets are sustainable. (see Section V).

Ii Competitive Market Model

In this section, we propose a utility theory based small cloud Walrasian competitive market model comprising of profit maximizing autonomous SCs, their customers, and a regulatory agency overseeing some functionality aspects of the SCs. A Walrasian competitive market [17] represents a pure exchange economy without production, where there are a finite number of agents, i.e., SCs in our work, endowed with a finite number of commodities, i.e., computing resources in our work, that gets traded with SC customers and peer SCs. The aim behind proposing the model is to pave the path for mathematically analyzing SC markets for market equilibrium properties, and derive their practical implications.

In this paper, we consider each SC customer to deal with three job types, where each job comprises multiple tasks: (i) Type I jobs that need to be serviced wholly/entirely when they arrive (e.g., a user could invoke a regular MapReduce batch job that defines a set of Mappers and Reducers to be executed for the job to complete in its entirety.), (ii) Type II jobs that can be curtailed to fewer tasks (e.g., an approximation job like the ones cited in [21]), the curtailment decision primarily arising from (a) the nature of VM instance prices, (b) the unnecessity of the customer to keep executing a job beyond a certain accuracy already achieved, and (c) the unnecessity of the customer to keep executing the job beyond a certain deadline, and (iii) Type III jobs where certain tasks can be shifted over time for future processing, the remaining job tasks requiring service as they arrive (e.g., analyzing a DNA sequence, re-running partially/entirely a current job later when it gets killed in a spot cloud environment due to momentary unavailability of resources.). Next, we model the stakeholders in the SC market.

Ii-a Modeling the SCs

Let there be autonomous profit maximizing SCs. Each SC can spread over multiple locations. Customer demand for SC is a set of processing tasks from its customers (both end-users and peer SCs) that require the use of virtual machines as the primary computing resources. In this regard, we assume that each SC reserves (allocates) a total of virtual machines (VMs) in its data center to service demands from its customers. We term such VMs as reserved VMs. The value of is pre-decided by SC based on the statistics of customer demand patterns observed over a period of time, as one of the factors. For simplicity, we will focus on VMs representing a single resource type in this paper. A justification is provided in Section 4 of the Appendix [22]. In the event that machines are insufficient to satisfy consumer demands, SC buys/borrows VMs from peer SCs. Here, is the number of borrowed VMs available to SC from its peers. In the event that both reserved and borrowed VMs are insufficient to meet customer demand, SC resorts to a public cloud for VM instances. We assume here that a public cloud is large enough to provide any required number of VM instances to SCs. We do not consider communication network bandwidth issues to be a bottleneck to customer service satisfaction in our work.

Let be the associated operating cost to SC for reserving virtual machines to serve its customers. We define via a separable equation of the following form.

(1)

where (a linear function) and (a non-linear function) are functions such that the marginal operating cost for SC is a general decreasing linear function of the number of VM instances, i.e., the additional operating cost, , due to a unit increase in the number of VMs required to service customer demand varies in a negative linear fashion with the number of VMs. We use this type of marginal cost functions in our work due to their popularity in economics to model diminishing costs/returns [17]. We approximate the number of VMs to be a non-discrete quantity. Specifically, for the purpose of analysis, we assume the cost function to be concave, quadratic, and twice continuously differentiable, i.e., the marginal costs become decreasing linear functions of the number of VM instances. We can define one such function as follows.

(2)

where (a positive value) and (a negative value) are SC ’s cost coefficients for its reserved resources, i.e., virtual machines, such that the marginal operating cost for SC is a negative linear function. The above quadratic form of the cost function, apart from satisfying the property of negative linear marginals, not only allows for tractable analysis, but also serves as a good second-order approximation for the broader class of concave payoffs [23]. We denote to be the profit that SC makes through its reserved VMs for servicing customers, and define the maximum profit that SDC can make, via the following optimization problem.

subject to

where is the per-unit VM instance price charged by SC to its customers, and and are the lower and upper bounds for the number of VM instances reserved by SC for its customers. We assume that each SC is small enough not to be able to exert market power over its peer SCs and strategically influence the prices they charge their customers. i.e., each SC is a price taker [17]. The prices that individual SCs charge their customers are determined by individual SCs in price competition with one another in the process of maximizing their own utilities and selling off their endowment.

Let be the associated operating cost to SC for borrowing virtual machines from peer SCs to serve customers, when the reserved VMs are not enough to satisfy customer service demands. Like in the case of formulating , we formulate in a manner such that the associated marginal operating costs for borrowing an additional VM instance decreases in a negative linear fashion with the number of VMs. Mathematically, we represent by the following equation.

(3)

where (a positive quantity) and ( a negative quantity) are SC ’s coefficients for its borrowed virtual machines. We denote by the profit that SC makes when borrowing VMs from peer SCs for servicing customers, and define the maximum profit that SC can make, via the following optimization problem.

subject to

Here, (i) and are the lower and upper bounds for the number of VM instances borrowed by SC for its customers, from peer SCs, (ii) is the cost to SC to offload VM instances worth of customer demand to a public cloud in the event that and VM instances together are not enough to service ’s total customer demand. We mathematically represent in the same manner as and , and express it via the following equation.

(4)

where (a positive quantity) and (a negative quantity) are SC ’s coefficients for the resources the public cloud uses to service ’s offloaded customer demand portions. We do not assume any constraints on the resources available to the public cloud for servicing offloading requests by SCs.

Ii-B Modeling SC Customers

For a customer who has a Type I job, we express this customer’s utility for that job as a concave, quadratic, and twice continuously differentiable separable function, , defined as follows.

(5)

where is the amount of VM instances required to process ’s entire job. Similar to the motivation and rationale behind the concave, quadratic cost functions for SCs, the utility function of an SC customer is designed such that the marginal utility for the customer is a decreasing linear function of the number of VM instances, i.e., the additional utility increase due to a unit increase in the number of VMs varies in a negative linear fashion with the number of VMs. (a positive quantity) and (a negative quantity) in the above equation are ’s utility coefficients.

Like in the case of a customer with a Type I job, for a customer who has a Type II job, we express his utility for that job as a quadratic twice continuously differentiable function, , defined as follows.

(6)

where is the amount of VM instances required to process ’s curtailed job, and is expressed as

Here, (a positive value) and (a negative value) are ’s utility coefficients for Type I jobs. The interpretation of is as follows: is the number of VMs required to accomplish ’s curtailed task, whereas is the additional number of unused VMs that contribute to ’s extra utility when its job is curtailed, and provides it with an overall perceived satisfaction greater than that obtained from the utility derived solely using used VMs for the curtailed job.

For a customer who has a Type III job, similar to the case of Type I and Type II jobs, we express his utility for those tasks as a quadratic twice continuously differentiable function, , defined as follows.

(7)

where is the amount of VM instances required to process ’s time-shiftable tasks, and ( a positive value) and (a negative value) are ’s utility coefficients for time-shiftable jobs.

A customer can have jobs of all three types. Thus, his aggregate tasks are worth VM instances. Therefore, customer ’s aggregate utility takes a similar form to his utility for a specific job type, and is given by

(8)

where (a positive quantity) and (a negative quantity) are ’s utility coefficients for his job aggregate.

We denote to be the net utility that customer generates through getting service for a given job type = from its contracted SC, and define the maximum net utility that customer can generate, via the following optimization problem.

subject to

Here, and are the lower and upper bounds for the number of VM instances used up by customer ’s job type (be it whole, curtailed, shifted, or aggregate). is the price paid by customer to his chosen SC per VM instance used for his job.

Ii-C Modeling the Regulator

The role of the regulator (e.g., the government, a federated agency) as applicable to our work is to ensure (i) good privacy practices between SC, (ii) the design of policies/mechanisms that enable autonomous SCs to price customers appropriately without making excessive profits through market exploitation, and (iii) an optimum level of social welfare allocation amongst the autonomous SCs at market equilibrium. (i) is specific to our problem setting and is one of the most important motivation for the presence of a regulator (see Section I) in the first place111In practice, using mechanism design theory, the regulator can devise efficient economic mechanisms that enable SCs to find it incentive compatible in protecting the privacy of their customers. However, we do not focus on the design of such mechanisms in this paper.. However, the presence of a regulator brings in other important benefits through (ii) and (iii). (ii) is necessary to prevent any SC from exploiting its customers on service costs. In this work we do not focus on the design of such mechanisms, and assume the existence of one222Economists Laffont and Tirole have proposed principal-agent models in this regard [24] which will enable autonomous SCs to charge appropriate prices to customers purely out of self-interest., whereas (iii) is important from an economic perspective as maximizing social welfare is a key objective in welfare economics because it leads to (a) a certain level of equitability of allocations (in resources or in net utility) amongst the stakeholders, (b) might guarantee Pareto efficiency at market equilibrium [17], and (c) an optimal social welfare state denotes the best possible operating point of an economic system. A Pareto efficient allocation of utilties amongst a set of stakeholders ensures that at market equilibrium none of the stakeholders can increase their net utility without decreasing any other stakeholder’s net utility. The notion of equitability is important in the context of autonomous SC markets because they often operate in a decentralized fashion, and ideally, we would want a social welfare allocation at market equilibrium that does not result in considerable disparity amongst the players’ allocations (despite being Pareto efficient).

In this paper, we define the social welfare function of the regulator to be the sum of the net utilities of the SCs and their customers at market equilibrium. We denote this function by SW, and express it as

(9)

where is the set of consumers, is the set of small clouds, the first term is the sum of the utility of the consumers, and the second term is the sum of the costs faced by the SCs in for servicing customer demands. The aforementioned social welfare expression is the standard Bergson-Samuelson utilitarian social welfare function in economics [17] whose optimality does not focus on equality of resource or utility allocations amongst each class of stakeholders, i.e., the SCs and the customers, but only on Pareto efficiency of resource allocations amongst the stakeholders, and equality of marginal utility allocations amongst the stakeholders. Note that due to our autonomous SC setting, the regulator in practice might not have enough say in welfare maximizing resource allocation, and can only expect to have the social welfare function maximized in the best case because it cannot directly enforce optimal strategy choices on the SCs like in a centralized control setting. The important question here is whether the utilitarian social welfare function is indeed the most appropriate choice for this work.

We choose to work with the utilitarian function over two other popular Bergson-Samuelson social welfare functions used in economic applications: the egalitarian function, and the Rawl’s function, for the following reasons:

  • The parameters corresponding to the unique optimal solution of the maximum utilitarian social welfare problem coincides with those obtained at the unique equilibrium of a purely distributed market comprising autonomous SC’s without the presence of a regulator, and are Pareto optimal. This result is due to Arrow-Debreu’s first and second fundamental theorems of welfare economics [17]. In addition, at market equilibrium, there is equitability in the marginal utilities of all the autonomous SCs (in case of SCs, the utility is represented by cost and is thus a negative utility) and their customers. The parameter coincidence property does not necessarily hold for non-utilitarian social welfare functions.

  • The Rawl’s social welfare function focusses on maximizing the minimum resource/utility allocation to any stakeholder (e.g., SC) within the class of market stakeholders. A major drawback of adopting this social welfare function is that it will in general discourage SCs from sharing their resources (even at Pareto optimal system settings) with other SCs (consequently affecting customer QoS satisfaction), thereby challenging the core philosophy behind an SC market, and will not likely be popular with either the SCs or the regulator. A maximin utility allocation among SCs would favor, for example, a regime that reduces every SC to complete “misery” if it promotes the well-being of the most “miserable” SC by even a very small amount.

  • The egalitarian social welfare function focusses on equalizing the utilities of all market stakeholders in the absolute sense. Similar to the case of Rawl’s function, it suffers from the major drawback that it will in general discourage SCs from sharing their resources (even at Pareto optimal system settings) with other SCs. Likewise, it is unlikely to be popular amongst either the regulator or autonomous SCs. For example, if we had to choose between two allocation policies, one under which all SCs would have a cardinal utility of 100, but one SC would have a utility of 99; the second policy under which every SC is “miserable” and will have a cardinal utility of 1 unit. The egalitarian regulator would prefer the latter because under this option, every SC has exactly the same utility level.

Iii Static Market Analysis

In this section we derive and analyze perfectly competitive SC market equilibria. We assume perfect competition amongst SCs due to their lack of economic power in influencing other SCs based on their quantity of VM availability. Since prices in perfect competition are strategic complements (in the terminology of Bulow, Geanakoplos and Klemperer [25]), i.e., the decrease in an SC’s customer price results in the decrease of customer prices charged by other SCs in competition, we are going to eventually converge to a stage where a single uniform customer price will prevail in the SC market [25]. We are interested to know whether such a price results in social welfare optimality. In this regard, we (a) formulate and solve an optimization problem for a regulator who wishes to achieve socially optimal market equilibria that maximizes utilitarian social welfare amongst the market stakeholders, (b) characterize market equilibria in the absence of a regulator and draw comparative relationships between the equilibria obtained, with socially optimal market equilibria.

Optimization Problem Formulation - Here, we formulate a regulator’s optimization problem so as to achieve socially optimal market equilibria. The primary goal of the formulation is to maximize the net utilities for the SC customers, and minimize the net cost of operation of SCs to reach a net maximum social welfare situation amongst the SCs and their customers. We define this problem mathematically as follows:

subject to

where the objective function is to maximize social welfare SW (see Equation 9 above) or equivalently to minimize the negative of social welfare (to have a convex objective function to fit the convex programming paradigm), and the constraint is the supply-demand balance equation, with representing total customer demand, and representing total SC supply. is the set of customers served by SC . A potential solution to the above optimization problem indicates the parameters at which the SC market can ideally operate and (i) make all stakeholders satisfied to a point that no one has an incentive to deviate, and (ii) maximize the total satisfaction of all the stakeholders together. We denote such an ideal state of market operation as a static socially efficient market equilibrium.

Dual Problem Formulation - We will solve OPT using the primal-dual approach [26]. The advantage of using the primal-dual approach is that the dual optimization problem of the primal is always convex [26], and its solution results in global optima which can be related back to the optimal solution of the primal problem. Before deriving the dual optimization problem, we first define the Lagrangian function of OPT as follows:

where is the vector of Lagrange multipliers for the constraint in OPT. The dual optimization problem, DOPT, is then defined as follows.

where , and are vectors of customer VM types and , and are vectors of SC VM types. Note that for any customer equals . Thus, the goal here is to find an optimal tuple that is an optimal solution to both OPT and its dual.

Solving the Dual The dual optimization problem is convex and its optimal solution is found by applying the Karush-Kuhn-Tucker (KKT) conditions [26] that are stated through equations (10a)-(10g). Solving these equations, we obtain the optimal solution to DOPT. Since OPT is convex, applying Slater’s conditions we obtain strong duality, i.e., a duality gap of zero[26], which implies that the optimal solution to OPT coincides with that of DOPT, and there is no loss in the value of the optimal solution by the transformation of the primal problem to its dual. The optimal solution to OPT/DOPT is unique, and is the static market equilibrium. We denote this solution by the tuple . We now state the KKT conditions in the form of equations (10a)-(10e) as follows.

(10a)
(10b)
(10c)
(10d)
(10e)
(10f)
(10g)

Equilibrium in Distributed Autonomous Settings - The solution to OPT is unique due to the convexity of the dual formulation. The key question is whether this solution can be realized as a market equilibria in a distributed autonomous setting. Based on the general equilibrium theory in microeconomics [17], market equilibria in a perfectly competitive autonomous setting of firms is known as Walrasian equilibria. It turns out from general equilibrium results in [17] that the unique optimal solution to OPT (i) is a competitive Walrasian equilibrium that is Pareto efficient, (ii) satisfies Arrow-Debreu’s first and second fundamental theorems of welfare economics that establishes the if and only if relation between the existence of a Walrasian equilibrium and its Pareto efficiency [17], (iii) maximizes utilitarian social welfare (again derived from Arrow-Debreu’s first and second fundamental theorems), and (iv) clears the market by balancing total SC resource supply with consumer and SC resource demand. Thus, in view of points (i) - (iv), a regulator’s social welfare maximization objective coincides with the welfare state obtained at market equilibrium in a distributed autonomous firm setting. We consider this unique equilibrium state to be the benchmark at which the SC market would be willing to always operate. However, in practice, for a perfectly competitive market with non-utilitarian social welfare functions, there may be multiple Pareto efficient Walrasian market equilbria that are not socially efficient.

Computing the Socially Optimal Market Equilibrium The optimal solution to the dual optimization problem, DOPT, can be obtained in an iterative manner using a gradient approach, the principle behind which is the Primal-Dual Interior Point Method [26]. We adopt the Primal-Dual Interior Point method in our work because it has a polynomial-time complexity to arrive at the optimal solution to convex programs [27]. The basis of the method is to progressively change the argument vector of DOPT so that minima-Lagrange multiplier satisifes the KKT conditions.

Denote by , DOPT’s argument vector sans the Lagrange multiplier , . Applying the Interior Point method to DOPT gives us the the following equations:

(11a)
(11b)

Here, and are positive scaling parameters which control the amount of change in the direction of the gradient. Letting , we get

(12a)
(12b)

where for . The Interior Point Method converges in polynomial time when the duality gap approaches zero, due to the linear and super-linear convergence rate of the method [26].

Iv Dynamic SC Markets

On Dynamic SC Markets - In practice, an SC market can be dynamic in nature due to the non-static nature of the supply of SC resources and variability of over time of customer demand. This dynamic nature of the SC market is likely to lead to frequent static market equilibrium perturbations, which in turn might (not always) lead to a state of market disequilibrium. Here, the term ‘disequilibrium’ refers to a state when market supply does not equal market demand due to perturbations in market parameters (e.g., customer prices), and as a result all stakeholders do not mutually satisfy their interests. In such a case, an important challenge is to design a stable market that is robust to perturbations and always returns to its equilibrium point(s) when market disequilibrium results. Inspired by the notion of disequilibrium process [3], we propose a dynamic market mechanism for SCs. The concept of disequilibrium pertains to a situation where a static market equilibrium is perturbed, potentially to a disequilibrium state, and the underlying players (stakeholders) work together to re-attain the equilibrium. The main idea behind the disequilibrium process is an iterative sequence of action and state profiles (see below), i.e., information exchange between the dominant market stakeholders, of VM instance supply and demand levels, and per-unit VM instance prices, to arrive at a desired static equilibrium. Such an iterative process essentially implies an overall dynamic model with feedback. Our proposed dynamic market mechanism can also be used to re-attain a specific preferred equilibrium point from a given equilibrium point. We first present our dynamic market model and then follow it up with its stability analysis.

Iv-a Dynamic Model

Our dynamic model of SC markets consist of a state space, , where each state, , is the profile of per-unit VM instance prices at each SC . The state dependent payoff, i.e., profit function for each SC from its reserved resources is given by

The state dependent payoff for each SC from its borrowed resources is given by

Similarly, state dependent payoff for each SC from resources borrowed from a public cloud is given by

The payoff function for the SC customers for a given job type , is given by

Each SC is assigned a state dependent action that permits the SCs and their customers to change their VM instance generation and consumption levels respectively. We assume a perfect competition [19] of VM instance prices amongst the SCs in competition, and following that the action for each SC consists of commiting a certain amount of VM instances that influences the market-clearing process. In this paper, we use the gradient play technique in game theory [18] to derive the state dependent actions of the SCs and their customers. Our use of the gradient play technique is motivated by the fact that in many practical market environments stakeholders (i) find it behaviorally difficult or computationally expensive to play their best responses [19], (ii) have zero or incomplete knowledge of the utilities of other stakeholders in the market, and (iii) cannot even observe the actions of other stakeholders in the worst case. In such environments, gradient play is a suitable technique to achieve static market equilibrium stability iteratively [20]. More specifically, for our market setting the occurrence of (i)-(iii) is quite likely. Gradient play also works when issues (i)-(iii) do not arise. The main idea behind the gradient play technique is the use of ordinary differential equations (ODEs) to describe the path of a perturbed system state to the static market equilibrium state. Using gradient play, the action for the the th SC is given by

(13a)
(13b)
(13c)

Here, the parameters , , and are time constants that describe the speed with which the action of VM instance commitment by SC can be adjusted, and are free parameters to be determined. The goal of SC ’s action is to drive the solution , , and to , , and , the solution to Equations 10(a)-10(c) at static market equilibrium. It can be seen that the RHSs of 10(a)-10(c) are proportional to the gradient , , and respectively, where is the Lagrangian of OPT. The suite of equations 10(a)-10(c) can be solved independently by SC . In a similar fashion, using gradient play, the state dependent action for any SC customer is given by

(14)

is a free parameter to be determined that denotes the speed with which the consumption action of SC customer can be adjusted. The goal of the SC customer action here is to drive the solution to , the solution to Equation 10(d) at static market equilibrium. It can be seen that the RHS of 15 is proportional to the gradient , , where is the Lagrangian of OPT. Equation 15 can be solved independently by each SC customer .

The dynamics of the pricing mechanism can be expressed via the following equation.

(15)

where the goal is to drive the solution to , the solution of 10(e) at static market equilibrium. Here, is the free parameter denoting the speed with which can be adjusted. Equations 13-15 represent a dynamic model of the overall SC market. It resembles a repeated negotiation process where SC responds with a commitment of , to suggested prices received from the regulator; SC customer responds with a consumption amount of , type , to the same prices. The regulator in turn adjusts its prices to these actions by the SCs and their customers, and returns new prices, , and the process continues till convergence to the static market equilibrium. A compact representation of the above-mentioned dynamic SC market is presented in Section 2 of the Appendix. This representation paves the way for analytically analyzing the stability of such markets.

Iv-A1 A Compact Representation

We need to compactly represent the above dynamic SC market model to pave the way for analytically analyzing the stability of such markets via the Arrow-Hurwicz criterion that is based on the theory of Lyapunov stability (see Section IV.B). Using Equations 13-15, our proposed dynamic market mechanism can be compactly represented in the matrix form via the following equation:

(16)

Definiton of Equation Parameters. We now describe the parameters of Equation 16. We have

that is a vector of dimension . Here, . We also have

and

,

We define matrices to as follows: , type . We assume that all for a given type, ’s are equal for all . , type , where . , type . , type , where . , where is an matrix of 1’s except for the 0 diagonal elements, is an matrix with all entries 1 except for entries of the form that take a value of zero, and is an matrix. , type . , type . , where is an matrix. .

The expression is a projection function onto the non-negative orthant, and is given by

(17)

where , being a rotating matrix. of dimensionality , and denotes a vector of maximum VM instances committed by each individual SC. The th row of the projection is denoted as

(18)

in Equation 16 represents the resource availability perturbations due to dynamics of the SC market caused by factors stated in Section IV.A. The value lies in a perturbation set , where is given by

(19)

Here,

where matrix is given by , type , and . Matrix is given by , and . We also have expressed via the following:

where is a finite constant. Similar to the expression for , we have

where the matrix is given by . We also have

where is a finite constant. Finally, we express as

where , and . We assume that for given , the values of and are equal for all .

Iv-B Stability Analysis of Dynamic Markets

In this section, we derive results regarding the stability of static market equilibria in a dynamic SC market setting. Specifically, (i) we derive the dynamic market equilibria obtained through gradient play mechanics and compare it with the socially efficient static market equilibria, and (ii) study the region of attraction around dynamic market equilibria to derive stability connotations.

Case - 1: We first consider stability aspects when equals zero, i.e., there are no curtailed jobs. In this case, the equilibria of the dynamic SC market described through Equations 10a - 10c (via the use of the gradient play technique), lies in the set

Let be an equilibrium point in set . We then have the following theorem stating the relationship between and the unique static SC market equilibrium obtained through Equations 10(a) - 10(e). The proof of the theorem is in the Appendix.

Theorem IV.1

The equilibrium is identical to the unique static market equilibrium obtained from the solution of OPT.

Theorem Implications. The theorem suggests that in the absence of curtailed jobs, the equilibrium in a dynamic market setting is unique, and converges to the static market equilibrium in which the market existed initially before it was perturbed. Intuitively, when the SC market is perturbed from its equilibrium setting, a disequilibrium state might result, which will get resolved due to our proposed gradient-play based approach that rolls back the disequilibrium state to the original socially optimal static equilibrium state. In this paper, we are able to roll back to the original state in theory because of our assumptions regarding the nature of utility functions. In practice, gradient play will guarantee a roll back of a disequilibrium market state to an equilibrium state not necessarily the original equilibrium state from which it was perturbed.

We now investigate the stability of the dynamic market equilibrium to find the region of attraction around itself. We introduce a few definitions in this regard. Let , . Denote by a scalar, positive definite Lyapunov function expressed as

(20)

where and are diagonal matrices. We use Lyapunov functions from control theory [28] as a standard to prove the stability of an equilibrium of a system represented via ordinary differential equations (ODEs), such as the ones arising in our work in Section IV.A. Let be expressed as

(21)

where denotes the minimum eigenvalue of ,

where is a rotating matrix, and , being the coefficient of the orthogonal vector to express as . We now have the following theorem characterizing stability of the dynamic market equilibrium. The proof of the theorem is in the Appendix.

Theorem IV.2

Let be Hurwitz. Then the equilibrium is asymptotically stable for all initial conditions in

such that

.

Theorem Implications. Intuitively, the theorem states that irrespective of any initial state the market is in, on being perturbed, it will always come back to an equilibrium state from a disequilibrium state. The Hurwitz (not the same as Hurwicz) nature of matrix is determined from the time constants in Equations 13-15. Most real systems satisfy the Hurwitz criterion in that will be a real square matrix constructed with coefficients of a real polynomial.

Case 2: We now consider stability aspects when does not equal zero. In this case, the equilibria of the dynamic SC market described through Equations 13(a) - 13(c), also lies in the set . We define , and as before but define as

(22)

where is the same as in Equation (21), and represent the supply demand perturbation matrices, and and are given by

(23a)
(23b)

We now have the following theorem characterizing market stability. The proof of the theorem is in the Appendix.

Theorem IV.3

Let be Hurwitz, and let

(24)

Then the equilibrium is asymptotically stable for all initial conditions in

such that .

Theorem Implications. Similar to the implications of Theorem 4.2, this theorem states that irrespective of any initial state the market is in, on being perturbed, it will always come back to an equilibrium state from a disequilibrium state.

V Numerical Evaluation

In this section, we numerically evaluate our dynamic market model to investigate (a) static market efficiency and average stakeholder utility under different Samuelson-Bergson welfare metrics, and (b) stability behavior of dynamic markets. In the absence of a theoretical study (not our focus in this paper), (a) is important to get an approximate idea of the gaps in net stakeholder utility achieved via different welfare functions. (b) is important to characterize the speed of convergence of an SC market to go from a state of disequilibrium to a state of equilibrium. The first part of this section describes the evaluation setting, and the second part analyzes the results.

V-a Evaluation Setup

As a representative numerical evaluation setting, we consider five SCs and 15 customers (not including other SCs). Each SC has five customers each and they are tied to the SCs throughout the entire duration of the experiment. Peer SCs are assumed to be altruistic w.r.t. VM borrowing. The market parameters for the SCs and the customers are shown in Tables 1 and 2 respectively. We simulate a perfect price competition game between the SCs, and use the tatonnement process (TP) [29] to converge to a static market equilibrium in practice for a distributed setting. Tatonnement is a trial-and-error process similar to the hill climbing approach in local search theory by which equilibrium is reached in competitive markets via a distributed fashion. As a measure of static market efficiency we investigate and compare the utilitarian SW function values at market equilibrium for utilitarian, egalitarian, and Rawlsian (see Section II.C for more details) regulators. Note that the utilitarian SW function reflects the net stakeholder utility, and out goal is to study the net stakeholder utility at market equilibrium for regulators with different utility equitability mindsets. For the parameter values in Tables 1 and 2, we run numerical evaluations for all possible permutations (instances) of values that are applicable to SCs and their customers, and report the mean value of the results obtained (with the exception of Figure 1c which reports (without loss of generality) on individual permutations). Note that each permutation of values can be considered as a different market setting. To experiment on dynamic markets, as a representative example, we fix to be the same for all SCs and vary it in the interval . Similarly we fix to be the same for all 15 customers and vary it in the interval . We also make and to be equal for all customers and vary it in the interval . To provide a rationale behind the values for the time constant, , first note that it represents the market time scale for the update of prices. Small values of implies a fast update in real-time price, which can introduce volatility. On the other hand, high values of contributes to reduced volatility. The time constant, , represents the reciprocal of consumer demand elasticity.

SC#
SC1 0 200 0.6 -0.3 90
SC2 0 200 0.2 -0.6 102
SC3 0 250 0.6 -0.25 80
SC4 0 250 0.6 -0.25 80
SC5 0 200 0.2 - 0.01 20
TABLE I: Parameters of Cost Functions for SCs
Customer(C)#
C1, C2 60 100 0.1 -0.5 168
C3, C4 60 100 0.1 -0.15 140
C5, C6 70 80 0.2 -0.35 140
C7, C8 20 60 0.2 -0.2 100
C9, C10 30 60 0.2 -0.3 120
C11, C12 20 40 0.2 -0.1 125
C13, C14, C15 30 60 0.2 -0.5 135
TABLE II: Parameters of Utility Functions for Customers

V-B Analysis of Evaluation Results

In the first part of this section, we analyze SC cost and customer utility allocations at market equilibrium under the utilitarian, egalitarian, and Rawlsian SW paradigms. In the second part, we analyze the stability of various dynamic market settings, and also how fast a dynamic market converges to a stable equilibrium.

Static Market Equilibrium Performance. Using TP and in the presence of a regulator with different social welfare (SW) mindsets, we arrive at a different single market equilibrium (ME) maximizing SW. Note here that ME might not be unique, and in this case TP will converge locally to a ME in a distributed manner. The regulator will then have the option to work upon the ME to maximize SW. We observe (as a mean of multiple instances) from Figures 1(a&b) that with respect to SC and customer allocation ratio equitability, Egalitarian MEs are the best as they ensure nearly identical cost and utility allocation ratios across all autonomous SCs and customers respectively, followed closely by Rawlsian MEs, and utilitarian MEs that are not very fair (equitable) in the utility allocation sense. Here, we define allocation ratio as the ratio of the cost (utility) of SC (customer) to the maximum cost (utility) of any SC (customer) at ME, for each given market type. On the other hand, we see that market equilibrium in utilitarian markets, MEs lead to a considerably greater additive stakeholder satisfaction (utility) (see Figure 1c.) when compared to egalitarian and Rawlsian markets., i.e., the utilitarian SW metric is highest in utilitarian markets. This is true from theory as marginal stakeholder utility at utilitarian ME is equal across all stakeholders. In addition, from theory, SW maximizing ME in utilitarian competitive markets are always Pareto optimal. In Figure 1c, , , denotes the utilitarian social welfare value at the optimal market situation of type , and is the ratio of the utilitarian social welfare value at the optimal market situation of type to the optimal utilitarian social welfare at utilitarian ME.

Dynamic Market Stability Performance. Through Figures 2, 3, and 4, we study dynamic markets for three different instances of pairs, for utilitarian, Rawlsian, and egalitarian market types. For each instance, and for any market type, we observe that low values of for a given instance correspond to market instability, i.e., a state of disequilibrium, because they imply a fast update in SC prices charged to customers, indicating market volatility in supply and demand as well. Here, stability is indicated through the maximum of the eigenvalues of Hurwitz matrix (see Section III) formed from the market instance, which are negative in the stable zone, and positive in the unstable zone. It is logical to expect that market instability can be reduced if the price update is slower, i.e., if is larger. In this regard, we observe that the reduction in market instability takes place the slowest for egalitarian market types because in such markets there is a strict requirement of absolute stakeholder utilities to be equal at ME. The reduction in market instability is the fastest for utilitarian market types. We also observe from figures 2-4 that market volatility is increased due to a decrease in values because the latter trend corresponds to the increase in demand elasticity which contributes to a market being volatile. We infer from the plots that it is possible to design an SC market where volatility (arising due to either low or low values) can be contained by increasing market latency, i.e., increasing values. With respect to the speed of convergence, from Figures 2-4, we observe in general that SC markets converge fast to the stable zone, i,e., even at low values of , but the speed of convergence increases with increasing values. This is because increasing values indicate more demand curtailment by SC customers, potentially leading to non-volatility in supply and prices.

Fig. 1: ME Performance (a) SC Costs (left), (b) Customer Utilities (middle), (c) Social Welfare Ratio (right) w.r.t to various SW metrics
Fig. 2: Market Stability Performance when (a) (left), (b) (middle), (c) (right) [Utilitarian]
Fig. 3: Market Stability Performance when (a) (left), (b) (middle), (c) (right)[Rawlsian]
Fig. 4: Market Stability Performance when (a) (left), (b) (middle), (c) (right)[Egalitarian]

Vi Related Work

We give an overview of efforts related to ours and highlight the relevant differences. Works on hybrid clouds [30, 31] are related as they allow private (or smaller-scale) clouds to outsource their requests to large-scale public providers. However, since that can potentially be costly for a small-scale provider, our work differs in that it focuses on a sharing framework, while minimizing cost of using public clouds.

Earlier efforts also study the competition and cooperation within a federated cloud. For instance, authors in [32, 33] characterize the cloud federation to help cloud providers maximize their profits via dynamic pricing models. Earlier efforts [34, 35, 36, 37] also study the competition and cooperation among cloud providers, but assume that each cloud provider has sufficient resources to serve all users’ requests, while [35] incorporates a penalty function to address the service delay penalty. Authors in [38] propose a hierarchical cooperative game theoretic model for better resources integration and achieving a higher profit in the federation. Similarly to our work, [39] studies a federation formation game but assumes that cloud providers share everything with others, while [40] adopts cooperative game theoretic approaches to model a cloud federation and study the motivation for cloud providers to participate in a federation.

Another line of work focuses on designing sharing policies in the federation to obtain higher profit. For instance, [41] proposes a decentralized cloud platform SpotCloud [42], a real-world system allowing customers or SCs to sell idle compute resources at specified prices, and presents a resource pricing scheme (resulting from a repeated seller game) plus an optimal resource provisioning algorithm. [43] employs various cooperation strategies under varying workloads, to reduce the request rejection rate (i.e., the efficiency metric in [43]). Another effort [44] combines resource outsourcing and rejection of less profitable requests in order to increase resource utilization and profit. [45] proposes to efficiently deploy distributed applications on federated clouds by considering security requirements, the cost of computing power, data storage and inter-cloud communication. [46] groups resources of various SCs into computational units, in order to serve customers’ requests. [47] proposes to incorporate both historical and expected future revenue into VM sharing decisions in order to maximize an SC’s profit.

Differences and Drawbacks. Our work is a necessarily important theoretical extension of a very recent analytical work in [2] that was the first of its kind in the analysis of small cloud markets. There, the authors considered consequences of performance (i.e., queueing theory) driven non-cooperative game-theoretic (with no SC willing to share its utility and capacity information with others, i.e., an incomplete information game-theoretic setting) resource sharing on the resulting performance delivered to customers at static market equilibrium, something not considered by any of the above-mentioned efforts. However, [2] does not consider the important problem of analyzing equilibrium stability under variations in SC resource availability, in a non-cooperative game-theoretic SC environment. Without showing the existence of a stable SC market, one, based on the existing results showing the existence of a market equilibrium, cannot not say much regarding the sustainability of SC markets in the future. A characterization of this scenario is an important contribution of this work. A major difference of our work with the one in [2], is the lack of a queuing-driven performance model to reduce the equilibrium search space. However, our work is orthogonal in the sense that, given the existence of (efficient) market equilibria, we investigate whether such a state is sustainable in the long run.

Vii Conclusion and Future Work

In this paper, we addressed the problem of effective resource sharing between small clouds (SCs). We modeled the problem as an efficient supply-demand market design task consisting of (i) autonomous SCs, (ii) their customers, and (iii) a regulator, as the market stakeholders. We first showed that a welfare allocation policy for the stakeholders by the regulator maximizes utilitarian social welfare at the static market equilibrium and results in the best/most efficient state at which the SC markets could operate. Fortunately, courtesy Arrow-Debreu welfare theorems in welfare economics, this unique optimal operating point is also achieved in a distributed manner by the autonomous SCs in perfect price competition with one another, thereby guaranteeing no efficiency loss in a non-centralized market setting. However, the optimal market equilibrium point is prone to perturbations due to the dynamic nature of the SC market, thereby potentially leading to market disequilibrium. In this context, we designed a dynamic market mechanism based on Arrow and Hurwicz’s disequilibrium process that uses the gradient play technique in game theory to converge upon the optimal static market efficient equilibrium from a disequilibrium state caused due to supply-demand perturbations, and results in market stability. We illustrated the stability and sustainability of dynamic SC markets and the high speed with which such markets converge to stable equilibria, through numerical experiments.

A comment on Heterogenous VMs - In out work, we have modeled a homogenous VM case. In practice, each cloud provider offers heterogeneous VM profiles (e.g., memory-optimized, CPU-optimized, or GPU-enabled), which reserve hardware resources on pre-specified machine pools shared by multiple VMs [48]. However, many cloud providers, such as Amazon LightSail, DigitalOcean, and Linode, offer VM configurations with very similar specifications (e.g., $10/month instances from Linode, DigitalOcean, and Amazon Lightsail currently provide 1 CPU core, 30 GB SSD, 2 TB data transfer/month, 1 or 2 GB of RAM). We believe that it is very likely that SCs would negotiate the sharing policies for each VM profile separately, given that these profiles correspond to different prices and capacities at each SC. In this case, our model of homogeneous resources can be applied repeatedly to each VM profile. Sharing policies for hardware resources (rather than VM profiles) would require the introduction of scheduling and packing algorithms within our performance model, which is beyond the scope of this work.

As part of future work, we plan to design provably fast distributed algorithms to allow markets to roll back to efficient equilibria when perturbed from an equilibrium state, and study dynamic SC markets under (i) a setting of imperfect competition between SCs, and (ii) a coalitional market setting where SCs have the capability to collude with one another.

References

  • [1] H. Zhuang, R. Rahman, and K. Aberer, “Decentralizing the cloud: How can small data centers cooperate,” in IEEE International Conference on Peer-to-Peer Computing, 2014.
  • [2] S.-H. Lin, R. Pal, M. Paolieri, and L. Golubchik, “Performance driven resource sharing markets for the small cloud,” in Distributed Computing Systems (ICDCS), 2017 IEEE 37th International Conference on.   IEEE, 2017, pp. 241–251.
  • [3] K. J. Arrow and L. Hurwicz, “On the stability of the competitive equilibrium i,” Econometrica, vol. 26, no. 4, 1958.
  • [4] K. J. Arrow, H. D. Block, and L. Hurwicz, “On the stability of the competitive equilibrium ii,” Econometrica, vol. 27, no. 1, 1959.
  • [5] M. Armbrust, A. Fox, R. Griffith, A. D. Joseph, R. H. Katz, A. Konwinski, G. Lee, D. A. Patterson, A. Rabkin, I. Stoica, and M. Zaharia, “A view of cloud computing,” Communications of the ACM, vol. 53, 2010.
  • [6] “Amazon AWS,” http://aws.amazon.com.
  • [7] “Google Compute Engine,” https://cloud.google.com/compute.
  • [8] “Microsoft Azure,” https://azure.microsoft.com.
  • [9] “Blippex - Why we moved away from AWS,”
    http://blippex.github.io/updates/2013/09/23/why-we-moved-away-from-aws.html.
  • [10] “As Private Cloud Grows, Rackspace Expands Options Inside Equinix,” https://blog.equinix.com/blog/2016/05/16/as-private-cloud-grows-rackspace-expands-options-inside-equinix/.
  • [11] J. C. Mogul and L. Popa, “What we talk about when we talk about cloud network performance,” ACM SIGCOMM Computer Communication Review, 2012.
  • [12] “It’s clear to Linode: There’s a market to bring cloud services to small companies,”
    http://www.njbiz.com/article/20131104/NJBIZ01/311019994/It.
  • [13] Wikipedia, Business Cluster.   http://en.wikipedia.org.
  • [14] N. Samaan, “A novel economic sharing model in a federation of selfish cloud providers,” IEEE Transactions on Parallel and Distributed Systems, vol. 25, no. 1, 2014.
  • [15] B. Rochwerger, D. Breitgand, A. Epstein, D. Hadas, I. Loy, K. Nagin, J. Tordsson, C. Ragusa, M. Villari, S. Clayman, E. Levy, A. Maraschini, P. Massonet, H. Munoz, and G. Toffetti, “Reservoir: When one cloud is not enough,” Computer, vol. 44, no. 3, 2011.
  • [16] B. Sotomayor, R. S. Montero, I. M. Llorente, and I. Foster, “Virtual infrastructure management in private and hybrid clouds,” IEEE Internet Computing, vol. 13, no. 5, 2009.
  • [17] A. Mas-Collel, M. D. Whinston, and J. R. Green, Microeconomic Theory.   Oxford University Press, 1995.
  • [18] J. S. Shamma and G. Arslan, “Dynamic fictitious play, dynamic gradient play, and distributed convergence to nash equilibria,” IEEE Transactions on Automatic Control, vol. 50, no. 3, 2005.
  • [19] D.Fudenberg and J.Tirole, Game Theory.   MIT Press, 1991.
  • [20] S. D. Flam, “Equilibrium, evolutionary stability, and gradient dynamics,” International Game Theory Review, vol. 4, no. 4, 2002.
  • [21] G. Ananthnarayanan, M. Hung, X. Ren, I. Stoica, A. Wierman, and M. Yu, “Grass: Trimming stragglers in approximation analytics,” in NSDI, 2014.
  • [22] R. Pal, S.-H. Lin, and L. Golubchik, “Submission appendix,” https://www.dropbox.com/s/as7nqibz9i9h0u8/appendixtnotes.pdf, 2017, [Online; accessed 17-Sept-2017].
  • [23] O. Candogan, K. Bimpikis, and A. Ozdaglar, “Optimal pricing in networks with externalities,” Operations Research, vol. 60, no. 4, 2012.
  • [24] J.-J. Laffont and J. Tirole, A Theory of Incentives in Procurement and Regulation.   MIT Press.
  • [25] J. Bulow, J. Geneakolos, and P. Klemperer, “Multi-market oligopoly: Strategic substitutes and complements,” Journal of Political Economy, vol. 93, 1985.
  • [26] S.Boyd and L.Vanderberghe, Convex Optimization.   Cambridge University Press, 2005.
  • [27] Y. Nesterov and A. Nemirovski, Interior-Point Polynomial Algorithms in Convex Programming.   SIAM Studies in Applied Mathematics, 1994.
  • [28] A. Bacciotti and L. Rosier, Liapunov Functions and Stability in Control Theory.   Springer, 2005.
  • [29] D. P. Bertsekas and J. Tsitsiklis, Parallel and Distributed Computation.   Prentice-Hall, 1992.
  • [30] H. Zhang, G. Jiang, K. Yoshihira, and H. Chen, “Proactive workload management in hybrid cloud computing,” IEEE Transactions on Network and Service Management, 2014.
  • [31] M. Shifrin, R. Atar, and I. Cidon, “Optimal scheduling in the hybrid-cloud,” in Integrated Network Management (IM 2013).   IEEE, 2013.
  • [32] Í. Goiri, J. Guitart, and J. Torres, “Economic model of a cloud provider operating in a federated cloud,” Information Systems Frontiers, 2012.