On the Posted Pricing in Crowdsourcing: Power of Bonus

On the Posted Pricing in Crowdsourcing: Power of Bonus

Suho Shin,  Jungseul Ok,  Hoyong Choi,   and Yung Yi Suho Shin, Hoyong Choi, and Yung Yi are with the Department of Electrical Engineering at Korea Advanced Institute of Science and Technology, Daejeon, South Korea (e-mail: {shinas327, chy0707, yiyung}@kaist.ac.kr). Jungseul Ok is with the Department of Electrical Engineering at KTH Royal Institute of Technology, Stockholm, Sweden (e-mail: ockjs1@gmail.com).
Abstract

In practical crowdsourcing systems such as Amazon Mechanical Turk, posted pricing is widely used due to its simplicity, where a task requester publishes a pricing rule a priori, on which workers decide whether to accept and perform the task or not, and are often paid according to the quality of their effort. One of the key ingredients of a good posted pricing lies in how to recruit more high-quality workers with less budget, for which the following two schemes are considered: (i) personalized pricing by profiling users in terms of their quality and cost, and (ii) additional bonus payment offered for more qualified task completion. Despite their potential benefits in crowdsourced pricing, it has been under-explored how much gain each or both of personalization and bonus payment actually provides to the requester. In this paper, we study four possible combinations of posted pricing made by pricing with/without personalization and bonus. We aim at analytically quantifying when and how much such two ideas contribute to the requester’s utility. To this end, we first derive the optimal personalized and common pricing schemes and analyze their computational tractability. Next, we quantify the gap in the utility between with and without bonus payment in both pricing schemes. We analytically prove that the impact of bonus is negligible in personalized pricing, whereas crucial in common pricing. Finally, we study the notion of Price of Agnosticity (PoA) that quantifies the utility gap between personalized and common pricing policies, where we show that PoA is not significant under many practical conditions. This implies that a complex personalized pricing with more privacy concerns can be replaced by a simple common pricing with bonus, if designed well. We validate our analytical findings through extensive simulations and real experiments done in Amazon Mechanical Turk, and provide additional implications that are useful in designing a pricing policy in crowdsourcing.

1 Introduction

1.1 Motivation

Crowdsourcing system is a popular tool to solve problems which involve huge dataset but can be decomposed into simple tasks, where the tasks are electronically distributed to numerous workers who are willing to perform the tasks at low cost. It is useful, in particular, when the tasks, e.g., image classifications, are easy for human but hard for computer. People have paid much attentions to the following two directions in crowdsourcing: (i) post-processing dataset and (ii) worker quality control . In (i), once a task is given to workers, since payment is typically low and tasks are tedious, error is common even among those who are willing. This is further complicated by abundant spammers trying to make easy money with little effort. To handle this, the requester adds redundancy by assigning the same task to multiple workers, and from these redundant answer data set, some nice statistical inference techniques such as majority voting or belief propagation are applied to infer the correct answer to the task, e.g., [1, 2, 3, 4]. However, there should be unavoidable limitations in terms of guaranteeing a target precision when workers inputs are highly erroneous [5, 6]. In (ii), it would be highly beneficial to recruit as many high-quality workers as possible, which is possible by incentivizing workers to produce qualified output, see e.g., [7, 8]. In crowdsourcing, workers get paid as a reward of effort on submitting answers. Many experiments reveal that not only the quantity of users’ labels [9, 10], but also the quality of their labels significantly improve [11, 12, 13], if the payment for incentivization is properly designed. There exist many researches on efficient mechanism design for pricing in crowdsourcing, ranging from sealed bid auction to posted pricing (see more details in Section 1.3).

Out of many pricing mechanisms, practical crowdsourcing systems such as Amazon Mechanical Turk popularly use posted pricing due to its simplicity and convenience to workers, where a task requester publishes a pricing rule a priori, based on which each worker decides on the task acceptance and its completion. As in other pricing mechanisms, in posted pricing, recruiting and discouraging high-quality and low-quality workers, respectively, is the key, for which the following two ideas are natural: (i) personalized pricing by offering individual price to each worker based on her quality and cost and (ii) giving additional bonus (in addition to base payment just for participation) to workers with more qualified task completion. However, it has been largely under-explored how much gain each or both of personalization and bonus actually provides to the task requester.

1.2 Main Contribution

Personalized pricing (PP) that individually treats workers based on their cost and quality profiles obviously seems superior to the pricing without personalization, referred to as common pricing (CP) in this paper, which offers a uniform pricing to the entire workers. However, PP is obviously more complex and more privacy-vulnerable, especially for the tasks with a large number of workers. A simple option of giving bonus for more qualified workers in both PP and CP should provide more power of controlling workers’ behaviors, so as to help in increasing the requester’s utility. In this paper, we essentially consider four possible combinations of posted pricing made by personalized/common pricing and those with/without bonus payment, from which we quantify when and how much personalization and bonus payment contribute to the requester’s utility. We summarize our main contributions in what follows:

  • Optimal personalized pricing. In Section 3, we first derive the optimal pricing schemes (or their structures) in PP and CP. In PP, we prove that computing an optimal scheme is equivalent to solving the generalized 0-1 knapsack problem (GKP), which is known to be NP-hard. Using their equivalence, it is possible to inherit the approximation algorithms in GKP, which were developed under some special conditions of the objective utility function in literature, and we apply them to develop an approximation algorithm in designing an efficient scheme in PP. However, our investigation of the utility functions from popular crowdsourcing tasks in practice motivates us to study the unexplored conditions in literature: subadditivity and Schur-convexity. Under these conditions, we prove that a greedy-type algorithm achieves 1/2-factor approximation of the optimal PP, which can also be applied to GKP with a slight modification.

  • Optimal common pricing. In Section 4, we characterize the structure of the optimal way of pricing in CP. To this end, we assume a certain relation between workers’ cost and quality, motivated by our empirical study in Section 7.2. This relation is divided into three regimes: effort-unresponsive, effort-subresponsive, and effort-responsive, depending on how workers’ quality grows with increasing effort. For all three regimes, we prove that the optimal pricing schemes are highly structured, which is crucial to develop three polynomial-time algorithms, being in sharp contrast to PP. The time complexities are at most where is the pricing budget that may scale with the number of workers

  • Power of bonus and PoA (Price of Agnosticity). Using the optimal pricing results of PP and CP in Sections 3 and 4, in Section 5, we first prove that bonus is a marginal device in the optimal PP, whereas it is of significant importance in the optimal CP. This is intuitive because personalization has sufficient power to control workers’ behavior in PP, but bonus may be the only way to differentiate high and low-quality workers in CP. Next, we draw an interesting result that CP with bonus is comparable to PP by quantifying the notion of Price-of-Agnosticity (PoA) that is the utility gap between the optimal PP and CP schemes. This is somewhat surprising because a simple way of unified (base, bonus) payments is practically enough to achieve a comparable performance to personalized pricing. This enables task requesters to ensure that a simple pricing such as common pricing, if properly designed, is able to provide a large amount utility.

  • Implications from simulation and real experiment. To validate our analytical findings, we provide extensive simulation results, and perform various experiments on a real-world platform, Amazon Mechanical Turk. Especially, from our real experiment reveals the significant importance of bonus in common pricing, where without bonus, workers do not take the task even with excessive base payment offered, whereas with only small bonus, workers invest a large amount of effort of performing given tasks. This has a good match with our theoretical result on the impact of bonus. Furthermore, we derive other useful implications, in particular, on efficient pricing policies that deliver quick and qualified responses from workers.

Major CS app. Model Paper
Contest All-pay auction [14, 15, 16, 17]
Ecommerce Sealed-bid auction [18, 19, 20, 21, 22]
Microtask Posted price [23, 24, 25, 26, 27, 28, 29]
Microtask Misc. [30, 31, 32, 33]
Table 1: Related work: CS (CrowdSourcing)

1.3 Related Work

Posted Pricing. Deriving the optimal pricing becomes the problem of solving the knapsack optimization problem for which the following prior work exists. The authors in [24, 25] suggest approximation algorithms in the bayesian setting, assuming (a) the complete knowledge of the probability distribution of each worker’s cost and quality, and (b) submodularity or additivity of the utility function. The authors in [27] consider the problem which is dual to [24], wherein with a given utility constraint the objective is to minimize the cost. The authors in [23] study the posted pricing in a large market setting where the cost of a single worker is very small compared to the budget. In [34], it is revealed that the online cloud resource problem can be expressed nicely by a posted pricing mechanism, showing its equivalence to a variant of an online knapsack problem and proposing an optimal posted pricing mechanism in terms of the competitive ratio. Other types of on-line setting for posted pricing have been studied, where the requester gradually adapts to heterogeneous worker qualities or task difficulties. In [35], the authors consider online worker arrivals with the objective of maximizing the number of solved tasks. The authors in [26, 28] study this problem in the context of posted pricing for online procurement auction. They again assume online arrivals of heterogeneous workers and formulate a problem in the multi-armed bandit framework, providing an analysis of UCB (Upper Confidence Bound) algorithm. The authors in [29] study a similar problem, but for homogeneous workers and heterogeneous tasks.

Auction-based pricing. All-pay auction has been studied as a good pricing [16, 15, 14, 22, 36] in crowdsourcing, where all workers obligatorily pay their bids (i.e., efforts) regardless of the allocated items, and then get paid from the task requester based on their efforts. They propose the algorithms with a certain approximation ratio of various bidding strategies, for different forms of revenue functions, especially in crowdsourcing contests. Another mechanism is the sealed-bid auction, where each worker first submits her bid for a set of tasks and the amount of charge she claims for performing such a set of tasks. Since the real world crowdsourcing system is not operated in such a one-shot manner, they fill such a gap using the concept of repetition of sealed-bid auctions. The requester tries to appropriately profile the task difficulties and worker qualities in an adaptive manner to maximize their revenue under the budget constraint, e.g., [22, 21, 20, 18]. To avoid the computational intractability of an optimal winning bid allocation for maximizing the utility, they suggest a greedy-style algorithm mechanism, and prove its truthfulness, individual rationality, and budget feasibility. The authors in [19] consider the hardness in the verification of submitted quality and propose a cost-effective mechanism with guaranteed symmetric Nash Equilibrium.

Bonus. There exists an array of prior work which studies bonus in various pricing scenarios. The authors in [37, 38] conduct a massive real-world experiment to measure the performance of various pricing mechanisms with bonus. In [30, 31], the problem of designing a crowdsourcing platform has been studied, assuming that workers try to align their output, and the authors analyze an equilibrium of the system regarding the proposed incentivizing algorithms. The authors in [32, 33] redefine the concept of incentive-compatibility, and propose an axiomatic approach to find an incentive-compatible mechanism.

2 Model

We consider a set of workers, where is the number of workers who are eligible to perform a target task given a certain time deadline. Each worker is associated with a profile if she can produce the output of (expected) quality at cost . In the worker ’s profile, quality corresponds to the expected individual contribution to the task and cost corresponds to the opportunity cost of the time to perform the task. The task is associated with a utility, which corresponds to the satisfaction given to the task requester and is a function of the expected contributions from workers, where we denote if worker participate the task and otherwise. More formally, we associate the task with the utility function for some , where , and is the element-wise product of two sequences of the same length, i.e., We assume is symmetric111A function is symmetric, when for all , if there exists an ordering such that for each . and non-decreasing222A function is non-decreasing, when for all , if for each .. This assumption means that worker’s contribution is equally and always welcome.

Example: Typo correction task. Consider a typo correction task with hidden typos, where a natural utility function is the number of expected typos corrected by at least one worker. Let be the probability of worker of correcting a single typo independently. If the requester has some fixed number and regards the quality as the probability of correcting at least typos, the mapping between and is such that with invertible function on . Thus, we have the following utility function:

(1)

where is the probability that worker fails to find and correct a single typo. Then, we can easily see that the above utility function is symmetric and non-decreasing, with increasing and . We also present other examples of utility functions in appendix.

Pricing policy with bonus. We consider a posted pricing scheme to recruit workers, where a fixed pricing policy is posted before workers perform the task, while the pricing policy may depend on a given worker profile . We particularly consider pricing policies with bonus, where bonus payment is additionally paid to workers who accomplish good-quality output. To formalize, we denote it by , where each worker is paid dollars as a base payment for participation, and at most dollars as a bonus payment proportional to the quality of her contribution. Hence, quality can be interpreted as the normalized expectation of bonus payment. We note that the pricing policy includes the worker qualification and thus the definition of quality or individual contribution, which often play a crucial role in designing a pricing policy.

Worker behavior. Under the belief that the requester has a mechanism to perfectly evaluate each worker’s contribution333This becomes possible in practice, by aggregating all workers’ answers and run some inference algorithms introduced in Section 1.3, whose inference accuracy increase as we have more redundancy, i.e., large ., and each worker knows her own profile with her payoff . We assume rational workers each of whom strategically maximizes her individual expected payoff, for given pricing policy. More formally, for given , the strategic decision of worker is denoted by :

(2)

where is the indicator function of such that if is true, and otherwise. We stress that the choice of quality indeed changes each worker’s decision. For example, recall the typo correction example, where each worker has the probability to correct a typo at , while quality depends on the definition of qualified worker, i.e., is a function of the minimum number of typo corrections required for bonus . Hence, even for the same value of and the same set of workers, the set of participating workers can change depending on the definition of quality.

Personalized pricing and common pricing. We classify post pricing policies into two groups: personalized pricing (PP) policies, and common pricing (CP) policies, depending on whether policy provides personalized pricing for each worker, or not. Mathematically, since a CP policy decides every worker’s payment using a common policy, it can be simplified into a single pair of base and bonus payments where and for each worker . Hence, the set of all CP policies is a subset of all PP policies. Despite PP policy’s more controllability on workers’ behavior and thus more expected utility, it seems useful to CP policy, since PP policy is often discouraged in practice, e.g., Amazon Mechanical-Turk [39]. Also, PP policy requires more sophisticated and detailed information on worker profile, leading to privacy concerns, whereas the CP policy needs only a summarizing property of the entire worker profiles rather than individual ones.

2.1 Problem Formulation

Our goal is to find an optimal pricing policy which maximizes the task utility given worker profile and budget constraint , i.e., the sum of expected payments to workers does not exceed . We first define the optimal PP policy as a solution of optimal personalized pricing problem (OPP) in the following:

(3a)
subject to (3b)
(3c)

where (3b) corresponds to the assumption on the rational workers, and (3c) refers to the budget constraint . We note that the control variables of (3) is the personalized pricing policy

In CP, the pricing policy is forced to be indistinguishable for all workers, while each worker can react differently since the expected worker utility still depends on individual profile . In this scenario, the control variables in (3a) are simplified into a single pair of base and bonus payments , and the following optimization problem is considered:

(4a)
subject to (4b)
(4c)

As mentioned earlier, note that the optimal PP policy outperforms the optimal CP one since the distribution of the CP policy is agnostic on workers, while that of the PP policy discriminates workers. However, OCP requires only partial information on worker profile, e.g., how many workers have some specific profile, whereas OPP needs to know the individual profile for each worker.

In both OPP and OCP, we have the following tensions: If the requester sets high base and low bonus, it may fail to recruit high-quality users and utilize only low-cost users. If bonus is set too high with low base, it should spend a huge amount of bonus on high quality users, so that the total number of recruited users shrinks. Thus, it is necessary to strike a good balance between base and bonus payments while satisfying the budget constraint to maximize the offered utility. This paper aims at addressing the following three main questions:

  1. Optimal pricing policy. What are the optimal base and bonus payments for both PP and CP policies? Are they easy to compute, and if hard, what are the approximate algorithms and how are their qualities?

  2. Power of bonus. We exploit the concept of bonus in posted pricing model. Does it actually improve the utility? How beneficial is the bonus payment in PP and CP policy?

  3. Price of Agnosticity. How much utility does the requester lose by pricing workers in a unified manner rather than by pricing them in a personalized manner?

3 Optimal Personalized Pricing

We begin with considering a well-known generalized 0-1 knapsack problem, described (GKP) in the following:

(5a)
subject to (5b)

We make the correspondence between solutions of OPP in (3) and GKP in (5), as stated in Theorem 3.1.

Theorem 3.1

If is a solution of OPP in (3), then a solution of GKP in (5) is given as in (2). Conversely, if is a solution of GKP in (5), then a solution of OPP in (3) is given as

(6)

where for each .

The proof of Theorem 3.1 is presented in Section 6. Note that by selecting in (6), we always find an optimal PP policy with zero bonus, i.e., . This implies that the bonus in PP policy is not mandatory since personalization in pricing is enough to control workers’ behavior without bonus payments. However, finding such optimal PP policy is computationally intractable for a general utility function. Indeed, it is well-known that even a special case of GKP, e.g., GKP with an additive utility, is NP-hard. Hence, recalling the equivalence between OPP and GKP in Theorem 3.1, OPP is also NP-hard. To make the problem tractable, we will consider utility functions that have some structures. We first review the following notion of majorization which compares two sequences of real numbers:

Definition 3.2

For , let denote the sequence with the same components, but sorted in descending order. Given , we say that weakly majorizes and denote if

(7)

If and additionally , we say that majorizes , and denote .

We now consider utility function with either of the following conditions:

  1. is subadditive, i.e., for all , and Schur-convex, i.e., for all such that

  2. is additive, i.e., for all .

Note that additivity in 2 implies subaddivity and Schur-convexity in 1, i.e., 1 is a more general condition than 2. Considering 1 and 2 is motivated by practical tasks such as typo correction task or labeling task. Subadditivity can be interpreted as diminishing returns of workers’ effort, which is often observed in practice since workers produce duplicated or redundant answers. Schur-convexity captures that a group of high-quality and low-quality workers perform no worse than a group of average-quality workers even though those groups have the same average worker quality, since the objective is often dominated by outputs from a few experts. Indeed, the utility function in (1) for the typo correction task with satisfies 1. In Appendix, we provide a proof of subadditivity and Schur-convexity of (1) for . Moreover, changing the utility function to the expected number of typos corrected by workers leads to an additive utility function. We remark that even the additive utility has a large set of applications such as survey or sensing.

3.1 Approximation Algorithm for OPP

We recall that under 1 or 2, OPP is still NP-hard, since GKP with 2 corresponds to the classical 0-1 knapsack problem, known to be NP-hard. However, it is reported that a greedy algorithm works well for the 0-1 knapsack problem with additive utility functions [40]. Inspired by this, we propose ModifiedGreedy in Algorithm 1 for OPP. In ModifiedGreedy, we use the notion of bang-per-buck which denotes the ratio of quality to cost. All the workers are aligned by their values of bang-per-buck, and the top workers or only one worker with the highest quality is recruited. ModifiedGreedy has polynomial complexity with respect to and the following guarantee under 1:

Theorem 3.3

Under 1 (and thus 2), ModifiedGreedy outputs a PP policy achieving at least , where is the optimal utility value of OPP in (3).

The proof is presented in Section 6.2. We note that under the stronger condition 2, the utility function becomes separable. The separability in 2 provides stronger approximation guarantee using fully polynomial-time approximation scheme (FPTAS) with approximation ratio of for the knapsack problem, see, e.g., [41, 42]. However, under 1, the objective function is not separable. For the 0-1 knapsack problem, such a non-separable objective makes the performance guarantee more difficult, and thus a somewhat restrictive class of objective functions is considered in prior work, e.g., exponential functions [43], quadratic functions [44], and submodular function with approximation ratio [45, 46]. Theorem 3.3 provides a constant-factor approximation for more general conditions of subadditivity and Schur-convexity for the 0-1 knapsack problem. This is of broad interest to the applications which can be formulated by the 0-1 knapsack problem.

0:  
1:  
2:  Obtain ordering of such that , where , i.e., sort in the descending order of bang-per-buck .
3:  while  do
4:     
5:     
6:  end while
7:  Find where is the sequence such that if and otherwise.
8:  Set if , or otherwise.
9:  return  
Algorithm 1 ModifiedGreedy

4 Optimal Common Pricing

In this section, we now study the optimal common pricing problem (OCP) in (4), that has much less freedom in choosing the price, yet used in many practical crowdsourcing systems. Despite the dimensional reduction of OCP, compared to OPP, we still find that solving OCP has a fundamental hardness due to the non-convex utility function and non-convex constraint even for an additive utility function. In order to handle such challenge in finding a global optimum in OCP, we introduce an assumption of one-to-one correspondence of worker’s quality and cost. In detail, we assume that a quality of any worker follows a monotone increasing mapping , where . Moreover, we assume that is twice-differentiable on its domain. This assumption is motivated by the empirical studies which reveal that according to the type of the tasks, induced profile distribution has quite specific form [38]. Following the notion of task types used in [38], we introduce the following regimes of workers’ profile:

Definition 4.1

We divide the class of function into the three regimes:

  • Effort-unresponsive. for .

  • Effort-subresponsive. and for .

  • Effort-responsive. and for .

Technically, the size of and decides whether the bang-per-buck value, is increasing with respect to or not, since .

In this context, the provided regimes can be interpreted using the behavior of bang-per-buck of workers. In effort-unresponsive regime, bang-per-buck tends to decrease as cost increases. This naturally describes a scenario that there exists no significant improvement in worker’s quality even when cost increases. This regime is considered to be highly practical, assuming the real world behavior of workers against tasks which is quite trivial for many people, as will be verified in our experiments (see Section 7.2). Some instances of task which reside in this regime include audio transcription task and handwriting recognition task [38]. In contrary, bang-per-buck could possibly be decreasing when is low, but it will eventually be increasing when becomes high. This oppositely describe the scenario of large improvement in worker’s quality with respect to increasing cost, e.g., a task where the quality of workers largely differs with respect to invested effort. Hence for effort-responsive regime, it is expected that high-quality workers need to be recruited. Proofreading task and spot-the-difference task can be the examples for such a regime [38]. In effort-subresponsive regime, bang-per-buck value possibly starts to increase as cost increases until it achieves some point, but it finally increases with cost. Following the similar logic of the previous regime, it is expected that recruiting workers within a medium range of quality would lead to a good pricing strategy in this regime.

4.1 Structure of Optimal Common Pricing

In OCP, our goal is to find the optimal CP policy, denoted by a sinlge pair of base and bonus payments. To this end, we first define two notions of pricing structure in CP, which facilitates the characterization of the optimal pricing policy, as described shortly.

Definition 4.2

Let denote the index of worker who has -th highest quality, i.e., .555For notational simplicity, we abuse the same notation here as that in ModifiedGreedy. Consider a CP policy . The policy is said to be -picking when if , and otherwise, i.e., only workers with quality between -th highest to -th highest values take the task. Conversely, the policy is said to be -blocking when if , and otherwise, i.e., only workers with quality between -th highest to -th highest values decline the task.

With the notions of -picking and -blocking in Definition 4.2, Theorem 4.3 states that under each cost-quality regime, any pricing policy induces some special structure in recruiting workers, which further sheds light on the structure of the optimal pricing policy for three regimes.

Theorem 4.3

Any pricing in CP is -picking in effort-unresponsive regime, -blocking in effort-responsive regime, and -picking in effort-subresponsive regime, for some and with , whose values differ under each underlying regime and depends on the pricing policy

We present the proof in Section 6.3. Theorem 4.3 provides characterization of pricing policies in CP for three different cost-quality regimes. This characterization not provide an analytical closed-form, yet giving us a useful clue on how CP policy should behave, conditioned on the relation between quality and cost. Since such structural regularity holds for any pricing policy, an optimal pricing policy naturally inherits it. This result enables us to develop algorithms for computing in polynomial time in Section 4.2. Particularly in the effort-unresponsive regime, we are able to obtain a more detailed characterization of the optimal CP policy, which also allows us to develop a much simpler algorithm to solve OCP, as we will describe shortly.

4.2 Algorithms for Optimal Common Price

We now present three algorithms, each of which outputs the optimal CP for three different regimes, for which the key ingredients are the optimal structures obtained as in Theorem 4.3. In this subsection, we focus on providing the key ideas and the main results of three algorithms, and the full algorithms descriptions are provided in Appendix for the readers’ convenience. We name three algorithms for effort-subresponsive, effort-unresponsive, and effort-responsive regimes as CP-SUBRES, CP-UNRES, and CP-RES.

Effort-subresponsive regimes. As stated in Theorem 4.3, this regime requires the optimal pricing that is -picking. Using this structural property, CP-SUBRES operated with the following skeleton:

 

Skeleton of CP-SUBRES

 

  1. Sort the entire workers in decreasing order of their qualities, and reassign the index. Then, initialize the upper worker index

  2. For the index over search the index that provides the maximum utility over the pricing schemes that induces -picking and is budget-feasible.

  3. If then set and go to Step (ii), otherwise go to Step (iv).

  4. Compute the maximum pricing as follows:

 

The algorithm operates in such a way that we iterate the upper worker index , and for each upper index , we find that the lower worker index such that the pricing inducing -picking with satisfying the budget constraint (Steps (i)-(iii)). Then, we produce the output pricing as the optimal one that maximizes the utility over all per- best pricing schemes with budget feasibility in (iv).

All steps seem straightforward except for (ii). Note that for a fixed upper worker index the utility is monotone-increasing with respect to the lower worker index , i.e. as decreases. Thus, assuming that we can easily find the pricing that induces -picking and check whether satisfies the budget constraint or not, it is possible for us to utilize an efficient search method, e.g., the binary search algorithm for fast searching. We now discuss how to compute and check whether it is budget-feasible or not. A natural way of handling this is that we aim at finding , such that for the worker index and for other index and the budget constraint . Since many well-known algorithms to solve an LP basically assumed that there is no strict inequality constraint, we firstly need to deal with the strict inequality constraints. In here, we propose a method to handle such difficulties using a special structural property coming from our assumed regime i.e. the function in is concave, and the fact that our problem is only to find a feasible point. This will be elaborated in the proof of Theorem  4.4.

Effort-unresponsive regime. As stated in Theorem 4.3, the optimal pricing in effort-unresponsive regime is -picking, which is a special case of -picking, where Thus, CP-SUBRES can be applied to this regime without any modification. However, the fact that gives rise to an additional property that no base payment is necessary, and only bonus payment is enough for the optimal pricing (see the proof of Theorem 4.4). This additional property significantly simplifies the search process of CP-SUBRES so that it suffices to find in -picking. Technically, the linear programming step in (ii) of CP-SUBRES is not required. We describe CP-UNRES in Appendix, and Theorem 4.4 presents the running time complexity of CP-UNRES.

Effort-responsive regime. This regime requires the optimal pricing to be -blocking, which can be interpreted as a “dual” of effort-subresponsive regime. However, the search of and in this regime is highly similar to that of CP-SUBRES, as explained in what follows. While fixing upper index , we search the index over that provides the maximum utility over the pricing schemes that induces -blocking and is budget-feasible. Then, we linearly probe the index and find the optimal scheme among the upper index . The full algorithm CP-RES in this regime is described in Appendix.

Theorem 4.4 states the correctness of three algorithms and their running time complexity. Note that the given budget may scale with the number of workers for which we include it in the running time analysis in the theorem.

Theorem 4.4

With given budget CP-SUBRES and CP-RES output their optimal CP pricing schemes in times, respectively, and CP-UNRES outputs the optimal CP scheme in time.

5 Power of Bonus and Price of Agnosticity

In this section, we provide theoretical comparison between personalized and common pricing policies in terms of power of bonus (Q2) and price of agnosticity (Q3).

5.1 Power of Bonus

The impact of bonus payment may differ for both personalized and common pricing policies. A natural intuition is that such a power of bonus is weaker in personalized pricing than in common one. This is because in common pricing, the bonus payment may be the only device to filter out and compensate for high-quality workers’ effort, whereas personalized pricing has full capability to manipulate individual user’s behavior by matching individual payment to targeted user’s cost. Indeed, recalling (6) in Theorem 3.1, we can always find an optimal PP policy without bonus, where each worker is offered only as base payment if she is targeted, i.e., , and is offered nothing if she is not targeted, i.e., . This implies that the bonus payment does not help in increasing the optimal utility of personalized pricing.

We now study impact of bonus in common pricing policies. Our intuition is that there exists a certain amount of gain from the bonus payment, since without it, there is almost no way to incentivize high-quality workers, and to prevent inefficient payment to low-quality workers or cherry pickers. We quantify this intuition by providing an example, where the ratio of optimal utility of common pricing policy without bonus to that without bonus can be arbitrarily small.

Theorem 5.1

For any given and , consider an additive utility function such that , and worker profile given by:

if (8a)
if (8b)
if . (8c)

Then, for budget , the following holds for :

(9)

where and are the optimal utilities in OCP and OCP with additional constraint , respectively.

In this example, the worker profile in (8c) has three groups: cherry pickers in the first half (8a), mid-quality workers in the third quarter (8b), and high-quality workers in the last quarter (8c), An optimal CP policy without bonus, obtained as , fails to attract qualified workers, and wastes budget for only cherry workers. However, an optimal PP policy, obtained as , fully exploits the bonus payment so that it is attractive to only high-quality workers. Such difference between two policies with and without bonus causes huge difference shown in (9). A formal proof of Theorem 5.1 is provided in Section 6.5.

5.2 Price of Agnosticity

We now address the question (Q3) by quantitatively comparing personalized and common pricing policies with bonus, which reveals how close a simple common pricing with bonus payment is to the personalized pricing. To do so, we first define the following notion of Price of Agnosticity (PoA).

Definition 5.2

For given , let and denote the optimal utilities in OPP (3) and OCP (4). Then, we say that PoA is for budget if

where and

The constants and capture the inefficiency of using common pricing instead of personalized pricing in terms of utility and budget, respectively. We note that PoA is when there exists a common pricing that achieves the utility of the optimal personalized pricing with the same budget. Hence, PoA close to implies that it is enough to use a common pricing policy, which is much simpler than a personalized one, if it is properly designed. In the following theorem, we quantify PoA under canonical assumptions.

Theorem 5.3

Consider a worker profile , where worker index is sorted in descending order of bang-per-buck , i.e., , and budget . Assume utility function satisfying that for all such that ,

(10)

where is the sequence such that and for all . Then, the followings hold:

  1. Under 1, PoA is ,

  2. Under 2, PoA is ,

where we define ,

The proof of Theorem 5.3 is provided in Section 6.6. We now briefly provide the interpretation of Theorem 5.3, followed by more in-depth description in the subsequent paragraphs. This theorem basically implies that under a mild condition as in (10), the optimal common policy (which can be computed in polynomial time) provides at least a half of the optimal utility of personalized policy as in (a), or more as in (b) with a small increase of allocated budget. Noting that the optimal personalized pricing is computationally intractable for a large number of workers , the actual utility due to common pricing would be closer to what a practical personalized pricing is able to provide. This shows that a simple bonus payment that are uniformly applied, not personalized, to all workers turns out to have a large power in recruiting high-quality workers and discourage low-quality ones. We also highlight that utility of common pricing without this bonus payment can be arbitrarily worse than that with bonus (see Theorem 5.1), which manifests that the role of bonus payment in common pricing is crucial.

Note that the condition in (10) means that the optimal utility of personalized pricing is simply twice larger than the utility from a pricing which recruits only a single user, which is readily satisfied when the number of workers in the system is reasonably large. In other words, we can find a sufficiently large , such that (10) is met. We now investigate how behaves with worker profiles . First, from the hypothesis that workers are sorted by bang-per-buck and the construction of the worker index we can easily check that that (i) and (ii) . The value of depends on how depends on i.e, the form of in To exemplify, we consider two types of one is concave (thus, effort-unresponsive) and one is convex (thus, effort-responsive) as in Figure (a)a. We uniformly generate workers respectively. Figure (b)b shows how changes for different values of for those two types of functions. We observe that when is concave, is maintained to be small.

(a) Two quality-cost regimes: effort-unresponsive and responsive ones
(b) Behavior of with increasing k
Figure 1: Behavior of budget approximation ratio for two cost-quality functions.

6 Proofs of Main Results

6.1 Proof of Theorem 3.1

We start with the proof of the first part by contradiction. Let , where is the solution of OPP, i.e., utility of is . Then it is straightforward to check that satisfies the constraint in (5b). Suppose is not a solution of GKP so that there exists a solution such that . We now construct a PP policy from . Since is a solution of GKP, policy satisfies the constraints in (3b) and (3c). The construction of contradicts to the optimality of since the utility of is strictly larger than that of This completes the proof of the first part.

Similarly to the proof of the first part, we also prove the second part by contradiction. Let be the pricing in (6) for a solution of GKP and some such that for all . It is not hard to check that satisfies the constraints in (3b) and (3c). Suppose the policy is not a solution of OPP. Then there exists a solution such that , where . This contradicts to the optimality of , and completes the proof of the second part, and thus Theorem 3.1.

6.2 Proof of Theorem 3.3

Without loss of generality, suppose that the workers are aligned in the descending order of bang-per-buck i.e., . Let be the output of ModifiedGreedy. Then, we will obtain a lower bound of for which consider the linear relaxation of GKP (GKP-relax) in the following:

(11a)
subject to (11b)

We obtain a solution of GKP-relax in the following lemma, whose proof will be presented at the end of this subsection:

Lemma 6.1

Let denote a -dimensional real-valued sequence such that if , if , and otherwise, where , and Then, is a solution of GKP-relax in (11).

With in Lemma 6.1, the following holds, because the optimal value of GKP is smaller than or equal to that of its relaxed problem GKP-relax:

(12)

Let be the sequence such that if and otherwise, where we have . Let . Then, from the construction of and , it follows that

(13)

where for the first, second and last inequalities, we use the subadditivity of in 1, , and is output of ModifiedGreedy, respectively. Combining (12) and (13), we complete the proof of Theorem 3.3. We remark that the subadditivity assumed in C2 can be loosen. To be precise, the condition for any two vectors and in can be loosen to, two vectors and with no index that and .

Proof of Lemma 6.1. We use the proof by contradiction. Suppose that is not the solution of GKP-relax, and consider with . Note that and due to the definition of and the non-decreasing property of . To complete the proof, we will construct a sequence of feasible instances of GKP-relax using a greedy iteration , where from given , is obtained as follows:

  1. Find and such that , , , and .

  2. , , and for .

Let , and let be those in the greedy iteration for . In the greedy iteration, we have

which implies . It is straightforward to check that the sequence of feasible instances from iteration of ) converges to after -th iteration. Thus, it suffices to show the following:

(14)

since the above implies , which contradicts to the hypothesis on the optimality of , and thus completes the proof of Lemma 6.1. To show (14), we compare and . Noting that , it follows that

which implies that comparing to , the amount of increment in -th element is larger than that of decrement in -th element. Hence, weakly majorizes , i.e., . Using this, the assumptions on , i.e., non-decreasing and Schur-convex, directly implies (14) This completes the proof of Lemma 6.1.

6.3 Proof of Theorem 4.3

For ease of presentation, we first define some notations. Let be the expected payment of worker if she accepts the task, i.e. , and let . Then, a worker ’s strategic decision can be expressed as . Hence, we are able to analyze the worker’s behavior by observing for . In order to see the dynamic of the function , we differentiate by as follows:

(15)

Since for in effort-unresponsive regime, (15) always negative. This means that is monotone decreasing in , implying that is also monotone decreasing in . This verifies that the resulting policy is always -picking for effort-unresponsive regime.

We now consider the case when for . We compute the sign of the second order derivative of (15) as follows:

(17)

Hence, is convex in effort-responsive regime, and concave in effort-subresponsive regime.

Note the basic property that any level set of a convex function is a convex set. Hence, in effort-responsive regime, the level set is a convex set. Recall that that , which means that for each worker with does not take the task. This implies that any pricing policy in effort-responsive regime is -blocking. In effort-subresponsive regime, where is concave, we take the similar argument to that in effort-responsive regime, it follows that any pricing policy is -picking. This completes the proof of Theorem  4.3.

6.4 Proof of Theorem  4.4

Effort-unresponsive regime. We first prove that CP-UNRES finds an optimal CP policy for effort-unresponsive regime in time. Note that CP-UNRES outputs a CP policy which is optimal among the one whose base payment equals to . For notational simplicity, without loss of generality, the workers are aligned in the descending order of cost. Consider a CP policy where , and suppose that it is -picking. Then, we have for , and the total expected payment to workers given by:

(18)

where (18) denotes the total payment to workers from to of policy . Hence, for any -picking pricing policy in effort-unresponsive regime, a pricing policy achieves the same amount of utility, while spends less budget. Using this fact, CP-UNRES (presented in Appendix) is designed to compute just the bonus payment which produces the largest utility, where it is easy to check the complexity of CP-UNRES needs time. This completes the proof.

Effort-subresponsive regime. We next prove that CP-SUBRES find an optimal CP policy for effort-subresponsive regime in time, where CP-SUBRES is presented in Appendix. We observe that CP-SUBRES solves a linear programming over where iterates from to while we perform the binary search for optimal . Hence, let the running time of linear programming be , and the entire running time can be expressed as .

We now prove that for given and , finding that induces -picking and is budget-feasible or deciding its non-existence takes time. Recall that any pricing policy in effort-subresponsive regime is -picking. Thus, the following inequalities are sufficient for a CP policy to be -picking:

(19)
(20)
(21)
(22)
(23)

which can be simply expressed as by loosening the strict inequality, where the matrices and is defined in Appendix. Then, we can check if there exists a budget-feasible policy in time [47]. Now suppose that the output of satisfies the equality on either of the (19) or (22). Assume that the equality holds for (19) on output of CP-SUBRES. We show that there exists a positive real value in which is also feasible to , and does not satisfy the equality on (19). Since decrementing to can possibly harm (20) or (21), it is sufficient to show that there exists such such that (20) and (21) are met. Suppose that the following holds for CP policy :

(24)

This means that , which can be expressed as the following:

(25)

In effort-subresponsive regime, is monotone increasing in Thus, in order for the equality to hold in (25), we must have that . Then, however, also holds, where it becomes impossible for any CP pricing to be -picking. Similarly, it can be shown that (19) and (21) cannot simultaneously satisfy the equality. Note that the equality in (22) cannot be satisfied. Elaborating these facts together, there exists a positive real value which makes CP policy to be -picking and budget-feasible. Recall that the LP can be solved in [47], which completes the proof for effort-subresponsive regime.

Effort-responsive regime. Due to high similar to effort-subresponsive regime, we omit this proof for brevity.

6.5 Proof of Theorem 5.1

Let