# False Discovery Rate Control and Statistical Quality Assessment of Annotators in Crowdsourced Ranking

## Abstract

With the rapid growth of crowdsourcing platforms it has become easy and relatively inexpensive to collect a dataset labeled by multiple annotators in a short time. However due to the lack of control over the quality of the annotators, some abnormal annotators may be affected by position bias which can potentially degrade the quality of the final consensus labels. In this paper we introduce a statistical framework to model and detect annotator’s position bias in order to control the false discovery rate (FDR) without a prior knowledge on the amount of biased annotators – the expected fraction of false discoveries among all discoveries being not too high, in order to assure that most of the discoveries are indeed true and replicable. The key technical development relies on some new knockoff filters adapted to our problem and new algorithms based on the Inverse Scale Space dynamics whose discretization is potentially suitable for large scale crowdsourcing data analysis. Our studies are supported by experiments with both simulated examples and real-world data. The proposed framework provides us a useful tool for quantitatively studying annotator’s abnormal behavior in crowdsourcing.

## 1Introduction

In applications, building good predictive models is challenging primarily due to the difficulties in obtaining annotated training data. A traditional way for data labeling is to hire a small group of experts to provide labels for the entire set of data. However, such an approach can be expensive and time consuming for large scale data. Thanks to the wide spread of crowdsourcing platforms (e.g., MTurk, Innocentive, CrowdFlower, CrowdRank, and Allourideas), a much more efficient way is to post unlabeled data to a crowdsourcing marketplace, where a big crowd of low-paid workers can be hired instantaneously to perform labeling tasks [27].

Despite of its high efficiency and immediate availability, crowd labeling raises many new challenges. Since typical crowdsourced tasks are tedious and annotators usually come from a diverse pool including genuine experts, novices, biased workers, and malicious annotators, labels generated by the crowd suffer from low quality. Thus, all crowdsourcers need strategies to ensure the reliability of answers. In other words, outlier detection is a critical task in order to achieve a robust labeling results. Various methods have been developed in literature for outlier detection, of which majority voting strategy [14] is the most typical one. In this setting, each pair is allocated to multiple annotators and their opinions are averaged over so as to identify and discard noisy data provided by unreliable raters. They thus require large amount of pairwise labels to be collected. More importantly as a local outlier detection method, majority voting is ineffective in identifying outliers that can cause global ranking inconsistencies [12]. The work in [30] attacks this problem and formulates the outlier detection as a LASSO problem based on sparse approximations of the cyclic ranking projection of paired comparison data in Hodge decomposition. Regularization paths of the LASSO problem could provide an order on samples tending to be outliers. However, these work all treat pairwise comparison judgements as independent random outliers, which are typically defined to be data samples that have unusual deviations from the remaining data.

In this paper, instead of modeling the random effect of sample-wise outliers, we are primarily interested in the fixed effect where the annotators are influenced by positions when labeling in pairwise comparison setting. Such an annotator’s position bias [10] is ubiquitous in uncontrolled crowdsourced ranking experiments. In our studies, annotator’s position bias typically arises from: i) ** the ugly**: one typically clicks one side more often than another. As some pairs are highly confusing or annotators get too tired, in these cases, some annotators tend to click one side hoping to simply raise their record to receive more payment; while for pairs with substantial differences, they click as usual. ii)

**: some extremely careless annotators, or robots pretending to be human annotators, actually do not look at the instances and click one side all the time to quickly receive pay for work. Such kinds of annotators may significantly deteriorate the quality of crowdsourcing data and increase the cost of acquiring annotations (since each raw feedback comes with a cost: the task requestor has to pay workers a pre-specified monetary reward for each labeling they provide, usually, regardless of the feedback correctness). Although it might be relatively easy to identify the bad annotators above by inspecting their inputs, it is impossible for eye inspection to pick up those ugly annotators with mixed behaviors. Therefore it is desired to design a statistical framework to quantitatively detect and eliminate annotator’s position bias for crowdsourcing platforms in market. Such a systematic study, up to the author’s knowledge, however has not been seen in literature.**

*the bad*In this paper, we propose a linear model with annotator’s position bias and new algorithms to find good estimates with an automatic control on the false discovery rate (FDR) – the expected fraction of false discoveries among all discoveries. To understand FDR, imagine that we have a detection method that has just made 100 discoveries. Then, if our method is known to control the FDR at the 10% level, this means that with high probability, we can expect at most 10 of these discoveries to be false and, therefore, at least 90 to be true and replicable. Such a FDR control is desired when we don’t have a prior knowledge about the amount of bad or ugly annotators and typical statistical estimates will lead to an over-identification of them.

Specifically, our contributions in this work are highlighted as follows:

A linear model with annotator’s position bias as fixed effects;

New algorithms to find good estimates of such position bias, etc., based on Inverse Scale Space dynamics and its discretization Linearized Bregman Iteration;

New knockoff filters for FDR control adapted to our setting, which aims to mimic the correlation structure found within the original features for position bias;

Extensive experimental validation based on one simulated and four real-world crowdsourced datasets.

## 2Methodology

In this section, we systematically introduce the methodology for annotator’s position bias estimation. Specifically, we first start from a basic linear model with different types of noise models, which have been successfully used widely in literature. Then we introduce a new dynamic approach with unbiased estimator called Inverse Scale Space (ISS). Based on this, we present the modified knockoff filter for FDR control in details.

### 2.1Basic Linear Model

Let be the set of nodes and be the set of edges, where is the set of all annotators. Suppose the pairwise ranking data is given as . means prefers to and otherwise. The magnitude of can represent the degree of preference and it varies in applications. It can be dichotomous choice , -point Likert scale (e.g. ), or even real values.

In this paper, consider the following linear model:

where is some common score on and the residue may have interesting structures in crowdsourcing settings.

The annotators might have different effects on the residues. While for most annotators, the deviations from the common score are due to random noise; occasionally the annotators deviate from the common behavior regularly – some careless ones always choose the left or the right candidate in comparisons, but others only do this when they get too confused to decide. Such behaviors can be modeled in the following way,

where measures an annotator’s position bias in a fixed effect, and the remainder measures the random effect in sampling which is assumed to be sub-gaussian noise. For example, a positive value of means the annotator is more likely to prefer the left choice. Under the random design of pairwise comparison experiments, a candidate should be placed on the left or the right randomly, so the position should not affect the choice of a careful (good) annotator. Therefore is assumed to be sparse, i.e., zero for most of annotators, and a nonzero position bias means the annotator is either always choosing one position over the other (bad) or occasionally incurring this when they get too confused or tired (ugly).

We note that should not be confused with recent studies in [12] on outlier detection problem, , where models sparse outliers for each sample , which only measures the random effect of samples rather than the fixed effect of annotators. By modeling the annotator’s fixed effect on position bias, one can systematically classify the annotators into the good, the ugly, and the bad according to their behaviors.

### 2.2Iss/lbi

Define the gradient operator by such that , and the annotator operator by , then the model above can be rewritten as:

In this case, detecting the annotators affected by position bias can be reformulated as: learning a sparse vector from given data . To solve such a problem, in this paper, we choose a new approach based on the following dynamics,

Its solution path can be easily solved by a sequence of nonnegative least squares, see [24] and references therein. In this paper we use the free R-package [29].

In [24], it has been shown that the dynamics above has several advantages over the traditional LASSO approach, which can be formulated as follows in our setting

First of all, the dynamics above is statistically equivalent to LASSO in terms of model selection consistency but may render oracle estimator which is bias-free, while the LASSO estimator is well-known biased. In this sense the ISS path can be better than the LASSO path. Here the solution path plays the same role of the regularization path of LASSO with roughly , where the important features (variables) are selected before the noisy ones. Following the tradition in image processing, such a dynamics is called *Inverse Scale Space* (ISS).

Beyond the charming statistical properties, ISS also admits an extremely simple discrete approximation, i.e., the Linearized Bregman Iteration (LBI), which has been widely used in image reconstruction with TV-regularization. Adapted to our setting, the discretized algorithm is illustrated in Algorithm ?, which is scalable, easy for parallelization, and particularly suitable for large scale crowdsourced ranking data analysis.

### 2.3FDR Control and New Knockoff Filter

A crucial question for LASSO and ISS is how to choose the regularization parameter and in real-world data. After all, different parameters can give different bad or ugly annotator sets. Traditional methods either require a prior knowledge on the amount of such annotators which is often unknown in practice, or some statistically optimal choice of such regularization parameters. Extensive studies in statistics have shown that such parameter tuning typically lead to an over estimation of the sparse signal, therefore False Discovery Rate (FDR) control is necessary [3] which is adopted in this paper.

FDR is defined as the expected proportion of false discoveries among the discoveries. Putting in a mathematical way, here we consider

To control the FDR means to control the accuracy of the bad/ugly annotators we detected to see if they are reasonable ones.

Recently, a new method called knockoff filter [3] is proposed to automatically control FDR in standard linear regression without a prior knowledge on the sparsity. In this paper, such an approach will be extended to our linear model (Equation 3) and the algorithms, where both non-sparse and sparse co-exist in the model. The extended method consists of the same three steps as in [3], where the key difference lies in the knockoff feature construction adapted to our setting.

Construct knockoff features: let be knockoff features that satisfy

where is positive and can be solved by SDP:

with . Let be an orthonormal matrix such that , which requires easily met in crowdsourcing. Then can be satisfied by defining

where satisfies .

Now define the extended design matrix and , then replace with and with in , or Alg. ?, we can get solution path (or ).Generate knockoff statistics for every original feature: define to be the first entering time for , i.e., for LASSO (or for ISS/LBI) and can be defined similarly. Then the knockoff statistics becomes

Choose variables based on the knockoff statistics: define the selected variable set , where

is knockoff cut and is knockoff+ cut.

It can be shown that the new knockoff filter above indeed controls FDR in the following sense, whose proof is similar to that of [3] (collected in Supplementary Materials for completeness).

Both knockoff filters above can choose variables with FDR control but the estimator consists of knockoff features, so we need to reestimate after bad annotator detection by passing to a least square while only keeping those nonzero parameters and features. Suppose that is the set of bad or ugly annotators given by knockoff filters, then one can find the final estimators by

## 3Experiments

In this section, five examples are exhibited with both simulated and real-world data to illustrate the validity of the analysis above and applications of the methodology proposed. The first example is with simulated data while the latter four exploit real-world data collected by crowdsourcing.

### 3.1Simulated Study

Settings

We first validate the proposed algorithm on simulated binary data labeled by 150 annotators. Of the 150 annotators we have 100 ** good** annotators (annotators 1 to 100 without position bias) and 50

**annotators (annotators 101 to 150 with position bias). We note that for good annotators, it does not mean that each worker always present the correct labels. Instead, it means that they also have the probability to make incorrect judgements due to certain reasons, rather than position effect.**

*bad/ugly*Specifically, we first create a random total order on candidates as the ground-truth and add paired comparison edges to graph until a complete graph, with the preference direction following the ground-truth order. Here we choose , which is consistent with the third real-world dataset with smallest node size. Then, for good annotators, they make judgements with an incorrect probability (i.e., of is reversed in preference direction), while for bad/ugly annotators, they are with a probability of disturbed by position effect.

Evaluation metrics

Two metrics are employed to evaluate the performance of the proposed algorithms. The first one is *Control of Actual FDR*, the second is *Number of True Discoveries*.

Experimental results

With different choices of and , the mean *Control of Actual FDR* and *Number of True Discoveries* with over 100 runs are shown in Table ? to measure the performance of knockoff filter via ISS in position biased annotator detection. It can be seen that via knockoff filter, ISS can provide an accurate detection of position biased annotators (indicated by *control of actual FDR* around and *Number of True Discoveries* around 50). Comparable results of LASSO with can be found in Table ?. It can be seen that via knockoff filter, both LASSO and ISS can provide an accurate detection of position biased annotators. This result is consistent with the theoretical comparison between LASSO and ISS discussed in [24], where ISS/LBI has similar theoretical guarantees as LASSO, but with bias-free and simpler implementation (the 3 line algorithm in Section 2.2) properties.

### 3.2Real-world Datasets

As there is no ground-truth for position biased annotators in real-world data, one can not compute *control of actual FDR* and *Number of True Discoveries* as in simulated data to evaluate the detection performance here. In this subsection, we inspect the annotators returned by knockoff filter via ISS/LASSO under to see if they are reasonably good position biased workers.

#### Human Age

In this dataset, 30 images from human age dataset FG-NET ^{1}

It is easy to see that these annotators can be divided into two types: (1) ** the bad**: click one side all the time (with ID in red); (2)

**: click one side with high probability (with ID in blue). Besides, the regularization paths of ISS can be found in Figure 1(a), where the position biased annotators detected mostly lie outside the majority of the paths. Note that since we allow a small percentage of false positives, some ugly annotators might be good in reality as well.**

*the ugly*To see the effect of position biased annotators on global ranking scores, Table ? shows the outcomes of two ranking algorithms, namely original and corrected. The original is calculated by least squares problems on all of the pairwise comparisons, while the corrected is obtained by the correction step via knockoff illustrated in Section 2.3. It is easy to see that the removal of position biased annotators often changes the orders of some competitive images, such as ID=11 and ID=21, ID=30 and ID=8, etc.

To see which ranking is more reasonable, Table ? shows the * groundtruth* ranking of these competitive images. We can find from this table that, compared with the original ranking, the corrected one is in more agreement with the groundtruth ranking, which further shows that: i) position biased annotators may disturb the ranking to a departure from the real ranking. ii) pairs with little differences are more likely to lead to position biased annotations. From this viewpoint, we can see that the knockoff-based FDR-controlling method indeed effectively selects the position biased annotators.

#### Reading Level

The second dataset is a subset of reading level dataset [9], which contains 490 documents. 8,000 pairwise comparisons are collected from 346 annotators using CrowdFlower crowdsourcing platform. More specifically, each annotator is asked to provide his/her opinion on which text is more challenging to read and understand. Table ? shows the position biased annotators detected from this dataset, together with the ISS regularization path shown in Figure 1(b). It is easy to see that LASSO and ISS picked out the same 6 annotators as position biased ones. In terms of the small number of bad annotators detected, we can say that the overall quality of annotators on this task is relatively high.

#### Image Quality Assessment

The third dataset is a pairwise comparison dataset for subjective image quality assessment (IQA), which contains 15 reference images and 15 distorted versions of each reference, for a total of 240 images which come from two publicly available datasets LIVE, [2] and IVC [1]. Totally, 342 observers, each of whom performs a varied number of comparisons via Internet, provide paired comparisons for crowdsourced subjective image quality assessment. Note that the number of responses each reference image received is different in this dataset.

To validate whether the annotators we detected are good position biased annotators or not, we randomly take reference image 1 as an illustrative example while other reference images exhibit similar results. Table ? shows the annotators with position bias picked by knockoff filter and the ISS regularization path is shown in Figure 1(c). In this dataset, the abnormal annotators picked out by LASSO and ISS are also exactly the same. It is easy to see that annotators picked out are mainly those clicking on one side almost all the time. Besides, it is interesting to see that all these bad annotators highlighted with red color in Table ? click the left side all the time. We then go back to the crowdsourcing platform and find out that the reason behind this is a default choice on the left button thus induces some lazy annotators cheat for the annotation task.

#### WorldCollege Ranking

We now apply the knockoff filter to the WorldCollege dataset, which is composed of 261 colleges. Using the Allourideas crowdsourcing platform, a total of 340 distinct annotators from various countries (e.g., USA, Canada, Spain, France, Japan) are shown randomly with pairs of these colleges, and asked to decide which of the two universities is more attractive to attend. Finally, we obtain a total of 8,823 pairwise comparisons. We then apply knockoff filter to the resulting dataset and find out that both LASSO and ISS selected 36 annotators as position biased ones, as is shown in Table ? and Figure 1(d). It is easy to see that similar to the human age dataset, the annotators picked out are either clicking one side all the time, or clicking one side with high probability.

### 3.3Discussion

Someone may argue that setting a threshold on the ratio of left/right answers can be an easy way to detect position biased annotators. To illustrate why simply setting a threshold does not work, Figure 2 shows the click counts of each side (i.e., X-axis: number of left clicks; Y-axis: number of right clicks), where each color represents one annotator. It is easy to see that there are indeed some overlaps between abnormal and normal annotators. For example, in reading level dataset, annotators with ID=69 and ID=57 both provide 6:0 on the ratio of left/right clicks. However, ID=69 is detected as abnormal annotator, while ID=57 as normal one. To figure out the reason behind this, we further compute the Match Ratio (MR) of these two annotators with the global ranking scores obtained by all pairwise comparisons and find that and . This indicates that the position biased annotator (i.e., ID=69) we picked out is the one not only with one-side click but also with a large deviation with the majority. Similar results can be easily found in other three datasets.

## 4Related Work

### 4.1Outlier Detection

Outliers are often referred to as abnormalities, discordants, deviants, or anomalies in data. Generally speaking, there can be two types of outliers: (1) samples as outliers; (2) subjects as outliers. Hawkins formally defined in [15] the concept of an outlier as follows: “An outlier is an observation which deviates so much from the other observations as to arouse suspicions that it was generated by a different mechanism.” Outliers are rare events, but once they have occurred, they may lead to a large instability of models estimated from the noisy data. For type (1), many methods have been developed for outlier detection, such as distribution-based [4], depth-based [19], distance-based [20], density-based [6], and clustering-based [17] methods. For subject-based outlier detection, some sophisticated methods have been proposed to model annotators’ judgements. Recently, [9] propose a Crowd-BT algorithm to detect spammers and malicious annotators: spammers assign random labels while malicious annotators assign the wrong label most of the time. Besides, [25] defines a score to rank the annotators for crowdsourced labeling tasks. Furthermore, [26] presents an empirical Bayesian algorithm called SpEM to eliminate the spammers and estimate the consensus labels based only on the good annotators. However, a phenomenon that has annoyed researchers who have used paired comparison tests is position bias or testing order effects. Until now, little work have been found for such kind of position biased annotator detection, which is our main focus in this paper.

### 4.2FDR Control and Knockoff Method

Most variable selection techniques in statistics such as LASSO suffer from over-selection as picking up too many false positives by leaving out few true positives. In order to offer guarantees on the accuracy of the selection, it is desired to control the false discovery rate (FDR) among all the selected variables. The Benjamini-Hochberg (BH) procedure [5] is a typical method known to control FDR under independence scenarios. Recently, [3] developed a new knockoff filter filtermethod for FDR control for general dependent features as long as the sample size is larger than that of parameters. In this paper, we extend this method to our setting with mixed parameters of both nonsparse and sparse ones to achieve the same FDR control.

### 4.3Inverse Scale Space and Linearized Bregman Iteration

Linearized Bregman Iteration (LBI) has been widely used in image processing and compressed sensing [23] even before its limit form as Inverse Scale Space (ISS) dynamics [7]. ISS/LBI at least have two advantages over the popular LASSO in variable selection: (1) ISS may give unbiased estimator [24], under nearly the same condition for model selection consistency as LASSO whose estimators are however always biased [11]. (2) LBI, regarded as a discretization of ISS dynamics, is an extremely simple algorithm which combines an iterative gradient descent algorithm together with a soft thresholding. It only runs in a single path and regularization is achieved by early stopping like boosting algorithms [24], which may save the computational cost greatly and thus suitable for large scale implementation [32].

## 5Conclusion

Annotator’s position bias is ubiquitous in crowdsourced ranking data, which, up to our knowledge, has not been systematically addressed in literature. In this paper, we propose a statistical model for annotator’s position bias with pairwise comparison data on graphs, together with new algorithms to reach statistically good estimates with a FDR control based on some new design of knockoff filters. FDR control here does not need a prior knowledge on the sparsity of position bias, i.e., the amount of bad or ugly annotators. Such a framework is valid for both traditional LASSO estimator and the new dynamic approach based on ISS/LBI with debiased estimator and scalable implementations which is desired for crowdsourcing experiments. Experimental studies are conducted with both simulated examples and real-world datasets. Our results suggest that the proposed methodology is an effective tool to investigate annotator’s abnormal behavior in modern crowdsourcing data.

## Acknowledgements

The research of Qianqian Xu was supported in part by National Natural Science Foundation of China (No. 61422213, 61402019, 61390514, 61572042), China Postdoctoral Science Foundation (2015T80025), “Strategic Priority Research Program” of the Chinese Academy of Sciences (XDA06010701), and National Program for Support of Top-notch Young Professionals. The research of Jiechao Xiong and Yuan Yao was supported in part by National Basic Research Program of China under grant 2015CB85600, 2012CB825501, and NSFC grant 61370004, 11421110001 (A3 project), as well as grants from Baidu and Microsoft Research-Asia. Xiaochun Cao and Yuan Yao are the corresponding authors. We would like to thank Yongyi Guo for helpful discussions and anonymous reviewers who gave valuable suggestions to help improve the manuscript.

## Supplementary Material

**Sketchy Proof of Theorem .**

Similar to the treatment in [3], we only need to prove that the knockoff statistics satisfy the following two properties:

sufficiency property

:

, which indicates depends only on and .antisymmetry property

:

Swapping and has the effect of switching the sign of .

The second property is obvious because is constructed using entering time difference. Now we go to prove the first property.

For ISS and LBI, the whole path is only determined by

which is only based on and , so is the entering time

The same reasoning holds for LASSO since

is equivalent to

So the entire path is determined by and .

**Proof of Theorem .**

Suppose is the knockoff statistics for , then it satisfies

Let , then and it can be verified

which means is a valid knockoff feature matrix for .

On the reverse, let be knockoff features for , it is also easy to verify satisfies condition . This establishes an injection between and .

The equivalence of knockoff statistics comes from the equivalence of solution paths in both approaches. To see this, actually means , plugging in , we get

This is equivalent to the ISS for the second procedure model in Remark 1. So in both approaches, the two ISS solution paths are identical.

The same reasoning holds for LASSO, the derivative of w.r.t. is zero at the optimal estimator which means

this is actually . So plugging in , we get

This is in fact the loss for the second procedure in Remark 1.

Finally identical paths lead to the same knockoff statistics which ends the proof.

### Footnotes

### References

Subjective quality assessment irccyn/ivc database. , 2005.

LIVE image & video quality assessment database. , 2008.**Controlling the false discovery rate via knockoffs.**

Barber, R. and Candès, E. The Annals of StatisticsOutliers in statistical data

Barnett, V. and Lewis, T. , volume 3.**Controlling the false discovery rate: a practical and powerful approach to multiple testing.**

Benjamini, Y. and Hochberg, Y. Journal of the Royal Statistical Society. Series B (Methodological)**LOF: identifying density-based local outliers.**

Breunig, M., Kriegel, H., Ng, R., and Sander, J. In*Proceedings of the ACM International Conference on Management of Data*, volume 29, pp. 93–104, 2000.Nonlinear inverse scale space methods for image restoration

Burger, M., Osher, S., Xu, J., and Gilboa, G. .**A crowdsourceable QoE evaluation framework for multimedia content.**

Chen, K., Wu, C., Chang, Y., and Lei, C. In*ACM International Conference on Multimedia*, pp. 491–500, 2009.**Pairwise ranking aggregation in a crowdsourced setting.**

Chen, X., Bennett, P., Collins-Thompson, K., and Horvitz, E. In*International Conference on Web Search and Data Mining*, pp. 193–202, 2013.**Position bias in paired product tests.**

Day, R. Journal of Marketing Research**Variable selection via nonconcave penalized likelihood and its oracle properties.**

Fan, J. and L, R. Journal of American Statistical Association**Interestingness prediction by robust learning to rank.**

Fu, Y., Hospedales, T., Xiang, T., Gong, S., and Yao, Y. In*European Conference on Computer Vision*, pp. 488–503. 2014.**Robust subjective visual property prediction from crowdsourced pairwise labels.**

Fu, Y., Hospedales, T., Xiang, T., Xiong, J., Gong, S., Wang, Y., and Yao, Y. IEEE Transactions on Pattern Analysis and Machine Intelligence**The interestingness of images.**

Gygli, M., Grabner, H., Riemenschneider, H., Nater, F., and Gool, L. In*IEEE International Conference on Computer Vision*, pp. 1633–1640, 2013.Identification of Outliers

Hawkins, D. , volume 11.**Data quality from crowdsourcing: a study of annotation selection criteria.**

Hsueh, P., Melville, P., and Sindhwani, V. In*NAACL HLT Workshop on Active Learning for Natural Language Processing*, pp. 27–35, 2009.**Data clustering: a review.**

Jain, A., Murty, M., and Flynn, P. ACM Computing Surveys**Understanding and predicting interestingness of videos.**

Jiang, Y., Wang, Y., Feng, R., Xue, X., Zheng, Y., and Yang, H. In*AAAI Conference on Artificial Intelligence*, volume 1, pp. 2, 2013.**Fast computation of 2-dimensional depth contours.**

Johnson, T., Kwok, I., and Ng, R. In*ACM International Conference on Knowledge Discovery and Data Mining*, pp. 224–228, 1998.**Finding intensional knowledge of distance-based outliers.**

Knorr, E. and Ng, R. In*International Conference on Very Large Data Bases*, pp. 211–222, 1999.**Distance-based outliers: algorithms and applications.**

Knorr, E., Ng, R., and Tucakov, V. International Journal on Very Large Data Bases**How reliable are annotations via crowdsourcing: a study about inter-annotator agreement for multi-label image annotation.**

Nowak, S. and Rüger, S. In*International Conference on Multimedia Information Retrieval*, pp. 557–566, 2010.**An iterative regularization method for total variation-based image restoration.**

Osher, S., Burger M. Goldfarb D. Xu J. and Yin, W. SIAM Journal on Multiscale Modeling and Simulation**Sparse recovery via differential inclusions.**

Osher, S., Ruan, F., Xiong, J., Yao, Y., and Yin, W. Applied and Computational Harmonic Analysis**Ranking annotators for crowdsourced labeling tasks.**

Raykar, V. and Yu, S. In*Advances in Neural Information Processing Systems*, pp. 1809–1817, 2011.**Eliminating spammers and ranking annotators for crowdsourced labeling tasks.**

Raykar, V. and Yu, S. The Journal of Machine Learning Research**Get another label? improving data quality and data mining using multiple, noisy labelers.**

Sheng, V., Provost, F., and Ipeirotis, P. In*ACM International Conference on Knowledge Discovery and Data Mining*, pp. 614–622, 2008.**Cheap and fast|but is it good?: evaluating non-expert annotations for natural language tasks.**

Snow, R., O’Connor, B., Jurafsky, D., and Ng, A. In*Conference on Empirical Methods in Natural Language Processing*, pp. 254–263, 2008.A Tutorial on Libra: R package for the Linearized Bregman Algorithm in high dimensional statistics

Xiong, J., Ruan, F., and Yao, Y. , 2016.**Robust evaluation for quality of experience in crowdsourcing.**

Xu, Q., Xiong, J., Huang, Q., and Yao, Y. In*ACM International Conference on Multimedia*, pp. 43–52, 2013.**Bregman iterative algorithms for compressed sensing and related problems.**

Yin, W., Osher, S., D., Jerome, and G., Donald. SIAM Journal on Imaging Sciences**A Linearized Bregman Algorithm for Decentralized Basis Pursuit.**

Yuan, K., Ling, Q., Yin, W., and Ribeiro, A. European Signal Processing Conference