LSCP: Locally Selective Combination in Parallel Outlier Ensembles
In unsupervised outlier ensembles, the absence of ground truth makes the combination of base detectors a challenging task. Specifically, existing parallel outlier ensembles lack a reliable way of selecting competent base detectors, affecting accuracy and stability, during model combination. In this paper, we propose a framework—called Locally Selective Combination in Parallel Outlier Ensembles (LSCP)—which addresses this issue by defining a local region around a test instance using the consensus of its nearest neighbors in randomly generated feature spaces. The top-performing base detectors in this local region are selected and combined as the model’s final output. Four variants of the LSCP framework are compared with six widely used combination algorithms for parallel ensembles. Experimental results demonstrate that one of these LSCP variants consistently outperforms baseline algorithms on the majority of eighteen real-world datasets.
LSCP: Locally Selective Combination in Parallel Outlier Ensembles
|Yue Zhao††thanks: Department of Computer Science, University of Toronto.|
|and Maciej K. Hryniewicki††thanks: Data Analytics, PricewaterhouseCoopers Canada.|
|and Zain Nasrullah11footnotemark: 1 22footnotemark: 2|
|and Zheng Li ††thanks: Toronto Campus, Northeastern University.|
Outlier detection methods aim to identify anomalous data objects from the general data distribution and are useful for problems such as credit card fraud prevention and network intrusion detection . Since the ground truth is often absent in outlier mining , unsupervised detection methods are commonly used for this task [5, 17, 8]. However, unsupervised approaches are susceptible to generating high false positive and false negative rates . To improve model accuracy and stability in these scenarios, recent research explores ensemble approaches to outlier detection [1, 3, 24, 30]. Ensemble learning combines multiple base estimators to achieve superior detection performance and reliability when compared to an individual estimator [12, 15, 29]. It is important to ensure that the combination process is robust because constituent estimators, if synthesized inappropriately, may be detrimental to the predictive capability of an ensemble[22, 23]. Similar to prior works in classification , an outlier ensemble may be characterized as parallel if detectors are generated independently or sequential if the detector generation, selection or combination is iterative.
Model combination is important for parallel ensembles to ensure diversity exists among base detectors; however, existing works have not jointly addressed two key limitations in this process. First, most parallel ensembles generically combine all detectors without considering selection. This limits the benefits of model combination since individual base detectors may not be proficient at identifying all outlier instances . For example, prior work has demonstrated that the value of good detectors can be neutralized by the inclusion of poor detectors in a generic averaging framework . Secondly, data locality is rarely emphasized in the context of detector selection and combination leading to potentially sub-optimal outcomes. While it is acknowledged that certain types of outliers are better identified by local data relationships , detectors are often evaluated at a global scale, where all training points are considered, instead of the local region related to a test instance.
To address the aforementioned limitations, we propose a fully unsupervised framework called Locally Selective Combination in Parallel Outlier Ensembles (LSCP) to selectively combine base detectors by emphasizing data locality. The idea is motivated by an established supervised ensemble framework known as Dynamic Classifier Selection (DCS) . DCS selects the best classifier for each test instance by evaluating base classifier competency at a local scale . The rationale behind this is that base classifiers will generally not excel at categorizing all unknown test instances and that an individual classifier is more likely to specialize in a specific local region [9, 16]. Similarly, LSCP first defines the local region of a test instance by the consensus of the nearest training points in randomly generated feature spaces, and then identifies the most competent base detector in this local region by measuring similarity relative to a pseudo ground truth (see [7, 23] for examples). To further improve algorithm stability and capacity, ensemble variations of LSCP are proposed where promising base detectors are kept for a second-phase combination instead of using the single most competent detector. Our technical contributions in this paper are:
We propose a novel combination framework which, to the best of our knowledge, is the first published effort to adapt DCS from supervised classification tasks to unsupervised parallel outlier ensembles.
As a general framework, LSCP is formulated to be compatible with different types of base detectors; we demonstrate its effectiveness with a homogeneous pool of Local Outlier Factor  detectors.
We employ various analysis methods to improve model interpretability. First, theoretical explanations and complexity are provided. Second, visualization techniques are used to intuitively explain why LSCP works and when to use it. Third, statistical tests are used to compare experimental results.
Effort has also been made to streamline the accessibility of LSCP. Hyperparameter selection and associated impacts in the context of the framework are discussed in detail. All source code, experiment results and figures are shared for reproduction111Repository: https://github.com/yzhao062/LSCP.
It should be noted that the purpose of LSCP is not to outperform the best unsupervised outlier detector, but rather to explore the use of detector selection and combination at the local level in unsupervised parallel outlier ensembles. However, extensive experiments on 18 real-world datasets show that LSCP consistently yields better performance than existing parallel combination methods. In summary, LSCP is intuitive, stable and effective for combining independent outlier detectors without supervision.
2 Related Works
2.1 Dynamic Classifier Selection and Dynamic Ensemble Selection.
Dynamic Classifier Selection (DCS) is an established combination framework for classification tasks. The technique was first proposed by Ho et al. in 1994  and then extended, under the name DCS Local Accuracy, by Woods et al. in 1997  to select the most accurate base classifier in a local region. The motivation behind this approach is that base classifiers often make distinctive errors and offer a degree of complementarity . Consequently, selectively combining base classifiers can result in a performance improvement over generic ensembles that use the majority vote of all base classifiers. Subsequent theoretical work by Giacinto and Roli validated that, under certain assumptions, the optimal Bayes classifier could be obtained by selecting non-optimal classifiers . DCS was later expanded by Ko et al. to Dynamic Ensemble Selection (DES) which selects multiple base classifiers for a second-phase combination given each test instance . By minimizing reliance on a single classifier and delegating the classification task to a group competent classifiers, the algorithm has demonstrated that it is more robust than DCS . Motivated by these approaches, LSCP adapts dynamic selection to unsupervised outlier detection tasks.
2.2 Data Locality in Outlier Detection.
The relationship among data objects is critical in outlier detection and existing algorithms can roughly be categorized as either global or local [17, 24, 25]. The former considers all objects during inference while the latter only considers a local selection of objects . In both cases, their applicability is dependent on the structure of the data. Global outlier detection algorithms, for example, offer superior performance when outliers are highly distinctive from the data distribution  but often fail to identify outliers in the local neighborhoods of high-dimensional data [5, 26]. Accordingly, global models also struggle with data represented by a mixture of distributions, where global characteristics do not necessarily represent the distribution of objects in local regions . To address these limitations, numerous works have explored local algorithms such as Local Outlier Factor (LOF) , Local Outlier Probabilities (LoOP)  and GLOSS . However, data locality is rarely considered in the context of detector combination; instead, most combination methods utilize all training data points, e.g., the weight calculation in weighted averaging . LSCP explores both global and local data relationships by training base detectors on the entire dataset and emphasizing data locality during detector combination.
2.3 Outlier Detector Combination.
Recently, studying outlier ensembles has become a popular research area [1, 2, 3, 30] resulting in numerous popular works including: (i) parallel ensembles such as Feature Bagging  and Isolation Forest ; (ii) sequential methods including CARE , SELECT  and BoostSelect  and (iii) hybrid approaches like BORE  and XGBOD . When the ground truth is unavailable, combining outlier models is challenging. Feature Bagging , an early work, generates a diversified set of base detectors by training on randomly selected feature subsets and statically combining their outlier scores. Existing unsupervised combination algorithms in parallel ensembles are often both generic and global (GG); a list of representative GG methods are described below (see [1, 2, 3, 30, 24] for details):
Averaging (GG_A): average scores of all detectors.
Maximization (GG_M): take the maximum score across all detectors.
Weighted Averaging (GG_WA): weight each base detector when averaging.
Threshold Sum (GG_TH): discard all scores below a threshold and sum over the remaining scores.
Average-of-Maximum (GG_AOM): divide base detectors into subgroups and take the maximum score for each subgroup. The final score is the average of all subgroup scores.
Maximum-of-Average (GG_MOA): divide base detectors into subgroups and take the average score for each subgroup. The final score is the maximum of all subgroup scores.
As discussed in §2.2, GG methods ignore the importance of data locality while evaluating and combining detectors, which may be inappropriate given the characteristics of outliers [5, 26]. Moreover, without a selection process, poor detectors may hurt the overall detection performance of an ensemble [23, 24]. All aforementioned GG algorithms are thus included as baselines.
There have been attempts to build selective outlier ensembles sequentially in a boosting style. Rayana and Akoglu introduced SELECT  and CARE  to pick promising detectors and exclude the underperforming ones iteratively, which yielded great results on both temporal graphs and multi-dimensional outlier data. Campos et al. further extend this idea by proposing an unsupervised boosting strategy BoostSelect for outlier ensemble selection . As an alternative to sequential selection models, this paper chooses to focus on parallel detector selection which stresses the importance of data locality. Compared with sequential detector selection methods, our approach can select detectors without iteration which may reduce computational cost.
3 Algorithm Design
LSCP starts with a group of diversified detectors to be combined. For each test instance, LSCP first defines its local region and then picks the most competent detector(s) locally. The selected detector(s) are used to generate the outlier score for the test instance. The workflow of all four proposed LSCP methods is shown in Fig. 1 and Algorithm 1.
3.1 Base Detector Generation.
An effective ensemble should be constructed with diversified base estimators [24, 30] to promote learning distinct characteristics in the data. With a group of homogeneous base detectors, diversity can be induced by subsampling the training set and feature space, or by varying model hyperparameters [6, 30]. In this study, we demonstrate the effectiveness of LSCP by using distinct hyperparameters to construct a pool of models with the same base algorithm. However, in practice, LSCP can also be used as a general framework with heterogeneous base detectors.
Let denote training data with points and features, and denote a test set with points. The algorithm first generates a pool of base detectors initialized with a range of hyperparameters, e.g., a group of LOF detectors with distinct . All base detectors are first trained on and then inference is performed on the same dataset. The results are combined into an outlier score matrix , formalized in Eq.(1), where denotes the score vector from the base detector. Each detector score is normalized using Z-normalization as per prior work[2, 30].
3.2 Pseudo Ground Truth Generation.
Since LSCP evaluates detector competency without ground truth labels, two methods are used for generating a pseudo ground truth (denoted ) with : (i) LSCP_A: averages base detector scores and (ii) LSCP_M: maximum score across detectors. This is generalized in Eq.(2) where represents the aggregation (average or max) taken across all base detectors.
It should be noted that the pseudo ground truth in LSCP is generated using training data and used solely for detector selection.
3.3 Local Region Definition.
The local region of a test instance is defined as the set of its k nearest training objects. Formally, this is denoted as:
where describes the set of a test instance’s nearest neighbours subject to an ensemble criteria. This variation of kNN, which is similar to Feature Bagging , is proposed to alleviate concerns involving the curse of dimensionality on kNN  while leveraging its better precision compared to clustering algorithms in DCS . The process is as follows: (i) groups of features are randomly selected to construct new feature spaces; (ii) the nearest training objects to in each group are identified using euclidean distance and (iii) training objects that have appeared more than times are added to thus defining the local region. The size of the region is not fixed because it is dependent on the number of training objects that meet the selection criteria.
The local region factor k decides the number of nearest neighbors to consider during this process; care is given to avoid selecting extreme values. Smaller values of k give more attention to local relationships which can result in instability, while large values of k may place too much emphasis on global relationships and have higher computational costs. While it is possible to experimentally determine an optimal k with cross-validation  when ground truth is available, a similar trivial approach does not exist in an unsupervised setting. For these reasons, we recommend setting , 10% of the training samples, bounded in the range of , which yielded good results in practice.
3.4 Model Selection and Combination.
For each test instance, the local pseudo ground truth can be obtained by retrieving values associated with the local region from :
where denotes the cardinality of . Similarly, the local training outlier scores can be retrieved from the pre-calculated training score matrix as:
Consequently, although the local region needs to be re-computed for each test instance, the local outlier scores and targets can be efficiently retrieved from pre-calculated values (see Fig. 1).
For evaluating base estimator competency in a local region, DCS measures the accuracy of base classifiers as the percentage of correctly classified points , while LSCP measures the similarity between base detector scores and the pseudo target instead. This distinction is motivated by the lack of direct and reliable ways to access binary labels in unsupervised outlier mining. Although converting pseudo outlier scores to binary labels is feasible, defining an accurate threshold for the conversion is challenging. Additionally, since imbalanced datasets are common in outlier detection tasks, it is more stable to use similarity measures over absolute accuracy for competency evaluation. Therefore, LSCP measures the local competency of each base detectors by the Pearson correlation between the local pseudo ground truth and the local detector score . The detector with the highest similarity is regarded as the most competent local detector for , and its outlier score can be considered the final score for the corresponding test sample.
3.5 Dynamic Outlier Ensemble Selection.
Selecting only one detector, even if it is most similar to the pseudo ground truth, can be risky in unsupervised learning. However, this risk can be mitigated by selecting a group of detectors for a second-phase combination. This idea can be viewed as an adaption of supervised DES  to outlier detection tasks; correspondingly, we introduce ensemble variations of LSCP which employ Maximum of Average (LSCP_MOA) and Average of Maximum (LSCP_AOM) ensembling methods. Specifically, when the psuedo ground truth is generated by , LSCP_MOA selects a subset of detectors with the highest outlier scores and then takes the maximum as the outlier score of the test instance. Inversely, LSCP_AOM computes the average of the selected subset of detectors as the outlier score when the pseudo target is generated with . Setting the group size of selected detectors equal to 1 is a special case of the ensembles yielding the original LSCP algorithms (LSCP_A and LSCP_M). Larger group sizes may be considered more global in their detector selection while a group size of results in a fully global algorithm. In response to this, we recommend using a group size selection process which includes some variance. Specifically, a histogram of detector Pearson correlation scores (to the pseudo ground truth) is built with equal intervals. The detectors belonging to the most frequent interval are kept for the second-phase combination. A large thus results in selecting fewer detectors which controls the strength of the group size of LSCP ensembles in a flexible way.
With the appropriate implementation of LSCP_A and LSCP_M, e.g., using a k-d tree, the time complexity for each test instance is : for the distance calculation and for summation and sorting . To combine the base detectors in LSCP_MOA and LSCP_AOM, an additional is needed resulting in a total time complexity of .
3.6 Theoretical Considerations.
Recently, Aggarwal and Sathe laid the theoretical foundation for outlier ensembles  using the bias-variance tradeoff, a widely used framework for analyzing generalization error in classification problems. The reducible generalization error in outlier ensembles may be minimized by either reducing squared bias or variance where a tradeoff between these two channels usually exists. A high variance detector is sensitive to data variation with high instability; a high bias detector is less sensitive to data variation but may fit complex data poorly. The goal of outlier ensembles is to control both bias and variance to reduce the overall generalization error. Various newly proposed algorithms have been analyzed using this new framework to enhance interpretability [23, 24, 28].
It has been shown that combining diversified base detectors, by averaging them for example, results in variance reduction [23, 24, 2]. However, a combination of all base detectors may also include inaccurate ones leading to higher bias. This explains why generic global averaging does not work well. Within Aggarwal’s bias-variance framework, LSCP possesses a combination of both variance and bias reduction. It induces diversity by initializing various base detectors with various hyperparameters and indirectly promotes variance reduction in the way that the pseudo ground truth is generated, e.g., averaging in LSCP_A. Furthermore, LSCP focuses on detector selection by local competency, which helps identify base detectors with conditionally low model bias. LSCP_M is also expected to be more stable than global maximization (GG_M) since the variance is reduced by using the most competent detector’s output rather than the global maximum values of all base detectors. LSCP_MOA and LSCP_AOM further decrease generalization error through bias reduction and variance reduction, respectively, through their second-phase combination. Although, LSCP may reduce the generalization error through both variance and bias channels, it is a heuristic framework with unpredictable results on pathological datasets.
4 Numerical Experiments
4.1 Datasets and Evaluation Metrics.
Table 1 summarizes the 18 public outlier detection benchmark datasets used in this study222ODDS Library: http://odds.cs.stonybrook.edu. In each experiment, 60% of the data is used for training and the remaining 40% is set aside for validation. Performance is evaluated by taking the average score of 20 independent trials using area under the receiver operating characteristic (ROC-AUC) and mean average precision (mAP). Both metrics are widely used in outlier research [3, 4, 21, 23, 28, 13] and statistical measures are used to analyze the results . Specifically, we use a non-parametric Friedman test followed by a post-hoc Nemenyi test. For these tests, is considered to be statistically significant.
4.2 Experimental Design.
This study compares the six GG algorithms introduced in §2.3 with the four proposed LSCP variations described in Algorithm 1. All models use a pool of 50 LOF base detectors ensuring consistency during performance evaluation. To induce diversity among base detectors, distinct initialization hyperparameters—specifically the number of neighbors () used in each LOF detector—are randomly selected in the range of . For GG_AOM and GG_MOA, the base detectors are divided into 5 subgroups and each group contains 10 base detectors selected without replacement. For all LSCP algorithms, the default hyperparameters mentioned in §3 are used.
4.3 Algorithm Performances.
Tables 2 and 3 summarize the ROC and mAP scores on the 18 datasets. Our experiments demonstrate that LSCP can bring consistent performance improvement over its GG counterparts, which is especially noticeable in the mAP scores. The Friedman test shows there is a statistically significant difference between the ten algorithms in both ROC-AUC and mAP ; however, the Nemenyi test fails to identify which pairs of algorithms are significantly different. The latter result is expected in an unsupervised setting due to the difficulty of this task relative to the limited number of datasets . In general, LSCP algorithms show great potential: they achieve the highest ROC-AUC scores on 14 datasets and the highest mAP scores on 16 datasets. While GG_M performs better on Vowels and Satellite, and GG_AOM achieves the highest mAP on Annthyroid, in all other cases, generic global algorithms are outperformed by a variant of LSCP. Specifically, LSCP_AOM is the best performing method and ranks highest on 11 datasets. It should be noted though that GG methods with a second-phase combination (GG_MOA and GG_AOM) demonstrate better performance than GG_A and better stability than GG_M. These observations agree with the conclusions in Aggarwalâs work [2, 3].
LSCP_A and LSCP_M do not demonstrate strong performance relative to their GG counterparts. Given that both pseudo ground truth generation methods are heuristic, they may result in poor local competency evaluation. For example, as discussed in §3.6, LSCP_A theoretically benefits from both the variance and bias reduction by averaging and focusing on locality. In practice though, by only selecting the single most competent detector, it’s possible that this approach yields weaker variance reduction compared to GG_A which uses all detector scores. As consequence, the variance reduction may not be able to sufficiently offset the bias inherent to the the pseudo ground truth generation process leading to diminished performance. Comparatively, when the ground truth is generated by taking the maximum among multiple detectors, it is observed that both LSCP_M and GG_M exhibit unstable behaviour. As discussed in [2, 3], selecting maximum scores across detectors yields high model variance which explains these results; a second-phase combination mitigates this risk.
A Friedman test confirms there is a significant difference among the four LSCP algorithms in both ROC-AUC and mAP . Correspondingly, the LSCP ensemble variations (LSCP_MOA and LSCP_AOM) show promise. Building on GG_AOM’s success as one of the most effective combination methods , LSCP_AOM averages the selected group of detectors which could be viewed as an additional reduction of model variance over LSCP_M. Moreover, LSCP_AOM’s concentration on the local competency evaluation may have improved model bias and the second-phase averaging should have decreased the model variance leading to better stability. The results show that LSCP_AOM outperforms all models on 11 datasets in terms of ROC-AUC and 12 datasets in terms of mAP. The latter improvement over GG methods is especially considerable on Cardio, Satimage-2 and Thyroid.
The benefit of taking the second-phase combination is less effective for LSCP_MOA, which did not outperform LSCP_A or GG_MOA. As discussed in [2, 3], it is less effective to do a second-phase combination after averaging as information has already been already lost due to blunting. The experiment results confirm that, in an LSCP scheme, the benefit from a second-phase maximization cannot offset the information loss due to the initial averaging. Overall, only LSCP_AOM is recommended for detector combination among the four LSCP algorithms due to its combined bias and variance reduction capabilities.
4.4 Visualization Analysis.
Figure 2 visually compares the performance of the best performing GG and LSCP methods on Cardio, Thyroid and Letter using t-distributed stochastic neighbor embedding (t-SNE) . The green and blue markers highlight objects that can only be correctly classified by either the GG or LSCP methods, respectively, to emphasize the mutual exclusivity of the two approaches. The visualizations of Cardio (left) and Thyroid (middle) illustrate that LSCP methods have an edge over GG methods in detecting local outliers when they cluster together (highlighted by red dotted circles in Fig. 2). Additionally, LSCP methods can contribute to classifying both outlying and normal points when locality is present in the data. However, the outlying data distribution in Letter (right) is more dispersed—outliers do not form local clusters but rather mix with normal points. This causes LSCP to perform worse than GG_M in terms of ROC-AUC despite showing an improvement in terms of mAP. Based on these visualizations, one could assume that LSCP is useful when outlying and normal objects are well separated, but less effective when they are interleaved and cannot easily form local clusters. Additionally, the size of the local region should be informed by the estimated proportion of outliers in the dataset. For instance, outliers account for only 3.43% and 6.25% of Vowels and Letter respectively, which may not be sufficient to form outlier clusters when the local region size is set to 10% of the training data. A smaller local region size is more appropriate when a small number of outliers is assumed.
4.5 Limitations and Future Directions.
Firstly, the local region definition depends on finding nearest neighbors by euclidean distance. However, it is not ideal due to: (i) high time complexity  and (ii) degraded performance when many irrelevant features are presented in high dimensional space . This may be improved by using prototype selection  or by defining the local region using advanced clustering methods . Secondly, only simple pseudo ground truth generation methods are explored (averaging or maximization) in this study; more accurate methods should be considered, such as actively pruning base detectors . Thirdly, only parallel combination methods are compared in this study; sequential methods discussed in §2.3 should be included in future works. Lastly, DCS has proven to work with heterogeneous base classifiers in classification problems [16, 9], which is pending for verification in LSCP. Significant improvement is expected since the base detectors used in this study are homogeneous with limited diversity.
In this work, we propose four variants of a novel unsupervised outlier detection framework called Locally Selective Combination in Parallel Outlier Ensembles (LSCP). Unlike traditional combination approaches, LSCP identifies the top-performing base detectors for each test instance relative to its local region. To validate the effectiveness of this approach, the proposed framework is assessed on 18 real-world datasets and observed to be superior to baseline algorithms. The ensemble approach LSCP_AOM demonstrated the best performance achieving the highest detection score on 11/18 datasets with respect to ROC-AUC and 12/18 datasets with respect to mAP. Theoretical considerations under the bias-variance framework are also provided for LSCP, alongside visualizations, to provide a holistic view of the framework. Since LSCP demonstrates the promise of data locality, we hope that future work extends this exploration by investigating the use of heterogeneous base detectors and more reliable pseudo ground truth generation methods. All source code, experimental results and figures used in this study are made publicly available333Repository: https://github.com/yzhao062/LSCP.
-  C. C. Aggarwal, Outlier ensembles: position paper, ACM SIGKDD Explorations, 14 (2013), pp. 49–58.
-  C. C. Aggarwal and S. Sathe, Theoretical Foundations and Algorithms for Outlier Ensembles, ACM SIGKDD Explorations, 17 (2015), pp. 24–47.
-  , Outlier ensembles: An introduction, Springer, 1st ed., 2017.
-  L. Akoglu, H. Tong, J. Vreeken, and C. Faloutsos, Fast and Reliable Anomaly Detection in Categorical Data, in CIKM, 2012.
-  M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. ö. r. Sander, LOF: Identifying Density-Based Local Outliers, ACM SIGMOD, (2000), pp. 1–12.
-  A. S. Britto, R. Sabourin, and L. E. Oliveira, Dynamic selection of classifiers - A comprehensive review, Pattern Recognition, 47 (2014), pp. 3665–3680.
-  G. O. Campos, A. Zimek, and W. Meira, An Unsupervised Boosting Strategy for Outlier Detection Ensembles, PAKDD, (2018), pp. 564–576.
-  V. Chandola, A. Banerjee, and V. Kumar, Anomaly detection: A survey, CSUR, 41 (2009), p. 15.
-  R. M. Cruz, R. Sabourin, and G. D. Cavalcanti, Dynamic classifier selection: Recent advances and perspectives, Information Fusion, 41 (2018), pp. 195–216.
-  S. Das, W.-K. Wong, T. Dietterich, A. Fern, and A. Emmott, Incorporating Expert Feedback into Active Anomaly Discovery, ICDM, (2016), pp. 853–858.
-  J. Dem š ar, Statistical Comparisons of Classifiers over Multiple Data Sets, JMLR, 7 (2006), pp. 1–30.
-  T. G. Dietterich, Ensemble Methods in Machine Learning, MCS, 1857 (2000), pp. 1–15.
-  A. Emmott, S. Das, T. Dietterich, A. Fern, and W.-k. Wong, A Meta-Analysis of the Anomaly Detection Problem, arXiv preprint, (2015).
-  G. Giacinto and F. Roli, A theoretical framework for dynamic classifier selection, ICPR, 2 (2000), pp. 0–3.
-  T. K. Ho, J. J. Hull, and S. N. Srihari, Decision Combination in Multiple Classifier Systems, TPAMI, 16 (1994), pp. 66–75.
-  A. H. Ko, R. Sabourin, and A. S. Britto, From dynamic classifier selection to dynamic ensemble selection, Pattern Recognition, 41 (2008), pp. 1735–1748.
-  H.-P. Kriegel, P. Kr ö ger, E. Schubert, and A. Zimek, LoOP: local outlier probabilities, CIKM, (2009), pp. 1649–1652.
-  A. Lazarevic and V. Kumar, Feature bagging for outlier detection, ACM SIGKDD, (2005), p. 157.
-  F. T. Liu, K. M. Ting, and Z. H. Zhou, Isolation forest, ICDM, (2008), pp. 413–422.
-  L. v. d. Maaten and G. Hinton, Visualizing data using t-sne, JMLR, 9 (2008), pp. 2579–2605.
-  B. Micenkov á, B. McWilliams, and I. Assent, Learning Representations for Outlier Detection on a Budget, arXiv preprint, (2015).
-  S. Rayana and L. Akoglu, An Ensemble Approach for Event Detection and Characterization in Dynamic Graphs, in ACM SIGKDD ODD Workshop, 2014.
-  , Less is More: Building Selective Anomaly Ensembles, TKDD, 10 (2016), pp. 1–33.
-  S. Rayana, W. Zhong, and L. Akoglu, Sequential ensemble learning for outlier detection: A bias-variance perspective, ICDM, (2017), pp. 1167–1172.
-  E. Schubert, A. Zimek, and H. P. Kriegel, Local outlier detection reconsidered: A generalized view on locality with applications to spatial, video, and network outlier detection, DMKD, 28 (2014), pp. 190–237.
-  B. van Stein, M. van Leeuwen, and T. B ä ck, Local subspace-based outlier detection using global neighbourhoods, IEEE International Conference on Big Data, (2016), pp. 1136–1142.
-  K. Woods, W. Kegelmeyer, and K. Bowyer, Combination of multiple classifiers using local accuracy estimates, TPAMI, 19 (1997), pp. 405–410.
-  Y. Zhao and M. K. Hryniewicki, XGBOD: Improving Supervised Outlier Detection with Unsupervised Representation Learning, IJCNN, (2018).
-  Y. Zhao, M. K. Hryniewicki, F. Cheng, B. Fu, and X. Zhu, Employee Turnover Prediction with Machine Learning: A Reliable Approach, IEEE Intelligent System Conference (Intellisys), (2018).
-  A. Zimek, R. J. G. B. Campello, and J. ö. r. Sander, Ensembles for unsupervised outlier detection: Challenges and research questions, ACM SIGKDD Explorations, 15 (2014), pp. 11–22.