Binary Classifier Calibration using an Ensemble of Near Isotonic Regression Models
Abstract
Learning accurate probabilistic models from data is crucial in many practical tasks in data mining. In this paper we present a new nonparametric calibration method called ensemble of near isotonic regression (ENIR). The method can be considered as an extension of BBQ [18], a recently proposed calibration method, as well as the commonly used calibration method based on isotonic regression. ENIR is designed to address the key limitation of isotonic regression which is the monotonicity assumption of the predictions. Similar to BBQ, the method postprocesses the output of a binary classifier to obtain calibrated probabilities. Thus it can be combined with many existing classification models. We demonstrate the performance of ENIR on synthetic and real datasets for the commonly used binary classification models. Experimental results show that the method outperforms several common binary classifier calibration methods. In particular on the real data, ENIR commonly performs statistically significantly better than the other methods, and never worse. It is able to improve the calibration power of classifiers, while retaining their discrimination power. The method is also computationally tractable for large scale datasets, as it is time, where is the number of samples.
I Introduction
This paper focuses on developing a new nonparametric calibration method for postprocessing the output of commonly used binary classification models to generate accurate probabilities. Obtaining accurate probabilities is crucial in many real world decision making and data mining problems. Decision theory provides a rationale basis for intelligent agents to make decisions [21]. Decision theory combines utilities and probabilities in determining the actions that maximize expected utility. In general, the probabilities need to be well calibrated in order to achieve this goal of finding the best actions.
Informally, we say that a classification model is well calibrated if events predicted to occur with probability p do occur about p fraction of the time, for all p. This concept applies to binary as well as multiclass classification problems. Figure 1 illustrates the binary calibration problem using a reliability curve [5, 17]. The curve shows the probability predicted by the classification model versus the actual fraction of positive outcomes for a hypothetical binary classification problem, where Z is the binary event being predicted. The curve shows that when the model predicts to have probability , the outcome occurs in about fraction of the time. The curve shows that the model is fairly well calibrated, but it tends to underestimate the actual probabilities. In general, the straight dashed line connecting to represents a perfectly calibrated model. The closer a calibration curve is to this line, the better calibrated is the associated prediction model. Deviations from perfect calibration are very common in practice and may vary widely depending on the binary classification model that is used [18].
Producing wellcalibrated probabilistic predictions is critical in many areas of science (e.g., determining which experiments to perform), medicine (e.g., deciding which therapy to give a patient), business (e.g., making investment decisions), and many others. In data mining problems, obtaining wellcalibrated classification models is crucial not only for decisionmaking, but also for combining output of different classification models [3]. It is also useful when we aim to use the output of a classifier not only to discriminate the instances but also to rank them [26, 14, 10]. Research on learning well calibrated models has not been explored in the data mining literature as extensively as, for example, learning models that have high discrimination (e.g., high accuracy).
There are two main approaches to obtaining wellcalibrated classification models. The first approach is to build a classification model that is intrinsically wellcalibrated ab initio. This approach will restrict the designer of the data mining model by requiring major changes in the objective function (e.g, using a different type of loss function) and could potentially increase the complexity and computational cost of the associated optimization program to learn the model. The other approach is to rely on the existing discriminative data mining models and then calibrate their output using postprocessing methods. This approach has the advantage that it is general, flexible, and it frees the designer of a data mining algorithm from modifying the learning procedure and the associated optimization method [18]. However, this approach has the potential to decrease discrimination while increasing calibration, if care is not taken.
The method we describe in this paper is shown empirically to improve calibration of different types of classifiers (e.g., LR, SVM, and NB) while maintaining their discrimination performance well. Existing postprocessing binary classifier calibration methods include Platt scaling [20], histogram binning [24], isotonic regression [25], and a recently proposed method BBQ which is a Bayesian extension of histogram binning [18]. In all these methods, the postprocessing step can be seen as a function that maps the outputs of a prediction model to probabilities that are intended to be wellcalibrated. Figure 1 shows an example of such a mapping.
In general, there are two main applications of postprocessing calibration methods. First, they can be used to convert the outputs of discriminative classification methods with no apparent probabilistic interpretation to posterior class probabilities [20]. An example is an SVM model that learns a discriminative model that does not have a direct probabilistic interpretation. In this paper, we show this use of calibration to map SVM outputs to wellcalibrated probabilities. Second, calibration methods can be applied to improve the calibration of predictions of a probabilistic model that is miscalibrated. For example, a naïve Bayes (NB) model is a probabilistic model, but its class posteriors are often miscalibrated due to unrealistic independence assumptions [17]. The method we describe is shown empirically to improve the calibration of NB models without reducing their discrimination. The method can also work well on calibrating models that are less egregiously miscalibrated than are NB models.
Ii Related work
Existing postprocessing binary classifier calibration models can be divided into parametric and nonparametric methods. Platt’s method is an example of the former; it uses a sigmoid transformation to map the output of a classifier into a calibrated probability [20]. The two parameters of the sigmoid function are learned in a maximumlikelihood framework using a modeltrust minimization algorithm [9]. The method was originally developed to transform the output of an SVM model into calibrated probabilities. It has also been used to calibrate other type of classifiers [17]. The method runs in at test time, and thus, it is fast. Its key disadvantage is the restrictive shape of sigmoid function that rarely fits the true distribution of the predictions [15].
A popular nonparametric calibration method is the equal frequency histogram binning model which is also known as quantile binning [24]. In quantile binning, predictions are partitioned into equal frequency bins. For each new prediction that falls into a specific bin, the associated frequency of observed positive instances will be used as the calibrated estimate for , where is the true label of an instance that is either or . Histogram binning can be implemented in a way that allows it to be applied to large scale data mining problems. Its limitations include (1) bins inherently âpigeonholeâ calibrated probabilities into only possibilities, (2) bin boundaries remain fixed over all predictions, and (3) there is uncertainty in the optimal number of the bins to use [25].
The most commonly used nonparametric classifier calibration method in machine learning and data mining applications is the isotonic regression based calibration (IsoRegC) model [25]. To build a mapping from the uncalibrated output of a classifier to the calibrated probability, IsoRegC assumes the mapping is an isotonic (monotonic) mapping following the ranking imposed by the base classifier. The commonly used algorithm for isotonic regression is the Pool Adjacent Violators Algorithm (PAVA), which is linear in the number of training data [2]. An IsoRegC model based on PAVA can be viewed as a histogram binning model [25] where the position of the boundaries are selected by fitting the best monotone approximation to the train data according to the ordering imposed by the classifier. There is also a variation of the isotonicregressionbased calibration method for predicting accurate probabilities with a ranking loss [16]. In addition, an extension to IsoRegC combines the outputs generated by multiple binary classifiers to obtain calibrated probabilities [27]. While IsoRegC can perform well on some real datasets, the monotonicity assumption it makes can fail in real data mining applications. This can specifically occur when we encounter large scale data mining problems in which we have to make simplifying assumptions to build the classification models. Thus, there is a need to relax the assumption, which is the focus of the current paper.
Adaptive calibration of predictions (ACP) is another extension to histogram binning [15]. ACP requires the derivation of a statistical confidence interval around each individual prediction to build the bins. It then sets the calibrated estimate to the observed frequency of the instances with positive class among all the predictions that fall within the bin. To date, ACP has been developed and evaluated using only logistic regression as the base classifier [15].
Recently, a new nonparametric calibration model called BBQ was proposed which is a refinement of the histogrambinning calibration method [18]. BBQ addresses the main drawbacks of the histogram binning model by considering multiple different equal frequency histogram binning models and their combination using a Bayesian scoring function [11]. However, BBQ has two disadvantages. First, as a post processing method, BBQ does not utilize a key prior knowledge in calibrating the output of a classifier. This prior knowledge is that, in most real data mining problems, the classifier should perform well in terms of discrimination measures (e.g., AUC measure), otherwise we are not interested in using it. Second, BBQ still selects the position and boundary of the bins by considering only equal frequency histogram binning models. A Bayesian nonparametric method called ABB addresses the later problem by considering Bayesian averaging over all possible binning models induced by the training instances [19]. The main drawback of ABB is that it is computationally intractable for most real world applications, as it requires computations for learning the model as well as computations for computing the calibrated estimate for each of the test instances^{1}^{1}1Note that the running time for the test instance can be reduced to in any postprocessing calibration model by using a simple caching technique that reduces calibration precision in order to decrease calibration time [19].
This paper presents a new binary classifier calibration method called ensemble of near isotonic regression (ENIR) that can post process the output generated by a wide variety of classification models. The essential idea in ENIR is to use the prior knowledge that the scores that are going to be calibrated are in fact generated by a well performing classifier in terms of discrimination. IsoRegC also uses such prior knowledge; however, it is biased by constraining the calibrated scores to obey the ranking imposed by the classifier. In the limit, this is equivalent to presuming the classifier has AUC equal to , which rarely happens in real world data mining applications. In contrast, BBQ does not make any assumptions about the correctness of classifier rankings. ENIR provides a balanced approach that spans between IsoRegC and BBQ. In particular, ENIR assumes that the mapping from uncalibrated scores to calibrated probabilities is a near isotonic (monotonic) mapping; It allows violations of the ordering imposed by the classifier and then penalizes them through the use of a regularization term. ENIR utilizes the path algorithm modified pool adjacent violators algorithm (mPAVA) that can find the solution path to near isotonic regression problem in , where is the number of training instances [23]. Finally, it uses the BIC scoring measure to combine the predictions made by these models to yield more robust calibrated predictions.
Iii Method
In this section we introduce the ensemble of near isotonic regression (ENIR) calibration method. ENIR utilizes the near isotonic regression method [23] that seeks a nearly monotone approximation for a sequence of data . The proposed calibration model extends the commonly used isotonic regression based calibration by a (approximate) selective Bayesian averaging of a set of nearly isotonic regression models. The set includes the isotonic regression model as an extreme member. From another viewpoint, ENIR can be considered as an extension to a recently proposed calibration model BBQ [18] by relaxing the assumption that probability estimates are independent inside the bins and finding the boundary of the bins automatically through an optimization algorithm.
Before getting into the details of the method, we define some notation. Let and define respectively an uncalibrated classifier prediction and the true class of the ’th instance. In this paper, we focus on calibrating a binary classifier’s output^{2}^{2}2For classifiers that output scores that are not in the unit interval (e.g. SVM), we use a simple sigmoid transformation to transform the scores into the unit interval., and thus, and . Let define the set of all training instances . Without loss of generality, we can assume that the instances are sorted based on the classifier scores , so we have , where is the total number of samples in the training data.
The standard isotonic regression based calibration model finds the calibrated probability estimates by solving the following optimization problem:
(1)  
s.t.  
where is the vector of calibrated probability estimates. The rationale behind the model is to assume that the base classifier ranks the instances correctly. To find the calibrated probability estimates, it seeks the best fit of the data that are consistent with the classifier’s ranking. A unique solution to the above convex optimization program exists and can be obtained by an inductive iterative algorithm called pool adjacent violator algorithm (PAVA) that runs in . Note, however, that isotonic regression calibration still needs computations, due to the fact that instances are required to be sorted based on the classifier scores . PAVA iteratively groups the consecutive instances that violate the ranking constraint and uses their average over (frequency of positive instances) as the calibrated estimate for all the instances within the group. We define the set of these consecutive instances that are located in the same group and attain the same predicted calibrated estimate, as a bin. Therefore, an isotonic regressionbased calibration can be viewed as a histogram binning method [25] where the position of boundaries are selected by fitting the best monotone approximation to the training data according to the ranking imposed by the classifier.
One can show that the second constraint in the optimization given by Equation 1 is redundant, and it is possible to rewrite the equation in the following equivalent form:
(2)  
s.t. 
where is the indicator function of ranking violation. Relaxing the equality constraint in the above optimization program leads to a new optimization problem called nearly isotonic regression [23].
(3) 
where is a positive real number that regulates the tradeoff between the monotonicity of the calibrated estimates with the goodness of fit by penalizing adjacent pairs that violate the ordering imposed by the base classifier. The above optimization problem is convex having a unique solution , where the use of the subscript emphasizes the dependency of the final solution to the value of .
The entire path of solutions for any value of of the near isotonic regression problem can be found using a similar algorithm to PAVA which is called modified pool adjacent violator algorithm (mPAVA) [23]. mPAVA finds the whole solution path in , and needs memory space. Briefly, the algorithm works as follows: It starts by constructing bins, each bin containing a single instance of the train data. Next, it finds the solution path by starting from the saturated fit , that corresponds to setting , and then increasing iteratively. As the increases the calibrated probability estimates , for each bin, will change linearly with respect to until the calibrated probability estimates of two consecutive bins attain equal value. At this stage, mPAVA merges the two bins that have the same calibrated estimate to build a larger bin, and it updates their corresponding estimate to a common value. The process continues until there is no change in the solution for a large enough value of that corresponds to finding the standard isotonic regression solution. The essential idea of mPAVA is based on a theorem stating that if two adjacent bins are merged on some value of to construct a larger bin, then the new bin will never split for all larger values of [23].
mPAVA yields a collection of nearly isotonic calibration models, with the over fitted calibration model at one end () and the isotonic regression solution at the other (), where is a large positive real number. Each of these models can be considered as a histogram binning model where the position of boundaries and the size of bins are selected according to how well the model trades off the goodness of fit with the preservation of the ranking generated by the classifier, which is governed by the value of , (As increases the model is more concerned to preserving the original ranking of the classifier, while for the small it prioritizes the goodness of fit.)
ENIR employs the approach just described to generate a collection of models (one for each value of ). It then uses the Bayesian Information Criterion (BIC) to score each of the models ^{3}^{3}3Note that we exclude the highly overfitted model that corresponds to from the set of models in ENIR. Assume mPAVA yields the binning models , where is the total number of models generated by mPAVA. For any new classifier output , the calibrated prediction in the ENIR model is defined using selective Bayesian model averaging [12]:
where is the probability estimate obtained using the binning model for the uncalibrated classifier output . Also, is defined using the BIC scoring function [22]. Next, for the sake of the completeness, we briefly describe the mPAVA algorithm; more detailed information about the algorithm and the derivations can be found in [23].
Iiia The modified PAV algorithm
Suppose at a value of we have bins, . We can represent the unconstrained optimization program given by Equation 3 as the following loss function that we seek to minimize :
(4)  
where defines the common estimated value for all the instances located at the bin . The loss function is always differentiable with respect to unless two calibrated probabilities are just being joined (which only happens if for some ). Assuming that is optimal, the partial derivative of has to be at , which implies:
(5) 
Rewriting the above equation, the optimum predicted value for each bin can be calculated as:
(6) 
While PAVA uses the frequency of instances in each bin as the calibrated estimate, Equation 6 shows that mPAVA uses a shrunken version of the frequencies by considering the estimates that are not following the ranking imposed by the base classifier. In Equation 5, taking derivatives with respect to yields:
(7) 
where we set for notational convenience. As we noted above, it has been proven that the optimal values of the instances located in the same bin are tied together and the only way that they can change is to merge two bins as they can never split apart as the increases [23]. Therefore, as we make changes in , the bins , and hence the values remain constant. This implies the term is a constant in Equation 7. Consequently, the solution path remains piecewise linear as increases, and the breakpoints happen when two bins merge together. Now, using the piecewise linearity of the solution path and assuming that the two bins and are the first two bins to merge by increasing , the value of at which the two bins and will merge is calculated as:
(8) 
where is the slope of the changes of with respect to according to Equation 7. Using the above identity, the at which the next breakpoint occurs is obtained using the following equation:
(9)  
where indicates the set of the indexes of the bins that will be merged by their consecutive bins changing the ^{4}^{4}4Note that there could be more than one bin achieving the minimum in Equation 9, so they should be all merged with the bins that are located next to them.. If then the algorithm will terminate since it has obtained the standard isotonic regression solution, and by increasing none of the existing bins will ever merge. Having the solutions of the near isotonic regression problem in Equation 3 at the breakpoints, and using the piecewise linearity property of the solution path, it is possible to recover the solution for any value of through interpolation. However, the current implementation of ENIR only uses the near isotonic regression based calibration models that corresponds to the value of at the breakpoints. The sketch of the algorithm is shown as Algorithm [1].
Iv Empirical Results
This section describes the set of experiments that we performed to evaluate the performance of in comparison to Isotonic Regression based Calibration (IsoRegC) [25], and a state of the art binary classifier calibration method called BBQ [18]. We use IsoRegC because it is one of the most commonly used calibration models showing promising performance on real world applications [17, 25]. Moreover is an extension of IsoRegC, and we are interested in evaluating whether it performs better than IsoRegC. We also include BBQ as a state of the art binary classifier calibration model, which is a Bayesian extension of the simple histogram binning model [18]. We did not include Platt’s method since it is a simple and restricted parametric model and there are prior works showing that IsoRegC and BBQ perform superior to Platt’s method [17, 25, 18]. We also did not include the ACP method since it requires not only probabilistic predictions, but also a statistical confidence interval () around each of those predictions, which makes it tailored to specific classifiers, such as LR [15]; this is counter to our goal of developing postprocessing methods that can be used with any existing classification models. Finally, we did not include ABB in our experiments mainly because it is not computationally tractable for real datasets that have more than couple of thousands instances. Moreover, even for small size datasets, we noticed that ABB performs quite similarly to BBQ. To evaluate the performance of the methods, we ran experiments on both simulated and on real data.
Iva Evaluation Measures
In order to evaluate the performance of the calibration models, we use different evaluation measures. We use Accuracy (Acc) and area under ROC curve (AUC) to evaluate how well the methods discriminate the positive and negative instances in the feature space. We also utilize the three measures of root mean square error (RMSE), maximum calibration error (MCE), and expected calibration error (ECE) to measure the calibration [18, 19]. MCE and ECE are two simple statistics of the reliability curve (Figure 1 shows a hypothetical example of such curve) computed by partitioning the output space of the binary classifier, which is the interval , into fixed number of bins ( in our experiments). The estimated probability for each instance will be located in one of the bins. For each bin we can define the associated calibration error as the absolute difference between the expected value of predictions and the actual observed frequency of positive instances. The calculates the maximum calibration error among the bins, and calculates expected calibration error over the bins, using empirical estimates as follows:
where is the empirical probability or the fraction of all instances that fall into bin , is the mean of the estimated probabilities for the instances in bin , and is the observed fraction of positive instances in bin . The lower the values of and , the better is the calibration of a model.
IvB Simulated Data
For the simulated data experiments, we used a binary classification dataset that was used in previous works [18, 19]. The scatter plot of the simulated dataset is shown in Figure 2. Pakdaman M. et al. developed this classification problem, which illustrates how IsoRegC can suffer from a violation of the isotonicity assumption. We repeated the experiments in [18, 19] to compare the performance of IsoRegC with our new method that assumes approximate isotonicity. In our experiments, the data is divided into instances for training and calibrating the prediction model, and instances for testing the models. We report the average of fold cross validation results for the simulated dataset.
To conduct the experiments with the simulated data, we used two extreme classifiers: support vector machines (SVM) with linear and quadratic kernels. The choice of SVM with a linear kernel allows us to see how ENIR perform when the classification model makes an over simplifying (linear) assumption. Also, to achieve good discrimination on the circular configuration data in Figure 2, SVM with a quadratic kernel is a reasonable choice (as is also evidenced qualitatively in Figure 2 and quantitatively in Table (b)b). So, the experiment using quadratic kernel SVM allows us to see how well ENIR performs when we use models that should discriminate well.
As seen in Table I, ENIR generally outperforms IsoRegC on the simulation dataset, especially when the linear SVM method is used as the base learner. This is due to the monotonicity assumption of IsoReC which presumes the best calibrated estimates will match the ordering imposed by the base classifier. When we use SVM with a linear kernel, this assumption is violated due to the nonlinarity of the data. Consequently, IsoRegC only provides limited improvement of the calibration and discrimination performance of the base classifier. ENIR performs very well in this case since it is using the ranking information of the base classifier, but it is not anchored to it. The violation of the monotonicity assumption can happen in real data as well, especially in large scale data mining problems in which we use simple classification models due to the computational constraints. As shown in Table (b)b, even when we apply a highly appropriate SVM classifier to classify the instances for which IsoRegC is expected to perform well (and indeed does so), ENIR performs as well or better than IsoRegC.


IvC Real Data
We used different baseline real data sets from the UCI and LibSVM repositories^{5}^{5}5The datasets used were as follows: spect, adult, breast, pageblocks, pendigits, ad, mamography, satimage, australian, code rna, colon cancer, covtype, letter unbalanced, letter balanced, diabetes, duke, fourclass, german numer, gisette scale, heart, ijcnn, ionosphere scale, liver disorders, mushrooms, sonar scale, splice, svmguide, svmguide, coil, balance, breast cancer, leu, wa, thyroid sick, scene, uscrime, solar, car, car , protein homology. [1, 4]. Five summary statistics of the size of the datasets and the percentage of the minority class are shown in Table II.
We used three common classifiers, Logistic Regression (LR), Support Vector Machines (SVM), and Naïve Bayes (NB) to evaluate the performance of the proposed calibration method. In the experiments we used the average over random runs of fold cross validation, and we always used the train data for calibrating the models. To compare the performance of the calibration models, we used the statistical test procedure recommended by Demsar [6]. More specifically, we used the Freidman nonparametric hypothesis testing method [8] followed by Holm’s stepdown procedure [13] to evaluate the performance of ENIR in comparison with IsoregC and BBQ, across the baseline datasets.
Min  Q1  Meadian  Q3  Max  

Size  42  683  1861  8973  581012 
Percent  0.009  0.076  0.340  0.443  0.500 
Tables [III,IV,V] show the results of the performance of in comparison with IsoRegC and BBQ. In these tables, we show the average rank of each method across the baseline datasets, where boldface indicates the best performing method. In these tables, the marker / indicates whether ENIR is statistically superior/inferior to the compared method using the Friedman test followed by Holm’s stepdown procedure at a significance level. For instance, Table IV shows the performance of the calibration models when we use SVM as the base classifier; the results show that ENIR achieves the best performance in terms of RMSE by having an average rank of across the baseline datasets. The result indicates that in terms of RMSE, ENIR is statistically superior to BBQ; however, it is not performing statistically differently than IsoRegC.
Table III shows the results of comparison when we use LR as the base classifier. As shown, the performance of ENIR is always superior to BBQ and IsoRegC except for MCE in which BBQ is superior to ENIR; however, the difference is not statistically significant for MCE. The results show that in terms of discrimination based on AUC, there is not a statistically significant difference between the performance of ENIR compared with BBQ and IsoRegC. However, ENIR performs statistically better than BBQ in terms of ACC. In terms of calibration measures, ENIR is statistically superior to both IsoRegC and BBQ in terms of RMSE. In terms of MCE, ENIR is statistically superior to IsoRegC.
Table IV shows the results when we use SVM as the base classifier. As shown, the performance of ENIR is always superior to BBQ and IsoRegC except for MCE in which BBQ performs better than ENIR; however, the difference is not statistically significant for MCE. The results show that although ENIR is superior to IsoRegC and BBQ in terms of discrimination measures, AUC and ACC, the difference is not statistically significant. In terms of calibration measures, ENIR performs statistically superior to BBQ in terms of RMSE and it is statistically superior to IsoRegC in terms of MCE.
Table V shows the results of comparison when we use NB as the base classifier. As shown, the performance of ENIR is always superior to BBQ and IsoRegC. In terms of discrimination, for AUC there is not a statistically significant difference between the performance of ENIR compared with BBQ and IsoRegC; however, in terms of ACC, ENIR is statistically superior to BBQ. In terms of calibration measures, ENIR is always statistically superior to IsoRegC. ENIR is also statistically superior to BBQ in terms of ECE and RMSE.
Overall, in terms of discrimination measured by AUC and ACC, the results show that the proposed calibration method either outperforms IsoRegC and BBQ, or has a performance that is not statistically significantly different. In terms of calibration measured by ECE, MCE, and RMSE, ENIR either outperforms other calibration methods, or it has a statistically equivalent performance to IsoRegC and BBQ.
IsoReg  BBQ  ENIR  

AUC  1.963  2.225  1.813 
ACC  1.675  2.663  1.663 
RMSE  1.925  2.625  1.450 
ECE  2.125  1.975  1.900 
MCE  2.475  1.750  1.775 
IsoReg  BBQ  ENIR  

AUC  1.988  2.025  1.988 
ACC  2.000  2.150  1.850 
RMSE  1.850  2.475  1.675 
ECE  2.075  2.025  1.900 
MCE  2.550  1.625  1.825 
IsoReg  BBQ  ENIR  

AUC  2.150  1.925  1.925 
ACC  1.963  2.375  1.663 
RMSE  2.200  2.375  1.425 
ECE  2.475  2.075  1.450 
MCE  2.563  1.850  1.588 
In addition to comparing the performance of ENIR with IsoRegC and BBQ, we also show in Table VI the confidence interval for the mean of the random variable , which is defined as the percentage of the gain (or loss) of ENIR with respect to the base classifier:
(10) 
where is one of the evaluation measures AUC, ACC, ECE, MCE, or RMSE. Also, denotes one of the choices of the base classifiers, namely, LR, SVM, or NB. For instance, Table VI shows that by postprocessing the output of SVM using ENIR, we are confident to gain anywhere from to average improvement in terms of RMSE. This could be a promising result, depending on the application, considering the CI for the AUC which shows that by using ENIR we are confident not to loose more than of the SVM discrimination power in terms of AUC (Note, however, that the CI includes zero, which indicates that there is not a statistically significant difference between the performance of SVM and ENIR in terms of AUC).
Overall, the results in Table VI show that there is not a statistically meaningful difference between the performance of ENIR and the base classifiers in terms of AUC. The results support at a confidence level that ENIR improves the performance of LR or NB in terms of ACC. Furthermore, the results in Table VI show that by postprocessing the output of LR, SVM, and NB using ENIR, we can make dramatic improvements in terms of calibration measured by RMSE, ECE, and MCE. For instance, the results indicate that at a confidence level, ENIR improved the average performance of NB in terms of ECE anywhere from to , which could be practically significant in many decisionmaking and data mining applications.
LR  SVM  NB  

AUC  [0.008 , 0.003]  [0.010 , 0.003]  [0.010 , 0.000] 
ACC  [0.002 , 0.016]  [0.001 , 0.010]  [0.012 , 0.068] 
RMSE  [0.124 , 0.016]  [0.310 , 0.176]  [0.196 , 0.100] 
ECE  [0.389 , 0.153]  [0.768 , 0.591]  [0.514 , 0.274] 
MCE  [0.313 , 0.064]  [0.591 , 0.340]  [0.552 , 0.305] 
Finally, Table VII shows a summary of the time complexity of different binary classifier calibration methods in learning for N training instances and the test time for only one instance.
Training Time  Testing Time  

Platt  
Hist  
IsoRegC  
ACP  
ABB  
BBQ  
ENIR 
V Conclusion
In this paper, we presented a new nonparametric binary classifier calibration method called ensemble of near isotonic regression (ENIR) that generalizes the isotonic regression based calibration method (IsoRegC) in two ways. First, ENIR makes a more realistic assumption compared to IsoRegC by assuming that the transformation from the uncalibrated output of a classifier to calibrated probability estimates is approximately (but not necessarily exactly) a monotonic function. Second, ENIR is an ensemble model that utilizes the BIC scoring function to perform selective model averaging over a set of near isotonic regression models that indeed includes IsoRegC as an extreme member. The method is computationally tractable, as it runs in for training instances. It can be used to calibrate many different types of binary classifiers, including logistic regression, support vector machines, naïve Bayes , and others. Our experiments show that by post processing the output of classifiers using ENIR, we can gain high calibration improvement in terms of RMSE, ECE, and MCE, without losing any statistically meaningful discrimination performance. Moreover, our experimental evaluation on a broad range of real datasets showed that ENIR outperforms IsoRegC and BBQ (i.e. a stateoftheart binary classifier calibration method [18]).
An important advantage of ENIR over BBQ is that it can be extended to a multiclass and multilabel calibration models similar to what has done for the standard IsoRegC [25]. This is an area of our current research. We also plan to investigate theoretical properties of ENIR. In particular, we are interested to investigate theoretical guarantees regarding the discrimination and calibration performance of ENIR, similar to what has been proved for the AUC guarantees of IsoRegC [7].
References
 [1] K. Bache and M. Lichman. UCI machine learning repository, 2013.
 [2] Richard E Barlow, David J Bartholomew, JM Bremner, and H Daniel Brunk. Statistical inference under order restrictions: The theory and application of isotonic regression. Wiley New York, 1972.
 [3] Antonio Bella, Cèsar Ferri, José HernándezOrallo, and María José RamírezQuintana. On the effect of calibration in classifier combination. Applied intelligence, 38(4):566–585, 2013.
 [4] ChihChung Chang and ChihJen Lin. Libsvm: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology (TIST), 2(3):27, 2011.
 [5] M.H. DeGroot and S.E. Fienberg. The comparison and evaluation of forecasters. The Statistician, pages 12–22, 1983.
 [6] Janez Demšar. Statistical comparisons of classifiers over multiple data sets. The Journal of Machine Learning Research, 7:1–30, 2006.
 [7] Tom Fawcett and Alexandru NiculescuMizil. Pav and the roc convex hull. Machine Learning, 68(1):97–106, 2007.
 [8] Milton Friedman. The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the American Statistical Association, 32(200):675–701, 1937.
 [9] Philip E Gill, Walter Murray, and Margaret H Wright. Practical optimization, volume 5. Academic press London, 1981.
 [10] Homa B. Hashemi, Nasser Yazdani, Azadeh Shakery, and Mahdi Pakdaman Naeini. Application of ensemble models in web ranking. In 5th International Symposium on Telecommunications (IST), pages 726–731. IEEE, 2010.
 [11] D. Heckerman, D. Geiger, and D.M. Chickering. Learning bayesian networks: The combination of knowledge and statistical data. Machine Learning, 20(3):197–243, 1995.
 [12] Jennifer A Hoeting, David Madigan, Adrian E Raftery, and Chris T Volinsky. Bayesian model averaging: a tutorial. Statistical Science, pages 382–401, 1999.
 [13] Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, pages 65–70, 1979.
 [14] Liangxiao Jiang, Harry Zhang, and Jiang Su. Learning knearest neighbor naive bayes for ranking. In Advanced Data Mining and Applications, pages 175–185. Springer, 2005.
 [15] X. Jiang, M. Osl, J. Kim, and L. OhnoMachado. Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association, 19(2):263–274, 2012.
 [16] Aditya Menon, Xiaoqian Jiang, Shankar Vembu, Charles Elkan, and Lucila OhnoMachado. Predicting accurate probabilities with a ranking loss. In Proceedings of the International Conference on Machine Learning, pages 703–710, 2012.
 [17] A. NiculescuMizil and R. Caruana. Predicting good probabilities with supervised learning. In Proceedings of the International Conference on Machine Learning, pages 625–632, 2005.
 [18] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. In TwentyNinth AAAI Conference on Artificial Intelligence, 2015.
 [19] Mahdi Pakdaman Naeini, Gregory F. Cooper, and Milos Hauskrecht. Binary classifier calibration using a bayesian nonparametric approach. In SIAM Data Mining (SDM), 2015.
 [20] John C Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in Large Margin Classifiers, 10(3):61–74, 1999.
 [21] Stuart Jonathan Russell, Peter Norvig, Ernest Davis, Stuart Jonathan Russell, and Stuart Jonathan Russell. Artificial intelligence: a modern approach, volume 2. Prentice hall Englewood Cliffs, 2010.
 [22] Gideon Schwarz et al. Estimating the dimension of a model. The annals of statistics, 6(2):461–464, 1978.
 [23] Ryan J Tibshirani, Holger Hoefling, and Robert Tibshirani. Nearlyisotonic regression. Technometrics, 53(1):54–61, 2011.
 [24] B. Zadrozny and C. Elkan. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In International Conference on Machine Learning, pages 609–616, 2001.
 [25] B. Zadrozny and C. Elkan. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 694–699, 2002.
 [26] Harry Zhang and Jiang Su. Naive bayesian classifiers for ranking. In Machine Learning: ECML 2004, pages 501–512. Springer, 2004.
 [27] Leon Wenliang Zhong and James T Kwok. Accurate probability calibration for multiple classifiers. In Proceedings of the TwentyThird international joint conference on Artificial Intelligence, pages 1939–1945. AAAI Press, 2013.