CONE: Zero-Calibration Accurate Confidence Estimation for Indoor Localization Systems
Accurate estimation of the confidence of an indoor localization system is crucial for a number of applications including crowd-sensing applications, map-matching services, and probabilistic location fusion techniques; all of which lead to an enhanced user experience. Current approaches for quantifying the output accuracy of a localization system in real-time either do not provide a distance metric, require an extensive training process, and/or are tailored to a specific localization system.
In this paper, we present the design, implementation, and evaluation of CONE: a novel calibration-free accurate confidence estimation system that can work in real-time with any location determination system. CONE builds on a sound theoretical model that allows it to trade the required user confidence with tight bound on the estimated confidence radius. We also introduce a new metric for evaluating confidence estimation systems that can capture new aspects of their performance.
Evaluation of CONE on Android phones in a typical testbed using the iBeacons BLE technology with a side-by-side comparison with traditional confidence estimation techniques shows that CONE can achieve a consistent median absolute error difference accuracy of less than 2.7m while estimating the user position more than 80% of the time within the confidence circle. This is significantly better than the state-of-the-art confidence estimation systems that are tailored to the specific localization system in use. Moreover, CONE does not require any calibration and therefore provides a scalable and ubiquitous confidence estimation system for pervasive applications.
978-1-5090-2425-4/16/$31.00 ©2016 IEEE
The recent advances in wireless technologies, automatic indoor floorplan construction [1, 2, 3, 4], and the wide-spread use of sensor-rich smart phones have sparked research in different indoor localization systems [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]. Such systems can achieve meter-level accuracy with minimal calibration overhead. Usually, a localization system is experimentally tested offline to quantify its performance against ground truth data. While this may be useful for validating the typical error of the localization system, it does not help end users understand the real-time error in their estimated location. Therefore, a real-time confidence measure in the predicted locations would enhance the system usability from end users’ perspective. This confidence measure is usually represented as a circle of ambiguity whose center is the estimated user location and radius is the confidence of the estimation (Figure 1). Such confidence estimation is crucial to enhance the performance of a growing number of applications including crowd-sensing applications , map-matching services [25, 26], and probabilistic location fusion techniques . In addition, it helps users make better use of the provided services .
To address the error estimation problem, a number of techniques have been proposed in literature [29, 30, 31, 32, 28, 30]. These systems, however, do not provide the estimated confidence as distance, provide a static/constant measure of error, require an extensive calibration process to provide the estimated confidence (which does not scale to large areas and whose accuracy degrades with the continuously varying dynamics of the indoor environment) and/or are tailored to specific localization systems. In addition, current systems only quantify the accuracy of the estimated confidence based on the difference between the estimated confidence radius and actual location, which does not capture all aspects of a good confidence measure, e.g. over estimation or under estimation of the estimated confidence.
In this paper, we propose a zero-calibration accurate real-time CONfidence Estimation (CONE) technique for indoor localization systems in the dynamically changing indoor environments. Our proposed method builds on a sound theoretical model for error estimation that can achieve tight error estimates in practical deployment scenarios and works with any localization system. We derive our analytical model and show how different error bounds can be achieved based on the required confidence in the user location. We also introduce a new metric for evaluating confidence estimation techniques that can better quantify the system accuracy.
Evaluation of the proposed technique on Android phones in a typical testbed using the iBeacons BLE technology with a side-by-side comparison with traditional confidence estimation techniques shows that CONE can achieve a median absolute error difference accuracy of 2.7m while localizing the user within the confidence circle more than 80% of the time. Compared to a state-of-the-art confidence estimation technique that is tailored to the specific localization system in use, CONE provides significantly tighter bounds on the estimated confidence without requiring any information from the localization system nor any calibration overhead.
In summary, our contribution in this paper is three-fold:
We present a theoretical model for estimating the confidence in the estimated user location that can provide different estimation granularities based on the required user-defined confidence. The proposed model does not require any collection of training data and can be added to any location estimation system with no modifications.
We propose a novel metric for evaluating confidence estimating systems that can provide fine-grained quantification of the estimated confidence.
We implement the proposed confidence estimation system on Android phones and evaluate its performance in a typical indoor environment with a side-by-side comparison with a state-of-the-art system.
2 Location Confidence Estimation
Figure 2 shows the basic system operation. CONE can be integrated with any of the current indoor location determination systems without modifications. It combines the estimated location with the required confidence level to estimate the radius of the ambiguity circle, i.e. the circle within which the device is located with the required confidence.
The basic idea CONE builds on is simple: if the tracked device is not moving, then the higher the variance of the estimated location over time the lower the confidence in its location. In the rest of this section, we provide theoretical analysis to justify this intuition as well as quantify the distance error as a function of the required confidence.
2.1 Confidence Radius Estimation
Without loss of generality, we assume that the tracked device is located in 2D. Let represent the unknown (deterministic) actual user location and be the random variable corresponding to the estimated user location. Let be a random variable that represents the distance error in the estimated location. Therefore, . We assume that follows a normal distribution with mean and variance . That is, . This assumption has been verified in our testbed with reasonable accuracy.
Therefore, given a list of the last estimated locations (), CONE estimates the parameters of based on Algorithm 1. In particular, the first step is to estimate the center of mass () of the set . This center of mass should converge to the actual user location as . Then, the distance error () from each location to the to is calculated. Finally, the mean and variance of are estimated as:
Once the parameters of are estimated, the required confidence radius () is estimated based on required confidence level (100%) using the quantile function:
CONE can be integrated with any of the current localization systems as it depends only on the estimated location, without the need for any calibration data.
Since the parameters of the distance error distribution are calculated based on the estimated location, the higher the variance in the estimated location, the higher the variance in the estimated error distance, and hence the higher the estimated confidence radius. This fits our initial intuition.
Similarly, when the user is mobile, the estimated location will be changing frequently, leading to a higher variance and hence to a higher confidence radius. This is intuitive as the user should expect higher ambiguity during movement. Once the person stops moving, her position estimation stabilizes, leading to a reduced confidence radius.
Apart from the required confidence level parameter (), which is specified by the user, CONE depends on only one parameter: the window size () of the last estimated locations. The dependence on only one system parameter makes CONE robust to different operation environments. We quantify the effect of on performance in the evaluation section.
3 Confidence Evaluation Metrics
To evaluate the performance of confidence estimation techniques, the absolute error difference (AED) has been the only metric usually used in literature [32, 28, 30]. The absolute error difference is estimated as the absolute difference between the actual localization error and the estimated error. Although AED is a good metric, it fails to capture other important performance aspects of confidence estimation techniques. In particular, it is important to quantify how a system overestimates and underestimates the localization system accuracy as depicted in Figure 1. Note that there is a natural tradeoff between the two aspects: reducing the underestimation error will increase the overestimation error and vice versa. A perfect confidence estimation technique should reduce both to zero, i.e. makes the actual error equal to the estimated error.
To capture these two aspects, we propose to use the signed error difference (SED) metric. Specifically, SED is estimated as:
Hence, a positive SED indicates that the confidence estimation system overestimates the localization system accuracy and that the actual location is outside the predicted confidence circle (Figure b). On the other hand, a negative SED indicates that the user’s actual location is inside the predicted confidence circle, underestimating the system accuracy (Figure a).
4 Performance Evaluation
In this section, we evaluate the performance of CONE in a typical indoor environment. We start by describing the experimental testbed followed by evaluating the effect of the window size parameter on performance. We finally compare the performance to a typical confidence estimation technique that is tailored to the used localization system.
4.1 Experimental Testbed
Our system was evaluated in the second floor of our campus building (Figure 3). The floor dimensions are 26m by 17m covering corridors, offices, labs, and classrooms. The ceiling height is about 3m.
Our experiments involved the iBeacons technology. Ten Kontact iBeacons were installed throughout the floor as shown in Figure 3 with an average density of one beacon every 44m. All beacons are installed at the same height of 2.7m.
For evaluating the accuracy of the system, ground truth data was collected using a Samsung Galaxy S5 Android phone. Test points were collected on a uniform grid with a 1m spacing. At each test point, beacons were scanned for 60 seconds in each of the four directions: north, east, south, and west, for a total of four minutes at each point.
The used localization system is the IncVoronoi calibration-free RF localization system [33, 34] that exploits constraints based on the mutual Received Signal Strength(RSS) relation between each pair of installed RF transmitters to estimate the user’s location. To reduce the computational overhead, IncVoronoi discretizes the area of interest into a grid and then estimates location based on the the top candidate grid points.
4.2 Effect of the Window Size Parameter on Performance
Figure 4 shows the effect of changing the window size of the last estimated locations (parameter ) on performance. The figure shows that increasing the window size leads to enhancing the two metrics due to the better estimation of the variance. However, this increases the latency of the estimation process. A value of balances the two factors.
4.3 Comparison with Other Systems
In this section, we compare CONE to a grid point (GP) based confidence estimation system tailored to the used localization system (we call it “GP-Tailored”) for different user-specified desired confidence levels. Specifically, since the IncVoronoi localization system [33, 34] discretizes the area of interest into a grid and then estimates location as a function of the top candidate grid points; in the baseline (GP) technique, the radius of the circle is estimated proportionally to the distance between the estimated user location and the furthest grid point within the top candidate grid points.
Figures 5 and 6 show the absolute and signed distance error CDFs respectively. Table 1 summarizes the results. A number of interesting results are revealed by the figures: First, for the signed distance error metric, there is a natural tradeoff between underestimating and overestimating the actual system error for the two techniques. The best technique is the closest to the line , reflecting the least negative and positive error.
Second, the two metrics reflect different aspects of the performance of the confidence estimation technique: for example, for CONE, setting the ambiguity radius to one gives the best absolute error difference (Figure a) while setting the radius to two gives the highest probability of estimated points inside the confidence circle (80% as in Figure a). Therefore, the designer should take into consideration the different metrics, as opposed to the absolute error metric only – typically used in literature, when making her decision.
Finally, CONE follows the theoretical limit trend with increasing and provides significantly better results for the two metrics. In particular, CONE can estimate the user’s actual error with a median absolute error difference less than m while estimating the user location within the confidence circle more than 80% of the time (). Therefore, it is more rigorous compared to the grid-based method. In addition, unlike the GP-tailored method, it does not require any information related to the internal working of the used localization system.
|Method||Median abs. error (m)||% of estimates inside circle||Median +ve dist. (m)||Med. -ve dist. (m)|
|GP-Tailored ( max. dist.)||2.32||10%||2.5515||-0.8187|
5 Related Work
Recently, researchers have proposed confidence estimation techniques for the different localization systems to improve the usability of their location predictions, e.g. GPS [29, 30], GNSS  and fingerprinting-based localization techniques [32, 28, 30].
Confidence estimation for the GPS is typically derived from the geometric dilution of precision, especially the horizontal dilution of precision (HDOP) [29, 30, 31]. The HDOP, however does not provide a measure of error in meters but rather a scaling factor based on the geometry of the visible satellites. An HDOP value of three or less indicates good satellite geometry and a relatively accurate location estimate . To obtain a confidence measure in meters, in  and  authors analyze the error characteristics of GPS/GNSS localization systems and build an error model to estimate the localization error.
For indoor localization, confidence estimation techniques typically try to leverage their war-driving/training data to estimate an accuracy measure along with the predicted location. For example, in  authors proposed to maintain a database of locations and their corresponding measured error to estimate the fingerprinting-based systems’ accuracy. Such a database is built off-line by running the localization system on a set of measurements with known actual locations and recording the predicted locations and their associated error in the database. Similarly, in , authors proposed to use the leave-one out method, instead of having separate traces to estimate the error and proposed other techniques to estimate the confidence including fingerprints clustering and signal strength variation.
In comparison to these fingerprinting-based and GPS and GNSS localization systems, CONE depends on the estimated location only, without requiring any internal information from the localization systems or extensive calibration, which may change over time affecting the confidence estimation system accuracy. This allows it to work with any location determination system under the dynamically changing indoor environment.
We presented the design, implementation, and evaluation of CONE, an accurate fingerprint-less confidence estimation technique for indoor localization systems. CONE works only on the estimated location to quantify the accuracy of the system, and hence can work with any localization system. We derived the accuracy bound as a function of the user desired confidence level as well as proposed a new signed distance error metric that can capture different performance aspects of confidence estimation systems than the traditional absolute distance error metric.
Evaluation of CONE in a typical testbed using the iBeacons technology shows that it can achieve a median distance error of less than 2.7m while estimating the user location within the confidence circle more than 80% of the time. This is better than a traditional state-of-the-art confidence estimation technique that is tailored to the localization system in use. In addition, CONE does not require any internal information about the performance of the localization system nor any calibration overhead.
Currently, we are expanding CONE in different directions including evaluation on other testbeds, integration with other localization systems, deriving a hybrid evaluation metric, among others.
This work is supported in part by NaviRize Inc. and in part by a grant from the Egyptian Information Technology Industry Development Agency (ITIDA).
- M. Alzantot and M. Youssef, “CrowdInside: Automatic construction of indoor floorplans,” in ACM SIGSPATIAL GIS 2012.
- M. Elhamshary and M. Youssef, “SemSense: Automatic construction of semantic indoor floorplans,” in IEEE IPIN 2015.
- M. Elhamshary et al., “TransitLabel: A crowd-sensing system for automatic labeling of transit stations semantics,” in ACM MobiSys 2016.
- M. Elhamshary and M. Youssef, “CheckInside: A fine-grained indoor location-based social network,” in ACM UbiComp 2014.
- H. Wang, S. Sen, A. Elgohary, M. Farid, M. Youssef, and R. R. Choudhury, “No need to war-drive: unsupervised indoor localization,” in ACM MobiSys 2012.
- H. Abdelnasser, R. Mohamed, A. Elgohary, M. F. Alzantot, H. Wang, S. Sen, R. R. Choudhury, and M. Youssef, “SemanticSLAM: Using environment landmarks for unsupervised indoor localization,” IEEE Transactions on Mobile Computing, vol. 15, no. 7, pp. 1770–1782, 2016.
- M. Youssef and A. Agrawala, “The Horus WLAN location determination system,” in ACM MobiSys 2005.
- M. Alzantot and M. Youssef, “Uptime: Ubiquitous pedestrian tracking using mobile phones,” in IEEE WCNC 2012.
- M. Youssef, “Towards truly ubiquitous indoor localization on a worldwide scale,” in ACM SIGSPATIAL GIS 2015.
- P. Bahl and V. N. Padmanabhan, “RADAR: An in-building RF-based user location and tracking system,” in IEEE INFOCOM 2000.
- H. Aly and M. Youssef, “An analysis of device-free and device-based WiFi-localization systems,” International Journal of Ambient Computing and Intelligence, vol. 6, no. 1, pp. 1–19, 2014.
- I. Sabek, M. Youssef, and A. V. Vasilakos, “ACE: An accurate and efficient multi-entity device-free WLAN localization system,” IEEE transactions on mobile computing, vol. 14, no. 2, pp. 261–273, 2015.
- A. Saeed, A. E. Kosba, and M. Youssef, “Ichnaea: A low-overhead robust WLAN device-free passive localization system,” IEEE Journal of selected topics in signal processing, vol. 8, no. 1, pp. 5–15, 2014.
- M. Seifeldin, A. Saeed, A. E. Kosba, A. El-Keyi, and M. Youssef, “Nuzzer: A large-scale device-free passive localization system for wireless environments,” IEEE Transactions on Mobile Computing, vol. 12, no. 7, pp. 1321–1334, 2013.
- M. Seifeldin and M. Youssef, “A deterministic large-scale device-free passive localization system for wireless environments,” in ACM PETRA 2010.
- A. E. Kosba, A. Abdelkader, and M. Youssef, “Analysis of a device-free passive tracking system in typical wireless environments,” in IEEE NTMS 2009.
- A. E. Kosba, A. Saeed, and M. Youssef, “Robust WLAN device-free passive motion detection,” in IEEE WCNC 2012.
- I. Sabek and M. Youssef, “Multi-entity device-free WLAN localization,” in IEEE GLOBECOM 2012.
- H. Abdel-Nasser, R. Samir, I. Sabek, and M. Youssef, “MonoPHY: Mono-stream-based device-free WLAN localization via physical layer information,” in IEEE WCNC 2013.
- M. Youssef, M. Abdallah, and A. Agrawala, “Multivariate analysis for probabilistic WLAN location determination systems,” in IEEE MobiQuitous 2005.
- K. El-Kafrawy, M. Youssef, A. El-Keyi, and A. Naguib, “Propagation modeling for accurate indoor WLAN RSS-based localization,” in IEEE VTC 2010-Fall.
- M. Youssef and A. Agrawala, “Location-clustering techniques for WLAN location determination systems,” International Journal of Computers and Applications, vol. 28, no. 3, pp. 278–284, 2006.
- A. Eleryan, M. Elsabagh, and M. Youssef, “Synthetic generation of radio maps for device-free passive localization,” in IEEE GLOBECOM 2011.
- H. Aly, A. Basalamah, and M. Youssef, “Map++: A crowd-sensing system for automatic map semantics identification,” in IEEE SECON 2014.
- R. Mohamed, H. Aly, and M. Youssef, “Accurate and efficient map matching for challenging environments,” in ACM SIGSPATIAL GIS 2014.
- H. Aly and M. Youssef, “SemMatch: road semantics-based accurate map matching for challenging positioning data,” in ACM SIGSPATIAL GIS 2015.
- D. Fox, J. Hightower, L. Liao, D. Schulz, and G. Borriello, “Bayesian filtering for location estimation,” IEEE Pervasive Computing, vol. 2, no. 3, pp. 24–33, July 2003.
- H. Lemelson, M. B. Kjærgaard, R. Hansen, and T. King, “Error estimation for indoor 802.11 location fingerprinting,” in International Symposium on Location-and Context-Awareness. Springer, 2009, pp. 138–155.
- N. M. Drawil, H. M. Amar, and O. A. Basir, “GPS localization accuracy classification: A context-based approach,” IEEE Transactions on Intelligent Transportation Systems, vol. 14, no. 1, pp. 262–273, 2013.
- V. Moghtadaiee, A. Dempster, and B. Li, “Accuracy indicator for fingerprinting localization systems,” in IEEE/ION Position Location and Navigation Symposium (PLANS), April 2012, pp. 1204–1208.
- X. Niu, Q. Chen, Q. Zhang, H. Zhang, J. Niu, K. Chen, C. Shi, and J. Liu, “Using Allan variance to analyze the error characteristics of GNSS positioning,” Springer GPS solutions, vol. 18, no. 2, pp. 231–242, 2014.
- D. Dearman, A. Varshavsky, E. De Lara, and K. N. Truong, “An exploration of location error estimation,” in International Conference on Ubiquitous Computing. Springer, 2007, pp. 181–198.
- R. Elbakly and M. Youssef, “A robust zero-calibration RF-based localization system for realistic environments,” in IEEE SECON 2016.
- ——, “A Calibration-free RF localization system,” in ACM SIGSPATIAL GIS 2015.