Curiosity Based Exploration for Learning Terrain Models
Abstract
We present a robotic exploration technique in which the goal is to learn to a visual model and be able to distinguish between different terrains and other visual components in an unknown environment. We use ROST, a realtime online spatiotemporal topic modeling framework to model these terrains using the observations made by the robot, and then use an information theoretic path planning technique to define the exploration path. We conduct experiments with aerial view and underwater datasets with millions of observations and varying path lengths, and find that paths which are biased towards locations with high topic perplexity produce more better terrain models with high discriminative power, especially with short paths of length close to the diameter of the world.
I Introduction
This work presents an exploration technique using a realtime online topic modeling framework, which models the cause of observations made by a robot with a latent variable (called topic) that is representative of different kinds of terrains or other visual constructs in the scene, and then uses a local planner to find an exploratory path through the world which would result in learning this topic model quickly.
We define curiosity as the unsupervised act of moving through the world in order to seek novel observations with high information content. We posit that observation data collected from such paths that seek novelty and maximize information gain would result in better terrain models. Computing information gain on low level sensor data, which in the case of vision corresponds to pixel colors or edges, might not work in many scenarios where we are interested in modeling more abstract visual constructs. Hence, we propose the use of a topic modeling framework, which have been shown to produce semantic labeling of text [1] and images [2], including satellite maps [3].
In this work we use a realtime online spatiotemporal topic modeling technique called ROST [4] that is suitable for use in the robotic exploration context. ROST allows for topic modeling of streaming data (observations made by a robot over time), while taking into account the spatial and temporal distribution of the data. Moreover, ROST can process the incoming observation data in real time, while providing a very close approximation to traditional batch implementations [4].
At each time step, we add the observations from the current location to the topic model, and compute the perplexity of the observations from the neighboring locations. This perplexity score, along with a repulsive potential from previously visited locations, is then used to bias the probability of next step in the path. Since observations with high perplexity have high information gain, we claim that this approach would results in faster learning of the terrain topic model, which would imply shorter exploration paths for the same accuracy in predicting terrain labels for unseen regions.
Ii Prior Work
Autonomous exploration is a well studied field in robotics, and there are different variants of the exploration problem.
Iia Exploration for Navigation
Navigating a robot through free space is a fundamental problem in robotics. Yamauchi [5] defined exploration as the “act of moving through an unknown environment while building a map that can be used for subsequent navigation”. Yamauchi’s proposed solution involved moving the robot towards the frontier regions in the map, which were described as the boundary between known free space and the uncharted territories.
If we have an inverse sensor model of the range sensor, it is possible to compute locations in the world which would maximize the utility of the sensor reading in resolving obstacle position and shape. Grabowski [6] proposed such an exploration strategy where the goal is to maximize the understanding of obstacles rather than the exposure to free space. In this approach, the robot identifies the location with next best view in space where a sonar sensor reading would have the greatest utility in improving the quality of representation of an obstacle.
If there is no external localizer available to the robot, then it is desirable that robot explores, maps and localizes in the environment at the same time [7][8][9][10]. Bourgault [2] and Stachniss [11] have proposed an exploration strategy which moves the robot to maximize the map information gain, while minimizing the robot’s pose uncertainty.
IiB Exploration for Monitoring Spatiotemporal Phenomenon
In underwater and aerial environments, obstacle avoidance is typically not the primary concern, but many different kinds of high level exploration tasks still exist.
Binney [12] has described an exploration technique to optimize the monitoring spatiotemporal phenomena by taking advantage of the submodularity of the objective function. Bender [13] has proposed a Gaussian process based exploration technique for benthic environments, which uses an experiment specific utility function. Das et al. [14] have presented techniques to autonomously observe oceanographic features in the open ocean. Hollinger et al. [15] have studied the problem of autonomously studying underwater ship hulls by maximizing the accuracy of sonar data stream. Smith et al. [16] have looked at computing robot trajectories which maximize information gained, while minimizing the deviation from the planned path. Girdhar et al. [17] have proposed a coral reef exploration algorithm by varying the robot speed based on a surprise score.
Iii Terrain Modeling
Iiia Topic Models
Topic modeling methods were originally developed for text analysis. Probabilistic Latent Semantic Analysis (PLSA) proposed by Hoffman [18], models the probability of observing a word in a given document as:
(1) 
where takes a value between , the vocabulary size, and is the hidden variable or topic label for . Topic label takes a value between , where is a much smaller than . The central idea here is the introduction of the latent variable , which models the underlying topic, or the context responsible for generating the word. Latent Dirichlet Allocations [1] improve upon PLSA by placing Dirichlet priors on and , which bias these distributions to be sparse, preventing overfitting.
For modeling visual data observed by the robot, instead of text words, we use two different kinds of visual words: Oriented BRIEF (ORB) [19] based visual words [20] that describing local visual features, and texton words [21] in Lab color space to describe texture properties of a region. Moreover, instead of documents, we compute the prior topic distribution for a given word by taking into account the topic distribution in its spatial neighborhood. We posit that the resulting topic labels modeled by the system then represent high level visual patterns that are representative of different terrain types in the world.
IiiB Generative Process for Observations
We assume the following generative process for observations produced by the spatial region being explored. The world is decomposed into cells, in which each cell is connected to its neighboring cells . In this paper, we only experiment with grid decomposition of the world, where each cell is connected to its four nearest neighbors. However, the general idea presented here are applicable to any topological decomposition of spacetime.
Each cell is modeled by a mixture of at most different kinds of terrain topics, each of which when observed by a robot can result in one of the different types of visual words. Let be the distribution of topics in and around the cell. Intuitively, we would like visual words with the same topic to cluster together in space. This phenomena can be modeled by placing a Dirichlet prior on .
Figure 2 shows random samples from the generative process used to describe these terrain topic distribution in a 2D map. As we vary the Dirichlet concentration parameter and the neighborhood size , we see that smaller results in fewer topics in the neighborhood of any given cell, and smaller results is clusters with less mixing.
1  2  5  

0.001  
0.01  
0.1  
1.0 
Similar to LDA, we describe each topic by a word distribution over different types of visual words, and is assumed to have a Dirichlet prior with parameter . This Dirichlet prior puts a constraint on the complexity of the terrain being described by this topic. Topic model is a matrix that encodes the global topic description information shared by all cells.
The overall generative process for a word in cell is thus described as following:

Word distribution for each topic :

Topic distribution for words in neighborhood of :

Topic label for :

Word label:
where implies that random variable is sampled from distribution .
IiiC Gibbs Sampling
At each time step, we add the observed words to their corresponding cells, and use a Gibbs sampler to update and refine the topic labels until the next time step.
Let , be the set of cells in the current path at time .
The posterior topic distribution for a word in cell is given by:
(2) 
where counts the number of words of type in topic , excluding the current word , and is the number of words with topic label in neighborhood of cell , excluding the current word , and are the Dirichlet hyperparameters. Note that for a neighborhood size of 0, , and the above Gibbs sampler is equivalent to the LDA Gibbs sampler proposed by Griffiths et al. [22].
Several different strategies exist in the literature to do online refinement of the topic label assignment on streaming data [23]. However, in this work, we are interested in the more constrained realtime version of the problem. After each new observation, we only have a constant amount of time to do topic label refinement, hence any online refinement algorithm that has computational complexity which increases with new data is not applicable.
We then must use a refinement strategy which only partially updates the topic labels after each time step. To ensure that the topic labels from the last observation converge before the next observation arrives, at each time step, for each refine iteration, we refine the last observation with probability , or a previous observation with probability . We pick the previous observation using age proportional random sampling. We found to work well in most cases, however on faster machines, could be set to a lower value, which would encourage better globally optimal topic labels. Algorithm 1 summarizes the proposed realtime topic refinement strategy.
Iv Curiosity based Exploration
At time , let the robot be in cell , and let be the set of cells in its neighborhood. We would like to compute a weight value for each , such that
(3) 
In this work we consider a four different weight functions.

Random Walk  Each cell in the neighborhood is equally likely to be the next step:
(4) 
Stochastic Coverage  Use a potential function to repel previously visited locations:
(5) where is the number of times we have visited cell , and is the Euclidean distance between these two cells.

Word Perplexity  Bias the next step towards cells which has high word perplexity:
(6) 
Topic Perplexity  Bias the next step towards cells which has high topic perplexity:
(7)
We compute the word perplexity of the words observed in by taking the inverse geometric mean of the probability of observing the words in the cell, given the current topic model and the topic distribution of the path thus far.
(8) 
where is the number of words observed in , is the probability of observing word if its topic label is , and is the probability of seeing topic label in the path executed by the robot thus far.
To compute topic perplexity of the words observed in , we first compute topic labels for these observed words by sampling them from the distribution in Eq. 2, without adding these words to the topic model. These temporary topic labels are then used to compute the perplexity of in topic space.
(9) 
V Experiments
To validate our hypothesis that biasing exploration towards high perplexity cells will result in a better terrain topic model of the environment, we conducted the following experiment. We considered three different maps, two aerial view, and one underwater coral reef map.
Dataset  width(px)  height(px)  n.cells  n.words 

Montreal1  1024  1024  4096  3,239,631 
Montreal2  1024  1024  4096  1,675,171 
SouthBellairs  2500  2500  6241  1,664,749 
We extracted ORB words describing local features, and texton words describing texture at every pixel (every second pixel for the SouthBellairs underwater dataset). ORB words had a dictionary size of 5000, and texton words had a dictionary size of 1000. The dictionary was computed by extracting features from every 30th frame of a completely unrelated movie^{1}^{1}1We used the documentary movie Baraka(1992) for extracting visual feature, because of its rich visuals from many different contexts.
Each of these maps were decomposed into square cells of width 16 (32 for SouthBellairs). Now for each weight function, we computed exploration paths of varying length, with 20 different random restart locations for each case. Each time step was fixed at 200 milliseconds to allow the topic model to converge. We limited the path length to 320 steps, which is about .
Each of these exploration runs returned a topic model , which we then used to compute topic labels for each pixel in the map in batch mode. Let be these topic labels. An example of this labeling for each of the three dataset is shown in Fig. 3 (j,k,l). We compared this topic labeling with two other labelings: human labeled groundtruth , and labels computed in batch mode , with random access to the entire map.
We then computed the mutual information between and , and , and plotted the results as a function of path length, as shown in Fig. 4.
Vi Results and Discussion
The results are both encouraging and surprising. As shown in Fig. 4, we see that topic perplexity based exploration (shown with blue squares) performs consistently better than all other weight functions, when compared against ground truth, or the batch results.
For paths of length 80, which is close to the width of the maps, we see that mutual information between topic perplexity based exploration and ground truth is 1.51, 1.20 and 1.05 times higher respectively for the three datasets, compared to the next best performing technique.
For long path lengths (320 steps or more), stochastic coverage (shown with orange circles) based exploration matches the mean performance of topic perplexity exploration. This is expected because the maps are bounded, and as the path length increases, the stochastic coverage algorithm is able to stumble across different terrains, even without a guiding function.
For short path lengths (40 steps or less), we do not see any statistical difference between the performance of different techniques.
Marked with purple triangles, we see the results of exploration using Brownian random motion. Although this strategy has a probabilistic guarantee of asymptotically complete coverage, but it does so at a lower rate that stochastic coverage exploration startegy. A random walk in two dimensions is expected to travel a distance of from start, where is the number of steps. Hence it is highly likely that it never visits different terrains. The resulting topic models from these paths are hence unable to resolve between these unseen terrains.
The performance of word perplexity exploration (shown with green diamonds) is surprisingly poor in most cases. We hypothesis that this poor performance is due to the algorithm getting pulled towards locations with terrain described by a more complex word distribution. This will cause the algorithm to stay in these complex terrains, and not explore as much as the other algorithms. In comparison, the topic perplexity exploration is not affected by the complexity of the distribution describing the topic, and is only attracted to topic rarity.
Vii Conclusion
We have presented a novel exploration technique that aims to learn a terrain model for the world by finding paths with high information content. The use of a realtime online topic modeling framework allows us to model incoming streams of low level observation data via the use of a latent variable representing the terrain. Given this online model, we measure the utility of the potential next steps in the path. We validated the effectiveness of the proposed exploration technique over candidate techniques by computing mutual information between the terrain maps generated through the use of the learned terrain model, and hand labeled ground truth, on three different datasets.
Acknowledgment
This work was supported by the Natural Sciences and Engineering Research Council (NSERC) through the NSERC Canadian Field Robotics Network (NCFRN).
References
 [1] D. M. Blei, A. Y. Ng, and M. I. Jordan, “Latent dirichlet allocation,” The Journal of Machine Learning Research, vol. 3, pp. 993–1022, 2003.
 [2] F. Bourgault, A. A. Makarenko, S. B. Williams, B. Grocholsky, and H. F. DurrantWhyte, “Information based adaptive robotic exploration,” in Intelligent Robots and Systems, 2002. IEEE/RSJ International Conference on, 2002, pp. 540—545. [Online]. Available: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=1041446
 [3] M. Lienou, H. Maitre, and M. Datcu, “Semantic Annotation of Satellite Images Using Latent Dirichlet Allocation,” Geoscience and Remote Sensing Letters, IEEE, vol. 7, no. 1, pp. 28–32, 2010. [Online]. Available: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5184874
 [4] Y. Girdhar, P. Giguere, and G. Dudek, “Autonomous Adaptive Exploration using Realtime Online Spatiotemporal Topic Modeling,” International Journal of Robotics Research (Accepted), 2013.
 [5] B. Yamauchi, “A frontierbased approach for autonomous exploration,” in Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA’97. ’Towards New Computational Principles for Robotics and Automation’. IEEE Comput. Soc. Press, 1997, pp. 146–151. [Online]. Available: http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=613851
 [6] R. Grabowski, P. Khosla, and H. Choset, “Autonomous exploration via regions of interest,” in Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003), vol. 2. IEEE, 2003, pp. 1691–1696. [Online]. Available: http://ieeexplore.ieee.org/lpdocs/epic03/wrapper.htm?arnumber=1248887
 [7] R. Sim, G. Dudek, and N. Roy, “Online control policy optimization for minimizing map uncertainty during exploration,” in Robotics and Automation, 2004. Proceedings. ICRA’04. 2004 IEEE International Conference on, vol. 2. IEEE, 2004, pp. 1758–1763.
 [8] R. Sim and J. J. Little, “Autonomous visionbased robotic exploration and mapping using hybrid maps and particle filters,” Image and Vision Computing, vol. 27, no. 1, pp. 167–177, 2009.
 [9] T. Kollar and N. Roy, “Efficient Optimization of InformationTheoretic Exploration in SLAM.” in AAAI, vol. 8, 2008, pp. 1369–1375.
 [10] F. Amigoni and V. Caglioti, “An informationbased exploration strategy for environment mapping with mobile robots,” Robotics and Autonomous Systems, vol. 58, no. 5, pp. 684–699, 2010.
 [11] C. Stachniss, G. Grisetti, and W. Burgard, “Information Gainbased Exploration Using RaoBlackwellized Particle Filters.” Robotics: Science and Systems, vol. 2, 2005. [Online]. Available: http://pdf.aminer.org/000/685/672/information_gain_based_exploration_using_rao_blackwellized_particle_filters.pdf
 [12] J. Binney, A. Krause, and G. S. Sukhatme, “Optimizing waypoints for monitoring spatiotemporal phenomena,” The International Journal of Robotics Research, vol. 32, no. 8, pp. 873–888, Jul. 2013. [Online]. Available: http://ijr.sagepub.com/cgi/doi/10.1177/0278364913488427
 [13] A. Bender, S. B. Williams, and O. Pizarro, “Autonomous Exploration of LargeScale Benthic Environments,” in IEEE International Conference on Robotics and Automation (ICRA), 2013, pp. 390–396.
 [14] J. Das, F. Py, T. Maughan, T. O’Reilly, M. Messie, J. Ryan, G. S. Sukhatme, and K. Rajan, “Coordinated sampling of dynamic oceanographic features with underwater vehicles and drifters,” The International Journal of Robotics Research, vol. 31, no. 5, pp. 626–646, Apr. 2012. [Online]. Available: http://ijr.sagepub.com/cgi/doi/10.1177/0278364912440736
 [15] G. a. Hollinger, B. Englot, F. S. Hover, U. Mitra, and G. S. Sukhatme, “Active planning for underwater inspection and the benefit of adaptivity,” The International Journal of Robotics Research, vol. 32, no. 1, pp. 3–18, Nov. 2012. [Online]. Available: http://ijr.sagepub.com/cgi/doi/10.1177/0278364912467485
 [16] R. N. Smith, M. Schwager, S. L. Smith, B. H. Jones, D. Rus, and G. S. Sukhatme, “Persistent Ocean Monitoring with Underwater Gliders: Adapting Sampling Resolution,” Journal of Field Robotics, vol. 28, no. 5, pp. 714–741, 2011.
 [17] Y. Girdhar, P. Giguère, and G. Dudek, “Autonomous Adaptive Underwater Exploration using Online Topic Modelling,” in International Symposium on Experimental Robotics (ISER), 2012. [Online]. Available: http://www.cim.mcgill.ca/~yogesh/iser2012
 [18] T. Hofmann, “Unsupervised Learning by Probabilistic Latent Semantic Analysis,” Machine Learning, vol. 42, no. 1, pp. 177–196, 2001.
 [19] E. Rublee, V. Rabaud, K. Konolige, and G. Bradski, “ORB : an efficient alternative to SIFT or SURF,” in IEEE International Conference on Computer Vision (ICCV), 2011, pp. 2564—2571.
 [20] J. Sivic and A. Zisserman, “Video Google: Efficient Visual Search of Videos,” in Toward CategoryLevel Object Recognition, ser. Lecture Notes in Computer Science, J. Ponce, M. Hebert, C. Schmid, and A. Zisserman, Eds. Springer Berlin / Heidelberg, 2006, vol. 4170, pp. 127–144. [Online]. Available: http://dx.doi.org/10.1007/11957959_7
 [21] M. Varma and A. Zisserman, “A Statistical Approach to Texture Classification from Single Images,” International Journal of Computer Vision, vol. 62, no. 1/2, pp. 61–81, Apr. 2005. [Online]. Available: http://link.springer.com/10.1023/B:VISI.0000046589.39864.ee
 [22] T. L. Griffiths and M. Steyvers, “Finding scientific topics,” Proceedings of the National Academy of Sciences of the United States of America, vol. 101, no. Suppl 1, pp. 5228–5235, 2004.
 [23] K. R. Canini, L. Shi, and T. L. Griffiths, “Online Inference of Topics with Latent Dirichlet Allocation,” Proceedings of the International Conference on Artificial Intelligence and Statistics, vol. 5, no. 1999, pp. 65–72, 2009.