GONet: A Semi-Supervised Deep Learning Approach
For Traversability Estimation
We present semi-supervised deep learning approaches for traversability estimation from fisheye images. Our method, GONet, and the proposed extensions leverage Generative Adversarial Networks (GANs) to effectively predict whether the area seen in the input image(s) is safe for a robot to traverse. These methods are trained with many positive images of traversable places, but just a small set of negative images depicting blocked and unsafe areas. This makes the proposed methods practical. Positive examples can be collected easily by simply operating a robot through traversable spaces, while obtaining negative examples is time consuming, costly, and potentially dangerous. Through extensive experiments and several demonstrations, we show that the proposed traversability estimation approaches are robust and can generalize to unseen scenarios. Further, we demonstrate that our methods are memory efficient and fast, allowing for real-time operation on a mobile robot with single or stereo fisheye cameras. As part of our contributions, we open-source two new datasets for traversability estimation. These datasets are composed of approximately 24h of videos from more than 25 indoor environments. Our methods outperform baseline approaches for traversability estimation on these new datasets.
Effective identification of non-traversable spaces is essential for long-term mobile robot operation. Autonomous service robots [1, 2, 3, 4], electric wheelchairs [5, 6], rolling walkers , and other smart mobile machines need to move safely in dynamic environments; otherwise they could damage themselves or even injure people .
Common approaches to recognize non-traversable paths during robot navigation use bumpers or depth measurements . Unfortunately, bumpers don’t prevent robots from falling off edges and can fail to detect small obstacles. Depth measurements can be obtained from 2D/3D lidars [10, 11, 12], depth cameras [13, 14], stereo camera pairs , or single cameras with 3D reconstruction methods [16, 17]. But lidar sensors are expensive and depth measurements can be affected by surface textures and materials. For example, lidars and depth cameras often have trouble sensing highly reflective surfaces and transparent objects, such as mirrors and glass doors. These problems have motivated alternative approaches for traversability estimation using RGB cameras. For instance, prior work has used images to classify terrain into traversable and non-traversable areas [18, 19].
We contribute semi-supervised computer vision methods for traversability estimation. These methods learn to distinguish traversable and non-traversable areas by using many positive examples of places that are safe to navigate through, but just a few negative images, e.g., displaying stairs in front of a robot or objects that a robot will imminently collide with. Being able to learn from such uneven data makes our approach practical because collecting positive examples is easy – we can drive a robot through safe areas [19, 12]. But collecting negative examples, as in  or , can be time-consuming, costly, and dangerous. We cannot afford potentially damaging collisions or causing injuries to nearby people. Our experiments show that the proposed methods outperform supervised baselines, and that small amounts of negative examples can improve traversability estimation in comparison to using only positive data, as in .
The first approach that we describe in this paper is GONet, a method that leverages powerful generative deep adversarial models  to estimate traversability. The model takes as input a view of the environment from a fisheye camera on a robot, and predicts whether the area seen through the camera is safe to traverse (Fig. 1). GONet is:
cheap, as it does not require expensive sensors, but just an off-the-shelf camera;
fast, because the structure of GONet is feed-forward, thus allowing for real-time operation on mobile platforms;
robust, because fisheye cameras efficiently capture every angle of the surrounding environment and our methods are capable of learning from a wide variety of data; and
scalable. Once trained on a given view, the model can run on other views from fisheye cameras positioned in a similar orientation with respect to the ground on the robot. For example, GONet can be used on a back-facing view after training on data from a forward-facing camera.
As part of our contributions, we propose two extensions to GONet. First, we show how it can be modified to enforce temporal consistency on traversability predictions. To facilitate training on image sequences, we propose an automatic annotation procedure that increases the amount of labeled data without human supervision. Then, we extend our approach to work with stereo fish-eye cameras. While these extensions require a slight increase in computational power over our vanilla GONet model, they improve both the smoothness of our traversability predictions and their accuracy in challenging scenarios, such as when strong shadows significantly alter the appearance of the environment.
We conduct extensive experiments to validate the proposed semi-supervised methods, and contribute new datasets for data-driven traversability estimation in indoor environments. We also illustrate the applicability of our approach using real-time robot demonstrations. We show how GONet and its extensions can serve as an automatic visual emergency stop switch, enabling a robot to avoid collisions or falling down a flight of stairs. We also describe how our approach can complement 2D lidar range measurements during mobile robot navigation.
Ii Preliminaries on Deep Learning
This section provides a brief introduction to Generative Adversarial Networks (GANs) , which are at the core of the proposed approaches for traversability estimation. Section III then delves into the details of our methods.
GANs are a framework for estimating generative models through an adversarial process. They simultaneously train two networks: a generator () that captures the distribution of the training data and can produce examples from its manifold from a latent variable ; and a discriminator () that tries to distinguish between samples generated by and actual samples from the training set. GANs are considered unsupervised methods as they don’t rely on labels, but use all the training data to learn both the generator and the discriminator. Training of is performed by means of a minimax two-player game, trying to maximize the probability of making a mistake.
In this work, we use a particular class of GANs known as Deep Convolutional Generative Adversarial Networks (DCGANs) . When the latent variable is sampled from some simple prior distribution, like a normal distribution, the generator of a DCGAN outputs an image that looks like the training data. Readers not familiar with GANs are encouraged to refer to  for an introductory tutorial.
Iii Semi-Supervised Traversability Estimation
This section introduces GONet, a new semi-supervised approach for traversability estimation from fisheye images. Then, we describe two extensions of the vanilla method: GONet+T, which enforces temporal consistency in the predictions; and GONet+TS, that uses stereo images while still considering temporal consistency.
Iii-a GONet: Single-View Traversability Classification
GONet takes as input an image from a fisheye camera on a robot and outputs whether the environment seen through the image is traversable or not. The model is designed to learn from an uneven dataset with large numbers of positive examples, but just a few negative images of non-traversable areas. This capability is important because collecting negative data can be time-consuming, costly, and dangerous.
- Model Architecture.
As shown in Fig. 2, GONet is composed of two modules: a Feature Extraction Module that is trained with automatically-labeled positive examples of traversable areas; and a Classification Module that is trained with both positive and negative examples. Intuitively, GONet works by first generating an image that is similar to the input and that looks as if it came from the manifold of positive examples, i.e., as if it belonged to the set of images of traversable areas. The image is generated with a DCGAN trained on positive examples only, as detailed in the next section. Then, GONet compares the input with the generated image to decide whether the area seen through the input image is traversable. The main assumption of the model is that when the input indeed shows a traversable area, the generated image would look very similar to it. But when the input depicts a non-traversable scenario, then the generated image would look different. More specifically, GONet estimates traversability from:
, the residual difference between the images,
, the difference between the discriminator features, and
, the discriminator features of the input image.
The features correspond to the last convolutional layer of the discriminator function of the DCGAN, which is trained to distinguish between the true input and the generated .
The features , , and are processed by the Classification Module of GONet to output the traversability probability of the input image. First, these features are processed independently by fully connected layers that each output a scalar. These outputs are then concatenated into a vector, and passed to another fully connected layer with a sigmoid activation function. We optimize the final output using mean squared error and back-propagation.
- Generating Images From the Positive Manifold.
GONet generates images from the positive manifold with a generator function from a DCGAN, as described in Sec. II. To ensure that the generated image resembles the input , we use another network to search for an appropriate variable that induces to be close to . Note that this approach is faster than iteratively searching for the appropriate with back-propagation, as previously proposed for anomaly detection by Schlegl et al. .
We create an auxiliary autoencoder  network as shown in the top part of Fig. 2 to find an appropriate variable for a given input image . This autoencoder is composed of two modules: an inverse generator that outputs , and the generator from GONet’s DCGAN. The inverse generator is structured with the same layers as , but in inverse order. The first four convolutional layers have dimensions , , and . The final layer is a fully connected layer with output size of (corresponding to the dimensionality of ). The auxiliary network for is trained by minimizing the loss , where is the residual loss for a given , and is the discriminator loss. The features are the output of last convolutional layer of the DCGAN’s discriminator. Note that, for training the we freeze the ’s weights. The parameter in trades-off between and , and is chosen empirically with a the validation set.
- Data Collection & Annotation.
To train GONet, we use data from a fisheye camera on a mobile robot. We drive this robot through safe traversable areas and automatically collect significant amounts of positive examples by inspecting the robot’s velocity. If the robot moves continuously for s at a minimum velocity of 0.3 m/s, then we assume that the image collected in the middle of this time interval (at s) depicts a traversable scenario. To gather a small set of negative examples, we carefully position the robot near obstacles and dangerous places. Section IV-A details the datasets that we used for our evaluation.
GONet is trained with back-propagation in 3 steps. First, we train a DCGAN with automatically annotated positive data. Through this GAN, we estimate the generator () and discriminator () of GONet. Second, we use the auxiliary autoencoder network shown at the top part of Fig. 2 to train the using positive examples. Third, we train the final fully connected layer of GONet with a small set of positive and negative examples. We use early stopping in this last step to prevent over-fitting.
Iii-B GONet+T: Enforcing Temporal Consistency
GONet assumes that every input image is independent and identically distributed; but this is not true for image sequences captured from a robot . Current observations are dependent on the past, and this past can often help understand whether the present state is traversable or not.
We propose GONet+T to account for the temporal nature of data gathered from a robot. GONet+T extends GONet by using a Long Short-Term Memory (LSTM) unit  to reduce the variance on traversability predictions over time. LSTMs are popular Recurrent Neural Network models capable of learning long-term dependencies. These recurrent models have shown great promise for sequential tasks .
- Model Architecture.
GONet+T first uses the Feature Extraction Module of GONet to compute , , and , as depicted in Fig. 3 for . The features are then transformed in GONet+T by three fully connected layers to reduce their dimensionality to -dimensional vectors each. These vectors are concatenated and, consequently, processed by a recurrent LSTM unit. The LSTM learns temporal dependencies in the data, and its output is finally passed to a fully connected layer with a sigmoid activation function to predict the traversability probability.
We optimize GONet+T with a multi-objective loss function: , where corresponds to the ground truth traversability probability at time , and are the predicted probabilities at times and , respectively, and is the length of the training image sequences – here we assume equal lengths for simplicity. The parameter trades-off between the prediction error and the smoothness term . We choose using the validation set.
- Data Collection & Annotation.
The use of an LSTM unit in GONet+T requires fully annotated sequences at training time. But annotating vast amounts of negative examples in these sequences is costly and very time consuming. Thus, we devised the following semi-automatic annotation procedure for fisheye image sequences collected from a robot:
We collect data by driving the robot through safe places. The robot is controlled to carefully approach and stop when its near obstacles and dangerous spaces.
We annotate vast amounts of positive data based on the robot’s velocity (as mentioned before), and manually label a small amount of randomly-chosen negative samples.
We train GONet (as in Sec. III-A) with the labeled data that we have so far. We use all positive data for GONet’s Feature Extraction Module, and a small set of positive and negative examples for the Classification Module.
We run GONet on the remaining images that were collected while the robot was static or moved at low speeds, and label these images with GONet’s output probability.
To prove that this annotation procedure would not negatively affect training (as GONet might naturally make mistakes), we conducted an empirical evaluation on the predictions of GONet before and after the re-evaluation. The mean difference between the two sets of predictions for the entire unlabeled dataset is calculated as . Where is the output probability of GONet, is the output probability of GONet after re-training on the re-annotated data, and is the set of all unlabeled data. After the re-annotation, the accuracy of our network on the positive training set increased because they are correctly labeled through the robot’s moving experience. We show that 0.04, showing that the predictions of GONet before and after training on the re-annotation dataset does not change.
We train GONet+T with the same procedure that we used to train GONet. Except that in this case we optimize the final output of the model using labeled image sequences.
Iii-C GONet+TS: Estimating Traversability With Stereo Views
While GONet and GONet+T worked well in different situations, their performance was sometimes affected by the appearance of the environment, e.g., strong shadows occasionally induced false negative predictions. As a result, we decided to extend GONet+T to reason about stereo fisheye images. Our hypothesis was that having two views of the environment would increase the robustness of our traversability estimation approach in challenging scenarios.
GONet+TS has the same architecture as GONet+T, except for the fact that it receives as input a 6 channel tensor, as illustrated in Fig. 3 for . The first three channels of the input tensor correspond to the RGB channels of the left image in the stereo pair; the last three channels correspond to the right image. We collect data and annotate it as in the previous section. We also train GONet+TS with the same loss and in the same fashion as GONet+T.
This section describes the experiments to evaluate the proposed methods in indoor environments. Results were computed using an NVidia GeForce GTX TITAN X GPU.
We collected two datasets to evaluate the proposed traversability estimation methods.111We will open-source our datasets upon acceptance. The first dataset, “Go Stanford 1” (GS1), was collected as described in Sec. III-A. We teleoperated a TurtleBot2 robot222https://www.turtlebot.com/turtlebot2/ with an onboard Ricoh THETA S fisheye camera333Although the THETA S has two fisheye cameras (one in the front and one in the back), we only use the front camera to capture the environment. The horizontal and vertical fields of view of this camera are both . in 15 different buildings at Stanford University (totaling 7.2 hours of sensor data). For each data collection session, we recorded the view from the robot’s fisheye camera in HD resolution and at 3Hz. We also collected the teleoperation commands, and robot’s velocity. The velocity was used to automatically annotate 35783 positive examples, out of a total of 78711 useful images. In our experiments, we split the data to , , and buildings for training, validation, and testing, respectively.
The second dataset, “Go Stanford 2” (GS2), was recorded with a TurtleBot2 with two fisheye THETA S cameras. The cameras were placed in front of the robot, with a baseline distance of m. We collected the same data as for GS1 in GS2, but this time we operated the robot in 27 campus buildings. GS2 is composed of a total of 16.74 hours of video ( useful images). We split the data to buildings for training, for validation, and for testing.
We annotated negative examples in GS1 and GS2 ( images for training, for validation, and for testing in each dataset). We then complemented this data with the same amount of automatically annotated positive images. We also augment GS1 by flipping images horizontally, and GS2 by swapping and flipping the stereo images. For all the experiments we resize the images to 3128128.
Iv-B Data efficiency
We first investigate the performance of GONet as the number of annotated positive and negative examples grows. We first train the Feature Extraction Module of GONet with the positive data from GS1. Then, we train GONet’s Classification Module with , , , , and training examples from GS1 and GS2. We evaluate the models on the unseen labeled images from GS1’s testing set.
As can be seen in Table I, GONet is data efficient. It quickly reaches accuracy using only 800 training examples ( positive and negative images). For this reason, we use images for training supervised models in the experiments in the following sections.
Iv-C GONet Evaluation on GS1
Using GS1, we performed an ablation study for GONet and evaluated it against fully supervised and unsupervised baselines for traversability estimation. The unsupervised models or components of the methods considered in this evaluation were all trained with the same positive data from GS1. The supervised models were trained on the same positive and negative training examples, and also used the same validation set. Classification thresholds for unsupervised methods were chosen using the validation set; the thresholds for supervised methods were optimized with the training data. We considered the following models:
Following , we use an autoencoder as an unsupervised method to classify traversability. The autoencoder has the same structure as the auxiliary network used to train (see Sec. III-A). However, we train this autoencoder from scratch to minimize the difference between the input and the generated images. Traversability is decided based on .
As a baseline, we try predicting traversability with state-of-the-art ResNet 50 and ResNet 152 features for image recognition . The ResNet models are pre-trained on ImageNet, and their features for a given input image are evaluated by a fully connected layer to predict traversability. To avoid over-fitting, we only train the final layer.
- Feature Extraction Module:
We compare the proposed approach to estimate in GONet’s Feature Extraction Module with the iterative back-propagation method of . We threshold image reconstruction errors for these models to classify the input images, as we did for the autoencoder.
We break-down the contributions of GONet’s components. Based on the features output by its Feature Extraction Module, we consider classifying traversability using the feature (R Model), the feature (D Model), the feature (F Model), and their combinations. Classification thresholds are chosen with a final fully connected layer appended to individual features. For their combinations, we append two fully connected layers as in Fig. 2.
Table II shows our results on GS1. The top section of the table shows the performance, speed, and memory requirements of the models trained with positive examples only (P); the bottom section shows results with both positive and negative examples (P+N). In general, using a small amount of negative examples helped improve accuracy.
By comparing rows 1-3 of Table II (excluding the header), we can observe that GONet’s Feature Extraction model outperforms the baseline autoencoder . While these two methods work in a similar spirit, our DCGAN with the inverse generator results in highest accuracy across the unsupervised methods that were trained with positive data only. In terms of accuracy and speed, the results also show the effectiveness of using the inverse generator  in GONet in contrast to using iterative back-propagation .
The results in the bottom section of Table II suggest that each feature output by GONet’s Feature Extraction Module and the components of its Classification Module contribute to the accuracy of the proposed approach. In addition, the full GONet architecture (R+D+F GONet Model in the last row) outperforms all other models evaluated on GS1. In contrast to ResNet, not only does GONet lead to higher accuracy, but it also reduces memory requirements and computation time significantly. The reason is that while ResNet shines at image recognition tasks due to its deep architecture, the network’s size also requires more computational resources. Since mobile robots must often depend on less powerful onboard computing hardware, lower memory usage and faster computation time are desirable qualities for our application.
|Trained on||Model||Accuracy ||Hz||Memory [MB]|
|Feat. Ext. Module||P||back-prop ||60.00||0.125||323|
|baseline||P+N||ResNet 50 ||91.63||34.46||705|
|P+N||ResNet 152 ||92.25||12.21||1357|
Iv-D Evaluation of GONet and its extentions on GS2
|Model||Sensor||Training Data||Neg. accu.||Pos. accu.||Accu. ||Prec. ||Hz||Mem. [MB]|
|(a): 4ch GONet||Kinect||Hand Labeled||89.60||87.20||88.40||89.34||75.75||416|
|(b): disparity map||two THETA S||Hand Labeled||66.20||75.20||70.70||68.99||-||-|
|(c): siamese net||two THETA S||Hand Labeled||88.40||94.20||91.30||88.98||8.63||3744|
|(d): GONet||one THETA S||Hand Labeled||90.60||94.65||92.55||90.63||110.80||339|
|(e): (d) + re-annotation||one THETA S||Re-annotation||90.6||96.20||93.40||91.10|
|(f): GONet+T||one THETA S||Re-annotation||91.7||97.20||94.45||92.05||107.41||347|
|(g): GONet+S||two THETA S||Hand Labeled||93.80||96.00||94.90||93.93||109.24||413|
|(h): (g) + re-annotation||two THETA S||Re-annotation||95.80||96.20||96.00||95.82|
|(i): GONet+TS||two THETA S||Re-annotation||96.20||97.60||96.90||96.26||86.761||429|
In this section, we evaluate GONet and its two extensions, GONet+T, and GONet+TS, against baselines in the GS2 dataset. We take advantage of the two cameras used to collect images in GS2 to perform comparisons across systems based on accuracy, speed, and memory requirements. All the models considered in this experiment are evaluated on GS2’s testing set. We choose classification thresholds 0.5 and model parameters based on the validation set. This evaluation considered the following models:
- 4ch GONet with Kinect:
We train a GONet with a 4 channel input tensor: the first 3 channels correspond to the Kinect’s RGB image and the last channel corresponds to the Kinect’s depth map.
- Disparity Map:
We compute disparity maps  using the left and right image pairs in GS2. The depth maps is reshaped into a vector, and passed to two fully connected layers to predict traversability.
- Simese Net:
In the spirit of , we explored using a siamese network with ResNet 152 features. We concatenate the ResNet features from the image pairs and pass them two three fully connected layers to classify traversability. In this baseline, we only train the fully connected layers.
- GONet Models:
We evaluate all the proposed methods (GONet, GONet+T, and GONet+TS) for traversability estimation after training on hand labeled annotations only ( positive and negative examples) and after re-annotated data as explained in Sec. III-B.
Table III shows the results on GS2. The annotation process proposed in Sec. III-B increased the accuracy. By comparing the results in rows (f) and (i) of Table III with the other rows, we can see that the use of LSTM units in GONet also increased performance. We attribute this success to the fact that GONet+T and GONet+TS are able to reason about the inter-dependency of the input images.
Our GONet model and its extensions outperformed the supervised baselines that used depth maps (rows (a) and (b)) and pre-trained features (row (c)). In particular, GONet+TS had the best performance of all the models on GS2. This provided support to our hypothesis that reasoning about stereo fisheye images can be help estimate traversability.
Iv-E Qualitative Analysis
- Generated Images.
Fig. 4 shows several example input images and corresponding generated images by GONet’s Feature Extraction Module. Because this module was trained only on positive examples, the obstacles in the generated images are often blurred out. Likewise, closed pathways in the input images tend to be converted to open spaces. These changes are what enables the proposed approaches to identify unsafe and non-traversable areas.
- Saliency Maps.
- Traversability Predictions.
Fig. 6 visualizes GONet, GONet+T and GONet+TS predictions in two challenging scenarios. The gray areas in the plots correspond to time periods in which the environment right in front of the robot was unsafe to traverse. In particular, Fig. 6[i] depicts a situation in which a robot enters a room with a glass door. At the beginning, the door is closed (images [a,b,c]) and, then, a person opens the door and holds it for the robot to pass. In general, all the methods tended to perform well in this situation. GONet+T and GONet+TS led to smoother predictions in comparison to GONet thanks to their LSTM.
Fig. 6[ii] shows a situation in which the robot moves through areas with strong shadows. At time [b], GONet mistakenly predicts that the area in front of the robot is unsafe to traverse. At time [d], both GONet and GONet+T output false negatives; only GONet+TS can correctly predict traversability. In our experiments, enforcing temporal consistency in the traversability predictions and using stereo vision often helped deal with such challenging scenarios.
|passing through the door||sun light and shade|
We demonstrate how the proposed approaches for traversability estimation can be used as an autonomous visual emergency stop mechanisms for mobile robots. We also argue that these approaches can increase the robustness of the standard (2D) lidar-based exploration and mapping methods. Finally, we demonstrate the versatility of GONet by showing how it can be used as a navigation aid for visually impaired people. Due to space constraints, we omit results with GONet+TS in this paper (even though we implemented systems equivalent to those discussed in the following sections with stereo cameras). We also mainly discuss GONet+T as it was more accurate than GONet in our quantitative evaluation (Sec. IV). Additional supplementary material for these demonstrations can be found in the accompanying video submission and online.444http://cvgl.stanford.edu/gonet
For all the demonstrations described in this Section, we ran GONet and GONet+T on a laptop with an Intel Core i7-6700HQ processor, 32GB of RAM, and an NVidia GeForce 980M GPU. The laptop was connected to a Ricoh THETA S fisheye camera positioned at the front of a TurtleBot2 robot (Sec. V-A & V-B) or manually held by a user (Sec. V-C). GONet and GONet+T could run on the laptop at the maximum frame rate of the camera (15Hz), but we limited how fast they processed images to match the frame rate used at training time (3Hz).
V-a Visual Emergency Stop (E-Stop) Switch
GONet+T ran on a teleoperated TurtleBot2 to signal unsafe areas in front of the robot. When these areas were detected, GONet+T overrode teleoperation commands to force the robot to stop. Our traversability estimation approach was able to prevent the robot from falling down stairs, and it stopped the robot from colliding with glass walls, fences, doors, and people in previously unseen environments. Moreover, we tested the ability of GONet+T to recognize new obstacles including a piece of aluminum, a jacket, and a binder that were suddenly thrown in front of the robot.
While GONet+T was able to prevent the robot from falling and colliding in many different situations, we observed an interesting failure case with a tangled wired on the ground. When the robot approached the wire, GONet+T was able to stop the TurtleBot, but it did so late. Part of the wire was already underneath the robot. A few other failure cases happened with very small objects, like a computer mouse lying on the ground that had a color similar to the carpet. GONet+T missed these few objects, and the TurtleBot pushed them.
V-B Mobile Robot Exploration & Mapping
We added a 2D lidar to the TurtleBot2 that we used for our applications to enable the robot to navigate and explore the environment with standard probabilistic mapping methods . We also added two more fisheye cameras on the robot to demonstrate the scalability of GONet at handling different views of the environment (see Fig. 7). Based on the traversability predictions from GONet, we automatically updated the robot’s costmap as it navigated inside a building in the test set of our GS1 dataset. GONet identified blocked and unsafe areas that the lidar missed, such as parts of a fence and stairs (Fig. 7). Because recognizing these critical areas is essential for autonomous robot operation, we believe that the proposed traversability estimation methods can increase the robustness of current navigation approaches. Investigating effective mechanisms to combine GONet with other sensing modalities is an interesting future research direction.
V-C Assistive Traversability for the Visually Impaired
We implemented a navigational aid system that produces an audible warning when an obstacle or drop-off is detected by GONet+T, and placed the laptop with our system in a backpack. We then asked a person to carry this backpack and to hold the fisheye camera that was connected to the laptop so that it could gather images of the environment in front of the person. As this person walked blindfolded through unseen indoor and outdoor environments, GONet+T recognized unsafe areas, and our system correctly issued audible warnings. Note that we never trained GONet+T in outdoor environments, nor with images gathered by a pedestrian. Nonetheless, our traversability estimation approach showed great generalization potential under new conditions.
As cameras and computational devices continue to shrink in size, such a warning system could be mounted in a small enclosure either on a belt or even integrated into a pair of glasses. We are excited about the porential of GONet to advance application areas beyond traditional robotics.
Vi Related Work
The traversability estimation approaches presented in this work were motivated by the success of deep learning on many visual tasks . By leveraging deep generative adversarial models [23, 24], our methods were able to process raw images to distinguish traversable and non-traversable areas. One advantage of these methods is that they do not rely on hand-tuned image features, as some prior efforts [19, 37, 38]. Rather, the proposed methods learn from data the relevant information for the traversability estimation task.
Several prior efforts suggested using only positive data for traversability estimation [19, 12, 22]. Inspired by this line of work, we collected vast amounts of positive examples by driving a robot through safe places, and used this data to train our models. However, in contrast to these other efforts, we also collected a small number of negative images. These negative examples increased the accuracy of GONet in comparison to using positive examples only, as in .
At a high level, the proposed approaches can be considered anomaly detection methods . GONet and its extensions expect to get as input an image that depicts a traversable area; negative examples are outliers – they correspond to non-conforming patterns. In contrast to prior work in deep learning on anomaly detection , our methods use a visual manipulation technique  to generate images from the positive manifold. The generated images not only look like traversable areas, but also resemble the input query. This makes our approach fast and practical for mobile robots. To the best of our knowledge, this is the first time that this image manipulation technique is used for traversability estimation.
It is worth noting that prior work involving learned motion control policies for robots have started to reason about the traversability of environments along with a robot’s next actions [40, 41, 22, 20]. While combining traversability estimation and policy learning is out of the scope of this paper, our experimental results suggest that GONet and its extensions can potentially facilitate learning motion behaviors for robots. For example, GONet could be combined with a convolutional network to predict steering angles, as in .
Vii Conclusion & Future Work
We presented semi-supervised methods for traversability estimation. At the core of these methods are powerful deep generative models that distinguish between traversable and non-traversable areas. The models are trained with vast amounts of positive images depicting safe areas that a robot can navigate through, but just a few negative examples. The ability to learn from such uneven data makes our methods more robust than supervised and fully unsupervised baselines while keeping them practical. GONet and its extensions are cheap and fast. They take as input, images from fisheye cameras and process them in a feed-forward fashion, thus allowing for real-time operation on mobile hardware. We demonstrated how the proposed methods can save a robot from dangerous situations, e.g., falling down stairs and colliding with glass. Moreover, this same capability can be used to complement 2D lidar range measurements during robot navigation and exploration. Finally, we demonstrated how the trained GONet can be removed from the robot and carried by a visually impaired individual to issue warnings when the user is heading toward obstacles or drop-offs.
Our experiments focused on traversability estimation in indoor environments. Future work should explore the accuracy of the proposed methods on outdoor settings. Another interesting avenue of future research is end-to-end joint traversability and policy learning. GONet and its extensions can potentially facilitate learning robust motion behaviors.
-  S. Thrun et al., “Minerva: A second-generation museum tour-guide robot,” in ICRA, 1999.
-  B. Mutlu and J. Forlizzi, “Robots in Organizations: The Role of Workflow, Social, and Environmental Factors in Human-robot Interaction,” in HRI, 2008.
-  N. Hirose, R. Tajima, and K. Sukigara, “Personal robot assisting transportation to support active human life – Posture stabilization based on feedback compensation of lateral acceleration –,” in IROS, 2013.
-  M. M. Veloso et al., “Cobots: Robust symbiotic autonomous mobile service robots.” in IJCAI, 2015, p. 4423.
-  S. Oh, N. Hata, and Y. Hori, “Integrated Motion Control of a Wheelchair in the Longitudinal, Lateral, and Pitch Directions,” IEEE Trans. Ind. Elec., vol. 55, no. 4, pp. 1855–1862, 2008.
-  N. Hirose et al., “Mode switching control for a personal mobility robot based on initial value compensation,” in IECON, 2010.
-  V. Kulyukin et al., “iwalker: Toward a rollator-mounted wayfinding system for the elderly,” in 2008 IEEE International Conference on RFID, 2008.
-  G. Wells, “Security Robot Suspended After Colliding With a Toddler ,” 2016.
-  P. Papadakis, “Terrain traversability analysis methods for unmanned ground vehicles: A survey,” Engineering Applications of Artificial Intelligence, vol. 26, no. 4, pp. 1373–1385, 2013.
-  M. Cinietal, “Real-time obstacle avoidance by an autonomous mobile robot using an active vision sensor and a vertically emitted laser slit,” in Intelligent Autonomous Systems, vol. 7, no. 1, 2002, pp. 301–309.
-  M. Pfeiffer et al., “From perception to decision: A data-driven approach to end-to-end motion planning for autonomous ground robots,” in ICRA, 2017.
-  B. Suger, B. Steder, and W. Burgard, “Traversability analysis for mobile robots in outdoor environments: A semi-supervised learning approach based on 3D-lidar data,” in ICRA, 2015.
-  F. Flacco et al., “A depth space approach to human-robot collision avoidance,” in ICRA, 2012.
-  I. Bogoslavskyi, O. Vysotska, J. Serafin, G. Grisetti, and C. Stachniss, “Efficient traversability analysis for mobile robots using the Kinect sensor,” in ECMR, 2013.
-  H. Koyasu, J. Miura, and Y. Shirai, “Real-time omnidirectional stereo for obstacle detection and tracking in dynamic environments,” in IROS, 2001.
-  S. Daftry, S. Zeng, A. Khan, D. Dey, N. Melik-Barkhudarov, J. A. Bagnell, and M. Hebert, “Robust monocular flight in cluttered outdoor environments,” arXiv:1604.04779, 2016.
-  H. Alvarez et al., “Collision avoidance for quadrotors with a monocular camera,” in Experimental Robotics. Springer, 2016, pp. 195–209.
-  D. Kim, S. M. Oh, and J. M. Rehg, “Traversability classification for UGV navigation: a comparison of patch and superpixel representations,” in IROS, 2007.
-  I. Ulrich and I. R. Nourbakhsh, “Appearance-Based Obstacle Detection with Monocular Color Vision,” in IAAI, 2000.
-  A. Loquercio et al., “DroNet: Learning to Fly by Driving,” IEEE RA-Letters, vol. 3, no. 2, pp. 1088–1095, 2018.
-  D. Gandhi, L. Pinto, and A. Gupta, “Learning to fly by crashing,” arXiv:1704.05588, 2017.
-  C. Richter and N. Roy, “Safe visual navigation via deep learning and novelty detection,” in R:SS, 2017.
-  I. Goodfellow et al., “Generative adversarial nets,” in Advances in neural information processing systems, 2014, pp. 2672–2680.
-  A. R. et al, “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks,” arXiv:1607.07539, 2015.
-  I. J. Goodfellow, “NIPS 2016 Tutorial: Generative Adversarial Networks,” arXiv abs/1701.00160, 2017.
-  J.-Y. Zhu et al., “Generative visual manipulation on the natural image manifold,” in ECCV, 2016.
-  T. Schlegl et al., “Unsupervised anomaly detection with generative adversarial networks to guide marker discovery,” in IPMI, 2017.
-  Y. Bengio, A. Courville, and P. Vincent, “Representation Learning: A Review and New Perspectives,” IEEE PAMI, vol. 35, no. 8, pp. 1798–1828, 2013.
-  T. Darrell et al., “Machine Learning with Interdependent and Non-identically Distributed Data (Dagstuhl Seminar 15152),” in Dagstuhl Reports, vol. 5, no. 4, 2015.
-  F. A. Gers, J. Schmidhuber, and F. Cummins, “Learning to forget: Continual prediction with LSTM,” 1999.
-  Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” nature, vol. 521, no. 7553, p. 436, 2015.
-  K. He et al., “Deep residual learning for image recognition,” in CVPR, 2016.
-  H. Hirschmuller, “Stereo processing by semiglobal matching and mutual information,” IEEE Transactions on pattern analysis and machine intelligence, vol. 30, no. 2, pp. 328–341, 2008.
-  A. Kendall et al., “End-to-end learning of geometry and context for deep stereo regression,” CoRR, vol. abs/1703.04309, 2017.
-  K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps,” arXiv, vol. abs/1312.6034, 2013.
-  S. Thrun, W. Burgard, and D. Fox, Probabilistic robotics. MIT press, 2005.
-  P. H. Batavia and S. Singh, “Obstacle detection using adaptive color segmentation and color stereo homography,” in ICRA, 2001.
-  S. Lenser and M. Veloso, “Visual sonar: Fast obstacle avoidance using monocular vision,” in IROS, 2003.
-  V. Chandola, A. Banerjee, and V. Kumar, “Anomaly Detection: A Survey,” ACM Comput. Surv., vol. 41, no. 3, pp. 15:1–15:58, 2009.
-  D. Dey et al., Vision and Learning for Deliberative Monocular Cluttered Flight, 2016.
-  A. Giusti et al., “A machine learning approach to visual perception of forest trails for mobile robots,” IEEE RA Letters, vol. 1, no. 2, pp. 661–667, 2016.