Unsupervised Learning of Depth and Ego-Motion from Video

Unsupervised Learning of Depth and Ego-Motion from Video

Abstract

We present an unsupervised learning framework for the task of monocular depth and camera motion estimation from unstructured video sequences. In common with recent work [10], we use an end-to-end learning approach with view synthesis as the supervisory signal. In contrast to the previous work, our method is completely unsupervised, requiring only monocular video sequences for training. Our method uses single-view depth and multi-view pose networks, with a loss based on warping nearby views to the target using the computed depth and pose. The networks are thus coupled by the loss during training, but can be applied independently at test time. Empirical evaluation on the KITTI dataset demonstrates the effectiveness of our approach: 1) monocular depth performs comparably with supervised methods that use either ground-truth pose or depth for training, and 2) pose estimation performs favorably compared to established SLAM systems under comparable input settings.

1Introduction

Humans are remarkably capable of inferring ego-motion and the 3D structure of a scene even over short timescales. For instance, in navigating along a street, we can easily locate obstacles and react quickly to avoid them. Years of research in geometric computer vision has failed to recreate similar modeling capabilities for real-world scenes (e.g., where non-rigidity, occlusion and lack of texture are present). So why do humans excel at this task? One hypothesis is that we develop a rich, structural understanding of the world through our past visual experience that has largely consisted of moving around and observing vast numbers of scenes and developing consistent modeling of our observations. From millions of such observations, we have learned about the regularities of the world—roads are flat, buildings are straight, cars are supported by roads , and we can apply this knowledge when perceiving a new scene, even from a single monocular image.

Figure 1: The training data to our system consists solely of unlabeled image sequences capturing scene appearance from different viewpoints, where the poses of the images are not provided. Our training procedure produces two models that operate independently, one for single-view depth prediction, and one for multi-view camera pose estimation.
Figure 1: The training data to our system consists solely of unlabeled image sequences capturing scene appearance from different viewpoints, where the poses of the images are not provided. Our training procedure produces two models that operate independently, one for single-view depth prediction, and one for multi-view camera pose estimation.

In this work, we mimic this approach by training a model that observes sequences of images and aims to explain its observations by predicting likely camera motion and the scene structure (as shown in Figure 1). We take an end-to-end approach in allowing the model to map directly from input pixels to an estimate of ego-motion (parameterized as 6-DoF transformation matrices) and the underlying scene structure (parameterized as per-pixel depth maps under a reference view). We are particularly inspired by prior work that has suggested view synthesis as a metric [44] and recent work that tackles the calibrated, multi-view 3D case in an end-to-end framework [10]. Our method is unsupervised, and can be trained simply using sequences of images with no manual labeling or even camera motion information.

Our approach builds upon the insight that a geometric view synthesis system only performs consistently well when its intermediate predictions of the scene geometry and the camera poses correspond to the physical ground-truth. While imperfect geometry and/or pose estimation can cheat with reasonable synthesized views for certain types of scenes (e.g., textureless), the same model would fail miserably when presented with another set of scenes with more diverse layout and appearance structures. Thus, our goal is to formulate the entire view synthesis pipeline as the inference procedure of a convolutional neural network, so that by training the network on large-scale video data for the ‘meta’-task of view synthesis the network is forced to learn about intermediate tasks of depth and camera pose estimation in order to come up with a consistent explanation of the visual world. Empirical evaluation on the KITTI [15] benchmark demonstrates the effectiveness of our approach on both single-view depth and camera pose estimation. Our code will be made available at https://github.com/tinghuiz/SfMLearner.

2Related work

Structure from motion The simultaneous estimation of structure and motion is a well studied problem with an established toolchain of techniques [12]. Whilst the traditional toolchain is effective and efficient in many cases, its reliance on accurate image correspondence can cause problems in areas of low texture, complex geometry/photometry, thin structures, and occlusions. To address these issues, several of the pipeline stages have been recently tackled using deep learning, e.g., feature matching [18], pose estimation [26], and stereo [10]. These learning-based techniques are attractive in that they are able to leverage external supervision during training, and potentially overcome the above issues when applied to test data.

Warping-based view synthesis One important application of geometric scene understanding is the task of novel view synthesis, where the goal is to synthesize the appearance of the scene seen from novel camera viewpoints. A classic paradigm for view synthesis is to first either estimate the underlying 3D geometry explicitly or establish pixel correspondence among input views, and then synthesize the novel views by compositing image patches from the input views (e.g., [4]). Recently, end-to-end learning has been applied to reconstruct novel views by transforming the input based on depth or flow, e.g., DeepStereo [10], Deep3D [51] and Appearance Flows [54]. In these methods, the underlying geometry is represented by quantized depth planes (DeepStereo), probabilistic disparity maps (Deep3D) and view-dependent flow fields (Appearance Flows), respectively. Unlike methods that directly map from input views to the target view (e.g., [45]), warping-based methods are forced to learn intermediate predictions of geometry and/or correspondence. In this work, we aim to distill such geometric reasoning capability from CNNs trained to perform warping-based view synthesis.

Learning single-view 3D from registered 2D views Our work is closely related to a line of recent research on learning single-view 3D inference from registered 2D observations. Garg [14] propose to learn a single-view depth estimation CNN using projection errors to a calibrated stereo twin for supervision. Concurrently, Deep3D [51] predicts a second stereo viewpoint from an input image using stereoscopic film footage as training data. A similar approach was taken by Godard [16], with the addition of a left-right consistency constraint, and a better architecture design that led to impressive performance. Like our approach, these techniques only learn from image observations of the world, unlike methods that require explicit depth for training, e.g., [20].

These techniques bear some resemblance to direct methods for structure and motion estimation [22], where the camera parameters and scene depth are adjusted to minimize a pixel-based error function. However, rather than directly minimizing the error to obtain the estimation, the CNN-based methods only take a gradient step for each batch of input instances, which allows the network to learn an implicit prior from a large corpus of related imagery. Several authors have explored building differentiable rendering operations into their models that are trained in this way, e.g., [19].

While most of the above techniques (including ours) are mainly focused on inferring depth maps as the scene geometry output, recent work (e.g., [13]) has also shown success in learning 3D volumetric representations from 2D observations based on similar principles of projective geometry. Fouhey [11] further show that it is even possible to learn 3D inference without 3D labels (or registered 2D views) by utilizing scene regularity.

Unsupervised/Self-supervised learning from video Another line of related work to ours is visual representation learning from video, where the general goal is to design pretext tasks for learning generic visual features from video data that can later be re-purposed for other vision tasks such as object detection and semantic segmentation. Such pretext tasks include ego-motion estimation [2], tracking [49], temporal coherence [17], temporal order verification [36], and object motion mask prediction [39]. While we focus on inferring the explicit scene geometry and ego-motion in this work, intuitively, the internal representation learned by the deep network (especially the single-view depth CNN) should capture some level of semantics that could generalize to other tasks as well.

Concurrent to our work, Vijayanarasimhan [48] independently propose a framework for joint training of depth, camera motion and scene motion from videos. While both methods are conceptually similar, ours is focused on the unsupervised aspect, whereas their framework adds the capability to incorporate supervision (e.g., depth, camera motion or scene motion). There are significant differences in how scene dynamics are modeled during training, in which they explicitly solve for object motion whereas our explainability mask discounts regions undergoing motion, occlusion and other factors.

3Approach

Here we propose a framework for jointly training a single-view depth CNN and a camera pose estimation CNN from unlabeled video sequences. Despite being jointly trained, the depth model and the pose estimation model can be used independently during test-time inference. Training examples to our model consist of short image sequences of scenes captured by a moving camera. While our training procedure is robust to some degree of scene motion, we assume that the scenes we are interested in are mostly rigid, i.e., the scene appearance change across different frames is dominated by the camera motion.

Figure 2:  Overview of the supervision pipeline based on view synthesis. The depth network takes only the target view as input, and outputs a per-pixel depth map \hat{D}_t. The pose network takes both the target view (I_t) and the nearby/source views (e.g., I_{t-1} and I_{t+1}) as input, and outputs the relative camera poses (\hat{T}_{t\rightarrow t-1}, \hat{T}_{t\rightarrow t+1}). The outputs of both networks are then used to inverse warp the source views (see Sec. ) to reconstruct the target view, and the photometric reconstruction loss is used for training the CNNs. By utilizing view synthesis as supervision, we are able to train the entire framework in an unsupervised manner from videos.
Figure 2: Overview of the supervision pipeline based on view synthesis. The depth network takes only the target view as input, and outputs a per-pixel depth map . The pose network takes both the target view () and the nearby/source views (e.g., and ) as input, and outputs the relative camera poses (). The outputs of both networks are then used to inverse warp the source views (see Sec. ) to reconstruct the target view, and the photometric reconstruction loss is used for training the CNNs. By utilizing view synthesis as supervision, we are able to train the entire framework in an unsupervised manner from videos.

3.1View synthesis as supervision

The key supervision signal for our depth and pose prediction CNNs comes from the task of novel view synthesis: given one input view of a scene, synthesize a new image of the scene seen from a different camera pose. We can synthesize a target view given a per-pixel depth in that image, plus the pose and visibility in a nearby view. As we will show next, this synthesis process can be implemented in a fully differentiable manner with CNNs as the geometry and pose estimation modules. Visibility can be handled, along with non-rigidity and other non-modeled factors, using an “explanability” mask, which we discuss later (Sec. Section 3.3).

Let us denote as a training image sequence with one of the frames being the target view and the rest being the source views . The view synthesis objective can be formulated as

where indexes over pixel coordinates, and is the source view warped to the target coordinate frame based on a depth image-based rendering module [8] (described in Section 3.2), taking the predicted depth , the predicted camera transformation matrix1 and the source view as input.

Figure 3: Illustration of the differentiable image warping process. For each point p_t in the target view, we first project it onto the source view based on the predicted depth and camera pose, and then use bilinear interpolation to obtain the value of the warped image \hat{I}_s at location p_t.
Figure 3: Illustration of the differentiable image warping process. For each point in the target view, we first project it onto the source view based on the predicted depth and camera pose, and then use bilinear interpolation to obtain the value of the warped image at location .

Note that the idea of view synthesis as supervision has also been recently explored for learning single-view depth estimation [14] and multi-view stereo [10]. However, to the best of our knowledge, all previous work requires posed image sets during training (and testing too in the case of DeepStereo), while our framework can be applied to standard videos without pose information. Furthermore, it predicts the poses as part of the learning framework. See Figure 2 for an illustration of our learning pipeline for depth and pose estimation.

3.2Differentiable depth image-based rendering

As indicated in Equation 1, a key component of our learning framework is a differentiable depth image-based renderer that reconstructs the target view by sampling pixels from a source view based on the predicted depth map and the relative pose .

Let denote the homogeneous coordinates of a pixel in the target view, and denote the camera intrinsics matrix. We can obtain ’s projected coordinates onto the source view by2

Notice that the projected coordinates are continuous values. To obtain for populating the value of (see Figure 3), we then use the differentiable bilinear sampling mechanism proposed in the spatial transformer networks [23] that linearly interpolates the values of the -pixel neighbors (top-left, top-right, bottom-left, and bottom-right) of to approximate , i.e. where is linearly proportional to the spatial proximity between and , and . A similar strategy is used in [54] for learning to directly warp between different views, while here the coordinates for pixel warping are obtained through projective geometry that enables the factorization of depth and camera pose.

Figure 4: Network architecture for our depth/pose/explainability prediction modules. The width and height of each rectangular block indicates the output channels and the spatial dimension of the feature map at the corresponding layer respectively, and each reduction/increase in size indicates a change by the factor of 2. (a) For single-view depth, we adopt the DispNet  architecture with multi-scale side predictions. The kernel size is 3 for all the layers except for the first 4 conv layers with 7, 7, 5, 5, respectively. The number of output channels for the first conv layer is 32. (b) The pose and explainabilty networks share the first few conv layers, and then branch out to predict 6-DoF relative pose and multi-scale explainability masks, respectively. The number of output channels for the first conv layer is 16, and the kernel size is 3 for all the layers except for the first two conv and the last two deconv/prediction layers where we use 7, 5, 5, 7, respectively. See Section  for more details.
Figure 4: Network architecture for our depth/pose/explainability prediction modules. The width and height of each rectangular block indicates the output channels and the spatial dimension of the feature map at the corresponding layer respectively, and each reduction/increase in size indicates a change by the factor of . (a) For single-view depth, we adopt the DispNet architecture with multi-scale side predictions. The kernel size is for all the layers except for the first conv layers with , respectively. The number of output channels for the first conv layer is . (b) The pose and explainabilty networks share the first few conv layers, and then branch out to predict 6-DoF relative pose and multi-scale explainability masks, respectively. The number of output channels for the first conv layer is , and the kernel size is for all the layers except for the first two conv and the last two deconv/prediction layers where we use , respectively. See Section for more details.

3.3Modeling the model limitation

Note that when applied to monocular videos the above view synthesis formulation implicitly assumes 1) the scene is static without moving objects; 2) there is no occlusion/disocclusion between the target view and the source views; 3) the surface is Lambertian so that the photo-consistency error is meaningful. If any of these assumptions are violated in a training sequence, the gradients could be corrupted and potentially inhibit training. To improve the robustness of our learning pipeline to these factors, we additionally train a explainability prediction network (jointly and simultaneously with the depth and pose networks) that outputs a per-pixel soft mask for each target-source pair, indicating the network’s belief in where direct view synthesis will be successfully modeled for each target pixel. Based on the predicted , the view synthesis objective is weighted correspondingly by

Since we do not have direct supervision for , training with the above loss would result in a trivial solution of the network always predicting to be zero, which perfectly minimizes the loss. To resolve this, we add a regularization term that encourages nonzero predictions by minimizing the cross-entropy loss with constant label at each pixel location. In other words, the network is encouraged to minimize the view synthesis objective, but allowed a certain amount of slack for discounting the factors not considered by the model.

3.4Overcoming the gradient locality

One remaining issue with the above learning pipeline is that the gradients are mainly derived from the pixel intensity difference between and the four neighbors of , which would inhibit training if the correct (projected using the ground-truth depth and pose) is located in a low-texture region or far from the current estimation. This is a well known issue in motion estimation [3]. Empirically, we found two strategies to be effective for overcoming this issue: 1) using a convolutional encoder-decoder architecture with a small bottleneck for the depth network that implicitly constrains the output to be globally smooth and facilitates gradients to propagate from meaningful regions to nearby regions; 2) explicit multi-scale and smoothness loss (e.g., as in [14]) that allows gradients to be derived from larger spatial regions directly. We adopt the second strategy in this work as it is less sensitive to architectural choices. For smoothness, we minimize the norm of the second-order gradients for the predicted depth maps (similar to [48]).

Our final objective becomes

where indexes over different image scales, indexes over source images, and and are the weighting for the depth smoothness loss and the explainability regularization, respectively.

3.5Network architecture

Single-view depth For single-view depth prediction, we adopt the DispNet architecture proposed in [35] that is mainly based on an encoder-decoder design with skip connections and multi-scale side predictions (see Figure 4). All conv layers are followed by ReLU activation except for the prediction layers, where we use with and to constrain the predicted depth to be always positive within a reasonable range. We also experimented with using multiple views as input to the depth network, but did not find this to improve the results. This is in line with the observations in [47], where optical flow constraints need to be enforced to utilize multiple views effectively.

Pose The input to the pose estimation network is the target view concatenated with all the source views (along the color channels), and the outputs are the relative poses between the target view and each of the source views. The network consists of stride-2 convolutions followed by a convolution with output channels (corresponding to Euler angles and -D translation for each source view). Finally, global average pooling is applied to aggregate predictions at all spatial locations. All conv layers are followed by ReLU except for the last layer where no nonlinear activation is applied.

Explainability mask The explainability prediction network shares the first five feature encoding layers with the pose network, followed by deconvolution layers with multi-scale side predictions. All conv/deconv layers are followed by ReLU except for the prediction layers with no nonlinear activation. The number of output channels for each prediction layer is , with every two channels normalized by softmax to obtain the explainability prediction for the corresponding source-target pair (the second channel after normalization is and used in computing the loss in Equation 3).

4Experiments

Here we evaluate the performance of our system, and compare with prior approaches on single-view depth as well as ego-motion estimation. We mainly use the KITTI dataset [15] for benchmarking, but also use the Make3D dataset [42] for evaluating cross-dataset generalization ability.

Training Details We implemented the system using the publicly available TensorFlow [1] framework. For all the experiments, we set ( is the downscaling factor for the corresponding scale) and . During training, we used batch normalization [21] for all the layers except for the output layers, and the Adam [28] optimizer with , , learning rate of and mini-batch size of . The training typically converges after about iterations. All the experiments are performed with image sequences captured with a monocular camera. We resize the images to during training, but both the depth and pose networks can be run fully-convolutionally for images of arbitrary size at test time.

Figure 5: Our sample predictions on the Cityscapes dataset using the model trained on Cityscapes only.
Figure 5: Our sample predictions on the Cityscapes dataset using the model trained on Cityscapes only.
Figure 6: Comparison of single-view depth estimation between Eigen   (with ground-truth depth supervision), Garg   (with ground-truth pose supervision), and ours (unsupervised). The ground-truth depth map is interpolated from sparse measurements for visualization purpose. The last two rows show typical failure cases of our model, which sometimes struggles in vast open scenes and objects close to the front of the camera.
Figure 6: Comparison of single-view depth estimation between Eigen (with ground-truth depth supervision), Garg (with ground-truth pose supervision), and ours (unsupervised). The ground-truth depth map is interpolated from sparse measurements for visualization purpose. The last two rows show typical failure cases of our model, which sometimes struggles in vast open scenes and objects close to the front of the camera.
Figure 7: Comparison of single-view depth predictions on the KITTI dataset by our initial Cityscapes model and the final model (pre-trained on Cityscapes and then fine-tuned on KITTI). The Cityscapes model sometimes makes structural mistakes (e.g. holes on car body) likely due to the domain gap between the two datasets.
Figure 7: Comparison of single-view depth predictions on the KITTI dataset by our initial Cityscapes model and the final model (pre-trained on Cityscapes and then fine-tuned on KITTI). The Cityscapes model sometimes makes structural mistakes (e.g. holes on car body) likely due to the domain gap between the two datasets.

4.1Single-view depth estimation

We train our system on the split provided by [7], and exclude all the frames from the testing scenes as well as static sequences with mean optical flow magnitude less than pixel for training. We fix the length of image sequences to be frames, and treat the central frame as the target view and the frames as the source views. We use images captured by both color cameras, but treated them independently when forming training sequences. This results in a total of sequences, out of which we use for training and for validation.

To the best of our knowledge, no previous systems exist that learn single-view depth estimation in an unsupervised manner from monocular videos. Nonetheless, here we provide comparison with prior methods with depth supervision [7] and recent methods that use calibrated stereo images (i.e. with pose supervision) for training [14]. Since the depth predicted by our method is defined up to a scale factor, for evaluation we multiply the predicted depth maps by a scalar that matches the median with the ground-truth, i.e. .

Similar to [16], we also experimented with first pre-training the system on the larger Cityscapes dataset [5] (sample predictions are shown in Figure 5), and then fine-tune on KITTI, which results in slight performance improvement.

Kitti Here we evaluate the single-view depth performance on the images from the test split of [7]. As shown in Table ?, our unsupervised method performs comparably with several supervised methods (e.g. Eigen [7] and Garg [14]), but falls short of concurrent work by Godard [16] that uses calibrated stereo images (i.e. with pose supervision) with left-right cycle consistency loss for training. For future work, it would be interesting to see if incorporating the similar cycle consistency loss into our framework could further improve the results. Figure 6 provides examples of visual comparison between our results and some supervised baselines over a variety of examples. One can see that although trained in an unsupervised manner, our results are comparable to that of the supervised baselines, and sometimes preserve the depth boundaries and thin structures such as trees and street lights better.

We show sample predictions made by our initial Cityscapes model and the final model (pre-trained on Cityscapes and then fine-tuned on KITTI) in Figure 7. Due to the domain gap between the two datasets, our Cityscapes model sometimes has difficulty in recovering the complete shape of the car/bushes, and mistakes them with distant objects.

We also performed an ablation study of the explainability modeling (see Table ?), which turns out only offering a modest performance boost. This is likely because 1) most of the KITTI scenes are static without significant scene motions, and 2) the occlusion/visibility effects only occur in small regions in sequences across a short time span (-frames), which make the explainability modeling less essential to the success of training. Nonetheless, our explainability prediction network does seem to capture the factors like scene motion and visibility well (see Section 4.3), and could potentially be more important for other more challenging datasets.

Make3D To evaluate the generalization ability of our single-view depth model, we directly apply our model trained on Cityscapes + KITTI to the Make3D dataset unseen during training. While there still remains a significant performance gap between our method and others supervised using Make3D ground-truth depth (see Table ?), our predictions are able to capture the global scene layout reasonably well without any training on the Make3D images (see Figure 8).

Figure 8: Our sample predictions on the Make3D dataset. Note that our model is trained on KITTI + Cityscapes only, and directly tested on Make3D.
Figure 8: Our sample predictions on the Make3D dataset. Note that our model is trained on KITTI + Cityscapes only, and directly tested on Make3D.

4.2Pose estimation

To evaluate the performance of our pose estimation network, we applied our system to the official KITTI odometry split (containing driving sequences with ground truth odometry obtained through the IMU/GPS readings, which we use for evaluation purpose only), and used sequences - for training and - for testing. In this experiment, we fix the length of input image sequences to our system to frames. We compare our ego-motion estimation with two variants of monocular ORB-SLAM [37] (a well-established SLAM system): 1) ORB-SLAM (full), which recovers odometry using all frames of the driving sequence (i.e. allowing loop closure and re-localization), and 2) ORB-SLAM (short), which runs on -frame snippets (same as our input setting). Another baseline we compare with is the dataset mean of car motion (using ground-truth odometry) for -frame snippets. To resolve scale ambiguity during evaluation, we first optimize the scaling factor for the predictions made by each method to best align with the ground truth, and then measure the Absolute Trajectory Error (ATE) [37] as the metric. ATE is computed on -frame snippets and averaged over the full sequence.3 As shown in Table 1 and Figure 9, our method outperforms both baselines (mean odometry and ORB-SLAM (short)) that share the same input setting as ours, but falls short of ORB-SLAM (full), which leverages whole sequences ( for seq. and for seq. ) for loop closure and re-localization.

For better understanding of our pose estimation results, we show in Figure 9 the ATE curve with varying amount of side-rotation by the car between the beginning and the end of a sequence. Figure 9 suggests that our method is significantly better than ORB-SLAM (short) when the side-rotation is small (i.e. car mostly driving forward), and comparable to ORB-SLAM (full) across the entire spectrum. The large performance gap between ours and ORB-SLAM (short) suggests that our learned ego-motion could potentially be used as an alternative to the local estimation modules in monocular SLAM systems.

Table 1: Absolute Trajectory Error (ATE) on the KITTI odometry split averaged over all -frame snippets (lower is better). Our method outperforms baselines with the same input setting, but falls short of ORB-SLAM (full) that uses strictly more data.

Method

Seq. Seq. ORB-SLAM (full)
Figure 9: Absolute Trajectory Error (ATE) at different left/right turning magnitude (coordinate difference in the side-direction between the start and ending frame of a testing sequence). Our method performs significantly better than ORB-SLAM (short) when side rotation is small, and is comparable with ORB-SLAM (full) across the entire spectrum.
Figure 9: Absolute Trajectory Error (ATE) at different left/right turning magnitude (coordinate difference in the side-direction between the start and ending frame of a testing sequence). Our method performs significantly better than ORB-SLAM (short) when side rotation is small, and is comparable with ORB-SLAM (full) across the entire spectrum.

4.3Visualizing the explainability prediction

We visualize example explainability masks predicted by our network in Figure 10. The first three rows suggest that the network has learned to identify dynamic objects in the scene as unexplainable by our model, and similarly, rows 4–5 are examples of objects that disappear from the frame in subsequent views. The last two rows demonstrate the potential downside of explainability-weighted loss: the depth CNN has low confidence in predicting thin structures well, and tends to mask them as unexplainable.

Figure 10: Sample visualizations of the explainability masks. Highlighted pixels are predicted to be unexplainable by the network due to motion (rows 1–3), occlusion/visibility (rows 4–5), or other factors (rows 7–8).
Figure 10: Sample visualizations of the explainability masks. Highlighted pixels are predicted to be unexplainable by the network due to motion (rows 1–3), occlusion/visibility (rows 4–5), or other factors (rows 7–8).

5Discussion

We have presented an end-to-end learning pipeline that utilizes the task of view synthesis for supervision of single-view depth and camera pose estimation. The system is trained on unlabeled videos, and yet performs comparably with approaches that require ground-truth depth or pose for training. Despite good performance on the benchmark evaluation, our method is by no means close to solving the general problem of unsupervised learning of 3D scene structure inference. A number of major challenges are yet to be addressed: 1) our current framework does not explicitly estimate scene dynamics and occlusions (although they are implicitly taken into account by the explainability masks), both of which are critical factors in 3D scene understanding. Direct modeling of scene dynamics through motion segmentation (e.g. [48]) could be a potential solution; 2) our framework assumes the camera intrinsics are given, which forbids the use of random Internet videos with unknown camera types/calibration – we plan to address this in future work; 3) depth maps are a simplified representation of the underlying 3D scene. It would be interesting to extend our framework to learn full 3D volumetric representations (e.g. [46]).

Another interesting area for future work would be to investigate in more detail the representation learned by our system. In particular, the pose network likely uses some form of image correspondence in estimating the camera motion, whereas the depth estimation network likely recognizes common structural features of scenes and objects. It would be interesting to probe these, and investigate the extent to which our network already performs, or could be re-purposed to perform, tasks such as object detection and semantic segmentation.

Acknowledgments: We thank our colleagues, Sudheendra Vijayanarasimhan, Susanna Ricco, Cordelia Schmid, Rahul Sukthankar, and Katerina Fragkiadaki for their help. We also thank the anonymous reviewers for their valuable comments. TZ would like to thank Shubham Tulsiani for helpful discussions, and Clement Godard for sharing the evaluation code. This work is also partially funded by Intel/NSF VEC award IIS-1539099.

Footnotes

  1. In practice, the CNN estimates the Euler angles and the 3D translation vector, which are then converted to the transformation matrix.
  2. For notation simplicity, we omit showing the necessary conversion to homogeneous coordinates along the steps of matrix multiplication.
  3. For evaluating ORB-SLAM (full) we break down the trajectory of the full sequence into -frame snippets with the reference coordinate frame adjusted to the central frame of each snippet.

References

  1. TensorFlow: Large-scale machine learning on heterogeneous distributed systems.
    M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, et al. arXiv preprint arXiv:1603.04467, 2016.
  2. Learning to see by moving.
    P. Agrawal, J. Carreira, and J. Malik. In Int. Conf. Computer Vision, 2015.
  3. Hierarchical model-based motion estimation.
    J. Bergen, P. Anandan, K. Hanna, and R. Hingorani. In Computer Vision—ECCV’92, pages 237–252. Springer, 1992.
  4. View interpolation for image synthesis.
    S. E. Chen and L. Williams. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques, pages 279–288. ACM, 1993.
  5. The Cityscapes dataset for semantic urban scene understanding.
    M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3213–3223, 2016.
  6. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach.
    P. E. Debevec, C. J. Taylor, and J. Malik. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 11–20. ACM, 1996.
  7. Depth map prediction from a single image using a multi-scale deep network.
    D. Eigen, C. Puhrsch, and R. Fergus. In Advances in Neural Information Processing Systems, 2014.
  8. Depth-image-based rendering (dibr), compression, and transmission for a new approach on 3d-tv.
    C. Fehn. In Electronic Imaging 2004, pages 93–104. International Society for Optics and Photonics, 2004.
  9. Image-based rendering using image-based priors.
    A. Fitzgibbon, Y. Wexler, and A. Zisserman. Int. Journal of Computer Vision, 63(2):141–151, 2005.
  10. DeepStereo: Learning to predict new views from the world’s imagery.
    J. Flynn, I. Neulander, J. Philbin, and N. Snavely. In Computer Vision and Pattern Recognition, 2016.
  11. Single image 3D without a single 3D image.
    D. F. Fouhey, W. Hussain, A. Gupta, and M. Hebert. In Proceedings of the IEEE International Conference on Computer Vision, pages 1053–1061, 2015.
  12. Towards internet-scale multi-view stereo.
    Y. Furukawa, B. Curless, S. M. Seitz, and R. Szeliski. In Computer Vision and Pattern Recognition, pages 1434–1441. IEEE, 2010.
  13. 3d shape induction from 2d views of multiple objects.
    M. Gadelha, S. Maji, and R. Wang. arXiv preprint arXiv:1612.05872, 2016.
  14. Unsupervised CNN for single view depth estimation: Geometry to the rescue.
    R. Garg, V. K. BG, G. Carneiro, and I. Reid. In European Conf. Computer Vision, 2016.
  15. Are we ready for autonomous driving? The KITTI vision benchmark suite.
    A. Geiger, P. Lenz, and R. Urtasun. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on, pages 3354–3361. IEEE, 2012.
  16. Unsupervised monocular depth estimation with left-right consistency.
    C. Godard, O. Mac Aodha, and G. J. Brostow. In Computer Vision and Pattern Recognition, 2017.
  17. Unsupervised learning of spatiotemporally coherent metrics.
    R. Goroshin, J. Bruna, J. Tompson, D. Eigen, and Y. LeCun. In Proceedings of the IEEE International Conference on Computer Vision, pages 4086–4093, 2015.
  18. MatchNet: Unifying feature and metric learning for patch-based matching.
    X. Han, T. Leung, Y. Jia, R. Sukthankar, and A. C. Berg. In Computer Vision and Pattern Recognition, pages 3279–3286, 2015.
  19. gvnn: Neural network library for geometric computer vision.
    A. Handa, M. Bloesch, V. Patraucean, S. Stent, J. McCormac, and A. Davison. arXiv preprint arXiv:1607.07405, 2016.
  20. Automatic photo pop-up.
    D. Hoiem, A. A. Efros, and M. Hebert . In Proc. SIGGRAPH, 2005.
  21. Batch normalization: Accelerating deep network training by reducing internal covariate shift.
    S. Ioffe and C. Szegedy. arXiv preprint arXiv:1502.03167, 2015.
  22. About direct methods.
    M. Irani and P. Anandan. In International Workshop on Vision Algorithms, pages 267–277. Springer, 1999.
  23. Spatial transformer networks.
    M. Jaderberg, K. Simonyan, A. Zisserman, et al. In Advances in Neural Information Processing Systems, pages 2017–2025, 2015.
  24. Learning image representations tied to egomotion.
    D. Jayaraman and K. Grauman. In Int. Conf. Computer Vision, 2015.
  25. Depth transfer: Depth extraction from video using non-parametric sampling.
    K. Karsch, C. Liu, and S. B. Kang. IEEE transactions on pattern analysis and machine intelligence, 36(11):2144–2158, 2014.
  26. PoseNet: A convolutional network for real-time 6-DOF camera relocalization.
    A. Kendall, M. Grimes, and R. Cipolla. In Int. Conf. Computer Vision, pages 2938–2946, 2015.
  27. End-to-end learning of geometry and context for deep stereo regression.
    A. Kendall, H. Martirosyan, S. Dasgupta, P. Henry, R. Kennedy, A. Bachrach, and A. Bry. arXiv preprint arXiv:1703.04309, 2017.
  28. Adam: A method for stochastic optimization.
    D. Kingma and J. Ba. arXiv preprint arXiv:1412.6980, 2014.
  29. Deep convolutional inverse graphics network.
    T. D. Kulkarni, W. F. Whitney, P. Kohli, and J. Tenenbaum. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems, pages 2539–2547. Curran Associates, Inc., 2015.
  30. Semi-supervised deep learning for monocular depth map prediction.
    Y. Kuznietsov, J. Stückler, and B. Leibe. arXiv preprint arXiv:1702.02706, 2017.
  31. Deeper depth prediction with fully convolutional residual networks.
    I. Laina, C. Rupprecht, V. Belagiannis, F. Tombari, and N. Navab. In 3D Vision (3DV), 2016 Fourth International Conference on, pages 239–248. IEEE, 2016.
  32. Learning depth from single monocular images using deep convolutional neural fields.
    F. Liu, C. Shen, G. Lin, and I. Reid. IEEE transactions on pattern analysis and machine intelligence, 38(10):2024–2039, 2016.
  33. Discrete-continuous depth estimation from a single image.
    M. Liu, M. Salzmann, and X. He. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 716–723, 2014.
  34. OpenDR: An approximate differentiable renderer.
    M. M. Loper and M. J. Black. In European Conf. Computer Vision, pages 154–169. Springer, 2014.
  35. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation.
    N. Mayer, E. Ilg, P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4040–4048, 2016.
  36. Shuffle and learn: unsupervised learning using temporal order verification.
    I. Misra, C. L. Zitnick, and M. Hebert. In European Conference on Computer Vision, pages 527–544. Springer, 2016.
  37. ORB-SLAM: a versatile and accurate monocular SLAM system.
    R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos. IEEE Transactions on Robotics, 31(5), 2015.
  38. DTAM: Dense tracking and mapping in real-time.
    R. A. Newcombe, S. J. Lovegrove, and A. J. Davison. In Int. Conf. Computer Vision, pages 2320–2327. IEEE, 2011.
  39. Learning features by watching objects move.
    D. Pathak, R. Girshick, P. Dollár, T. Darrell, and B. Hariharan. In CVPR, 2017.
  40. Dense monocular depth estimation in complex dynamic scenes.
    R. Ranftl, V. Vineet, Q. Chen, and V. Koltun. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4058–4066, 2016.
  41. Unsupervised learning of 3d structure from images.
    D. J. Rezende, S. A. Eslami, S. Mohamed, P. Battaglia, M. Jaderberg, and N. Heess. In Advances In Neural Information Processing Systems, pages 4997–5005, 2016.
  42. Make3D: Learning 3D scene structure from a single still image.
    A. Saxena, M. Sun, and A. Y. Ng. Pattern Analysis and Machine Intelligence, 31(5):824–840, May 2009.
  43. View morphing.
    S. M. Seitz and C. R. Dyer. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 21–30. ACM, 1996.
  44. Prediction error as a quality metric for motion and stereo.
    R. Szeliski. In Int. Conf. Computer Vision, volume 2, pages 781–788. IEEE, 1999.
  45. Multi-view 3d models from single images with a convolutional network.
    M. Tatarchenko, A. Dosovitskiy, and T. Brox. In European Conference on Computer Vision, pages 322–337. Springer, 2016.
  46. Multi-view supervision for single-view reconstruction via differentiable ray consistency.
    S. Tulsiani, T. Zhou, A. A. Efros, and J. Malik. In Computer Vision and Pattern Recognition, 2017.
  47. DeMoN: Depth and motion network for learning monocular stereo.
    B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, E. Ilg, A. Dosovitskiy, and T. Brox. arXiv preprint arXiv:1612.02401, 2016.
  48. SfM-Net: Learning of structure and motion from video.
    S. Vijayanarasimhan, S. Ricco, C. Schmid, R. Sukthankar, and K. Fragkiadaki. arXiv preprint, 2017.
  49. Unsupervised learning of visual representations using videos.
    X. Wang and A. Gupta. In Proceedings of the IEEE International Conference on Computer Vision, pages 2794–2802, 2015.
  50. VisualSFM: A visual structure from motion system.
    C. Wu. http://ccwu.me/vsfm, 2011.
  51. Deep3D: Fully automatic 2D-to-3D video conversion with deep convolutional neural networks.
    J. Xie, R. B. Girshick, and A. Farhadi. In European Conf. Computer Vision, 2016.
  52. Perspective transformer nets: Learning single-view 3d object reconstruction without 3d supervision.
    X. Yan, J. Yang, E. Yumer, Y. Guo, and H. Lee. In Advances in Neural Information Processing Systems, pages 1696–1704, 2016.
  53. Stereo matching by training a convolutional neural network to compare image patches.
    J. Zbontar and Y. LeCun. Journal of Machine Learning Research, 17(1-32):2, 2016.
  54. View synthesis by appearance flow.
    T. Zhou, S. Tulsiani, W. Sun, J. Malik, and A. A. Efros. In European Conference on Computer Vision, pages 286–301. Springer, 2016.
  55. High-quality video view interpolation using a layered representation.
    C. L. Zitnick, S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski. In ACM Transactions on Graphics (TOG), volume 23, pages 600–608. ACM, 2004.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
2489
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description