Unsupervised Depth Completion from Visual Inertial Odometry
Abstract
We describe a method to infer dense depth from camera motion and sparse depth as estimated using a visualinertial odometry system. Unlike other scenarios using point clouds from lidar or structured light sensors, we have few hundreds to few thousand points, insufficient to inform the topology of the scene. Our method first constructs a piecewise planar scaffolding of the scene, and then uses it to infer dense depth using the image along with the sparse points. We use a predictive crossmodal criterion, akin to “selfsupervision,” measuring photometric consistency across time, forwardbackward pose consistency, and geometric compatibility with the sparse point cloud. We also present the first visualinertial + depth dataset, which we hope will foster additional exploration into combining the complementary strengths of visual and inertial sensors. To compare our method to prior work, we adopt the unsupervised KITTI depth completion benchmark, where we achieve stateoftheart performance.
I Introduction
A sequence of images is a rich source of information about both the threedimensional (3D) shape of the environment and the motion of the sensor within. Motion can be inferred at most up to a scale and a global Euclidean reference frame, provided sufficient parallax and a number of visually discriminative Lambertian regions that are fixed in the environment and visible from the camera. The position of such regions in the scene defines the Euclidean reference frame, with respect to which motion is estimated. Scale, as well as two directions of orientation, can be further identified by fusion with inertial measurements (accelerometers and gyroscopes) and, if available, a magnetometer can fix the last (Gauge) degree of freedom.
Because the regions defining the reference frame have to be visually distinctive (“features”), they are typically sparse. In theory, three points are sufficient to define a Euclidean Gauge if visible at all times. In practice, because of occlusions, any Structure From Motion (SFM) or simultaneous localization and mapping (SLAM) system maintains an estimate of the location of a sparse set of features, or “sparse point cloud,” typically in the hundreds to thousands. These are sufficient to support a pointestimate of motion, but a rather poor representation of shape as they do not reveal the topology of the scene: The empty space between points could be empty, or occupied by a solid with a smooth surface radiance (appearance). Attempts to densify the sparse point cloud, by interpolation or regularization with generic priors such as smoothness, piecewise planarity and the like, typically fail since SFM yields far too sparse a reconstruction to inform topology. This is where the image comes back in.
Inferring shape is illposed, even if the point cloud was generated with a lidar or structured light sensor. Filling the gaps relies on assumptions about the environment. Rather than designing adhoc priors, we wish to use the image to inform and restrict the set of possible scenes that are compatible with the given sparse points.
Summary of contributions
We use a predictive crossmodal criterion to score dense depth from images and sparse depth. This kind of approach is sometimes referred to as “selfsupervised.” Specifically, our method (i) exploits a set of constraints from temporal consistency (a.k.a. photometric consistency across temporally adjacent frames) to pose (forwardbackward) consistency in a combination that has not been previously explored. To enable our pose consistency term, we introduce (ii) a set of logarithmic and exponential mapping layers for our network to represent motion using exponential coordinates, which we found to improve reconstruction compared to other parametrizations.
The challenge in using sparse depth as a supervisory (feedback) signal is precisely that it is sparse. Information at the points does not propagate to fill the domain where depth is defined. Some computational mechanism to “diffuse the information” from the sparse points to their neighbors is needed. Our approach proposes (iii) a simple method akin to using a piecewise planar “scaffolding” of the scene, sufficient to transfer the supervisory signal from sparse points to their neighbors. This yields a twostage approach, where the sparse points are first processed to design the scaffolding (“meshing and interpolation”) and then “refined” using the images as well as priors from the constraints just described.
One additional contribution of our approach is (iv) to introduce the first visualinertial + depth dataset. The role of inertials is to enable reconstruction in metric scale, which is critical for robotic applications. Although scale can be obtained via other sensors, e.g., stereo, lidar, and RGBD, we note they are not as widely available as monocular cameras with inertials (almost every modern phone has it) and consume more power. Since inertial sensors are now ubiquitous and typically colocated with cameras in many mobile devices from phones to cars, we hope this dataset will foster additional exploration into combining the complementary strengths of visual and inertial sensors.
To evaluate our method, since no other visualinertial + depth benchmark is available, and to facilitate comparison with similar methods, we adopt the KITTI benchmark, where a Velodyne (lidar) sensor provides sparse points with scale, unlike monocular SFM, but like visualinertial odometry (VIO). Although the biases in lidar are different from VIO, this can be considered a baseline. Note that we only use the monocular stream of KITTI (not stereo) for fair comparison.
The result is a (v) twostage approach of scaffolding and refining with a network that contains much fewer parameters than competing methods, yet achieves stateoftheart performance in the “unsupervised” KITTI benchmark (a misnomer). The supervision in the KITTI benchmark is really fusion from separate sensory channels, combined with adhoc interpolation and extrapolation. It is unclear whether the benefit from having such data is outweighed by the biases it induces on the estimate, and in any case such supervision does not scale; hence, we forgo (pseudo) ground truth annotations altogether.
Ii Related Work
Supervised Depth Completion minimizes the discrepancy between ground truth depth and depth predicted from an RGB image and sparse depth measurements. Methods focus on network topology [16, 26, 29], optimization [4, 5, 31], and modeling [6, 10]. To handle sparse depth, [16] employed early fusion, where the image and sparse depth are convolved separately and the results concatenated as the input to a ResNet encoder. [11] proposed late fusion via a Unet containing two NASNet encoders for image and sparse depth and jointly learned depth and semantic segmentation, whereas [29] used ResNet encoders for late fusion. [6] proposed a normalized convolutional layer to propagate sparse depth and used a binary validity map as a confidence measure. [10] proposed an upsampling layer and joint concatenation and convolution to deal with sparse inputs. All these methods require perpixel groundtruth annotation. What is called “ground truth” in the benchmarks is actually the result of data processing and aggregation of many consecutive frames. We skip such supervision and just infer dense depth by learning the crossmodal fusion from the virtually infinite volume of unannotated data.
Unsupervised Depth Completion methods, such as [16, 23, 29] predict depth by minimizing the discrepancy between prediction and sparse depth input as well as the photometric error between the input image and its reconstruction from other viewpoints available only during training. [16] used PerspectivenPoint (PnP) [15] and Random Sample Consensus (RANSAC) [8] to align monocular image sequences for their photometric term with a secondorder smoothness prior. Yet, [16] does not generalize well to indoor scenes that contains many textureless regions (e.g. walls), where PnP with RANSAC may fail. [23] used a local smoothness term, but instead minimized the photometric error between rectified stereopairs where pose is known. [29] also leveraged stereo pairs and a more sophisticated photometric loss [28]. [29] replaced the generic smoothness term with a learned prior to regularize their prediction. To accomplish this, [29] requires a conditional prior network (CPN) that is trained on an additional dataset (representative of the depth completion dataset) in a fullysupervised manner using groundtruth depth. The CPN does not generalize well outside its training domain (e.g. one cannot use a CPN trained on outdoors scenes to regularize depth predictions for indoors). Hence, [29] is essentially not unsupervised and has limited applicability. In contrast, our method is trained on monocular sequences, is fully unsupervised and does not use any auxiliary groundtruth supervision. Unlike previous works, our method does not require large networks ([11, 16, 23, 29]) nor any complex network operations ([6, 10]). Moreover, our method outperforms [16, 29] on the unsupervised KITTI depth completion benchmark [26] while using fewer parameters.
Rotation Parameterization. To construct the photometric consistency loss during training, an auxiliary pose network is needed if no camera poses are available. While the translational part of the relative pose can be modeled as , the rotational part belongs to the special orthogonal group [17], which is represented by a matrix. [13] uses quaternions, which require an additional norm constraint; this is a soft constraint imposed in the loss function, and thus is not guaranteed. [7, 30, 32] use Euler angles which may result in a nonorthogonal rotation matrix due to rounding error from the multiplication of many sine and cosine terms. We use the exponential map on to map the output of the pose network to a rotation matrix. Though theoretically similar, we empirically found that the exponential map is more beneficial than the Euler angles in Sec. VII.
Our contributions are a simple, yet effective twostage approach resulting in a large reduction in network parameters while achieving stateoftheart performance on the unsupervised KITTI depth completion benchmark; using exponential parameterization of rotation for our pose network; a pose consistency term that enforces forward and backward motion to be the inverse of each other; and finally a new depth completion benchmark for visualinertial odometry systems with indoor and outdoor scenes and challenging motion.
Iii Method Formulation
We reconstruct a 3D scene given an RGB image and the associated set of sparse depth measurements .
We begin by assuming that world surfaces are graphs of smooth functions (charts) locally supported on a piecewise planar domain (scaffolding). We construct the scaffolding from the sparse point cloud (“Scaffolding” in Fig. 3) to obtain , then learn a completion model refining by leveraging the monocular sequences (), of frames before and after the given time , and the sparse depth . We compose a surrogate loss (Eqn. 2) for driving the training process, using an encoderdecoder architecture parameterized by weights , where the input is an image with its scaffolding , and the output is the dense depth . Fig. 2 shows an overview of our approach.
Iiia A TwoStage Approach
Depth completion is a challenging problem due to the sparsity level of the depth input, . As the density of sparse depth measurements covers of the image plane for the outdoor selfdriving scenario (Sec. VA) and less than for the indoor setting (Sec. VIIC), generally only a single measurement will be present within a local neighborhood and in most instances none. This renders conventional convolutions ineffective as each sparse depth measurement can be seen as a Dirac delta and convolving a kernel over the entire sparse depth input will give mostly zero activations. Hence, [6], [10], and [26] proposed specialized operations to propagate the information from the sparse depth input through the network. We, instead, propose a twostage approach that circumvents this problem by first approximating a coarse scene geometry with scaffolding and training a network to refine the approximation.
IiiB Scaffolding
Given sparse depth measurements , our goal is to create a coarse approximation of the scene; yet, the topology of the scene is not informed by . Hence, we must rely on a prior or an assumption – that surfaces are locally smooth and piecewise planar. We begin by applying the lifting transform [3] to , mapping from 2d to 3d space. We then compute its convex hull [2], of which the lower envelope is taken as the Delaunay triangulation of the points in – resulting in a triangular mesh in Barycentric coordinates.
To form the tessellation of the triangular mesh, we approximate each surface using linear interpolation within the Barycentric coordinates and the resulting scaffolding is projected back onto the image plane to produce . For a given triangle, simple interpolation is sufficient for recovering the plane as a linear combination of the coplanar points. For sets of points not coplanar, interpolation will give an approximation, with which we refine using a network. We note that our approximation cannot be achieved by simply filtering (e.g. Gaussian) to propagate depth values as the filter would produce mostly zeros and even destroy the sparse depth signal.
IiiC Refinement
Given an RGB image and its corresponding piecewise planar scaffolding , we train a network to recover the 3d scene by refining based on information from . Our network learns to refine without groundtruth supervision by minimizing Eqn. 2 (see Sec. IV).
Network Architecture. We propose two encoderdecoder architectures with skip connections following the late fusion paradigm [11, 29]. Each encoder has an image branch and a depth branch, where each contains and of the total encoder parameters, respectively. The latent representation of the branches are concatenated and fed to the decoder. We propose a VGG11 encoder (M parameters) containing 8 convolution layers for each branch as our best performing model, and a VGG8 encoder (M parameters) containing only 5 convolution layers for each branch as our lightweight model. This is in contrast to other unsupervised methods [16] (early fusion) and [29] (late fusion) – both of which use ResNet34 encoders with M and M parameters, respectively. [16, 29] and our approach share the same decoder architecture containing M parameters. We show in Sec. VII that despite having and fewer encoder parameters than [16] and [29], our VGG11 model outperforms both [16] and [29]. Moreover, performance does not degrade by much from VGG11 to VGG8 and VGG8 still surpasses [16] and [29] while having a and reduction in the encoder parameters. Unlike [16, 29], which requires high energy consumption hardware, our approach is computationally cheap, and can be deployed to lowpowered agents using an Nvidia Jetson.
Logarithmic and Exponential Map Layers. To construct our objective (Eqn. 2), we leverage a pose network [13] to regress the relative camera poses . We present a novel logarithmic map layer: , where is the tangent space of , and an exponential map layer: – for mapping between and . We use the logarithmic map to construct the pose consistency loss (Eqn. 6), and the exponential to map the output of the pose network as coordinates in to a rotation matrix:
(1) 
where the hat operator maps to a skewsymmetric matrix [17]. We train of our pose network using a surrogate loss (Eqn. 3) without explicit supervision. Ablation studies on the use of exponential coordinates and pose consistency for depth completion can be found in Table III and IV.
Our approach contains two stages: (i) we generate a coarse piecewise planar approximation of the scene from the sparse depth inputs via scaffolding and (ii) we feed the resulting depth map along with the associated RGB image to our network for refinement (Fig. 3). This approach alleviates the network from the need of learning from sparse inputs, for which [16] and [29] compensated with parameters. We show the effectiveness of this approach by achieving the stateoftheart on the unsupervised KITTI depth completion benchmark with half as many parameters as the priorart.
Iv Loss Function
Our loss function is a linear combination of four terms that constrain (i) the photometric consistency between the observed image and its reconstructions from the monocular sequence, (ii) the predicted depth to be similar to that of the associated available sparse depth, (iii) the composition of the predicted forward and backward relative poses to be the identity, and (iv) the prediction to adhere to local smoothness.
(2) 
where denotes photometric consistency, sparse depth consistency, pose consistency, and local smoothness. Each loss term is described in the next subsections and the associated weight in Sec. VI.
Iva Photometric Consistency
We enforce temporal consistency by minimizing the discrepancy between each observed image and its reconstruction from temporally adjacent images , where :
(3) 
where are the homogeneous coordinates of , is the relative pose of the camera from time to , denotes the camera intrinsics, and refers to the perspective projection.
IvB Sparse Depth Consistency
Our sparse depth consistency term provides our predictions with metric scale by encouraging the predictions to be similar to that of the metric sparse depth available from lidar in KITTI dataset (Sec. VA) and sparse reconstruction in our visualinertial dataset (Sec. VB). Our sparse depth consistency loss is the norm of the difference between the predicted depth and the sparse depth averaged over (the support of the sparse depth):
(5) 
IvC Pose Consistency
A pose network takes an ordered pair of images and outputs the relative pose (forward pose). When a temporally swapped pair is fed to the network, the network is expected to output (backward pose) – the inverse of , i.e., . The forwardbackward pose consistency thus penalizes the deviation of the composed pose from the identity:
(6) 
where is the logarithmic map.
IvD Local Smoothness
We impose a smoothness loss on the predicted depth by applying an penalty to the gradients in both the x and y directions of the predicted depth :
(7) 
where and are the edgeawareness weights to allow for discontinuities in regions corresponding to object boundaries.
IvE The Role of Inertials
Although inertials are not directly present in the loss, their role in metric depth completion is crucial. Without inertials, a SLAM system cannot produce sparse point clouds in metric scale, which are then used as both the input to the scaffolding stage (Sec. IIIB) and a supervisory signal (Eqn. 5).
V Datasets
Va KITTI Benchmark
We evaluate our approach on the KITTI depth completion benchmark [26]. The dataset provides raw image frames and associated sparse depth maps. The sparse depth maps are the raw output from the Velodyne lidar sensor, each with a density of . The groundtruth depth map is created by accumulating the neighbouring 11 raw lidar scans, with dense depth corresponding to the bottom of the images. We use the officially selected 1,000 samples for validation and we apply our method to 1,000 testing samples, with which we submit to the official KITTI website for evaluation. The results are reported in Table II.
VB VOID Benchmark
While KITTI provides a standard benchmark for evaluating depth completion in the driving scenario, there exists no standard depth completion benchmark for the indoor scenario. [16, 29] used NYUv2 [19] – an RGBD dataset – to develop and evaluate their models on indoor scenes. Yet, each performs a different evaluation protocol with different sparse depth samples – varying densities of depth values were randomly sampled from the depth frame, preventing direct comparisons between methods. Though this is reasonable as a proof of concept, it is not realistic in the sense that no sensor measures depth at random locations.
The VOID dataset. We propose a new publicly available dataset for a real world use case of depth completion by bootstrapping sparse reconstruction in metric space from a SLAM system. While it is well known that metric scale is not observable in the purely imagebased SLAM and SFM setting, it has been resolved by the recent advances in VIO [12, 18], where metric pose and structure estimation can be realized in a gravityaligned and scaled reference frame using a inertial measurement unit (IMU).
To this end, we leverage an offtheshelf VIO system
Our dataset is dubbed “Visual Odometry with Inertial and Depth” or “VOID” for short and is comprised of RGB video streams and inertial measurements for metric reconstruction along with perframe dense depth for crossmodal validation.
Data acquisition. Our data was collected using the latest Intel RealSense D435i camera
The SLAM system we use is based on [12] – an EKFbased VIO model. While the VIO recursively estimates a joint posterior of the state of the sensor platform (e.g. pose, velocity, sensor biases, and cameratoIMU alignment) and a small set of reliable feature points, the 3D structure it estimates is extremely sparse – typically feature points (instate features). To facilitate 3D reconstruction, we track a moderate amount of outofstate features in addition to the instate ones, and estimate the depth of the feature points using auxiliary depth subfilters [17].
The benchmark. We evaluate our method on the VOID depth completion benchmark, which contains 56 sequences in total, both indoor and outdoor with challenging motion. Typical scenes include classrooms, offices, stairwells, laboratories, and gardens. Of the 56 sequences, 48 sequences ( frames) are designated for training and 8 sequences for testing, from which we sampled frames to construct the testing set. Our benchmark provides sparse depth maps at three density levels. We configured our SLAM system to track and estimate depth of 1500, 500 and 150 feature points, corresponding to and density of VGA size, which are then used in the depth completion task.
Vi Implementation Details
Our approach was implemented using TensorFlow [1]. With a Nvidia GTX 1080Ti, training takes hours for our VGG11 model and hours for our VGG8 model on KITTI depth completion benchmark (Sec. VA) for 30 epochs; whereas training takes hours and hours on the VOID benchmark (Sec. VB) for 10 epochs. Inference takes 22 ms per image. We used Adam [14] with and to optimize our network endtoend with a base learning rates of for KITTI and for VOID . We decrease the learning rate by half after 18 epochs for KITTI and 6 epochs for VOID , and again after 24 epochs and 8 epochs, respectively. We train our network with a batch size of 8 using a resolution for KITTI and for VOID . We are able to achieve our results on the KITTI benchmark using the following set of weights for each term in our loss function: , , , , and . For the VOID benchmark, we increased to and to . We do not use any data augmentation.
Vii Experiments and Results
Viia KITTI Depth Completion Benchmark
We show quantitative and qualitative comparisons on the unsupervised KITTI depth completion benchmark in Table II and Fig. 4, respectively. The results of the methods listed are taken directly from their papers. We note that [29] only reported their result in their paper and do have have an entry in KITTI depth completion benchmark for their unsupervised model. Hence, we compare qualitatively with the priorart [16]. Our VGG11 model outperforms the stateoftheart [29] on every metric by as much as while using fewer parameters. Our lightweight VGG8 model also outperforms [29] on MAE, RMSE, and iMAE while [29] beat our VGG8 by 2.2% on iRMSE. We note that [29] trains a separate network, using ground truth, to supervise their depth completion model. Moreover, [29] exploits rectified stereoimagery where the pose of the cameras is known; whereas, we learn our pose by jointly training the pose network with our depth predictor. In comparison to [16] (who also uses monocular videos), both our VGG11 and VGG8 model outperforms them on every metric while using much fewer paramters. We also note that the qualitative results of [16] contains artifacts such as apparent scanlines of the Velodyne and “circles” in far regions.
As an introspective exercise, we plot the mean error of our model at varying distances on the KITTI validation set (Fig. 6) and overlay it with the ground truth depth distribution to show that our model performs very well in distances that matter in reallife scenarios. Our performance begins to degrade at distances larger than 80 meters; this is due to the lack of sparse measurements and insufficient parallax – problems that plague methods relying on multiview supervision.
Model  Encoder  Rot.  MAE  RMSE  iMAE  iRMSE 

Scaffolding      443.57  1990.68  1.72  6.43 
+ + (vanilla)  VGG11  Eul.  347.14  1330.88  1.46  4.22 
+ +  VGG11  Eul.  327.84  1262.46  1.31  3.87 
+ +  VGG11  Exp.  312.10  1255.21  1.28  3.86 
+ + +  VGG11  Exp.  305.06  1239.06  1.21  3.71 
+ + +  VGG8  Exp.  308.81  1230.85  1.29  3.84 

We compare variants of our model on the KITTI depth completion validation set. Each model is denoted by its loss function. Regions with missing depth in Scaffolding Only is assigned average depth. It is clear that scaffolding alone (row 1) and our baseline model trained without scaffolding (row 2) do poorly compared to our models that combine both (rows 36). Our full model using VGG11 produces the best overall results and achieves stateoftheart on the test set Table II. Our approach is robust, our lightweight VGG8 model achieves similar performance to our VGG11 model.
ViiB KITTI Depth Completion Ablation Study
We analyze the effect brought by each of our contributions through a quantitative evaluation on the KITTI depth completion validation set (Table III). Our two baseline models, scaffolding and vanilla model trained without scaffolding, perform poorly in comparison to the models that are trained with scaffolding – showcasing the effectiveness of our refinement approach. Although the loss functions are identical, exponential parameterization consistently improves over Euler angles across all metrics. We believe this is due to the regularity of the derivatives of the exponential map [9] compared to other parameterizations – resulting in faster convergence and wider minima during training. While [7, 27, 30] train their pose network using the photometric error with no additional constraint, we show that it is beneficial to impose our pose consistency term (Sec. 6). By constraining the forward and backward poses to be inverse of each other, we obtain a more accurate pose resulting in better depth prediction. Our experiments verify this claim as we see an improvement in across all metrics in Table III. We note that the improvement does not seem significant on KITTI as the motion is mostly planar; however, when predicting nontrivial 6 DoF motion (Sec. VIID), we see a significant boost when employing this term. Our model trained with the full loss function produces the best results (bolded in Table II) and is the stateoftheart for unsupervised KITTI depth completion benchmark. We further propose a VGG8 model that only contains M parameters. Despite having fewer paramters than VGG11, the performance of VGG8 does not degrade by much (see Table II, III, V).
ViiC VOID Depth Completion Benchmark
We evaluate our method on the VOID depth completion benchmark for all three density levels (Table V) using error metrics in Table I. As the photometric loss (Eqn. 4) is largely dependent on obtaining the correct pose, we additionally propose a hybrid model, where the relative camera poses from our visualinertial SLAM system are used to construct the photometric loss to show an upper bound on performance. In contrast to the KITTI, which provides sparse depth density concentrated on the bottom 30% of the image, the VOID benchmark only provides , and densities in sparse depth. Yet, our method is still able to produce reasonable results for indoor scenes with a MAE of cm on density and cm when given only . Since most scenes contain textureless regions, sparse depth supervision becomes important as photometric reconstruction is unreliable. Hence, performance degrades as density decreases. Yet, we degrade gracefully: as density decreases by 10X, our error only doubles. We note that the scaffolding may poorly represent the scene. In the worst case, where it provides no extra information, our method becomes the common depth completion approach. Also, we observe systematic performance improvement in all the evaluation metrics (Table V) when replacing the pose network with SLAM pose. This can be largely attributed to the necessity for the correct pose to minimize photometric error during training. Our pose network may not be able to consistently predict the correct pose due to the challenging motion of the dataset. Fig. 7 shows two sample RGB images with the densified depth images backprojected to 3D, colored, and viewed from a different vantage point.
ViiD VOID Depth Completion Ablation Study
To better understand the effect of rotation parameterization and our pose consistency loss (Eqn. 6) on the depth completion task, we compare variants of our model and again replace the pose network with SLAM pose to show an upperbound on performance. Although exponential outperforms Euler parameterization, we note that both perform much worse than using SLAM pose. However, we observe a performance boost when applying our pose consistency term and our model improves over exponential without pose consistency by as much as . Moreover, it approaches the performance of our model trained using SLAM pose. This trend still holds when density decreases (Table V). This suggests that despite the additional constraint, the pose network still has some difficulties predicting the pose due to the challenging motion. This finding, along with results from Table V, highlights the strength of classical SLAM systems in the deep learning era, which also urges us to develop and test pose networks on the VOID dataset which features nontrivial 6 DoF motion – much more challenging than the mostlyplanar motion in KITTI.
Method  MAE  RMSE  iMAE  iRMSE 

Ma [16]  198.76  260.67  88.07  114.96 
Yang [29]  151.86  222.36  74.59  112.36 
VGG11 PoseNet + Eul.  108.97  212.16  64.54  142.64 
VGG11 PoseNet + Exp.  103.31  179.05  63.88  131.06 
VGG11 PoseNet + Exp. +  85.05  169.79  48.92  104.02 
VGG11 SLAM Pose  73.14  146.40  42.55  93.16 
VGG8 PoseNet + Exp. +  94.33  168.92  56.01  111.54 

We compare the variants of our pose network. SLAM Pose replaces the output of pose network with SLAM estimated pose to gauge an upper bound in performance. When using our pose consistency term with exponential parameterization, our method approaches the performance of our method when using SLAM pose. Note: we trained [16] from scratch using groundtruth pose and adapted [26] to train on monocular sequences. The conditional prior network used in [29] is trained on ground truth from NYUv2 [19].
Density  Pose From  MAE  RMSE  iMAE  iRMSE 

PoseNet  85.05  169.79  48.92  104.02  
SLAM  73.14  146.40  42.55  93.16  
PoseNet  124.11  217.43  66.95  121.23  
SLAM  118.01  195.32  59.29  101.72  
PoseNet  179.66  281.09  95.27  151.66  
SLAM  174.04  253.14  87.39  126.30 

The VOID dataset contains VGA size images () of both indoor and outdoor scenes with challenging motion. For “Pose From”, SLAM refers to relative poses estimated by a SLAM system, and PoseNet refers to relative poses predicted by a pose network.
Viii Discussion
While deep networks have attracted a lot of attention as a general framework to solve an array of problems, we must note that pose may be difficult to learn on datasets with nontrivial 6 DoF motion – which the SLAM community has studied for decades. We hope that VOID will serve as a platform to develop models that can handle challenging motion and further foster fusion of multisensory data. Furthermore, we show that a network can recover the scene geometry from extremely sparse point clouds (e.g. features tracked by SLAM). We also show that improvements can be obtained by leveraging pose from a SLAM system instead of a pose network. These findings motivate a possible mutually beneficial marriage between classical methods and deep learning.
Appendix A VOID Dataset
In the main paper, we introduced the “Visual Odometry with Inertial and Depth” (VOID) dataset with which we propose a new depth completion benchmark. We described the data acquisition process, benchmark setup, and evaluation protocols in Sec. VB and Sec. VIIC. To give some flavor of the VOID dataset, Fig. 9 shows a set of images (top inset) sampled from video sequences in VOID, and output of our visualinertial odometry (VIO) system (bottom), where the blue pointcloud is the sparse reconstruction of the underlying scene and the yellow trace is the estimated camera trajectory.
Two rows of chairs in a classroom  “L” shape formed by desks in a mechanical laboratory 
a brick wall with plants on the ground underneath  stairs 
Appendix B More results on VOID Dataset
In the main paper, we evaluated our approach on the VOID depth completion benchmark in Sec. VIIC, and Sec. VIID provided quantitative results in Table V and IV and qualitative results in Fig. 7. Here, we provide additional qualitative results in Fig. 10 to show how our approach performs on a variety of scenes – both indoor and outdoor – from the VOID dataset. The figure is arranged in two panels of grids, where each panel contains a sample RGB image (left) that is fed to our depth completion network as input, and the corresponding colored pointcloud (right) produced by our approach, viewed at a different vantage point. The pointclouds are obtained by backprojecting the color pixels to the estimated depth. We used an input sparse depth density level of to produce the results. Our approach can provide detailed reconstructions of scenes from both indoor (e.g. right panel, last row: equipment from mechanical lab) and outdoor settings (e.g. left panel: flowers and leaves of plants in garden). It is also able to recover small objects such as the mouse on the desk in the mechanical lab, and structures at very close range (e.g. left panel, last row: staircase located less than half a meter from the camera).
Appendix C Pose Ablation Study
In the main paper, we focus on the depth completion task and hence we evaluate the effects of different pose parameterizations and our pose consistency term by computing error metrics relevant to the recovery of the 3D scene on both the VOID and KITTI depth completion benchmarks. Here, we focus specifically on pose by directly evaluating the pose network on the KITTI odometry dataset in Table VI. We show qualitative results on the trajectory obtained by chaining pairwise camera poses estimated by each pose network in Fig. 11 and provide an analysis of the results in Sec. CB.
Ca Pose Evaluation Metrics
To evaluate the performance of the pose network and its variants, we adopt two most widely used metrics in evaluating simultaneous localization and mapping (SLAM) systems: absolute trajectory error (ATE) and relative pose error (RPE) [25] along with two novel metrics tailored to the evaluation of pose networks.
Given a list of estimated camera poses , where , relative to a fixed world frame, and the list of corresponding ground truth poses , where , ATE reads
(8) 
where the function extracts the translational part of a rigid body transformation. ATE is essentially the root mean square error (RMSE) of the translational part of the estimated pose over all time indices. [32] proposed a “5frame” version of ATE (ATE5F) – the root mean square of ATE of a 5frame sliding window over all time indices, which we also incorporate.
While ATE measures the overall estimation accuracy of the whole trajectory – suitable for evaluating fullfledged SLAM systems where a loop closure module presents, it does not faithfully reflect the accuracy of our pose network since 1) our pose network is designed to estimate pairwise poses, and 2) thus by simply chaining the pose estimates overtime, the pose errors at earlier time instants are more pronounced. Therefore, we also adopt RPE to measure the estimation accuracy locally:
(9) 
which is essentially the endpoint relative pose error of a sliding window averaged over time. By measuring the endpoint relative pose , where , over a sliding window , we are able to focus more on the relative pose estimator (the pose network) itself rather than the overall localization accuracy. In our evaluation, we choose a sliding window of size 1, i.e., . However, RPE is affected only by the accuracy of the translational part of the estimated pose, as we expand the relative pose error:
(10)  
(11) 
leading to , where the rotational part of the estimated pose disappears! Therefore, to better evaluate the rotation estimation, and, more importantly, to study the effect of different rotation parameterization and the pose consistency term, we propose the relative rotation error (RRE) metric:
(12) 
where extracts the rotational part of a rigid body transformation, and is the logarithmic map for rotations.
CB Ablation Study on KITTI Odometry
We perform an ablation study on the effects of our pose parameterizations and our pose consistency in Table VI and provide qualitative results showing the trajectory predicted by our pose network in Fig. 11. We jointly trained our depth completion network and our pose network on the KITTI depth completion dataset and evaluate the pose network on sequence 09 and 10 of the KITTI Odometry dataset.
For sequence 09, our pose network using exponential parameterization performs comparably to Euler angles on the ATE5F and RPE metrics while outperforming Euler by on ATE and on RRE. This result suggests that while within a small window Euler and exponential perform comparably on translation, exponential is a better pose parameterization and globally more correct. We additionally see that exponential outperforms Euler angles on all metrics in sequence 10.
Our best results are achieved using exponential parameterization with our pose consistency term (Eqn. 6): on sequence 09, it outperformed Euler and exponential without pose consistency by and on ATE, and on RPE, and on RRE, respectively, and both by on ATE5F. On sequence 10, it outperformed Euler and exponential by and on ATE, and on RPE, and and on RRE, respectively. It also beat Euler by on RPE and is comparable to exponential on the metric.
Appendix D More Results on KITTI Depth Completion Benchmark
In the main paper, we evaluated our approach on the KITTI depth completion benchmark test set in Sec. VIIA and performed an ablation study on the validation set in Sec. VIIB. Quantitative results are shown in Table II, III and qualitative results in Fig. 4. However, as the KITTI online depth completion benchmark only shows the first 20 samples from the test set, we provide additional qualitative results on a variety of scenes in Fig. 12 to better represent our performance on the test set.
The results in Fig. 12 were produced by our VGG11 model trained using the full loss function (Eqn. 2) with exponential parameterization for rotation. Our method is able to recover pedestrians and thin structures well (e.g. the guard rails, and street poles). Additionally, our network is also able to recover structures that do not have any associated sparse lidar points (e.g structures located on the upper half of the image domain). This can be attributed to our photometric datafidelity term (Sec. IVA). As show in Fig. 3, our network first learns to copy the input scaffolding and to output it as the prediction. It later learns to fuse information from the input image to produce a prediction that includes elements from the scene that is missing from the scaffolding.
Appendix E Network Architecture
We trained our model using two network architectures (Fig. 13) following the late fusion paradigm: (i) our main model using a VGG11 [24] encoder (Table VII), and (ii) our light weight model using a VGG8 [24] encoder (Table VIII). Both encoders use the same decoder (Table IX).
VGG11 Encoder  kernel  channels  resolution  
layer  size  stride  in  out  in  out  # params  input 
Image Branch  
conv1_image  5  2  3  48  1  1/2  3.6K  image 
conv2_image  3  2  48  96  1/2  1/4  41K  conv1_image 
conv3_image  3  1  96  192  1/4  1/4  166K  conv2_image 
conv3b_image  3  1  192  192  1/4  1/4  331K  conv3_image 
conv4_image  3  1  192  384  1/8  1/8  663K  conv3b_image 
conv4b_image  3  1  384  384  1/8  1/8  1.3M  conv4_image 
conv5_image  3  1  384  384  1/16  1/16  1.3M  conv4b_image 
conv5b_image  3  2  384  384  1/16  1/32  1.3M  conv5_image 
Depth Branch  
conv1_depth  5  2  2  16  1  1/2  0.8K  depth 
conv2_depth  3  2  16  32  1/2  1/4  4.6K  conv1_depth 
conv3_depth  3  1  32  64  1/4  1/4  18K  conv2_depth 
conv3b_depth  3  1  64  64  1/4  1/4  37K  conv3_depth 
conv4_depth  3  1  64  128  1/8  1/8  74K  conv3b_depth 
conv4b_depth  3  1  128  128  1/8  1/8  147K  conv4_depth 
conv5_depth  3  1  128  128  1/16  1/16  147K  conv4b_depth 
conv5b_depth  3  2  128  128  1/16  1/32  147K  conv5_depth 
Latent Encoding  
latent      384+128  512  1/32  1/32  0  conv5b_image conv5b_depth 
Total Parameters  5.7M 
VGG8 Encoder  kernel  channels  resolution  

layer  size  stride  in  out  in  out  # params  input 
Image Branch  
conv1_image  5  2  3  48  1  1/2  3.6K  image 
conv2_image  3  2  48  96  1/2  1/4  41K  conv1_image 
conv3b_image  3  2  96  192  1/4  1/8  166K  conv2_image 
conv4b_image  3  2  192  384  1/8  1/16  663K  conv3b_image 
conv5b_image  3  2  384  384  1/16  1/32  1.3M  conv4b_image 
Depth Branch  
conv1_depth  5  2  2  16  1  1/2  0.8K  depth 
conv2_depth  3  2  16  32  1/2  1/4  4.6K  conv1_depth 
conv3b_depth  3  1  32  64  1/4  1/4  18K  conv2_depth 
conv4b_depth  3  1  64  128  1/8  1/16  74K  conv3b_depth 
conv5b_depth  3  2  128  128  1/16  1/32  147K  conv4b_depth 
Latent Encoding  
latent      384+128  512  1/32  1/32  0  conv5b_image conv5b_depth 
Total Parameters  2.4M 

Our lightweight VGG8 [24] encoder following the late fusion paradigm [11, 29] contains only 2.4M parameters as opposed to the 23.8M and 14.8M parameters used by [16] and [29], respectively. The symbol denotes concatenation. Resolution ratio with respect to image size. Note that our lightweight model performs similarly to our VGG11 model.
Decoder  kernel  channels  resolution  
layer  size  stride  in  out  in  out  # params  input 
deconv5  3  2  512  256  1/32  1/16  1.2M  latent 
concat5      256+384+128  768  1/16  1/16  0  deconv5conv4b_imageconv4b_depth 
conv5  3  1  768  256  1/16  1/16  1.8M  concat5 
deconv4  3  2  256  128  1/16  1/8  295K  conv5 
concat4      128+192+64  384  1/8  1/8  0  deconv4conv3b_imageconv3b_depth 
conv4  3  1  384  128  1/8  1/8  442M  concat4 
deconv3  3  2  128  128  1/8  1/4  147K  conv4 
concat3      128+96+32  256  1/4  1/4  0  deconv3conv2_imageconv2_depth 
conv3  3  1  256  64  1/4  1/4  147K  concat3 
deconv2  3  2  64  64  1/4  1/2  37K  conv3 
concat2      64+48+16  128  1/2  1/2  0  deconv2conv1_imageconv1_depth 
conv2  3  1  128  1  1/2  1/2  1.2K  concat2 
output          1/2  1  0  conv2 
Total Parameters  4M 

Our decoder contains 4M parameters. The symbol denotes concatenation and the symbol denotes upsampling. Resolution ratio with respect to image size.
Depth completion networks. Our VGG11 and VGG8 model (Fig. 13) contain a total of M and M parameters, respectively. In comparison to [16] with M parameters and [29] with M, our VGG11 model have a and reduction in parameters over [16] and [29], respectively; our VGG8 model have a and reduction over [16] and [29]. The image and depth branches of the encoder process the image and depth inputs separately – weights are not shared. The results of the encoders are concatenated as the latent representation and passed to the decoder for depth completion. The decoder makes the prediction at 1/2 resolution. The final layer of the decoder is an upsampling layer.
Pose Network. Our pose network takes a pair of images as input and regresses the relative pose between the images. Reversing the order of the image will reverse the relative pose as well. We take the average across the width and height dimensions of the pose network output to produce a 6 element vector. We use 3 elements to model rotation and the rest to model translation.
Including Pose Network in Total Parameters. We follow the network parameter computations of [29] who employs an additional network trained on ground truth for regularization during training. Our pose network (Table X) is an auxiliary network that is only used in training, and not during inference. Hence, we do not include it in the total number of parameters. However, even if we do, our pose network has 1M parameters, making our total for VGG11 to be 10.7M and VGG8 to be 7.4M. Our VGG11 model is still has a reduction in parameter, and our VGG8 a over the 27.8M parameters used by [16]. If we include the auxiliary prior network of [29], containing 10.1M parameters, that is used for regularization during training, then [29] has a total of 28.8M parameters. Our VGG11 model, therefore, has a reduction in parameters over [29] and our VGG8 has a reduction.
Pose Network  kernel  channels  resolution  

layer  size  stride  in  out  in  out  # params  input 
conv1  7  2  6  16  1  1/2  4.7K  image pair 
conv2  5  2  16  32  1/2  1/4  13K  conv1 
conv3  3  2  32  64  1/4  1/8  18K  conv2 
conv4  3  2  64  128  1/8  1/16  74K  conv3 
conv5  3  2  128  256  1/16  1/32  295K  conv4 
conv6  3  2  256  256  1/32  1/64  295K  conv5 
conv7  3  2  256  256  1/64  1/128  295K  conv6 
output  3  1  256  6  1/128  1/128  14K  conv7 
Total Parameters  1M 

Our auxiliary pose network contains 1M parameters and is only used during training to construct the photometric and pose consistency loss (Eqn. 4, 6). The output is averaged along its width and height dimensions to result in a 6 element vector – of which 3 elements are used to compose rotation and the rest for translation.
Footnotes
References
 (2016) TensorFlow: a system for largescale machine learning.. In OSDI, Vol. 16, pp. 265–283. Cited by: §VI.
 (1996) The quickhull algorithm for convex hulls. ACM Transactions on Mathematical Software (TOMS) 22 (4), pp. 469–483. Cited by: §IIIB.
 (1979) Voronoi diagrams from convex hulls. Information processing letters 9 (5), pp. 223–228. Cited by: §IIIB.
 (2018) Deep Convolutional Compressed Sensing for LiDAR Depth Completion. In Asian Conference on Computer Vision (ACCV), Cited by: §II.
 (2018) Learning morphological operators for depth completion. In Advanced Concepts for Intelligent Vision Systems, (eng). Cited by: §II.
 (2018) Propagating confidences through cnns for sparse data regression. In Proceedings of British Machine Vision Conference (BMVC), Cited by: §II, §II, §IIIA.
 (2019) Geosupervised visual depth prediction. IEEE Robotics and Automation Letters 4 (2). Cited by: §II, §VIIB.
 (1981) Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Communications of the ACM 24 (6). Cited by: §II.
 (2015) A compact formula for the derivative of a 3d rotation in exponential coordinates. Journal of Mathematical Imaging and Vision 51 (3), pp. 378–384. Cited by: §VIIB.
 (2019) Hmsnet: hierarchical multiscale sparsityinvariant network for sparse depth completion. IEEE Transactions on Image Processing. Cited by: §II, §II, §IIIA.
 (2018) Sparse and dense data with cnns: depth completion and semantic segmentation. In International Conference on 3D Vision (3DV), Cited by: TABLE VII, TABLE VIII, §II, §II, §IIIC.
 (201101) Visualinertial navigation, mapping and localization: a scalable realtime causal approach. International Journal of Robotics Research. Cited by: §VB, §VB.
 (2015) Posenet: a convolutional network for realtime 6dof camera relocalization. In Proceedings of the IEEE international conference on computer vision, Cited by: §II, §IIIC.
 (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §VI.
 (2009) Epnp: an accurate o (n) solution to the pnp problem. International journal of computer vision 81 (2), pp. 155. Cited by: §II.
 (2019) Selfsupervised sparsetodense: selfsupervised depth completion from lidar and monocular camera. In International Conference on Robotics and Automation (ICRA), Cited by: Fig. 13, TABLE VII, TABLE VIII, Appendix E, Appendix E, §II, §II, §IIIC, §IIIC, Fig. 4, §VB, TABLE II, TABLE IV, §VIIA, TABLE II, TABLE IV.
 (2012) An invitation to 3d vision: from images to geometric models. Vol. 26, Springer Science & Business Media. Cited by: §II, §IIIC, §VB.
 (2007) A multistate constraint kalman filter for visionaided inertial navigation. In Robotics and automation, 2007 IEEE international conference on, pp. 3565–3572. Cited by: §VB.
 (2012) Indoor segmentation and support inference from rgbd images. In ECCV, Cited by: §VB, TABLE IV.
 (2017) PennCOSYVIO: A challenging visual inertial odometry benchmark. In 2017 IEEE International Conference on Robotics and Automation, ICRA 2017, Singapore, Singapore, May 29  June 3, 2017, pp. 3847–3854. Cited by: §VB.
 (2016) Semantically guided depth upsampling. In German Conference on Pattern Recognition, Cited by: TABLE II, TABLE II.
 (2018) The tum vi benchmark for evaluating visualinertial odometry. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1680–1687. Cited by: §VB.
 (2019) Dfusenet: deep fusion of rgb and sparse depth information for image guided dense depth completion. In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pp. 13–20. Cited by: §II.
 (2014) Very deep convolutional networks for largescale image recognition. arXiv preprint arXiv:1409.1556. Cited by: TABLE VII, TABLE VIII, Appendix E.
 (2012) A benchmark for the evaluation of rgbd slam systems. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 573–580. Cited by: §CA.
 (2017) Sparsity invariant cnns. In 2017 International Conference on 3D Vision (3DV), pp. 11–20. Cited by: §II, §II, §IIIA, §VA, TABLE II.
 (2018) Learning depth from monocular videos using direct methods. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2022–2030. Cited by: §VIIB.
 (2004) Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13 (4), pp. 600–612. Cited by: §II, §IVA.
 (2019) Dense depth posterior (ddp) from single image and sparse range. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Cited by: Fig. 13, TABLE VII, TABLE VIII, Appendix E, Appendix E, §II, §II, §IIIC, §IIIC, §VB, TABLE II, TABLE IV, §VIIA, TABLE II, TABLE IV.
 (2018) Geonet: unsupervised learning of dense depth, optical flow and camera pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1983–1992. Cited by: §II, §VIIB.
 (2018) Deep depth completion of a single rgbd image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 175–185. Cited by: §II.
 (2017) Unsupervised learning of depth and egomotion from video. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Cited by: §CA, §II.