Know What Your Neighbors Do:
3D Semantic Segmentation of Point Clouds
In this paper, we present a deep learning architecture which addresses the problem of 3D semantic segmentation of unstructured point clouds. Compared to previous work, we introduce grouping techniques which define point neighborhoods in the initial world space and the learned feature space. Neighborhoods are important as they allow to compute local or global point features depending on the spatial extend of the neighborhood. Additionally, we incorporate dedicated loss functions to further structure the learned point feature space: the pairwise distance loss and the centroid loss. We show how to apply these mechanisms to the task of 3D semantic segmentation of point clouds and report state-of-the-art performance on indoor and outdoor datasets.
In the field of 3D scene understanding, semantic segmentation of 3D point clouds becomes increasingly relevant. Point cloud analysis has found its application in indoor scene understanding and more recently has become an essential component of outdoor applications . This is due to the increasing availability and affordability of 3D sensors such as LiDAR or the Matterport scanner.
In the 2D image domain, for many tasks (including semantic segmentation) convolutional neural networks dominate the field. 2D convolutions allow processing large datasets with high resolution images by taking advantage of the locality of the convolutional operator. They reduce the number of model parameters allowing for deeper and more complex models while being efficient [3, 17, 19, 30].
However point clouds have no inherent order, such as pixel neighborhoods. They are generally sparse in 3D space and the density varies with the distance to the sensor. Moreover, the number of points in a cloud can easily exceed the number of pixels in a high resolution image by multiple orders of magnitude. All these properties make it difficult to process point clouds directly with traditional convolutional neural networks.
Recently, a lot of effort has been put into bridging the success from 2D scene understanding into the 3D world [24, 25, 21, 27, 20, 22, 16, 23, 7]. In this work, we aim to further narrow down the gap between 2D and 3D semantic scene understanding. The straightforward approach of applying CNNs in the 3D space is implemented by preprocessing the point cloud into a voxel representation first in order to apply 3D convolutions on that new representation . However 3D convolutions have drawbacks. Memory and computational time grows cubicly on the number of voxels, restricting approaches to use coarse voxels grids. However, by doing so, one then introduces discretization artifacts (especially for thin structures) and loose geometric information such as point density. Methods directly operating on the point cloud representation (e.g. [20, 22]) produce promising results. However, in these methods, the point neighborhoods over which point features are aggregated are either global  or defined in a static coarse-to-fine approach . Either way, the inherent way of convolutions to capture the local structure has only been transferred in a limited fashion.
In this work, we propose to define neighborhoods in an adaptive manner that is primarily sensitive to the local geometry by using K-means clustering on the input point cloud features in the world space and secondly defining dynamic neighborhoods in the learned feature space using nearest neighbors (kNN). Next comes the observation that a well structured feature space is essential for learning on point clouds. Thus, we add dedicated loss functions which help shaping the feature space at multiple locations in the network.
We showcase the effectiveness of our approach on the task of semantic segmentation of 3D point clouds. We present a comprehensive ablation study evaluating all introduced mechanisms. We apply our method in different scenarios: indoor data from the Stanford 3D Indoor Scene dataset  and ScanNet dataset  as well as outdoor data from the Virtual KITTI 3D dataset [7, 9].
2 Related Work
Before the introduction of deep learning methods, there have been numerous traditional approaches [10, 15, 18, 32] applied to the task of semantically labelling 3D point clouds. Since then, methods relying on deep learning can be roughly split into two groups: methods that impose structure on the unstructured 3D point cloud (by voxelization or projection) followed by standard convolutions, and methods that operate directly on the 3D point clouds:
2.0.1 Voxelized Methods
Up until recently, the standard method to perform semantic segmentation of 3D data involved voxelization. Voxelization approaches transform the unstructured 3D point clouds into regular volumetric 3D grids (voxels). By doing so, 3D convolutions can be directly applied to the voxels [29, 6]. Alternatively, projection approaches map the 3D points into 2D images as seen by virtual cameras. Then, 2D convolutions are applied on the projections [2, 21]. These methods suffer from major disadvantages: the mapping from a sparse representation to a dense one leads to an increased memory footprint. Moreover, the fixed grid resolution results in discretization artifacts and loss of information.
2.0.2 Point Cloud Methods
A new set of methods started with the work of PointNet . PointNet operates directly on 3D points. The key idea is the extraction of point features through a sequence of MLPs processing the points individually (point features) followed by a max-pooling operation that describes the points globally (global features). Point- and global-representations are fused (concatenation + MLP) before making the final label predictions. Many methods followed the PointNet paradigm to operate directly on point clouds. Where PointNet partitions the space into cuboidal blocks of fixed arbitrary size, others use octrees  or kd-trees  to partition the space in a more meaningful way. Furthermore, PointNet does not take into consideration the local geometry and surface information. Clustering has been used in many classical approaches as a way of imposing structure, mostly as a prepossessing step [32, 31]. So  and  were introduced trying to apply hierarchical grouping of the points and incorporate local structure information. The former used farthest point sampling and the latter kd-trees. The authors of  generalize the convolution operator on a spatial neighborhood. Taking it further from local neighborhoods,  organizes the points into superpoints of homogeneous elements and defines relationships between them with the use of graph neural networks on their so-called superpoint graph. In  also cuboidal blocks are used, which act as superpoints and update their respective global features based on the surrounding blocks in space or scale using GRUs.
We now compare our method to the recent PointNet++  which is an hierarchical extension of the original PointNet . PointNet (PN) globally aggregates point features using max-pooling. PN++ forms local groups (or neighborhoods) based on the metric world space and collapses each group onto a single representative point. This technique is repeated to increase the receptive field in each iteration. In our work, we follow a similar approach by iteratively applying feature-space neighborhoods (-modules, introduced later): In every iteration, each point is updated with the aggregated feature information from its kNNs. Repeating this procedure allows the information to flow over many points, one hop per iteration. Unlike , we build the neighborhood based on the feature space, this allows the network to learn the grouping. In PN++, neighborhoods are statically defined by a metric distance.
2.0.3 Feature Networks
As a first step on our network, we learn strong features using a dedicated feature network. The idea of extracting initial strong features is prominent in the field: In , features are learned in 2D image space using CNN for the task of 2.5 semantic segmentation. For the task of object detection, VoxelNet  uses a cascade of PointNet like architectures named Voxel Feature Encoding to obtain a more meaningful feature representation.
3 Our Approach
In the following, we describe the main components of our network. Starting from the initial point features (e.g. position and color), we learn more powerful feature representations using a new Feature Network as described in Sec. 3.1. Next, we define two kinds of neighborhoods (Sec. 3.2.1) within the point cloud, one defined on the learned feature space and one on the input world space. Based on these groupings, we learn regional descriptors which we use to inform the feature points about their neighborhood. Finally, we further enforce structure on the learned feature space by defining two dedicated loss functions (Sec. 3.3.1).
3.1 Feature Network
In this section, we describe our simple yet powerful architecture to learn point features. The goal of this component is to transform input features - such as position and color - into stronger learned features. It can be seen as the distilation of important elements from previous works, in particular PointNet  and ConsolidationUnits . A schematic visualization is shown in Fig. 2.
The network is built from a sequence of feature blocks. Each feature block performs the following tasks: Starting from a set of points, each with feature dimension , it produces refined point features by passing the incoming features through a multi layer perceptron (MLP). Then, a global representation is computed by aggregating all point features using max-pooling. This global feature is again passed through an MLP. Finally, after vertically stacking the global feature times, it is concatenated with the point features. Thus, a single feature block corresponds to a simplified PointNet. An important distinction is that feature blocks can be stacked to arbitrary depth.
In addition to the feature blocks, we introduce pathway connections which allow the individual feature blocks to consult features from previous layers. We distinguish between the point features (local point pathway) and global features (global pathway). Inspired by DenseNet  and ResNet , these features can be combined either by concatenation or summation. Our findings are that concatenation gives slightly inferior results over addition with the cost of a higher memory footprint. At the same time, increasing the number of feature blocks in the network is even more important. Thus, in the interest of scalability, in our final feature network we prefer addition over concatenation and use 17 feature blocks. Our experiments on different number of feature blocks and aggregation functions are summarized in Tab. 6. As a result, the feature network provides us with strong features required for the subsequent components.
We employ two different grouping mechanism to define neighborhoods over the point cloud: The feature space neighborhood is obtained by computing the nearest neighbors (kNN) for each point in the learned feature space, and the world space neighborhood is obtained by clustering points using K-means in the world space. In this context, the world space corresponds to the features of the input point cloud, such as position and color. In the following, we explain the two neighborhoods in more detail and show how they are used to update each point feature.
3.2.1 Feature space neighborhood
(See Fig. 3) From the set of input features of dimensionality , we compute an similarity matrix based on the pairwise -distance between the feature points x. We concatenate the features of each point with the features of its nearest neighbors to construct a kNN tensor. Each slice in the tensor corresponds to an -neighborhood of a feature point . Next, we learn a representation of this neighborhood using an MLP and we generate the updated feature point by applying max-pooling. This procedure is equivalent for each input feature and can be efficiently implemented using convolutions. We will refer to this architecture as an -module.
As such, an -module updates the local feature of a point based on its neighborhood in the feature space. By concatenating multiple -modules, we can increase the receptive field of the operation, one hop at a time, which is comparable to applying multiple convolutions in the image space.
3.2.2 World space neighborhood
Unlike kNN, K-means assigns a variable number of points to a neighborhood. K-means clustering is an iterative method, it alternatively assigns points to the nearest mean which represents the cluster center. Then it recomputes the means based on the assigned points. When applied to the world space, K-means can be seen as a pooling operation which reduces the input space and increases the receptive field by capturing long-range dependencies. Additionally, we are offered a feature point representative per cluster by averaging over all cluster members in the feature space.
We use this functionality in the -module: we perform K-means clustering in the world space, and represent each cluster by the average over all feature points in the cluster. Next, we concatenate this average to all the feature points within the same cluster. We then again apply max-pooling which produces a regional descriptor for this cluster. A visualization is shown in Fig. 5.
3.3 Loss Functions
In this section, we define the loss function that is minimized during the training of our network. The classification loss at the end of our network is realized as the cross entropy between predicted per-class probabilities and one-hot encoded ground truth semantic labels. Beside the classification loss, we introduce two additional losses and which further help to shape the feature space. The final loss is computed as the sum: .
3.3.1 Pairwise Similarity Loss
So far, we assumed that points from the same semantic class are likely to be nearby in the feature space.
The pairwise similarity loss, described in this section, explicitly enforces this assumption.
Similar to , we notice that semantic similarity can be measured directly as a distance in the feature space.
By minimizing pairwise distances, we can learn an embedding where two points sampled from the same object produce nearby points in the feature space.
Equivalently, two points originating from different objects have a large pairwise distance in the feature space.
This goal is illustrated with a 2D embedding in Fig. 4.
All we need is a pairwise distance matrix, which we already compute in the -module (Sec. 3.2). Hence, the distance loss is a natural extension of our network and comes at almost no additional memory cost. For a pair of points with features and , the loss is defined as follows:
where and are threshold values and is the semantic class of point . Finally, the loss is computed as the sum over all pairwise losses .
3.3.2 Centroid Loss
This loss reduces the within-class distance by minimizing the distance between point features and a corresponding representative feature (centroid). It makes the features in the feature space more compact. During training, the representative feature can be computed as the mean feature over all points from the same semantic class. An illustration is shown in Fig. 4. We define the centroid loss as the sum over all , pairs:
where is the total number of points. As distance measure , we found the cosine distance to be more effective than the or distance measures.
4 Implementation Details
In this section, we describe the integration of the aforementioned components into a deep learning architecture. The complete model is depicted in Fig. 5. We start by learning strong features using our Feature Network (Sec. 3.1) producing -dimensional features. See Tab. 6 for an evaluation and discussion on the architecture. We then feed these features into three stacked -modules. The subsequent -module computes a regional descriptors for each cluster (based on world space with descriptors form the feature space). We concatenate the regional descriptors to its corresponding feature points of the second and third -module. The concatenated features are passed through another MLP after which we compute the centroid loss. Finally, we reduce the feature points to 13 dimensions corresponding to the number of semantic classes in our datasets. The pairwise distance loss is computed in the beginning in the network. This informs the networks as early as possible which points should have similar features. This provides early layers a stronger signal of what should be learned and simplifies gradient propagation as the gradient is passed through fewer layers . Although the distance loss could be appended at each point where a similarity matrix is computed, we found it most effective to add it to the second -module. An ablation study is provided in Tab. 1 and shows the contribution of each component in the performance.
|Feature Network (FN) (Sec. 3.1)||82.43||56.96||47.25|
|*3 (Sec. 3.2.1)||81.70||55.14||47.37|
|*3 + (Sec. 3.3.1)||82.51||58.20||49.41|
|FN + *3 +||83.84||58.29||50.56|
|FN + *3 + + (Sec. 3.2.2)||84.31||59.18||50.95|
|FN + *3 + + + (Sec. 3.3.2)||84.19||60.59||51.56|
In this section, we evaluate the performance of our approach on the task of 3D semantic segmentation. We show qualitative and quantitative results and compare them to previous methods. We evaluate our method on multiple datasets: two indoor and one outdoor dataset showing the versatility of our approach. For each dataset, we report the overall accuracy (oAcc), the mean class accuracy (mAcc) and the mean class intersection-over-union (mIoU).
5.1 Indoor Datasets
We evaluate our model on the Stanford Large-Scale 3D Indoor Spaces (S3DIS) dataset  and the ScanNet dataset . Both datasets have recently become popular to evaluate 3D semantic segmentation methods. The S3DIS dataset consists of 6 different indoor areas, totaling to 272 rooms. Each point is labeled as one of 13 semantic classes as shown in Fig. 6. The ScanNet dataset contains 1523 RGB-D scans labeled with 20 different semantic classes.
5.2 Outdoor Dataset
We apply our approach to the vKITTI3D dataset, a large-scale outdoor data set in an autonomous driving setting. Introduced in , this dataset is an adaptation of the synthetic Virtual KITTI dataset  for the task of semantic segmentation of 3D point clouds. It is split in 6 different sequences containing 13 semantic classes listed in Fig. 7.
|S3DIS : 6-fold CV||oAcc||mAcc||mIoU|
5.3 Training Details
For the experiments on the S3DIS dataset, we follow a similar training procedure as  i.e. we split the rooms in blocks of 1 m on the ground plane. From each block we randomly sample 4096 points. During evaluation, we predict class labels for all points. Additionally, we add translation augmentation to the block positions. Each point is represented by a 9D feature vector consisting of the position , color and normalized coordinates as in . The hyperparameters of our method are set as follows: for the kNN-clustering, we set and use the -distance measure. For K-means we dynamically set where N is the number of points per block. We report scores on a 6-fold cross validation across all areas in Tab. 2 along with the detailed scores of per class IoU in Tab. 4. Additionally, we provide scores for Area 5 in Tab. 3 to compare ourself to  and .
On the ScanNet dataset , we use the reference implementation of PointNet++ to train and evaluate our model. This approach allows us to focus on the comparison of the models while abstracting from the training procedures. All hyperparameters remain the same. The results are shown in Tab. 5
On the VKITTI3D dataset, we follow again the same training procedure as on the S3DIS dataset. The scenes are split into blocks of 9 m on the ground plane. Since the dataset is much sparser, we set the number of points sampled per block to N = 256. Training on a 6-fold cross validation is performed as in . We use the same input features as in the indoor dataset and additionally, we analyze how well our method performs if we take into consideration only geometric features (xyz-position) while leaving out color information. This is an interesting experiment, as color is not always imminently available e.g. point clouds from laser scanners. We show quantitative results in Tab. 7. Qualitative results are shown in Fig. 7.
We have presented a deep learning framework for 3D semantic segmentation of point clouds. Its main components are - and -modules. They allow to incorporate neighborhood information from the feature space and from the world space. We have also introduced the pairwise distance loss and the centroid loss in the context of 3D semantic segmentation. The presented method produces state-of-the-art results on current indoor and outdoor datasets.
This project was funded by the ERC Consolidator Grant DeeViSe (ERC-2017-CoG-773161).
|VKITTI3D : 6-fold CV||oAcc||mAcc||mIoU|
|PointNet  from ||63.3||29.9||17.9|
|Ours ( + color)||79.69||57.59||35.59|
-  Adam W Harley, Konstantinos G. Derpanis, I.K.: Segmentation-Aware Convolutional Networks Using Local Attention Masks. In: IEEE International Conference on Computer Vision (ICCV) (2017)
-  Boulch, A., Saux, B.L., Audebert, N.: Unstructured Point Cloud Semantic Labeling Using Deep Segmentation Networks. In: Eurographics Workshop on 3D Object Retrieval (2017)
-  Chen, L., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. arXiv preprint arXiv:1606.00915 (2016)
-  Chopra, S., Hadsell, R., LeCun, Y.: Learning a Similarity Metric Discriminatively, with Application to Face Verification. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2005)
-  Dai, A., Chang, A.X., Savva, M., Halber, M., Funkhouser, T., Nießner, M.: ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. In: Proc. Computer Vision and Pattern Recognition (CVPR), IEEE (2017)
-  Daniel Maturana, S.S.: VoxNet: A 3D Convolutional Neural Network for Real-Time Object Recognition. In: IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2015)
-  Engelmann, F., Kontogianni, T., Hermans, A., Leibe, B.: Exploring Spatial Context for 3D Semantic Segmentation of Point Clouds. In: IEEE International Conference on Computer Vision, 3DRMS (ICCV) Workshop (2017)
-  Engelmann, F., Stückler, J., Leibe, B.: SAMP: shape and motion priors for 4d vehicle reconstruction. In: IEEE Winter Conference on Applications of Computer Vision, WACV (2017)
-  Gaidon, A., Wang, Q., Cabon, Y., Vig, E.: Virtual Worlds as Proxy for Multi-Object Tracking Analysis. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
-  Hackel, T., Wegner, J.D., Schindler, K.: Fast Semantic Segmentation of 3D Points Clouds with Strongly Varying Density. ISPRS (2016)
-  He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
-  Huang, G., Liu, Z., van der Maaten, L., Weinberger, K.Q.: Densely Connected Convolutional Networks. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
-  Huang, Q., Wang, W., Neumann, U.: Recurrent Slice Networks for 3D Segmentation on Point Clouds. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
-  Iro, A., Ozan, S., Amir, R.Z., Helen, J., Ioannis, B., Martin, F., Silvio, S.: 3D Semantic Parsing of Large-Scale Indoor Spaces. In: IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
-  Lai, K., Bo, L., Fox, D.: Unsupervised feature learning for 3D scene labeling. In: IEEE International Conference on Robotics and Automation (ICRA) (2014)
-  landrieu, L., Simonovsky, M.: Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)
-  Long, J., Shelhamer, E., Darrell, T.: Fully Convolutional Networks for Semantic Segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
-  Munoz, D., Vandapel, N., Hebert, M.: Directional Associative Markov Network for 3-D Point Cloud Classification. In: International Symposium on 3D Data Processing, Visualization and Transmission (2008)
-  Noh, H., Hong, S., Han, B.: Learning Deconvolution Network for Semantic Segmentation. In: IEEE International Conference on Computer Vision (ICCV) (2015)
-  Qi, C.R., Su, H., Mo, K., Guibas, L.J.: PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
-  Qi, C.R., Su, H., NieÃner, M., Dai, A., Yan, M., Guibas, L.J.: Volumetric and Multi-View CNNs for Object Classification on 3D Data. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
-  Qi, C.R., Yi, L., Su, H., Guibas, L.J.: PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. Conference on Neural Information Processing Systems (NIPS) (2017)
-  Qi, X., Liao, R., Ya, J., Fidler, S., Urtasun, R.: 3D Graph Neural Networks for RGBD Semantic Segmentation. In: IEEE International Conference on Computer Vision (ICCV) (2017)
-  Roman, K., Victor, L.: Escape from Cells: Deep Kd-Networks for the Recognition of 3D Point Cloud Models. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
-  Simonovsky, M., Komodakis, N.: Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017)
-  Tatarchenko, M., Dosovitskiy, A., Brox, T.: Octree Generating Networks: Efficient Convolutional Architectures for High-resolution 3D Outputs. IEEE International Conference on Computer Vision (ICCV) (2017)
-  Tchapmi, L.P., Choy, C.B., Armeni, I., Gwak, J., Savarese, S.: SEGCloud: Semantic Segmentation of 3D Point Clouds. In: International Conference on 3D Vision (3DV) (2017)
-  Wang, Y., Sun, Y., Liu, Z., Sarma, S.E., Bronstein, M.M., Solomon, J.M.: Dynamic Graph CNN for Learning on Point Clouds. arXiv preprint arXiv:1801.07829 (2018)
-  Wu, Z., Song, S., Khosla, A., Yu, F., Zhang, L., Tang, X., Xiao, J.: 3D ShapeNets: A Deep Representation for Volumetric Shape Modeling. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
-  Wu, Z., Shen, C., van den Hengel, A.: High-performance Semantic Segmentation Using Very Deep Fully Convolutional Networks. arXiv preprint arXiv:1604.04339 (2016)
-  Xiong, X., Munoz, D., Andrew, J., Hebert, B.M.: 3-D Scene Analysis via Sequenced Predictions over Points and Regions. In: IEEE International Conference on Robotics and Automation (ICRA) (2011)
-  Xiong, X., Munoz, D., Bagnell, J.A., Hebert, M.: 3-D Scene Analysis via Sequenced Predictions over Points and Regions. In: IEEE International Conference on Robotics and Automation (ICRA) (2011)
-  Zhou, Y., Tuzel, O.: VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection. IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2018)