A-CNN: Annularly Convolutional Neural Networks on Point Clouds
Analyzing the geometric and semantic properties of 3D point clouds through the deep networks is still challenging due to the irregularity and sparsity of samplings of their geometric structures. This paper presents a new method to define and compute convolution directly on 3D point clouds by the proposed annular convolution. This new convolution operator can better capture the local neighborhood geometry of each point by specifying the (regular and dilated) ring-shaped structures and directions in the computation. It can adapt to the geometric variability and scalability at the signal processing level. We apply it to the developed hierarchical neural networks for object classification, part segmentation, and semantic segmentation in large-scale scenes. The extensive experiments and comparisons demonstrate that our approach outperforms the state-of-the-art methods on a variety of standard benchmark datasets (e.g., ModelNet10, ModelNet40, ShapeNet-part, S3DIS, and ScanNet).
Nowadays, the ability to understand and analyze 3D data is becoming increasingly important in computer vision and computer graphics communities. During the past few years, the researchers have applied deep learning methods to analyze 3D objects inspired by the successes of these techniques in 2D images and 1D texts. Traditional low-level handcrafted shape descriptors suffer from not being able to learn the discriminative and sufficient features from 3D shapes . Recently, deep learning techniques have been applied to extract hierarchical and effective information from 3D shape features captured by low-level descriptors [20, 6]. 3D deep learning methods are widely used in shape classification, segmentation, and recognition, etc. But all these methods are still constrained by the representation power of the shape descriptors.
One of the main challenges to directly apply deep learning methods to 3D data is that 3D objects can be represented in different formats, i.e., regular / structured representation (e.g., multi-view images and volumes), and irregular / unstructured representation (e.g., point clouds and meshes). There are extensive approaches based on regular / structured representation, such as multi-view convolutional neural networks (CNNs) [34, 27, 10] and 3D volumetric / grid CNN methods and its variants [40, 27, 29, 37, 38, 16, 9]. These methods can be conveniently developed and implemented in 3D data structure, but they easily suffer from the heavy computation and large memory expense. So it is better to define the deep learning computations based on 3D shapes directly, i.e., irregular / unstructured representation, such as point cloud based methods [26, 28, 13, 32, 3, 18, 19, 35, 17, 44, 36, 8, 42]. However, defining the convolution on the irregular / unstructured representation of 3D objects is not an easy task. Very few methods on point clouds have defined an effective and efficient convolution on each point. Meanwhile, several approaches have been proposed to develop convolutional networks on 2D manifolds [22, 4, 24, 41]. Their representations (e.g., 3D surface meshes) have point positions as well as connectivities, which makes it relatively easier to define the convolution operator on them.
In this work, we present a new method to define and compute convolutions directly on 3D point clouds effectively and efficiently by the proposed annular convolutions. This new convolution operator can better capture local neighborhood geometry of each point by specifying the (regular and dilated) ring-shaped structures and directions in the computation. It can adapt to the geometric variability and scalability at the signal processing level. Then, we apply it along with the developed hierarchical neural networks to object classification, part segmentation, and semantic segmentation in large-scale scene as shown in Fig. 1. The key contributions of our work are as follows:
We propose a new approach to define convolutions on point cloud. The proposed annular convolutions can define arbitrary kernel sizes on each local ring-shaped region, and help to capture better geometric representations of 3D shapes;
We propose a new multi-level hierarchical method based on dilated rings, which leads to better capturing and abstracting shape geometric details. The new dilated strategy on point clouds benefits our proposed closed-loop convolutions and poolings;
Our proposed network models present new state-of-the-art performance on object classification, part segmentation, and semantic segmentation of large-scale scenes using a variety of standard benchmark datasets.
2 Related Work
Due to the scope of our work, we focus only on recently related deep learning methods, which are proposed on different 3D shape representations.
Volumetric Methods. One traditional way to analyze a 3D shape is to convert it into the regular volumetric occupancy grid and then apply 3D CNNs [40, 27]. The major limitation of these approaches is that 3D convolutions are more expensive in computations than 2D cases. In order to make the computation affordable, the volume grid size is usually in a low resolution. However, lower resolution means loosing some shape geometric information, especially in analyzing large-scale 3D shapes / scenes. To overcome these problems, octree-based methods [29, 37, 38] have been proposed to allow applying 3D CNNs on higher / adaptive resolution grids. PointGrid  is a 3D CNN that incorporates a constant number of points within each grid cell and allows it to learn better local geometric details. Similarly, Hua et al.  presented a 3D convolution operator based on a uniform grid kernel for semantic segmentation and object recognition on point clouds.
Point Cloud based Methods. PointNet  is the first attempt of applying deep learning directly on point clouds. PointNet model is invariant to the order of points, but it considers each point independently without including local region information. PointNet++  is a hierarchical extension of PointNet model and learns local structures of point clouds at different scales. But  still considers every point in its local region independently. In our work, we address the aforementioned issues by defining the convolution operator that learns the relationship between neighboring points in a local region, which helps to better capture the local geometric properties of the 3D object.
Klokov et al.  proposed a new deep learning architecture called Kd-networks, which uses kd-tree structure to construct a computational graph on point clouds. KCNet  improves PointNet model by considering the local neighborhood information. It defines a set of learnable point-set kernels for local neighboring points and presents a pooling method based on a nearest-neighbor graph. PCNN  is another method to apply convolutional neural networks to point clouds by defining extension and restriction operators, and mapping point cloud functions to volumetric functions. SO-Net  is a permutation invariant network that utilizes spatial distribution of point clouds by building a self-organizing map. There are also some spectral convolution methods on point clouds, such as SyncSpecCNN  and spectral graph convolution . Point2Sequence  learns the correlation of different areas in a local region by using attention mechanism, but it does not propose a convolution on point clouds. PointCNN  is a different method that proposes to transform neighboring points to the canonical order and then apply convolution.
Recently, there are several approaches proposed to process and analyze large-scale point clouds from indoor and outdoor environments. Engelmann et al.  extended PointNet model to exploit the large-scale spatial context. Ye et al.  proposed a pointwise pyramid pooling to aggregate features at local neighborhoods as well as two-directional hierarchical recurrent neural networks (RNNs) to learn spatial contexts. However, these methods do not define convolutions on large-scale point clouds to learn geometric features in the local neighborhoods. TangentConv  is another method that defines the convolution on point clouds by projecting the neighboring points on tangent planes and applying 2D convolutions on them. The orientation of the tangent image is estimated according to the local point / shape curvature, but as we know the curvature computation on the local region of the point clouds is not stable and not robust (see the discussion in Sec. 3.4), which makes it orientation-dependent. Instead, our method proposes an annular convolution, which is invariant to the orientations of local patches. Also, ours does not require additional input features while theirs needs such features (e.g., depth, height, etc.).
Mesh based Methods. Besides point cloud based methods, several approaches have been proposed to develop convolutional networks on 3D meshes for shape analysis. Geodesic CNN  is an extension of the Euclidean CNNs to non-Euclidean domains and is based on a local geodesic system of polar coordinates to extract local patches. Anisotropic CNN  is another generalization of Euclidean CNNs to non-Euclidean domains, where classical convolutions are replaced by projections over a set of oriented anisotropic diffusion kernels. Mixture Model Networks (MoNet)  generalizes deep learning methods to non-Euclidean domains (graphs and manifolds) by combining previous methods, e.g., classical Euclidean CNN, Geodesic CNN, and Anisotropic CNN. MoNet proposes a new type of kernel in parametric construction. Directionally Convolutional Networks (DCN)  applies convolution operation on the triangular mesh of 3D shapes to address part segmentation problem by combining local and global features. Lastly, Surface Networks  propose upgrades to Graph Neural Networks to leverage extrinsic differential geometry properties of 3D surfaces for increasing their modeling power.
In this work, we propose a new end-to-end framework named as annularly convolutional neural networks (A-CNN) that leverages the neighborhood information to better capture local geometric features of 3D point clouds. In this section, we introduce main technique components of the A-CNN model on point clouds that include: regular and dilated rings, constraint-based k-nearest neighbors (k-NN) search, ordering neighbors, annular convolution, and pooling on rings.
3.1 Regular and Dilated Rings on Point Clouds
To extract local spatial context of the 3D shape, PointNet++  proposes multi-scale architecture. The major limitation of this approach is that multiple scaled regions may have overlaps (i.e., same neighboring points could be duplicately included in different scaled regions), which reduces the performance of the computational architecture. Overlapped points at different scales lead to redundant information at the local region, which limits a network to learn more discriminative features.
In order to address the above issue, our proposed framework is aimed to leverage a neighborhood at different scales more wisely. We propose two ring-based schemes, i.e., regular rings and dilated rings. Comparing to multi-scale strategy, the ring-based structure does not have overlaps (no duplicated neighboring points) at the query point’s neighborhood, so that each ring contains its own unique points, as illustrated in Sec. A of Supplementary Material.
The difference between regular rings and dilated rings is that dilated rings have empty space between rings. The idea of proposed dilated rings is inspired by dilated convolutions on image processing , which benefits from aggregating multi-scale contextual information. Although each ring may define the same number of computation / operation parameters (e.g., number of neighboring points), the coverage area of each ring is different (i.e., dilated rings will have larger coverage than the regular rings) as depicted in Fig. 6. Regular rings can be considered as dilated rings with the dilation factor equal to 0.
The proposed regular rings and dilated rings will contribute to neighboring point search, convolution, and pooling in the follow-up processes. First, for k-NN algorithm, we constrain search areas in the local ring-shaped neighborhood to ensure no overlap. Second, the convolutions defined on rings cover larger areas with the same kernel sizes without increasing the number of convolution parameters. Third, the regular / dilated ring architectures will help to aggregate more discriminative features after applying max-pooling at each ring of the local region. We will discuss them in more detail in the following subsections.
To justify the aforementioned statements, we will compare multi-scale approach with our proposed multi-ring scheme on object classification task in the ablation study (Sec. 5.4). The results show that ring-based structure captures better local geometric features than previous multi-scale method, since it achieves higher accuracy.
3.2 Constraint-based K-NN Search
In the original PointNet++ model, the ball query algorithm returns the first neighbors found inside a search ball specified by a radius and query point , so that it cannot guarantee that the closest points will always be found. However, our proposed k-NN search algorithm guarantees returning closest points inside the searching area by using the Euclidean metric. Each ring is defined by two parameters: the inner radius and the outer radius (in Fig. 6); therefore, the constraint-based k-NN search ensures that the closest and unique points will be found in each ring.
3.3 Ordering Neighbors
In order to learn relationships between neighboring points in a local regions, we need first to order points in a clockwise / counterclockwise manner and then apply annular convolutions. Our proposed ordering operator consists of two main steps: projection and ordering. The importance of the projection before ordering is that the dot product has its restriction in ordering points. By projecting points on a tangent plane at a query point , we effectively order neighbors in clockwise / counterclockwise direction by taking use of cross product and dot product together. The detailed explanations of normal estimation, orthogonal projection, and ordering are given in the following subsections.
3.3.1 Normal Estimation on Point Clouds
Normal is an important geometric property of a 3D shape. We use it as a tool for projecting and ordering neighboring points at a local domain. The simplest normal estimation method approximates the normal at the given point by calculating the normal of the local tangent plane at that point, which becomes a least-square plane fitting estimation problem . To calculate normal , one needs to compute eigenvalues and eigenvectors of the covariance matrix as:
where is the number of neighboring points s around query point (e.g., = 10 in our experiments), and are the eigenvalue and eigenvector of the covariance matrix , respectively. The covariance matrix is symmetric and positive semi-definite. The eigenvectors form an orthogonal frame, in respect to the local tangent plane . The eigenvector that corresponds to the smallest eigenvalue is the estimated normal .
3.3.2 Orthogonal Projection
3.3.3 Counterclockwise Ordering
Firstly, we use the geometric definition of the dot product to compute the angle between two vectors (i.e., starts from the query point and connects with a randomly starting point, such as ) and (i.e., starts from the query point and connects with other neighboring points ):
We know that lies in , which corresponds to angles between . In order to sort the neighboring points around the query point between , we must to decide which semicircle the considered point belongs to as follows:
where is , and is .
Then, we can recompute the cosine value of the angle as:
Now the values of the angles lie in , which maps angles between .
Finally, we sort neighboring points by descending the value of to obtain the counterclockwise order. Fig. 3 (b) illustrates the process of ordering in a local neighborhood. The neighboring points can be ordered in the clockwise manner, if we sort neighboring points by ascending the value of .
Our experiments show in Sec. 5.4 that ordering points in the local regions is an important step in our framework and our model achieves better classification accuracy with ordered points than without ordering them.
3.4 Annular Convolution on Rings
Through the previous computation, we have the ordered neighbors represented as an array . In order to develop the annular convolution, we need to loop the array of neighbors with respect to the size of the kernel (e.g., , , …) on each ring. For example, if the convolutional kernel size is , we need to take the first two neighbors and concatenate them with the ending elements in the original array to construct a new circular array . Then, we can perform the standard convolutions on this array as shown in Fig. 3 (c).
There are some nice properties of the proposed annular convolutions as follows: (1) The annular convolution is invariant to the orientation of the local patch. That is because the neighbors are organized and ordered in a closed loop in each ring by concatenating the beginning with the end of the neighboring points’ sequence. Therefore, we can order neighbors based on any random starting position, which does not negatively affect the convolution results. Compared with some previous convolutions defined on 3D shapes [4, 41, 35], they all need to compute the real principal curvature direction as the reference direction to define the local patch operator, which is not robust and cumbersome. In particular, 3D shapes have large areas of flat and spherical regions, where the curvature directions are arbitrary. (2) As we know, in reality, the normal direction flipping issues are widely existing in point clouds, especially the large-scale scene datasets. Under the annular convolution strategy, no matter the neighboring points are ordered in clockwise or counterclockwise manner, the results are the same. (3) Another advantage of annular convolution is that we can define an arbitrary kernel size, instead of just kernels [26, 28]. Therefore, the annular convolution can provide the ability to learn the relationship between ordered points inside each ring as shown in Fig. 3 (c).
Annular convolutions can be applied on both regular and dilated rings. By applying annular convolutions with the same kernel size on different rings, we can cover and convolve larger areas by using the dilated structure, which helps us to learn larger spatial contextual information in the local regions. The importance of annular convolutions is shown in the ablation study in Sec. 5.4.
3.5 Pooling on Rings
After applying a set of annular convolutions sequentially, the resulting convolved features encode information about its closest neighbors in each ring as well as spatial remoteness from a query point. Then we aggregate the convolved features across all neighbors on each ring separately. We apply the max-pooling strategy in our framework. Our proposed ring-based scheme allows us to aggregate more discriminative features. The extracted max-pooled features contain the encoded information about neighbors and the relationship between them in the local region, unlike the pooling scheme in PointNet++ , where each neighbor is considered independently from its neighbors. In our pooling process, the non-overlapped regions (rings) will aggregate different types of features in each ring, which can uniquely describe each local region (ring) around the query point. The multi-scale approach in PointNet++ does not guarantee this and might aggregate the same features at different scales, which is redundant information for a network. The (regular and dilated) ring-based scheme helps to avoid extracting duplicate information but rather promotes extracting multi-level information from different regions (rings). This provides a network with more diverse features to learn from. After aggregating features at different rings, we concatenate and feed them to another abstract layer to further learn hierarchical features.
4 A-CNN Architecture
Our proposed A-CNN model follows a design where the hierarchical structure is composed of a set of abstract layers. Each abstract layer consists of several operations performed sequentially and produces a subset of input points with newly learned features. Firstly, we subsample points by using Farthest Point Sampling (FPS) algorithm  to extract centroids randomly distributed on the surface of each object. Secondly, our constraint-based k-NN extracts neighbors of a centroid for each local region (i.e., regular / dilated rings) and then we order neighbors in a counterclockwise manner using projection. Finally, we apply sequentially a set of annular convolutions on the ordered points and max-pool features across neighbors to produce new feature vectors, which uniquely describe each local region.
Given the point clouds of 3D shapes, our proposed end-to-end network is able to classify and segment the objects. In the following, we discuss the classification and segmentation network architectures on 3D point clouds.
4.1 Classification Network
The classification network is illustrated at the top of Fig. 4. It consists of two major parts: encoder and classification. The encoder extracts features from each ring independently inside every layer and concatenates them at the end to process further to extract high-level features. The proposed architecture includes both regular rings and dilated rings. We end up using two rings per layer, because it gives us pretty good experimental results as shown in the Sec. 5. It can be easily extended to more than two rings per layer, if necessary.
We use regular rings in the first layer and dilated rings in the second layer in the encoder. Annular convolutions with the kernel sizes and stride 1 are applied in the first two layers, followed by a batch normalization  (BN) and a rectified linear unit  (ReLU). Different rings of the same query point are processed in parallel. Then, the aggregated features from each ring concatenate together to propagate to the next layer. The last layer in the encoder performs convolutions with kernel sizes followed by BN and ReLU layers, where only spatial positions of the sampled points are considered. After that aggregated high-level features are fed to the set of fully-connected layers with integrated dropout  and ReLU layers to calculate probability of each class. The output size of the classification network is equal to the number of classes in the dataset.
4.2 Segmentation Network
The segmentation network shares encoder part with the classification network as shown in Figure 4. In order to predict the segmentation label per point, we need to upsample the sampled points in the encoder back to the original point cloud size. As pointed out by , the consecutive feature propagation proposed by  is not the most efficient approach. Inspired from , we propagate features from different levels from the encoder directly to the original point cloud size, and concatenate them by allowing the network to learn the most important features from different levels as well as to learn the relationship between them.
The output of each level has different sizes due to the hierarchical feature extractions, so we have to restore hierarchical features from each level back to the original point size by using an interpolation method . The interpolation method is based on the inverse squared Euclidean distance weighted average of the three nearest neighbors as:
where is an inverse squared Euclidean distance weight.
Then, we concatenate upsampled features from different levels and pass them through convolution to reduce feature space and learn the relationship between features from different levels. Finally, the segmentation class distribution for each point is calculated.
We evaluate our A-CNN model on various tasks such as point cloud classification, part segmentation, and large-scale scene segmentation. In the following subsections, we demonstrate more details on each task. It is noted that for the comparison experiments, best results in the tables are shown in bold font.
All models in this paper are trained on a single NVIDIA Titan Xp GPU with 12 GB GDDR5X. The training time of our model is faster than that of PointNet++ model. More details about the network configurations, training settings and timings in our experiments can be found in Sec. B and Tab. 5 of Supplementary Material. The source code of the framework will be made available later.
5.1 Point Cloud Classification
We evaluate our classification model on two datasets: ModelNet10 and ModelNet40 . ModelNet is a large-scale 3D CAD model dataset. ModelNet10 is a subset of ModelNet dataset that consists of 10 different classes with 3991 training and 908 testing objects. ModelNet40 includes 40 different classes with 9843 objects for training and 2468 objects for testing. Point clouds with 10,000 points and normals are sampled from meshes, normalized into a unit sphere, and provided by .
For experiments on ModelNet10 and ModelNet40, we sample 1024 points with normals, where normals are only used to order points in the local region. For data augmentation, we randomly scale object sizes, shift object positions, and perturb point locations. For better generalization, we apply point shuffling in order to generate different centroids for the same object at different epochs.
In Tab. 1, we compare our method with several state-of-the-art methods in the shape classification results on both ModelNet10 and ModelNet40 datasets. Our model achieves better accuracy among the point cloud based methods (with 1024 points), such as PointNet , PointNet++  (5K points + normals), Kd-Net (depth 15) , Pointwise CNN , KCNet , PointGrid , PCNN , and PointCNN . Our model is slightly better than Point2Sequence  on ModelNet10 and shows comparable performance on ModelNet40.
Meanwhile, our model performs better than other volumetric approaches, such as O-CNN  and AO-CNN ; while we are a little worse than SO-Net , which uses denser input points, i.e., 5000 points with normals as the input (1024 points in our A-CNN); MVCNN-MultiRes , which uses multi-view 3D volumes to represent an object (i.e., 20 views of volume); the VRN Ensemble , which involves an ensemble of six models.
We also provide some feature visualization results in Sec. C of Supplementary Material, including global feature (e.g., t-SNE clustering) visualization and local feature (e.g., the magnitude of the gradient per point) visualization.
|different methods with additional input or more points|
|VRN Ensemble ||-||97.1||-||95.5|
|point cloud based methods with 1024 points|
|Kd-Net (depth 15) ||93.5||94.0||88.5||91.8|
|Pointwise CNN ||-||-||81.4||86.1|
5.2 Point Cloud Segmentation
We evaluate our segmentation model on ShapeNet-part  dataset. The dataset contains 16,881 shapes from 16 different categories with 50 label parts in total. The main challenge of this dataset is that all categories are highly imbalanced. There are 2048 points sampled for each shape from the dataset, where most shapes contain less than six parts. We follow the same training and testing splits provided in [26, 43]. For data augmentation, we perturb point locations with the point shuffling for better generalization.
We evaluate our segmentation model with two different inputs. One of the models is trained without feeding normals as additional features and the other model is trained with normals as additional features. The quantitative results are provided in Tab. 2, where mean IoU (Intersection-over-Union) is reported. The qualitative results are visualized in Fig. -6. Our approach with point locations only as input outperforms PointNet , Kd-Net , KCNet , and PCNN ; and shows slightly worse performance comparing to PointGrid  (volumetric method) and PointCNN . Meanwhile, our model achieves the best performance with the input of point locations and normals, compared with PointNet++ , SyncSpecCNN , SO-Net , SGPN , O-CNN , RSNet , and Point2Sequence . The more detailed quantitative results (e.g., per-category IoUs) and more visualization results are provided in Sec. E of Supplementary Material.
5.3 Semantic Segmentation in Scenes
We also evaluate our segmentation model on two large-scale indoor datasets Stanford 3D Large-Scale Indoor Spaces (S3DIS)  and ScanNet . S3DIS contains 6 large-scale indoor areas with 271 rooms sampled from 3 different buildings, where each point has the semantic label that belongs to one of the 13 categories. ScanNet includes 1513 scanned indoor point clouds, where each voxel has been labeled with one of the 21 categories.
We employ the same training and testing strategies as PointNet  on S3DIS, where we use 6-fold cross validation over all six areas. The evaluation results are reported in Tab. 2, and qualitative results are visualized in Fig. 5. Our model demonstrates better segmentation results compared with PointNet , MS+CU (2) , G+RCU , 3P-RNN , SPGraph , and TangentConv . However, our model performs slightly worse than PointCNN  due to their non-overlapping block sampling strategy with paddings which we do not use. Meanwhile, our approach shows the best segmentation results on ScanNet  and achieves the state-of-the-art performance, compared with PointNet , PointNet++ , TangentConv , and PointCNN  according to Tab. 2.
|MS+CU (2) ||-||-||79.2||-|
Note: * TangentConv  OA on S3DIS Area 5 is 82.5 (as reported in their paper), which is worse compared with our OA of 85.5.
5.4 Ablation Study
The goal of our ablation study is to show the importance of the proposed technique components (in Sec. 3) in our A-CNN model. We evaluate three proposed components, such as rings without overlaps (Sec. 3.1), ordering (Sec. 3.3), and annular convolution (Sec. 3.4) on the classification task of ModelNet40 dataset as shown in Tab. 4. In the first experiment, we replace our proposed constraint-based k-NN on ring regions with ball query in , but keep ordering and annular convolutions on. In the second and third experiments, we turn off either annular convolutions or ordering, respectively; and keep the rest two components on. Our experiments show that the proposed ring-shaped scheme contributes the most to our model. It is because multi-level rings positively affect annular convolutions. Finally, A-CNN model with all three components (i.e., rings without overlaps, ordering, and annular convolutions) achieves the best results. We also discover that reducing overlap / redundancy in multi-scale scheme can improve existing methods. We evaluate the original PointNet++  with and without overlap as shown in Sec. A of Supplementary Material.
|A-CNN (without rings / with overlap)||89.2||91.7|
|A-CNN (without annular conv.)||89.2||91.8|
|A-CNN (without ordering)||89.6||92.0|
|A-CNN (with all components)||90.3||92.6|
In this work, we propose a new A-CNN framework on point clouds, which can better capture local geometric information of 3D shapes. Through extensive experiments on several benchmark datasets, our method has achieved the state-of-the-art performance on point cloud classification, part segmentation, and large-scale semantic segmentation tasks. Since our work does not solely focus on large-scale scene datasets, we will explore some new deep learning architectures to improve the current results. We will also investigate to apply the proposed framework on large-scale outdoor datasets in our future work.
Acknowledgment. We would like to thank the reviewers for their valuable comments. This work was partially supported by the NSF IIS-1816511, CNS-1647200, OAC-1657364, OAC-1845962, Wayne State University Subaward 4207299A of CNS-1821962, NIH 1R56AG060822-01A1, and ZJNSF LZ16F020002.
-  E. Ahmed, A. Saint, A. Shabayek, K. Cherenkova, R. Das, G. Gusev, D. Aouada, and B. Ottersten. Deep learning advances on different 3D data representations: A survey. arXiv preprint arXiv:1808.01462, 2018.
-  I. Armeni, O. Sener, A. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese. 3D semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1534–1543, 2016.
-  M. Atzmon, H. Maron, and Y. Lipman. Point convolutional neural networks by extension operators. ACM Transactions on Graphics, 37(4):71:1–71:12, 2018.
-  D. Boscaini, J. Masci, E. Rodolà, and M. Bronstein. Learning shape correspondence with anisotropic convolutional neural networks. In Advances in Neural Information Processing Systems, pages 3189–3197, 2016.
-  A. Brock, T. Lim, J. Ritchie, and N. Weston. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint arXiv:1608.04236, 2016.
-  S. Bu, P. Han, Z. Liu, J. Han, and H. Lin. Local deep feature learning framework for 3D shape. Computers & Graphics, 46:117–129, 2015.
-  A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. ScanNet: Richly-annotated 3D reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5828–5839, 2017.
-  F. Engelmann, T. Kontogianni, A. Hermans, and B. Leibe. Exploring spatial context for 3D semantic segmentation of point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 716–724, 2017.
-  B.-S. Hua, M.-K. Tran, and S.-K. Yeung. Pointwise convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 984–993, 2018.
-  H. Huang, E. Kalogerakis, S. Chaudhuri, D. Ceylan, V. G. Kim, and E. Yumer. Learning local shape descriptors from part correspondences with multiview convolutional networks. ACM Transactions on Graphics, 37(1):6, 2018.
-  Q. Huang, W. Wang, and U. Neumann. Recurrent slice networks for 3D segmentation of point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2626–2635, 2018.
-  S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
-  R. Klokov and V. Lempitsky. Escape from cells: Deep Kd-networks for the recognition of 3D point cloud models. In Proceedings of the IEEE International Conference on Computer Vision, pages 863–872, 2017.
-  I. Kostrikov, Z. Jiang, D. Panozzo, D. Zorin, and J. Bruna. Surface networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2540–2548, 2018.
-  L. Landrieu and M. Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4558–4567, 2018.
-  T. Le and Y. Duan. PointGrid: A deep network for 3D shape understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9204–9214, 2018.
-  J. Li, B. M. Chen, and G. H. Lee. SO-Net: Self-organizing network for point cloud analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 9397–9406, 2018.
-  Y. Li, R. Bu, M. Sun, W. Wu, X. Di, and B. Chen. PointCNN: Convolution on X-transformed points. In Advances in Neural Information Processing Systems, pages 828–838, 2018.
-  X. Liu, Z. Han, Y.-S. Liu, and M. Zwicker. Point2Sequence: Learning the shape representation of 3D point clouds with an attention-based sequence to sequence network. In Association for the Advancement of Artificial Intelligence, 2019.
-  Z. Liu, S. Chen, S. Bu, and K. Li. High-level semantic feature for 3D shape based on deep belief networks. In Proceedings of the IEEE International Conference on Multimedia and Expo, pages 1–6, 2014.
-  L. Maaten and G. Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(Nov):2579–2605, 2008.
-  J. Masci, D. Boscaini, M. Bronstein, and P. Vandergheynst. Geodesic convolutional neural networks on Riemannian manifolds. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 37–45, 2015.
-  C. Moenning and N. A. Dodgson. Fast marching farthest point sampling. Technical report, University of Cambridge, Computer Laboratory, 2003.
-  F. Monti, D. Boscaini, J. Masci, E. Rodola, J. Svoboda, and M. Bronstein. Geometric deep learning on graphs and manifolds using mixture model CNNs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5115–1543, 2017.
-  V. Nair and G. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the International Conference on Machine Learning, pages 807–814, 2010.
-  C. Qi, H. Su, K. Mo, and L. Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 652–660, 2017.
-  C. Qi, H. Su, M. Nießner, A. Dai, M. Yan, and L. Guibas. Volumetric and multi-view CNNs for object classification on 3D data. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5648–5656, 2016.
-  C. Qi, L. Yi, H. Su, and L. Guibas. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems, pages 5105–5114, 2017.
-  G. Riegler, A. Ulusoy, and A. Geiger. OctNet: Learning deep 3D representations at high resolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3577–3586, 2017.
-  R. Rusu. Semantic 3D Object Maps for Everyday Manipulation in Human Living Environments. PhD thesis, Computer Science department, Technische Universitaet Muenchen, Germany, October 2009.
-  R. Rusu and S. Cousins. 3D is here: Point cloud library (PCL). In Proceedings of the IEEE International Conference on Robotics and Automation, pages 1–4, 2011.
-  Y. Shen, C. Feng, Y. Yang, and D. Tian. Mining point cloud local structures by kernel correlation and graph pooling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4548–4557, 2018.
-  N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014.
-  H. Su, S. Maji, E. Kalogerakis, and E. Learned-Miller. Multi-view convolutional neural networks for 3D shape recognition. In Proceedings of the IEEE International Conference on Computer Vision, pages 945–953, 2015.
-  M. Tatarchenko, J. Park, V. Koltun, and Q.-Y. Zhou. Tangent convolutions for dense prediction in 3D. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3887–3896, 2018.
-  C. Wang, B. Samari, and K. Siddiqi. Local spectral graph convolution for point set feature learning. In Proceedings of The European Conference on Computer Vision, September 2018.
-  P.-S. Wang, Y. Liu, Y.-X. Guo, C.-Y. Sun, and X. Tong. O-CNN: Octree-based convolutional neural networks for 3D shape analysis. ACM Transactions on Graphics, 36(4):72, 2017.
-  P.-S. Wang, C.-Y. Sun, Y. Liu, and X. Tong. Adaptive O-CNN: A patch-based deep representation of 3D shapes. ACM Transactions on Graphics, 37(6), 2018.
-  W. Wang, R. Yu, Q. Huang, and U. Neumann. SGPN: Similarity group proposal network for 3D point cloud instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2569–2578, 2018.
-  Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 3D ShapeNets: A deep representation for volumetric shapes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1912–1920, 2015.
-  H. Xu, M. Dong, and Z. Zhong. Directionally convolutional networks for 3D shape segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pages 2698–2707, 2017.
-  X. Ye, J. Li, H. Huang, L. Du, and X. Zhang. 3D recurrent neural networks with context fusion for point cloud semantic segmentation. In Proceedings of The European Conference on Computer Vision, September 2018.
-  L. Yi, V. G. Kim, D. Ceylan, I. Shen, M. Yan, H. Su, C. Lu, Q. Huang, A. Sheffer, L. Guibas, et al. A scalable active framework for region annotation in 3D shape collections. ACM Transactions on Graphics, 35(6):210, 2016.
-  L. Yi, H. Su, X. Guo, and L. Guibas. SyncSpecCNN: Synchronized spectral CNN for 3D shape segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6584–6592, 2017.
-  F. Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions. In International Conference on Learning Representations, 2016.
-  L. Yu, X. Li, C.-W. Fu, D. Cohen-Or, and P.-A. Heng. PU-Net: Point cloud upsampling network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2790–2799, 2018.
A-CNN: Annularly Convolutional Neural Networks on Point Clouds
Appendix A Ball Query vs Ring-based Scheme
The comparison of multi-scale method proposed in  and our ring-based scheme is depicted in Fig. 6. It is noted that comparing to multi-scale regions, the ring-based structure does not have overlaps (no neighboring point duplication) at the query point’s neighborhood. It means that each ring contains its own unique points.
|PointNet++ (multi-scale / with overlap)||86.5||90.2|
|PointNet++ (multi-ring / without overlap)||87.3||90.6|
|A-CNN (with all components)||90.3||92.6|
We have discovered that reducing redundancy can improve the existing multi-scale approach in . We test redundancy issue on original PointNet++ model  with and without overlap / redundancy. We compare the original PointNet++ multi-scale model with ball queries (with redundant points) against PointNet++ with our proposed regular rings (without redundant points). Our experiments show that the proposed multi-ring (i.e., without redundant points) outperforms the multi-scale scheme (i.e., with redundant points) on ModelNet40 according to Tab. 4.
Appendix B Training Details
We use A-CNN-3L network configuration in Tab. 5 for all experiments on point cloud classification tasks and A-CNN-4L network configuration in Tab. 5 for both part segmentation and semantic segmentation tasks. We use regular rings in and dilated rings in in our A-CNN-3L architecture. Similarly, we use regular rings in and dilated rings in and in our A-CNN-4L architecture.
We use Adam optimization method with learning rate 0.001 and decay rate 0.7 in classification and decay 0.5 in segmentation tasks. We have trained our classification model for 250 epochs, our part segmentation model for 200 epochs, and our large-scale semantic segmentation models for 50 epochs on each area of S3DIS and for 200 epochs on ScanNet. The training time of our model is faster than that of PointNet++ model, since we use ring-based neighboring search, which is more efficient and effective than ball query in PointNet++ model. For instance, the training time on the segmentation model for 200 epochs is about 19 hours on a single NVIDIA Titan Xp GPU with 12 GB GDDR5X, and PointNet++ model needs about 32 hours for the same task. The size of our trained model is 22.3 MB and the size of PointNet++ model is 22.1 MB.
|rings||[[0.0, 0.1], [0.1, 0.2]]||-||-|
|k||[16, 48]||[16, 48]||128||-|
|F||[[32,32,64], [64,64,128]]||[[64,64,128], [128,128,256]]||[256,512,1024]||-|
|rings||[[0.0, 0.1], [0.1, 0.2]]||[[0.1, 0.2], [0.3, 0.4]]||[[0.2, 0.4], [0.6, 0.8]]||-|
|k||[16, 48]||[16, 48]||[16, 48]||32|
|F||[[32,32,64], [64,64,128]]||[[64,64,128], [128,128,256]]||[[128,128,256], [256,256,512]]||[512,768,1024]|
Note: Both of the models represent encoder part. A-CNN-3L model consists of three layers. A-CNN-4L model consists of four layers. For each layer, is the number of centroids, is the inner and outer radiuses of a ring: , is number of neighbors, is feature map size. For example, our A-CNN-4L model at the first layer has 512 centroids; two regular rings where first ring constrained by radiuses of 0.0 and 0.1 and the second ring has radiuses of 0.1 and 0.2; k-NN search returns points in the first ring, and points in the second ring; the feature map size in the first ring is equal to and in the second ring is . Convolutional kernel size across different rings and layers is the same and equal to . Also, we have to double the number of centroids in each layer in model A-CNN-4L on ScanNet as the number of points in each block is twice more than that in S3DIS.
Appendix C Feature Visualization
Local Feature Visualization. Fig. -8 and Fig. -11 visualize the magnitude of the gradient per point in the classification task on ModelNet10 and ModelNet40 datasets. Blue color represents low magnitude of the gradients and red color represents high magnitude of the gradients. The points with higher magnitudes get greater updates during training and the learning contribution of them is higher. Therefore, this feature visualization could be thought as the object saliency. For example, in ModelNet40 dataset our model considers wings and tails as important regions to classify an object as an airplane; bottle neck is important for a bottle; the flowers and leaves are important for a plant; tube or middle part (usually narrow parts) is important for a lamp; legs are important to classify an object as a stool.
Global Feature Visualization. Fig. -10 and Fig. -9 shows the t-SNE clustering visualization  of the learned global shape features from the proposed A-CNN model for the shape classification tasks in ModelNet10 and ModelNet40 test splits. We reduce 1024-dim feature vectors to 2-dim features. We can see that similar shapes are well clustered together according to their semantic categories. For example, in ModelNet10 dataset the clusters of desk, dresser, night stand, and table classes are closer and even intersect with each other, because the objects from these classes look similar. The perplexity parameters for ModelNet10 and ModelNet40 datasets are set as 15 and 50, respectively, to reduce spare space between clusters.
Appendix D Data Preparation Details
|mean||areo||bag||cap||car||chair||ear phone||guitar||knife||lamp||laptop||motor||mug||pistol||rocket||skate board||table|
|mean||areo||bag||cap||car||chair||ear phone||guitar||knife||lamp||laptop||motor||mug||pistol||rocket||skate board||table|
|O-CNN (+ CRF) ||85.9||85.5||87.1||84.7||77.0||91.1||85.1||91.9||87.4||83.3||95.4||56.9||96.2||81.6||53.5||74.1||84.4|
Note: “CRF” stands for conditional random field method for final result refinement in O-CNN method.
S3DIS data preparation. To prepare training and testing datasets, we divide every room into blocks with a size of 1 1 2 and with a stride of 0.5 . We has sampled 4096 points from each block. The height of each block is scaled to 2 to ensure that our constraint-based k-NN search works optimally with the provided radiuses. In total, the prepared dataset contains 23,585 blocks across all six areas. Each point is represented as a 6D vector (: normalized global point coordinates and centered at origin, : colors). We do not use the relative position of the block in the room scaled between 0 and 1 as used in , because our model already achieves better results without using this additional information. We calculate point normals for each room by using the Point Cloud Library (PCL) library . The calculated normals are only used to order points in the local region. For data augmentation, we use the same data augmentation strategy as used in the point cloud segmentation on ShapeNet-part dataset which is point perturbation with point shuffling.
ScanNet data preparation. ScanNet divides original 1513 scanned scenes in 1201 and 312 for training and testing, respectively. We sample blocks from the scenes following the same procedure as in , where every block has a size of 1.5 1.5 with 8192 points. We estimate point normals using the PCL library . Each point is represented as a 6D vector (: coordinates of the block centered at origin, : normals) without information. For data augmentation, we use the point perturbation with point shuffling.
Appendix E More Experimental Results
Point Cloud Segmentation. Tab. 6 and Tab. 7 show the quantitative results of part segmentation on ShapeNet-part dataset with two different inputs. Tab. 6 reports results when the input is point position only. Tab. 7 reports results when the input is point position with its normals.
|MS+CU (2) ||79.2||47.8||88.6||95.8||67.3||36.9||24.9||48.6||52.3||51.9||45.1||10.6||36.8||24.7||37.5|
For ShapeNet-part dataset, we visualize more results (besides the segmentation results shown in the paper) in Fig. -31. We compare our results with PointNet++ , and our A-CNN model can produce better segmentation results than PointNet++ model.
Semantic Segmentation in Scenes. For S3DIS dataset, we pick rooms from all six areas: area 1 (row 1), area 2 (row 2), area 3 (row 3), area 4 (row 4), area 5 (row 5), and area 6 (row 6); and compare them with PointNet  results and ground truth. The results are shown in Fig. -7. The detailed quantitative evaluation results for each shape class are reported in Tab. 8. Our model demonstrates good semantic segmentation results and achieves the state-of-the-art performance on segmenting walls and chairs. Meanwhile, our model performs slightly worse than PointCNN  on other categories due to their non-overlapping block sampling strategy with paddings which we do not use. Supplementary Video is included for dynamically visualizing each area in detail.
For ScanNet dataset, we pick six challenging scenes and visualize the results of our A-CNN model, PointNet++ , and ground truth side by side. The visualization results are provided in Fig. -6. Our approach outperforms PointNet++  and other baseline methods, such as PointNet , TangentConv , and PointCNN  according to Tab. 2 in the main paper.