ContourCNN:convolutional neural network for contour data classification

ContourCNN: convolutional neural network for contour data classification

Abstract

This paper proposes a novel Convolutional Neural Network model for contour data analysis (ContourCNN) and shape classification. A contour is a circular sequence of points representing a closed shape. For handling the cyclical property of the contour representation, we employ circular convolution layers. Contours are often represented sparsely. To address information sparsity, we introduce priority pooling layers that select features based on their magnitudes. Priority pooling layers pool features with low magnitudes while leaving the rest unchanged. We evaluated the proposed model using letters and digits shapes extracted from the EMNIST dataset and obtained a high classification accuracy.

Convolutional neural netwrok, CNN, contour, classification, circular data, priority pool

I Introduction

In many computer vision tasks such as object detection, classification, and instance segmentation, object’s shape offers vital information for the task at hand.

This paper focuses on 2D objects, which often represented using its bounding contour. Bounding contours are commonly represented as circular sequences of points or polygons. Such a representation plays an important role in many shape analysis tasks such as shape matching, classification, and segmentation. However, a significant challenge of representing shapes using contour is the lack of common space, i.e. a shape can be represented by a huge number of contour representations, since the origin point can be defined anywhere on the contour of the shape, and the contour can have an arbitrarily sparsity level of points on the shape’s contour.

Over the years a number of shape analysis methods were proposed that make the use of a designed smart features such as shape context for matching [3, 11]. Others such as [13, 2] combine local and global features using the Bag of Words model for shape classification. In addition, there has been a number methods that uses machine learning algorithms such as SVM [13] and deep Bayesian networks [10] to classify shapes using their features.

Over the last decades, convolutional neural networks (CNNs) have proved their effectiveness in addressing complex tasks such as classification, detection, and segmentation. CNNs can learn and extract features as opposed to traditional learning free methods that rely on hand-crafted features. However, CNNs are geared toward regular representation, such as vectors and matrices. Due to the irregular representation of contours, CNNs have not been used in shape analysis tasks.

In this paper, we propose a new CNN for contour analysis. The proposed network handles the lack of common space of the contour data mentioned above. It includes a circular convolutional layers and a priority pooling layers.

The circular convolutional layers deal with the circularity of the contour representation. To deal with the varying sparsity level of the contours, we introduce a number of priority pooling layers inspired by the pooling layer used in MeshCNN [6], in which only points with low feature magnitude are pooled. To show the effectiveness of the proposed network, we apply it to classify shapes of digits and letters extracted from the EMNIST dataset .

The remainder of this paper is organised as follows: In section II we summarize a number of related work done in the area of shape classification. We describe the proposed model in section III. Training datasets and data representation are discussed in section IV. In section V, experimental results are presented and discussed. Finally, we conclude and discuss future works in section VI

Ii Related work

Existing solutions to the problem of shape classification can be split into two categories, learning and non-learning based solutions. Huttenlocher et al. [7] and Liu et al. [12] compare and match shapes by calculating the similarity between two shapes using the Hausdorff and the Chamfer distance function, respectively. Other approaches extract shape descriptors such as the Shape Context [3], Inner Distance Shape Context [11] and Height Functions [16] to match and classify shape objects. Felzenszwalb et al. [5] represent shapes’ contours as a hierarchical structure of its fragments. This representation is used to classify the shape objects by a dynamic programming based matching algorithm. The Bag of Words (BoW) model was also applied to pool local features and generate a global descriptors for shape matching [13, 2].

Several learning based approaches were proposed to tackle shape matching. Approaches such as [13, 17, 14] pool shape features using the BoW model and classify them using a SVM. Rameshet et al. [13] uses the spectral magnitude of the log-polar transform as local feature and bi-grams from the spatial co-occurrence matrix as contextual features, while [17] describes a shape as local shape context features of its fragments. Rameshet al. [14] combines contour and skeleton information into local features to describe a shape object. Li et al. [10] uses a deep Bayesian network trained by Markov chain Monte Carlo (MCMC) based optimization to classify shape descriptors of height features introduces by [16].

Iii Model

We propose a novel convolutional neural network that includes a custom-built 1D convolutional and pooling layers. Where the input of this network is a sequence of points representing a one layer contour. We introduce a 1D circular convolution layer, and a number of priority pooling layers.

Iii-a 1D Circular convolutional layer

A sequence of circular data (such as a contour) is roll invariant, i.e. the sequence can be shifted (in a circular manner) and still represent the same data. Regular convolutional layers assume a unique start and end points in the input data, which removes the circularity of the data sequence. Thus, we apply circular convolutional layer.

The circular convolution operation is defined as follow:

(1)

where is the input 1D tensor with length and depth , i.e. , where each is of size . is a 1D kernel of size and depth with bias , and is the 1D output tensor of the convolution operation. As can be seen in Figure 1, circular convolution work similar to regular convolution, but handles the end vertices differently. When the kernel is on a boundary of the input it wraps around to the other end.

Fig. 1: Circular convolution

Iii-B Priority pooling layers

The different vertices on a contour carry various levels of geometric information. A vertex that lies on a straight line which is defined by its immediate neighbours has little information; its position can be inferred from its two adjacent vertices.

Fig. 2: Illustration of the regular pooling operation with kernel of size three and stride value equal to two

Typical pooling operations, such as average and max, are applied uniformally in a local manner, regardless of the global relative importance of the input vertices (as illustrated in Figure 2). To overcome this limitation and apply irregular global pooling, we introduce priority pooling layers, that reduce the dimensions of an input tensor by iteratively considering the magnitude of the features.

Fig. 3: Illustration of the Remove One priority pooling operation. The colored bars represent the magnitude of each one of the feature vectors.

Remove One priority pooling accepts an input tensor of dimensions , where is the length of the input and is its depth (the number of features for each data point). The magnitude of each feature vertex is calculated using and utilized to prioritized the list of vertices, then the vertex with the lowest priority is removed (see Figure 3). This operation is repeated iteratively until the input tensor reaches a desired dimension. The desired dimensions we used are shown in Table I.

Fig. 4: Illustration of the Max and Average priority pooling operation with kernel of size three. The colored bars represent the magnitude of each one of the feature vectors.

Max and Average priority pooling layers work similar to the Remove One priority pooling. However, instead of removing the feature vertex with the lowest priority, the vertex with the lowest priority is pooled (either average or max). After each pooling operation the features’ magnitudes of the remaining vertices are recalculated (see Figure 3). Similar to the Remove One, this priority pooling is applied iteratively until the input tensor reaches a desired dimension.

Iii-C Network architecture

Similar to a typical 2D classification CNNs, our network architecture consists of two sub-networks: a feature extraction convolutional network, and a fully connected classification network.

The feature extraction network consists of four blocks of circular convolutional layers, a batch normalization layer, and a priority pooling layer. The extracted features are average pooled and fed to a fully connected classification network.
The architecture and configuration of the network can be seen in Table I. Such architecture were used to imitate AlexNet’s and the parameters were further tuned through experiments.

CircConv
PriorityPool
CircConv
PriorityPool
CircConv
PriorityPool
GlobalAvgPool
FCN
FCN
TABLE I: Network configuration. CircConv is a circular convolutional layer, where is the number of features (the depth) of the input, and is the output. PriorityPool is a priority pooling layer, where is the length of the output. FCN is a fully connected layer with input neurons and output neurons. is the number of features of the input of the network and is the number of outputs, i.e. number of classes

Iv Dataset

Existing shapes dataset such as MPEG-7 [9] (1400 shapes of 70 classes) and Animal Shapes [1] (2000 shapes of 20 classes) contain insufficient number of samples to train CNNs models. In order to test the proposed model, we needed a sufficiently large dataset suitable for training deep-learning models. We chose to extract the contours from the EMNIST dataset [4] which contain a large number of hand-written digits, uppercase and lowercase letter images. The contours were extracted using the algorithm [15] implemented in the OpenCV library.
Our experiments were conducted on two separate subset of the EMNIST dataset. The subsets of digits consisting of images from 10 digit classes (0-9), a total of 402,953 images (344,307 for training and 58,646 for testing); and the subset of uppercase letters consisting of images from 26 classes (A-Z), a total of 220,304 images (208,364 for training and 11,941 for testing). We chose to split those two subset to avoid ambiguity between some of the letters and some of the digits, for example, in some cases the digits 1 and 0 cannot be differentiated from the letters I and O respectively.

Iv-a Data representation

We experimented with two different representation of the contour data. The first is a normalized Cartesian representation- each contour is represented as a sequence of the normalized position points of its vertices. In the second, each contour is represented as a sequence of an angles and a line lengths, we’ll refer to this representation as the ”Polar representation”. Each angle and line length represents a vertex on the contour. As shown in Figure 5, in each vertex representation, equals the angle between the two adjacent lines, and is the length of the line between the given vertex and the next vertex in the sequence.

Fig. 5: Polar representation of a contour. The vertex is represented by the angle (the angle between the two lines), and the line length the length of the line between the vertex and the next vertex in the sequence

V Experimental Evaluation

To evaluate the performance of our model, we conducted several experiments comparing activation functions, the different priority pooling layers, and the Cartesian and Polar representations (discussed in Section IV-A). Furthermore, we compare the performance of our model with AlexNet. Those comparison were conducted on the letters subset, because it offer more complex and varied data than the digits subset. Finally, we report the results of the best performing model on the letter and the digits subsets from EMNIST. In each one of the experiments the model was trained and tested using the official train and test split of EMNIST.

Activation function

In this experiment we compared the performance of the model with different activation functions. We trained three networks with the activation functions, Sgimoid, TanH, and ReLU using the letters subset, the Cartesian representation, and with the Remove One priority pooling layers.

Figure 6 shows the test loss and accuracy of each network during the training process. We can see that the two networks using Sigmoid and ReLU activation function reached similar accuracy and loss values. Using the ReLU activation function the network reached a higher accuracy faster than using the Sigmoid function. While using TanH activation function the network was unable to learn and got stuck in a local minima.

Fig. 6: The loss and accuracy of a number of networks using different activation functions on the test set during the training process

Priority pooling layers

In this experiment we compare the different priority pooling layer introduced in Section III-B. Similar to the previous experiment, three networks with different priority pooling layers and ReLU activation function were trained using the letters subset and the Cartesian representation. Note that this compression only affect the first three priority pooling layers, the global average pooling layer preceding the full connected layers remains unchanged.

Figure 7 shows the accuracy and loss of the networks on the test subset. As can be seen the networks which include the Remove One and the Max priority pooling layers reached similar accuracy. However, the two networks achieve different loss values on the test set. This suggests, the network with the Remove One priority pooling is more confident concern its classification, i.e. it gives a higher probability to the correct classification. As seen, the networks with the average priority pooling layer converges and obtained around 90% accuracy rate on the test subset, but it did not reach a higher accuracy as the other two networks.

Fig. 7: The loss and accuracy of networks using different priority pooling layers on the test set during the training process

Experimenting with data representations

In the following experiment we trained two identical networks with Remove One priority pooling layers and ReLU activation function on the letters subsets with the Cartesian and Polar data representations mentioned in Section IV-A.

Figure 8 shows the accuracy of the two networks on the test subset during training. Unsurprisingly, the network trained using the Cartesian representation reached higher accuracy faster than the network using the Polar representation. There will always be ambiguity between certain letters in the Polar representation. As can be seen in Figure 9, the letters M, W, and E have an almost identical Polar representation, since this representation is rotation invariant, rotating the letter M results in the letter W with the same Polar representation as the letter M. Such ambiguity can be seen in the confusion matrix of the model in Figure 10.

Shape simplification effects the priority pooling layers pools/removes a selected features corresponding to points on input contour. Thus, we can visualize the progress of the network by redrawing the contour after each priority pooling without the pooled points. Interestingly, we observed that the network trained using the Polar representation seemingly simplifies the contour shape. As can be seen in Figure 11, when drawing the contour after each priority pooling layer in the network, the contour of the letters is simplified as it traverse down the network’s pooling layers. This can be attributed to the fact that the activation of vertices that contain more information are larger than the activations of those with fewer information. Therefore, the priority pooling layers will first pool/remove redundant contour vertices or those with little information.

Fig. 8: The accuracy of two networks on the test set during training. The two networks were trained using two different data representations, the Cartesian representation and the Polar representation
Fig. 9: Example contour instances of the letters M, W and E in the test subset
Fig. 10: The confusion matrix of the network trained using the Polar representation. The color of each cell represent the number of samples classified in it. Lighter color means more samples (as shown in the color bar to the right)
Fig. 11: This figure show the simplification effect happening in the network. The row shows the input contour of the letter, the resulting contour after each pooling layer. The coloured circles represent the activations of each contour vertices; the size and colour of the circles reflect the magnitude of each activation

Comparing with regular CNN classification model

Here we compare our approach with a more established 2D classification CNN. Since the ContourCNN model are able to process one layer contours, training a 2D CNN using the original data would give it an unfair advantage. Because the original images include information missing from the contour data (e.g. the holes inside the letters A,B, D, and O). Therefore, in order to compare the two approaches, we generated 2D images by drawing the filled polygon represented by the extracted contour data (the silhouette of the letters) as can be seen in Figure 12. This way we insure that the two approaches are trained in a fair manner, i.e. using the same level of detail/information. Since AlexNet [8] has a similar architecture to our model, we chose to train it on the generated 2D images. The configuration of the network can be seen in Table II.

As can be seen in Figure 13 the two approaches reach similar accuracy of around 96%. This results shows the effectiveness of the proposed model as it managed to achieve similar accuracy results to a more established model such as AlexNet.

Fig. 12: Example of the generated 2D images from the contour data of the letters O, S, and T
Conv2D
MaxPool2D
Conv2D
MaxPool2D
Conv2D
Conv2D
MaxPool2D
AvgPool
FCN
FCN
TABLE II: AlexNet configuration
Fig. 13: The test accuracy of two approaches; ContourCNN: training using contour data, AlexNet: CNN training using the generated 2D images.

Results of the best model on the letters and digits subsets

The best results were reached by the model with Remove One priority pooling layer, ReLU activation function, and trained using the Cartesian representation. This model achieved around a 96% accuracy rate on the uppercase letters subset and round a 96.7% accuracy on the digits subset. The Figures 14 and 15 show the confusion matrices for the results of the trained network on the uppercase letters and digits subsets.

Fig. 14: The confusion matrix of the classification results of the network trained on the letters subset. The color of each cell represent the number of samples classified in it. Lighter color means more samples (as shown in the color bar to the right)
Fig. 15: The confusion matrix of the classification results of the network trained on the digits subset. The color of a cell indicates the number of samples were used for classification. Lighter color means more samples (as shown in the color bar to the right)

Vi Conclusion & Future work

This paper presents ContourCNN, a CNN for contour data analysis. We introduced 1D circular convectional layer for convolution on a circular data representation, which preserves the circularity of the data after the operation. To deal with the varying sparsity level of information of the contour data we introduced a number of priority pooling layers that pools only the vertices with the lowest feature magnitude. We showed the effectiveness of the model by classifying the contours of the digits and letters from the EMNIST dataset and compared the results to the similarly structured CNN, AlexNet. In addition, we provided a comparison of the accuracy and loss between the different priority pooling layers, activation function, and the input data representation.

In future work, we plan to explore the use of ContourCNN for contour segmentation using an encoder-decoder architecture of the model, where the pooled vertices are remembered in the down-sampling stage of the encoder and unpooled in the up-sampling stage of the decoder. We will also consider extending the ContourCNN to process nested contour.

Lastly, we would like to end on the following note: while we used the proposed model to analyse contours, we realized it can be used to process other similar data forms (data with circular structure or data sequences that include redundant information).

References

  1. X. Bai, W. Liu and Z. Tu (2009) Integrating contour and skeleton for shape classification. In 2009 IEEE 12th international conference on computer vision workshops, ICCV workshops, pp. 360–367. Cited by: §IV.
  2. X. Bai, C. Rao and X. Wang (2014) Shape vocabulary: a robust and efficient shape representation for shape matching. IEEE Transactions on Image Processing 23 (9), pp. 3935–3949. Cited by: §I, §II.
  3. S. Belongie, J. Malik and J. Puzicha (2002) Shape matching and object recognition using shape contexts. IEEE Transactions on Pattern Analysis & Machine Intelligence (4), pp. 509–522. Cited by: §I, §II.
  4. G. Cohen, S. Afshar, J. Tapson and A. van Schaik (2017) EMNIST: an extension of mnist to handwritten letters. arXiv preprint arXiv:1702.05373. Cited by: §IV.
  5. P. F. Felzenszwalb, R. B. Girshick, D. McAllester and D. Ramanan (2009) Object detection with discriminatively trained part-based models. IEEE transactions on pattern analysis and machine intelligence 32 (9), pp. 1627–1645. Cited by: §II.
  6. R. Hanocka, A. Hertz, N. Fish, R. Giryes, S. Fleishman and D. Cohen-Or (2019) MeshCNN: a network with an edge. ACM Transactions on Graphics (TOG) 38 (4), pp. 90. Cited by: §I.
  7. D. P. Huttenlocher, G. A. Klanderman and W. J. Rucklidge (1993) Comparing images using the hausdorff distance. IEEE Transactions on pattern analysis and machine intelligence 15 (9), pp. 850–863. Cited by: §II.
  8. A. Krizhevsky, I. Sutskever and G. E. Hinton (2012) Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105. Cited by: §V-4.
  9. L. J. Latecki, R. Lakamper and T. Eckhardt (2000) Shape descriptors for non-rigid shapes with a single closed contour. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662), Vol. 1, pp. 424–429. Cited by: §IV.
  10. C. Li, A. Stevens, C. Chen, Y. Pu, Z. Gan and L. Carin (2016) Learning weight uncertainty with stochastic gradient mcmc for shape classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5666–5675. Cited by: §I, §II.
  11. H. Ling and D. W. Jacobs (2007) Shape classification using the inner-distance. IEEE transactions on pattern analysis and machine intelligence 29 (2), pp. 286–299. Cited by: §I, §II.
  12. M. Liu, O. Tuzel, A. Veeraraghavan and R. Chellappa (2010) Fast directional chamfer matching. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 1696–1703. Cited by: §II.
  13. B. Ramesh, C. Xiang and T. H. Lee (2015) Shape classification using invariant features and contextual information in the bag-of-words model. Pattern Recognition 48 (3), pp. 894–906. Cited by: §I, §II, §II.
  14. W. Shen, Y. Jiang, W. Gao, D. Zeng and X. Wang (2016) Shape recognition by bag of skeleton-associated contour parts. Pattern Recognition Letters 83, pp. 321–329. Cited by: §II.
  15. S. Suzuki (1985) Topological structural analysis of digitized binary images by border following. Computer vision, graphics, and image processing 30 (1), pp. 32–46. Cited by: §IV.
  16. J. Wang, X. Bai, X. You, W. Liu and L. J. Latecki (2012) Shape matching and classification using height functions. Pattern Recognition Letters 33 (2), pp. 134–143. Cited by: §II, §II.
  17. X. Wang, B. Feng, X. Bai, W. Liu and L. J. Latecki (2014) Bag of contour fragments for robust shape classification. Pattern Recognition 47 (6), pp. 2116–2125. Cited by: §II.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
414421
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description