A Robust Indoor Scene Recognition Method based on Sparse Representation
In this paper, we present a robust method for scene recognition, which leverages Convolutional Neural Networks (CNNs) features and Sparse Coding setting by creating a new representation of indoor scenes. Although CNNs highly benefited the fields of computer vision and pattern recognition, convolutional layers adjust weights on a global-approach, which might lead to losing important local details such as objects and small structures. Our proposed scene representation relies on both: global features that mostly refers to environment’s structure, and local features that are sparsely combined to capture characteristics of common objects of a given scene. This new representation is based on fragments of the scene and leverages features extracted by CNNs. The experimental evaluation shows that the resulting representation outperforms previous scene recognition methods on Scene15 and MIT67 datasets, and performs competitively on SUN397, while being highly robust to perturbations in the input image such as noise and occlusion.
ndoor Scene Recognition, Sparse Coding, Convolutional Neural Networks
Scene recognition is one of the most challenging tasks on image classification, since the characterization of a scene is performed by using visual features of the objects that compose it and its spatial layout. In other words, a scene is the result of objects compositions. A given environment is more likely to be classified as a bathroom when equipped with a bath and a shower. This is especially true when considering indoor images, which are the focus of this work. The rationale is that besides the constituent objects, the image of a room (i.e., an indoor scene) is similar to every scene and it is hard to distinguish among them.
In this paper, we propose a novel method, which leverages features extracted by Convolutional Neural Networks (CNNs) using a sparse coding technique to represent discriminative regions of the scene. Specifically, we propose to combine global features from CNNs and local sparse feature vectors mixed with max spatial pooling. We built an over-complete dictionary whose base vectors are feature vectors extracted from fragments of a scene. This approach makes our image representation less sensitive to noise, occlusion and local spatial translations of regions and provides discriminative vectors with global and local features.
The main contributions of this paper are: i) a robust method that simultaneously leverages global and local features to the scene recognition task, and ii) a thorough set of experiments to validate our requirements and the proposed scene recognition method. We evaluate our method on three different datasets (Scene15 , MIT67  and SUN397 ) comparing it to previously proposed methods in the literature, and perform a robustness test against the work of Herranz et al. . The experimental results show that our method outperforms the current state of the art on Scene15 and MIT67, while performing competitively on SUN397. Additionally, when subjected to image perturbations (i.e., noise and occlusion), our proposal outperforms Herranz et al.  on all three datasets, surpassing it by a large margin on the most challenging one, SUN397.
Most solutions proposed in the past decade for scene recognition exploited handcrafted local  and global descriptors . Methods to combine these descriptors and to build an image representation varied from Fisher Vectors  to Sparse Representation . Sparse representation methods reached a great performance on image recognition, becoming popular in the last few years. Yang et al.  and Gao et al.  encoded SIFT  descriptors into a single sparse feature vector with the max-pooling method, achieving the best results on the Caltech-101 and the Scene-15 datasets.
In recent years, methods based on CNNs achieved state of the art on the task of scene recognition. In Zhou et al. , the authors introduce the Places dataset and present a CNN method to learn deep features for scene recognition. The authors reached an accuracy of on MIT67. Another line of investigation is to combine features from CNNs. Dixit et al.  use a network to extract posterior probability vectors from local image patches and combine them using a Gaussian mixture model. Features extracted from networks trained on ImageNet  and Places  are combined, achieving accuracy on MIT67.
Herranz et al.  analyzed the impact of object features in different scales in combination with global holistic features provided by scene features. According to the authors, the aggregation of local features brings robustness to semantic information in scene categorization. Herranz et al.  held the state of the art before Wang et al.  propose the architecture called PatchNet. This architecture models local representations from scene patches and aggregates them into a global representation using a newly semantic encoding method called VSAD.
Differently from previous works, our approach is flexible to being used with any feature representation, not being restricted to a single dataset or network architecture. This is a key advantage, since the use of different sources of features (e.g., ImageNet for object features and Places for structure) cannot be easily handled by training a single CNN or an autoencoder representation. Additionally, our method is attractive because it presents a small number of hyperparameters (the sparsity and dictionary size), which makes it much easier to train. Typical CNN or autoencoder approaches require selecting the batch size, learning rate and momentum, architecture, optimization algorithm, activation functions, the number of neurons in the hidden layer and dropout regularization.
Our methodology is composed of four steps, namely: i) feature extraction and dictionary building; ii) feature sparse coding; iii) pooling, and iv) concatenation. The final feature vector feeds a Linear SVM used to classify the image. The three first steps are illustrated in Fig. 1.
The overall idea of our approach is to extract features from two semantic levels of the image. Firstly, a global representation is computed from the entire image, encoding the structural features of the scene. Then, we move a sliding window over the image with and window sizes, where represent the image dimensions, computing a feature vector for each fragment of the scene. These fragments contain local features that are potentially from objects or object’s parts. The stride was fixed to half of the window dimensions in both scales. The rationale is that features extracted from smaller regions will convey information regarding the constituent objects of the scenes. Thus, by combining these features we can represent an indoor scene as a composition of objects.
Feature Extraction and Dictionary Building.
Although our method can be instantiated with any feature extractor, such as a bag-of-features model, we use Convolutional Neural Networks to extract the features. We extract semantic features from the fc7 layer of VGGNet-16 . For global information, the CNN was trained on Places , while two types of local information were extracted: the same model trained on Places to acquire information regarding local structures, and a second model trained on ImageNet to encode object features.
The feature extraction step provided a feature vector for each fragment of the image. This process is performed for all samples, grouping the vectors into clusters using the -means algorithm. Hence, a dictionary for scale is represented by , where represents the th cluster in scale . We define the matrix as the concatenation of multiple scales matrices : .
It is worth noting that, since we are considering a set of over-complete basis, we need , where is the number of scale matrices in the dictionary and is the dimension of each feature vector.
We concatenate the dictionaries built on different patch sizes into a single dictionary, respecting the nature of the feature. This leaves us with two dictionaries, one built from features extracted using the model trained on Places, and the other using the model trained on ImageNet. The idea is improving the probability of finding a match in different scales, which offers a better discrimination and repeatability power. Once we have built the initial dictionary , we adjust the dictionary to matrix by solving
where is the -norm, is the -norm, is a regularization parameter for the dictionary learning and the vector is the column of matrix . We applied a dictionary learning algorithm , which solves Eq. 1 alternating between and as variables, i.e., it minimizes over while keeping fixed. The dictionary is used to start the optimization process.
Despite the large number of possible objects that can compose a scene, each class of indoor environments has a small number of characteristic types of objects. The dictionary provides a linear representation of each image fragment using a small number of basis functions. Thus, we find the sparse vector by modeling the composition of a scene as a sparse coding problem.
Considering the new domain of representation defined by the dictionary , the representation of a feature vector extracted from a sample of indoor class , can be rewritten as , where is a vector whose entries are zero except those associated with the fragments of class .
We use as a basis for the vector representation the columns of a dictionary that is mixed with weights to infer the vector , which is the vector that reconstructs the input with the smallest error. Each column is a descriptor representing an image fragment of the scene. We use a sparsity regularization term to select a small number of columns in the dictionary .
Let be a descriptor extracted from an image patch and (). The coding can be modeled as the following optimization problem:
where is the -norm, indicating the number of nonzero values, and controls the sparsity of vector . The final vector is the set of basis weights that represents the descriptor as a linear and sparse combination of fragments of scenes.
Although the minimization problem of Eq. 2 is NP-hard, greedy approaches, such as Orthogonal Matching Pursuit (OMP)  or norm relaxation, also know as LASSO , can be used to effectively solve it. In this paper, we use OMP because it achieved the best results on our tests.
The pooling process refers to the final step of the diagram presented in Fig. 1. To create the vector encoding the scene features, we compute the final feature vector by a pooling function , where is defined on each column of . The rows of matrix represent the sparse vectors of each feature vector extracted from sliding-windows. We create -dimensional vectors according to the maximum pooling function, since according to Yang et al. , it gives the best results for sparse coded local features.
These steps are executed for both scales, using both the model trained on Places and ImageNet. At the end, five features vectors are concatenated into one: the global features as originally output by the CNN, and four local features.
We evaluated our approach on the standard datasets for scene recognition benchmark: Scene15, MIT67 and SUN397. The specific attributes of each dataset allowed us to analyze different properties of our method. We trained the models on Places for global and local descriptors, and ImageNet for local descriptors, with VGGNet-16 which has shown the lowest classification error . When considering features from the VGGNet-16, we used PCA to reduce the descriptions from to on the second and third scales.
To compose the dictionary, we set words for Scene15, words for MIT67 and words for SUN397, according to the number of regions extracted from both scales and the number of classes of each dataset. The regularization factor for the dictionary learning (Eq. 1) is set to . We set the same sparsity controller value (Eq. 2) for all configurations when executing OMP. It was set to non-zeros, where is the number of columns of the dictionary. All vectors were normalized after each step and after concatenation. We used a Linear Support Vector Machine to perform the classification.
We verified the descriptor robustness against occlusion and noise. For this purpose, we automatically generated squares randomly positioned, to simulate occlusions (black squares) and noise (granular squares), as illustrated in Fig. 2. Experiments were performed for different sizes of windows , where represents the dimensions of the image, with varying values.
Table 1 presents the results for occlusion and noise robustness tested against Herranz et al. . Our methodology performs better in all cases, indicating that our method is less sensitive to perturbations on the image. This behaviour is even more evident for SUN397, by far the most challenging of all three datasets, where the Herranz et al.’s methodology was inferior, on average for occlusion, and when noise was added.
|Occlusion||Herranz et al.|
|Noise||Herranz et al.|
Besides being highly robust to perturbations in the input image, one can clearly see in Table 2 that our methodology also leads the performance on Scene15 and MIT67, while performing competitively on SUN397. It is worth highlighting that our method surpasses human accuracy, which was measured for SUN397 at . To evaluate our assumption that local information can greatly benefit the task of indoor scene classification, Table 2 also presents the average accuracy achieved by a model trained solely on features from VGGNet-16 trained on Places, which composes the global portion of our methodology. On all datasets, global features by themselves showed inferior performance relative to our representation, revealing the complementary nature of local features.
We also performed a detailed performance assessment by comparing the accuracies among each indoor class on MIT67. The relative accuracies considering VGGNet-16 as a baseline are shown in Fig. 3. As we can see, for the classes that present a large number of object information (e.g., children room, emphasized in Fig. 3) we have a significant increase in accuracy. We can draw the following two observations. First, since the class of these environments strongly depends on the object configuration, the image fragments containing features of common objects can be represented by the sparse features on different scales. Second, for environments such as movie theater (Fig. 3), the proposed approach is less effective, since such environments are strongly distinguished by their overall structure, and not necessarily by the constituent objects.
We proposed a robust method that combines both global and local features to compose a high discriminative representation of indoor scenes. Our method improves the accuracy of CNN features by composing local features using Sparse-Coding and a max-pooling technique, by creating an indoor scene representation based on an over-complete dictionary, which is built from several image fragments. Our sparse coding-based composition approach was capable of encoding local patterns and structures for indoor environments, and in cases of strong occlusion and noise, we lead the performance on scene recognition, which is explained by the advantages of sparse representations. Our representation outperformed VGGNet-16 in all cases, reflecting the complementary nature of local features, and outperformed the current state of the art on Scene15 and MIT67, while being competitive on SUN397.
-  Dixit, M., Chen, S., Gao, D., Rasiwasia, N., Vasconcelos, N.: Scene classification with semantic fisher vectors. In: IEEE Conf. on Comp. Vision and Pattern Recog. pp. 2974–2983. IEEE (2015)
-  Gao, S., Tsang, I.W.H., Chia, L.T., Zhao, P.: Local features are not lonely - laplacian sparse coding for image classification. In: IEEE Conf. on Comp. Vision and Pattern Recog. pp. 3555–3561. IEEE Computer Society (2010)
-  Herranz, L., Jiang, S., Li, X.: Scene recognition with cnns: Objects, scales and dataset bias. In: IEEE Conf. on Comp. Vision and Pattern Recog. (June 2016)
-  Jaakkola, T., Haussler, D.: Exploiting generative models in discriminative classifiers. In: In Advances in Neural Information Processing Systems 11. pp. 487–493. MIT Press (1998)
-  Lazebnik, S., Schmid, C., Ponce, J.: Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories. In: IEEE Conf. on Comp. Vision and Pattern Recog. pp. 2169–2178. IEEE Computer Society (2006)
-  Li, F.F., Perona, P.: A bayesian hierarchical model for learning natural scene categories. In: IEEE Conf. on Comp. Vision and Pattern Recog. pp. 524–531. IEEE Computer Society, Washington, DC, USA (2005)
-  Lowe, D.G.: Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision 60(2), 91–110 (Nov 2004)
-  Mairal, J., Bach, F., Ponce, J., Sapiro, G.: Online dictionary learning for sparse coding. In: Proceedings of the 26th Annual International Conference on Machine Learning. pp. 689–696. ICML ’09, ACM, New York, NY, USA (2009)
-  Olga, Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision 115(3), 211–252 (2015)
-  Oliva, A., Torralba, A.: Modeling the shape of the scene: A holistic representation of the spatial envelope. International Journal of Computer Vision 42(3), 145–175 (May 2001)
-  Oliveira, G., Nascimento, E., Vieira, A., Campos, M.: Sparse spatial coding: A novel approach to visual recognition. IEEE Transactions on Image Processing 23(6), 2719–2731 (June 2014)
-  Quattoni, A., Torralba, A.: Recognizing indoor scenes. IEEE Conf. on Comp. Vision and Pattern Recog. 0, 413–420 (2009)
-  Sánchez, J., Perronnin, F., Mensink, T., Verbeek, J.: Image classification with the fisher vector: Theory and practice. International Journal of Computer Vision 105(3), 222–245 (2013)
-  Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. CoRR abs/1409.1556 (2014)
-  Tibshirani, R.: Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society, Series B 58, 267–288 (1994)
-  Tropp, J.A., Gilbert, A.C.: Signal recovery from random measurements via orthogonal matching pursuit. IEEE Transactions on Information Theory 53(12), 4655–4666 (Dec 2007)
-  Wang, Z., Wang, L., Wang, Y., Zhang, B., Qiao, Y.: Weakly supervised patchnets: Describing and aggregating local patches for scene recognition. IEEE Transactions on Image Processing (2017)
-  Xiao, J., Hays, J., Ehinger, K.A., Oliva, A., Torralba, A.: Sun database: Large-scale scene recognition from abbey to zoo. In: IEEE Conf. on Comp. Vision and Pattern Recog. pp. 3485–3492. IEEE Computer Society (2010)
-  Yang, J., Yu, K., Gong, Y., Huang, T.: Linear spatial pyramid matching using sparse coding for image classification. In: IEEE Conf. on Comp. Vision and Pattern Recog. (2009)
-  Yunchao Gong, Liwei Wang, R.G.S.L.: Multi-scale orderless pooling of deep convolutional activation features. CoRR abs/1403.1840 (2014)
-  Zhou, B., Lapedriza, A., Xiao, J., Torralba, A., Oliva, A.: Learning deep features for scene recognition using places database. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 27, pp. 487–495. Curran Associates, Inc. (2014)