An Information-rich Sampling Technique over Spatio-Temporal CNN for Classification of Human Actions in Videos

An Information-rich Sampling Technique over Spatio-Temporal CNN for Classification of Human Actions in Videos

Abstract

We propose a novel scheme for human action recognition in videos, using a 3-dimensional Convolutional Neural Network (3D CNN) based classifier. Traditionally in deep learning based human activity recognition approaches, either a few random frames or every frame of the video is considered for training the 3D CNN, where is a small positive integer, like , , or . This kind of sampling reduces the volume of the input data, which speeds-up training of the network and also avoids over-fitting to some extent, thus enhancing the performance of the 3D CNN model. In the proposed video sampling technique, consecutive frames of a video are aggregated into a single frame by computing a Gaussian-weighted summation of the frames. The resulting frame (aggregated frame) preserves the information in a better way than the conventional approaches and experimentally shown to perform better. In this paper, a 3D CNN architecture is proposed to extract the spatio-temporal features and follows Long Short-Term Memory (LSTM) to recognize the human actions. The proposed 3D CNN architecture is capable of handling the videos where the camera is placed at a distance from the performer. Experiments are performed with KTH and WEIZMANN human actions datasets, whereby it is shown to produce comparable results with the state-of-the-art techniques.

I Introduction and Related Works

Human action recognition in videos has been an active area of research, gaining the attention of Computer Vision and Machine Learning researchers during the last decade due to its potential applications in various domains, including intelligent video surveillance systems, viz., Human-Computer Interaction (HCI), robotics, elderly and child monitoring systems and several other real-world applications. However, recognizing human actions in the real world remains a challenging task due to several challenges involved in real-life videos, including cluttered backgrounds, viewpoint variations, occlusions, varying lighting conditions and many more. This paper proposes a technique for human activity recognition in videos, where the videos are captured by a camera placed at a distance from the performer.

The approaches for recognizing human actions from videos, found in the literature, can be broadly classified into two categories [40]. The first, make use of motion-related features (low, mid, and high level) for human action recognition [23, 9]. The other set of approaches experiment to learn a proper representation of the spatio-temporal features the during action using deep neural networks [34, 31, 38, 2].

Handcrafted features played a key role in various approaches for activity recognition [29]. Semantic features ease to identify similar activities that vary visually but have common semantics. Semantic features during an action contain human body parts (posture and poselet), background, motion and other features incorporating human perceptual knowledge about the activities. A study by Ziaeefard et al. [40] examined human action recognition approaches using semantic features. Malgireddy et al. [26] proposed a hierarchical Bayesian model which interconnects low-level features in videos with postures, motion patterns, and categories of activities. Very recently, Nazir et al. [29] proposed a Bag of Expression (BOE) framework for activity recognition.

The most common handcrafted feature, used for action recognition, is optical flow [6, 28, 27, 37]. Chaudhry et al. [6] introduced the concept of Histogram of Oriented Optical Flow (HOOF) for action recognition, where the optical flow direction is divided into octants. Mukherjee et al. [27] proposed Gradient-Weighted Optical Flow (GWOF) to limit the effect of camera shaking, where the optical flow of every frame is multiplied by the image gradient. Wang et al. [37] introduced another approach to reduce the camera shaking effect, called Warped Optical Flow (WOF), where gradient is computed on the optical flow matrix. In [28], the effect of background clutter is reduced by multiplying Weighted Optical Flow (WOF) features with the image gradients. Optical flow based approaches help in dissecting the motion, but gives too much unnecessary information such as, motion information at all the background pixels, which reduces the efficacy of the action recognition system in many cases.

Spatio Temporal Interest Points (STIP) introduced by [24], identifies spatio-temporal interest points based on the extension of Harris Corner Detection approach [14] towards the temporal domain. Several researchers have shown interest to recognize human actions with the help of some other variants of spatio-temporal features like Motion- Scale Invariant Feature Transform (MoSIFT) [7] and sparse features [9]. A study on STIP based human activity recognition methods is published by Dawn et al. [8]. However, such spatio-temporal features are unable to handle the videos taken in real-world which suffers from background clutter and camera shake. Buddubariki et al. [5] combined the benefits of GWOF and STIP features by calculating GWOF on the STIP points. In [1], combination of 3-dimensional SIFT and HOOF features are used along with support vector machine (SVM) for classifying human actions.

Recently, deep learning based models are gaining the interest of researchers for recognizing human actions [31, 2, 17, 35, 36, 19]. Taylor et al. [35] proposed a multi-stage network, where in a Convolutional Restricted Boltzmann Machine (ConvRBM) retrieves motion-related information from each pair of successive frames at the initial layer. In [31], a two-stream convolutional network is proposed that comprises spatial-stream ConvNet and temporal-stream ConvNet. Ji et al. [17] introduced a 3-dimensional CNN architecture for action recognition, where a 3 dimensional convolutions are used to extract the spatio-temporal features. Tran et al. [36] enhanced 3D CNN model by applying Fisher vector encoding scheme on the learned features. Karpathy et al. [19] proposed a deep neural network for spatio-temporal resolutions: high and low resolutions, then merged them to train the CNN. Kar et al. [18] proposed a technique for temporal frame pooling in a video for human activity recognition. A survey by Herath et al. [15] discusses both engineered and deep learning based human action recognition techniques.

In the literature of human action recognition, researchers have used either the fully observed video or a portion of the video to train the deep neural networks. Training the models using a portion of the video will take less amount of training time compared to training the model using entire video. However, considering a portion of the video (considering 9 frames from the entire video as in [2] and 7 frames as in [17]) results in information loss. Srivastava et al. [33] used multi layer LSTM network to learn the representations of video sequences. Recently, Bilen et al. [4] introduced dynamic image, a very compact representation of video used for analyzing the video with CNNs. However, dynamic images eventually dilute the importance of spatial information during action. The proposed sampling technique for video frames preserves both spatial and temporal information together.

Baccouche et al. [2] proposed a completely automated deep learning architecture for KTH dataset [30], which figures out how to characterize human activities with no earlier information. This 3D CNN architecture learns spatio-temporal features automatically, then LSTM network [11] is used to classify the learned features. Motivated by the method introduced in [2], we propose a 3D CNN to learn spatio-temporal features and then apply LSTM to classify human actions. The proposed method uses small sized filters throughout the 3D CNN architecture, which helps to learn minute information present in the videos, which can help in recognizing the action of performers appearing very small in the video, due to the distance of the camera.

Our contributions in this paper are two-folds. First, a novel sampling technique is introduced to aggregate the entire video into a fewer number of frames. Second, a 3D CNN architecture is proposed for better classification of human actions in videos where the performer looks significantly small. The choice of smaller filter size enables the proposed model work well in such scenarios where the performer looks small due to distance from the camera. We experiment with the proposed deep learning model with transfer learning technique, by transferring the knowledge learned from KTH dataset to fine-tune over WEIZMANN dataset and vice versa.

The proposed pre-processing method is presented in Section II. Section III illustrates the proposed 3D CNN architecture. The experiments and results are described in Section IV. Finally, Section V concludes and provides scope for future research.

Ii Pre-processing using An Information Sampling Approach

The primary objective of this pre-processing step is to reduce the amount of training time and at the same time motion information should be given utmost importance. We propose a novel sampling technique to aggregate a large number of frames into a fewer set of frames using Gaussian Weighing Function (GWF), which minimizes the information loss. The proposed video pre-processing scheme is shown in Figure 1.

Fig. 1: The proposed pre-processing procedure using Gaussian Weighing Function. An entire video (collection of all frames) is represented as an exhaustive non-overlapping sequence , which further has sub-sequences {, , }. A single pre-processed frame (for example ) is obtained by performing weighted summation of consecutive five frames (for instance , , , , and belongs to sub-sequence ) as shown in equation 2.

Gaussian Weighing Function (GWF) is used to aggregate the entire video into a fewer number of frames. Let us consider , an exhaustive non-overlapping sequence (collection of all frames of a video), which is given by

(1)

where is the sub-sequence of and . Mathematically, Gaussian Weighing Function , for a sub-sequence , is given as follows:

(2)

The function takes a sub-sequence , and Gaussian weight vector as input, and aggregates the information into a single frame. Here represents the element of Gaussian weight vector . For example, if the size of the Gaussian weight vector (i.e., ) is and the sub-sequence is , which has five frames of the video. The vector is given by = . A single frame is obtained by performing weighted summation of the five frames belonging to the sub-sequence as shown in equation (2). In other words, five frames are aggregated into a single frame using Gaussian weighing function. Similarly, the same process is repeated for subsequent five frames belonging to the next sub-sequence and so on. This sampling approach reduces the volume of data for training the deep learning model and also preserves the information in better way which helps to obtain better results in human activity recognition.

Iii Spatio-Temporal Features Extraction using Deep Learning Models

In this section, initially we describe 2-D CNNs, and then we present a detailed discussion about the proposed 3-D CNN architecture, which learns the spatio-temporal features.

Iii-a Convolutional Neural Networks

There are two major problems with Artificial Neural Networks (ANN) while dealing with real world data like images, videos, and any other high-dimensional data.

  • ANNs do not maintain the local relationship among the neighboring pixels in a frame.

  • Since full connectivity is maintained throughout the network, the number of parameters are proportional to the input size.

To address these problems, Lecun et al. [25] introduced Convolutional Neural Networks (CNN), which are also called ConvNets. Extensive amount of research is being carried out on images using CNN architectures to solve many problems in computer vision and machine learning. However, their application in video stream classification is comparatively a less explored area of research. In this paper, we performed 3D convolutions in the convolutional layers of proposed 3D CNN architecture to extract the spatial and temporal features.

Iii-B Proposed 3D CNN Model: Extracting Spatio-Temporal Features

In 2-D CNNs, features are computed by applying the convolutions spatially over images. Whereas in case of videos, we have to consider the temporal information along with spatial features. So, it is required to extract the motion information encoded in contiguous frames using 3D convolutions. The proposed 3-dimensional CNN architecture, shown in Fig. 2, uses 3D convolutions.

Fig. 2: Proposed 3-dimensional CNN for spatio-temporal feature construction (KTH dataset). The first two convolution layers and , both have feature maps of dimension and , respectively. The and layers are followed by and , to reduce the spatial dimension by half. and layers have 32 feature maps of dimension and . Finally, a fully connected layer has neurons.

Initially, the Gaussian Weighing function is used to aggregate the entire video into frames (considered 100 frames from each video throughout our experiments). To reduce the memory overhead, person centered bounding boxes are retrieved as in [16, 17], which results in frames of spatial dimension and in case of KTH [30] and WEIZMANN [13] datasets, respectively.

In this paper, a 3D CNN model is proposed to extract spatio-temporal features, which is shown in Fig. 2. The proposed model considers the input of dimension , corresponding to frames (encoded using GWF) of pixels each. The proposed 3D CNN architecture has learnable layers, viz., , , , , and . and max pooling layers are applied after and to reduce the spatial dimension of the feature maps by half. The layer generates feature maps of size by convolving 3-D kernels of size . layer down samples the feature maps by half, after applying sub-sampling operation with a receptive field of , which results in a dimensional feature vector. The layer results in a dimensional feature map by convolving filters of size . The layer produces a dimensional feature vector, by applying sub-sampling operation with a receptive field of . The convolution layer () produces feature maps of dimension , which is obtained by convolving kernels of dimension . The layer generates feature maps of dimension , which is obtained by convolving filters of dimension . The feature maps produced by layer are flattened into a single feature vector of dimension , which is given as input to the fully connected layer (). Finally, the layer produces dimensional feature vector. The 3D CNN architecture proposed for spatio-temporal feature extraction, consists a total of trainable parameters.

For WEIZMANN dataset, we used same architecture with necessary modifications. However, throughout the architecture same hyper-parameters (number of filters, filter size) are maintained as in the case of KTH dataset. The 3D CNN model proposed for WEIZMANN dataset takes input of dimension . This model has four Conv layers (, , , and ) and two max-Pooling layers (, ) layers, and towards the end one fully connected layer (). The layer results in feature maps of dimension , which is obtained by convolving kernels of size . The layer generates reduce the spatial dimension by half, after applying sub-sampling with a receptive field of , which generates dimensional feature vector. The layer generates feature maps of dimension , this is obtained by applying filters of size . The layer generates a dimensional feature vector by sub-sampling with a receptive field of . The layer does not consider the right and bottom border feature values to avoid the dimension mismatch between input and filter size. The layer results in a dimensional feature vector, which is obtained by convolving filters of size . The layer results in feature maps of dimension , which is obtained by convolving 32 filters of dimension . The output of layer is rolled into a single column vector of dimension . At the end of the architecture, layer has neurons, which results in a dimensional feature vector. The proposed 3D CNN architecture for WEIZMANN human action dataset consists of number of learnable parameters. The learned spatio-temporal features are given as input to LSTM model to learn the label of the entire sequence.

Iii-C Classification using Long Short-Term Memory (LSTM)

Fig. 3: The proposed two-steps deep neural network approach. Encoded frames are given as input to the 3D CNN model to extract spatio-temporal features as discussed in secton III-B . The proposed 3D CNN model generates dimensional feature vector, which is given as input the LSTM model to classify human actions. The LSTM has one hidden layer with cells, that accumulates the individual decisions corresponding to small temporal neighborhood ( frames ) of the video.

Once the 3D-CNN architecture is trained, it learns the spatio-temporal features automatically. The learned features are provided as input to an LSTM architecture (a Recurrent Neural Networks (RNN)) for classification. RNNs are widely used deep learning models to accumulate the individual decisions related to small temporal neighborhood of the video. RNNs make use of recurrent connections to analyze the temporal data. However, RNNs able to learn the information which are about short duration. To learn the class label of the entire sequence, Long Short-Term Memory (LSTM) [11] is employed, which accumulates the individual decisions corresponds to each small temporal neighborhood. To obtain a sequence, we have considered every frames as a temporal neighborhood. To classify human actions, we employ an RNN model having a hidden layer of LSTM cells. Figure 3 shows the overview of the proposed two-steps learning process. The input to this RNN architecture is features per time step. These dimensional input features are fully connected with LSTM cells. The number of LSTM cells considered are as in [2]. The training details of the proposed 3D CNN architecture is presented in section IV-C1.

Iv Experiments, Results and Discussions

As the proposed method aims to classify human actions in a video, where the videos are captured at a distance from the performer, we trained and evaluated the proposed 3D CNN model on KTH and WEIZMANN datasets. Also we experimented with transfer learning techniques, where proposed method is trained with KTH and then tested on WEIZMANN dataset, and vice versa.

Iv-a KTH dataset

KTH dataset [30] is one among the popular datasets in human action recognition. This dataset consists of six actions, viz., walking, jogging, running, boxing, hand-waving, and hand-clapping which were carried out by persons and the videos were recorded in four different scenarios (outdoor, variations in scale, variations in cloths, and indoor). The spatial dimension of each frame is pixels and the rate of frames per second (fps) is . This dataset has videos. All the videos were captured from a distance from the performer. As a result, the area covered by the person is less than of the whole frame.

Iv-B WEIZMANN dataset

The WEIZMANN human activity recognition dataset [13] consists of videos that correspond to ten actions, which were performed by nine different people. The ten actions are gallop sideways (Side), jumping-back (jack), bending, one-hand-waving (Wave1), two-hands-waving (Wave2), walking, skipping, jumping in place (Pjump), jumping-forward (jump), and running. The spatial dimension of each frame is , and is at frames per second (fps). The area covered by the person is less than of the entire frame, due to the reason that videos were captured from a distance from the performer.

Iv-C Experimental Results

To validate the performance of the proposed 3-D CNN model, throughout our experiments, we have considered videos up to seconds length ( frames) and aggregated them into frames using Gaussian Weighing Function as discussed in Section II. To reduce the memory consumption, we have used the person-centered bounding boxes as in [16, 17]. Apart from these simple preprocessing steps we have not performed any other complex preprocessing like optical flow, gradients, etc.

Training Setup

To train the proposed 3-D CNN architectures, ReLU [22] is used as the activation function after every and layers (except output layer). Initially learning rate is considered as . The value of the learning rate reduced with a factor of after every epochs. The developed models are trained for epochs using Adam optimizer [20] with , , and decay = . The of entire data is used to train the 3D CNN model and remaining data is utilized to test the performance of the model. After employing Gaussian Weighting function, we obtained frames corresponding to an entire video. To reduce the amount of over-fitting, we generated and videos (of length frames) for KTH and WEIZMANN datasets, respectively, using data-augmentation techniques like vertical flip, horizontal flip, rotation by . We also employed dropout [32] (after each , layers except final layer with a rate of , after is applied) along with data augmentation to reduce the amount of over-fitting.

S.No. Method Features Classification Accuracy
1 Baccouche et al. [2] 3D CNN features
2 Gorelick et al. [13] Space-time saliency, Action dynamics 97.83

3
Fathi et al. [10] Mid-level motion features 100
4 Bilen et al. [4] 2D CNN features

5
Proposed method 3D CNN features 95.78 0.58
6 Proposed method applying Transfer Learning* 3D CNN features 96.53 0.07
  • Fine-tuning the last two layers of pre-trained model, which is trained on KTH dataset.

TABLE I: A performance comparison of state-of-the-art methods on WEIZMANN dataset with proposed 3D CNN model using -folds cross validation test.

Results and Discussions

The obtained results are compared with the state-of-the-art methods as shown in Tables I, II on WEIZMANN and KTH datasets, respectively. [2] reported accuracy over KTH dataset using a 3D CNN architecture having five trainable layers. However, they have not evaluated their model on WEIZMANN dataset, we obtained accuracy through our experiment (input dimension is 64x48x9) using the same architecture (same w.r.t number of features, filter size, number of neurons in layers) as in [2]. After employing the proposed scheme of generating aggregated video to the 3D CNN model proposed in [2], we observed that the model outperforming the original model. However, the Dynamic Image Network introduced by Bilen et al. [4] results in high amount of over-fitting due to which it produces only , accuracies for KTH, WEIZMANN datasets. The proposed 3D CNN model produces and accuracies on KTH and WEIZMANN datasets, respectively, when the size of Gaussian weight vector is . From Table I and Table II, we can observe that the proposed 3D CNN model outperforming other deep learning based models on both the datasets.

S.No. Method Features Classification Accuracy
1 Nazir et al. [29] Bag of Expressions (BoE) 99.51
2 Baccouche et al. [2] 3D CNN features 94.39
3 Abdulmunem et al. [1] Bag of Visual Words 97.20
4 Ji et al. [17] 3D CNN features 90.20
5 Wang et al. [37] Dense Trajectories and motion boundary descriptor 95.00
6 Gilbert et al. [12] Mined Hierarchical compound features 94.50
7 Yang et al. [39] Multi-scale oriented neighborhood features 96.50
8 Kovashka et al. [21] Hierarchical Space time neighborhood features 94.53
9 Bilen et al. [4] 2D CNN features
10 Baccouche et al. [2]# 3D CNN features 94.78 0.11
11 Proposed Method 3D CNN features 95.27 0.45
12 Proposed Method applying Transfer learning$ 3D CNN features 95.86 0.3
  • Encoded frames are given as input to the 3D CNN model proposed in [2]. (the size of Gaussian vector is ).

  • Fine-tuning the last two layers of pre-trained model, which is trained on WEIZMANN dataset.

TABLE II: Comparing the state-of-the-art human action recognition approaches on KTH dataset with the proposed 3D CNN model using -folds cross validation test.

When compared with human action recognition methods involving hand-crafted features, our method produces comparable results with state-of-the-art on both KTH and WEIZMANN datasets. We also experimented the performance of our model by varying the size of Gaussian weight vector in the range from to . The performance variation of proposed model is shown in Figure 4 by varying the size of Gaussian weight vector . We observe that the proposed 3D CNN architecture is showing the best accuracy, when the size of Gaussian weight vector = . Based on the results depicted in Table I and Table II, we can conclude that, our 3D CNN architecture outperforms the state-of-the-art deep learning architectures. However, due to the small size of the available dataset of such kind, the proposed deep learning based method could not outperform the hand-crafted feature based methods (although showing a comparable result).

Fig. 4: A performance comparison of proposed 3D CNN model by varying the size of Gaussian weight vector. The size of the Gaussian weight vector is considered as , , , , , and in our experiments.

Basha et al. [3] shown the necessity of the fully connected layers based on the depth of the CNN. Motivated by their work, experiments are conducted by varying the number of trainable layers in the proposed 3D CNN architecture. The amount of over-fitting increases in the context of both the datasets after inclusion of more layers. The performance of the proposed 3D CNN architecture with varying number of trainable layers is depicted in Figure 5.

Fig. 5: Comparing the Training and Testing accuracies of both the datasets by varying the number of trainable layers (, , , and ) in the proposed 3D CNN architecture.

A common practice in deep learning community (especially to deal with small datasets) is that, using the pre-trained models to reduce the training time and obtaining competitive results by training the models for a fewer number of epochs. Generally, these pre-trained models work as feature extractors. With this motivation, we utilized the pre-trained model of KTH dataset to fine-tune over WEIZMANN dataset and vice-versa. The last two layers (, ) of the proposed 3D CNN model are fine-tuned in both the cases. Results of the above experiments are reported in the last rows of the Table I and Table II, respectively. We can observe a little increase in the classification accuracy for both the datasets, after applying the above scheme.

V Conclusion

We introduced an information-rich sampling technique using Gaussian weighing function as a pre-processing step before giving it as input to any deep learning model, for better classification of human actions from videos. The proposed scheme aggregates consecutive frames into a single frame by applying a Gaussian weighted summation of the frames. We further proposed a 3D CNN model that learns and extracts spatio-temporal features by performing 3D convolutions. The classification of the human actions are performed using LSTM. Experimental results on both KTH and WEIZMANN datasets show that proposed model produces comparable results, among the state-of-the art. Whereas, the proposed 3D CNN model outperforms the state-of-the-art deep CNN models. Learning the weights for frame aggregation may be a potential future research direction.

Acknowledgments

We acknowledge the support of NVIDIA with the donation of the GeForce Titan XP GPU used for this research.

References

  1. A. Abdulmunem, Y. Lai and X. Sun (2016) Saliency guided local and global descriptors for effective action recognition. Computational Visual Media 2 (1), pp. 97–106. Cited by: §I, TABLE II.
  2. M. Baccouche, F. Mamalet, C. Wolf, C. Garcia and A. Baskurt (2011) Sequential deep learning for human action recognition. In International Workshop on Human Behavior Understanding, pp. 29–39. Cited by: §I, §I, §I, §I, §III-C, item #, §IV-C2, TABLE I, TABLE II.
  3. S. S. Basha, S. R. Dubey, V. Pulabaigari and S. Mukherjee (2020) Impact of fully connected layers on performance of convolutional neural networks for image classification. Neurocomputing 378, pp. 112–119. Cited by: §IV-C2.
  4. H. Bilen, B. Fernando, E. Gavves, A. Vedaldi and S. Gould (2016) Dynamic image networks for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3034–3042. Cited by: §I, §IV-C2, TABLE I, TABLE II.
  5. V. Buddubariki, S. G. Tulluri and S. Mukherjee (2016) Event recognition in egocentric videos using a novel trajectory based feature. In Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image Processing, pp. 76. Cited by: §I.
  6. R. Chaudhry, A. Ravichandran, G. Hager and R. Vidal (2009) Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1932–1939. Cited by: §I.
  7. M. Chen and A. Hauptmann (2009) Mosift: recognizing human actions in surveillance videos. Cited by: §I.
  8. D. D. Dawn and S. H. Shaikh (2016) A comprehensive survey of human action recognition with spatio-temporal interest point (stip) detector. The Visual Computer 32 (3), pp. 289–306. Cited by: §I.
  9. P. Dollár, V. Rabaud, G. Cottrell and S. Belongie (2005) Behavior recognition via sparse spatio-temporal features. In Visual Surveillance and Performance Evaluation of Tracking and Surveillance, 2005. 2nd Joint IEEE International Workshop on, pp. 65–72. Cited by: §I, §I.
  10. A. Fathi and G. Mori (2008) Action recognition by learning mid-level motion features. In Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on, pp. 1–8. Cited by: TABLE I.
  11. F. A. Gers, N. N. Schraudolph and J. Schmidhuber (2002) Learning precise timing with lstm recurrent networks. Journal of machine learning research 3 (Aug), pp. 115–143. Cited by: §I, §III-C.
  12. A. Gilbert, J. Illingworth and R. Bowden (2011) Action recognition using mined hierarchical compound features. IEEE Transactions on Pattern Analysis and Machine Intelligence 33 (5), pp. 883–897. Cited by: TABLE II.
  13. L. Gorelick, M. Blank, E. Shechtman, M. Irani and R. Basri (2007) Actions as space-time shapes. IEEE transactions on pattern analysis and machine intelligence 29 (12), pp. 2247–2253. Cited by: §III-B, §IV-B, TABLE I.
  14. C. G. Harris and M. Stephens (1988) A combined corner and edge detector.. In Alvey vision conference, Vol. 15, pp. 10–5244. Cited by: §I.
  15. S. Herath, M. Harandi and F. Porikli (2017) Going deeper into action recognition: a survey. Image and vision computing 60, pp. 4–21. Cited by: §I.
  16. H. Jhuang, T. Serre, L. Wolf and T. Poggio (2007) A biologically inspired system for action recognition. In Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, pp. 1–8. Cited by: §III-B, §IV-C.
  17. S. Ji, W. Xu, M. Yang and K. Yu (2013) 3D convolutional neural networks for human action recognition. IEEE transactions on pattern analysis and machine intelligence 35 (1), pp. 221–231. Cited by: §I, §I, §III-B, §IV-C, TABLE II.
  18. A. Kar, N. Rai, K. Sikka and G. Sharma (2017) Adascan: adaptive scan pooling in deep convolutional neural networks for human action recognition in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3376–3385. Cited by: §I.
  19. A. Karpathy, G. Toderici, S. Shetty, T. Leung, R. Sukthankar and L. Fei-Fei (2014) Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1725–1732. Cited by: §I.
  20. D. P. Kingma and J. Ba (2014) Adam: a method for stochastic optimization. arXiv preprint arXiv:1412.6980. Cited by: §IV-C1.
  21. A. Kovashka and K. Grauman (2010) Learning a hierarchy of discriminative space-time neighborhood features for human action recognition. In Computer Vision and Pattern Recognition (CVPR), 2010 IEEE Conference on, pp. 2046–2053. Cited by: TABLE II.
  22. A. Krizhevsky, I. Sutskever and G. E. Hinton (2012) Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105. Cited by: §IV-C1.
  23. I. Laptev and P. Pérez (2007) Retrieving actions in movies. In Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on, pp. 1–8. Cited by: §I.
  24. I. Laptev (2005) On space-time interest points. International journal of computer vision 64 (2-3), pp. 107–123. Cited by: §I.
  25. Y. LeCun, L. Bottou, Y. Bengio and P. Haffner (1998) Gradient-based learning applied to document recognition. Proceedings of the IEEE 86 (11), pp. 2278–2324. Cited by: §III-A.
  26. M. R. Malgireddy, I. Nwogu and V. Govindaraju (2013) Language-motivated approaches to action recognition. The Journal of Machine Learning Research 14 (1), pp. 2189–2212. Cited by: §I.
  27. S. Mukherjee, S. K. Biswas and D. P. Mukherjee (2014) Recognizing interactions between human performers by ‘dominating pose doublet’. Machine Vision and Applications 25 (4), pp. 1033–1052. Cited by: §I.
  28. S. Mukherjee (2015) Human action recognition using dominant pose duplet. In International Conference on Computer Vision Systems, pp. 488–497. Cited by: §I.
  29. S. Nazir, M. H. Yousaf, J. Nebel and S. A. Velastin (2018) A bag of expression framework for improved human action recognition. Pattern Recognition Letters. Cited by: §I, TABLE II.
  30. C. Schuldt, I. Laptev and B. Caputo (2004) Recognizing human actions: a local svm approach. In Pattern Recognition, 2004. ICPR 2004. Proceedings of the 17th International Conference on, Vol. 3, pp. 32–36. Cited by: §I, §III-B, §IV-A.
  31. K. Simonyan and A. Zisserman (2014) Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems, pp. 568–576. Cited by: §I, §I.
  32. N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever and R. Salakhutdinov (2014) Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research 15 (1), pp. 1929–1958. Cited by: §IV-C1.
  33. N. Srivastava, E. Mansimov and R. Salakhudinov (2015) Unsupervised learning of video representations using lstms. In International conference on machine learning, pp. 843–852. Cited by: §I.
  34. L. Sun, K. Jia, K. Chen, D. Yeung, B. E. Shi and S. Savarese (2017-10) Lattice long short-term memory for human action recognition. In The IEEE International Conference on Computer Vision (ICCV), Cited by: §I.
  35. G. W. Taylor, R. Fergus, Y. LeCun and C. Bregler (2010) Convolutional learning of spatio-temporal features. In European conference on computer vision, pp. 140–153. Cited by: §I.
  36. D. Tran, L. Bourdev, R. Fergus, L. Torresani and M. Paluri (2015) Learning spatiotemporal features with 3d convolutional networks. In Computer Vision (ICCV), 2015 IEEE International Conference on, pp. 4489–4497. Cited by: §I.
  37. H. Wang, A. Kläser, C. Schmid and C. Liu (2013) Dense trajectories and motion boundary descriptors for action recognition. International journal of computer vision 103 (1), pp. 60–79. Cited by: §I, TABLE II.
  38. L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang and L. Van Gool (2016) Temporal segment networks: towards good practices for deep action recognition. In European Conference on Computer Vision, pp. 20–36. Cited by: §I.
  39. J. Yang, Z. Ma and M. Xie (2015) Action recognition based on multi-scale oriented neighborhood features. International Journal of Signal Processing, Image Processing and Pattern Recognition 8 (1), pp. 241–254. Cited by: TABLE II.
  40. M. Ziaeefard and R. Bergevin (2015) Semantic human activity recognition: a literature review. Pattern Recognition 48 (8), pp. 2329–2345. Cited by: §I, §I.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
407124
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description