Three-Stream Fusion Network for First-Person Interaction Recognition

Three-Stream Fusion Network for First-Person Interaction Recognition

Abstract

First-person interaction recognition is a challenging task because of unstable video conditions resulting from the camera wearer’s movement. For human interaction recognition from a first-person viewpoint, this paper proposes a three-stream fusion network with two main parts: three-stream architecture and three-stream correlation fusion. The three-stream architecture captures the characteristics of the target appearance, target motion, and camera ego-motion. Meanwhile the three-stream correlation fusion combines the feature map of each of the three streams to consider the correlations among the target appearance, target motion, and camera ego-motion. The fused feature vector is robust to the camera movement and compensates for the noise of the camera ego-motion. Short-term intervals are modeled using the fused feature vector, and a long short-term memory(LSTM) model considers the temporal dynamics of the video. We evaluated the proposed method on two public benchmark datasets to validate the effectiveness of our approach. The experimental results show that the proposed fusion method successfully generated a discriminative feature vector, and our network outperformed all competing activity recognition methods in first-person videos where considerable camera ego-motion occurs.

keywords:
First-person vision, first-person interaction recognition, three-stream fusion network, three-stream correlation fusion, camera ego-motion.
1

1 Introduction

Despite the increasing research on computer vision, the task of understanding human activity in videos remains a challenging task. Recent approaches based on deep learning techniques have achieved significant progress in third-person activity recognition Duta et al. (2017); Feichtenhofer et al. (2016a, b); Simonyan and Zisserman (2014); Wang et al. (2017). In third-person videos, the camera is fixed and at a large distance from people and objects. Many researchers have also studied human activity recognition in first-person video Kitani et al. (2011); Kwon et al. (2018); Ma et al. (2016); Ryoo and Matthies (2013); Zaki et al. (2017). First-person video is captured by a camera mounted on a person or object. The video has a unique characteristics called camera ego-motion, which is not usually seen in third-person video. When the video includes camera ego-motion, the appearance of the target subjects is easily distorted and the motion vectors are disordered. Therefore, the recognition of activities in first-person video requires an appropriate approach tailored to these particular characteristics. In this paper, we analyze first-person video frames and focus especially on the interaction between a camera wearer and a human subject.

Figure 1: Punch action in third-person and first-person videos. The two video clips have different characteristics in terms of appearance and motion. (a) The optical flow is extracted from a third-person video, where the camera is fixed. (b) The optical flow is extracted from a first-person video where the camera shakes considerably.

In Figure 1, the optical flow extracted from two consecutive RGB frames shows various motion vectors in each third-person and first-person video. As shown in Figure 1(a), motion vectors in the third-person video appear around the region where people are positioned because the camera is fixed and stationary. However, Figure 1(b) shows complex motion vectors for the target and the camera wearer. The motion vectors appear close to the region where the target is positioned and the entire region of optical flow. Camera ego-motion renders analyzing the appearance and motion characteristics of the target in first-person video more challenging for three reasons: (1) it is difficult to build discriminative motion features from the complex motion vector in first-person video; (2) target appearance, which is an important feature for analyzing the target’s behavior, is severely distorted because of the camera wearer’s movement; (3) camera ego-motion can occur when the camera wearer moves, regardless of the target’s actions, and its data can be noisy when first-person interactions are analyzed.

This paper proposes a three-stream fusion network to recognize interactions in first-person video where large amounts of camera ego-motion occur. The proposed method is composed of two main parts: three-stream architecture and three-stream correlation fusion (TSCF). The three-stream architecture consists of target appearance stream, target motion stream, and ego-motion stream. The target appearance stream and the motion stream capture the appearance features and motion features of the target, respectively, and the ego-motion stream captures features of the camera wearer’s movement. The camera ego-motion is an important clue to the camera wearer’s movement because the wearer’s pose in first-person video is unknown. To generate robust features for the camera ego-motion, the TSCF considers two types of correlations. It considers the correlation between the target appearance and motion to utilize the spatiotemporal relationship. This relationship complements the target’s appearance and motion features, which are distorted by the camera wearer’s movement. To consider the problem of the camera wearer’s movement being noisy, the TSCF also uses the correlation between the target and camera ego-motion to determine whether the camera wearer’s movement is caused by the target’s action. In other words, we can determine whether the camera ego-motion is an important clue for analyzing first-person interactions between the target and the camera wearer.

The main contributions of this paper are summarized as follows: (1) we propose a novel deep learning framework called the three-stream fusion network. The proposed network is specialized in extracting discriminative features and considers camera ego-motion an important feature for analyzing the camera wearer’s movement; (2) we also introduce a fusion method called TSCF, which considers the correlations between the target’s appearance, motion and the camera ego-motion. The proposed fusion method creates robust features mitigate the effects of the camera ego-motion; (3)wWe show that our proposed method outperforms state-of-the-art activity recognition methods using the JPL First-Person Interaction dataset and the UTKinect-FirstPerson dataset.

2 Related Works

There has been a great deal of progress in human activity recognition in video captured from a third-person viewpoint. Early work contributed hand-craft features to feature representation for activity recognition Dalal and Triggs (2005); Park et al. (2004); Laptev et al. (2008); Sun et al. (2015); Roh et al. (2010); Suk et al. (2008); Roh and Lee (2000); Suk et al. (2011). Some studies suggested various methods, such as support vector machine (SVM) Schuldt et al. (2004); Xi et al. (2002), unsupervised learning Du et al. (2016), and multi-label learning Du et al. (2017a) to improve recognition performance. In more recent research, a significant performance increment was achieved using deep ConvNet Duta et al. (2017); Feichtenhofer et al. (2016a, b); Lin et al. (2018); Xu et al. (2018). Simonyan et al. Simonyan and Zisserman (2014) proposed a two-stream architecture composed of a spatial and a temporal stream to capture appearance and motion features separately. Feichtenhofer et al. Feichtenhofer et al. (2016b) proposed a number of fusion methods to combine the convolutional neural networks (CNNs) of the two-stream architecture. The spatial and temporal streams were combined at the convolutional layer of the CNNs to take advantage of spatiotemporal features. Other recent research studies on activity recognition utilized long short-term memory (LSTM) to handle the time-series information of video Donahue et al. (2015); Du et al. (2017b); Shi and Kim (2017). To overcome the viewpoint dependency, Roh et al. Roh et al. (2010) proposed a volume motion template (VMT) and a projected motion template (PMT). However, these templates are not optimized for first-person video, which has different characteristics, such as camera ego-motion, multiple visual scales, and proceeding events (e.g. after the target delivers a punch, the camera wearer falls down).

The research on activity recognition in first-person video can be divided into three categories. (1) The action recognition of a camera wearer Kwon et al. (2018); Iwashita et al. (2014); Possas et al. (2018). This research is focused on actions related to “What am I doing alone?” such as walking, running, standing, and going up the stairs. Ryoo et al. Ryoo et al. (2015a) proposed a novel feature representation to capture camera ego-motion and continuously track detailed changes while noisy data are suppressed. Abebe et al. Abebe and Cavallaro (2017) used stacked spectrograms obtained from mean grid-optical flow vectors and the displacement vectors of the intensity centroid to represent various motions in video. (2) The interaction recognition between a camera wearer’s hand and objects Shen et al. (2018); Singh et al. (2016, 2017); Yan et al. (2015); Zhou et al. (2016). This research is focused on actions related to “How do I interact with what type of objects?” such as take jam, close freezer, and cut mushroom. Ma et al. Ma et al. (2016) proposed an egocentric activity deep network using ObjectNet and ActionNet, where ObjectNet captures the object of an interesting region and the ActionNet captures the motion of both the camera wearer and the handled objects. Li et al. Li et al. (2015) proposed a set of egocentric features, such as hands, gaze, object features, and head motion, and showed the manner in which they can be combined. (3) The interaction recognition between a camera wearer and a human Ryoo and Matthies (2013); Zaki et al. (2017); Özkan et al. (2017); Ryoo et al. (2015b); Ryoo and Matthies (2016); Sudhakaran and Lanz (2017); Yonetani et al. (2016). This research is focused on action related to “What is he doing to me?” such as hand shake, pet, punch, hug, and throw actions. In this paper, we especially focus on the interaction between a camera wearer and a human. Ryoo et al. Ryoo and Matthies (2013) used hand-crafted features in first-person video, such as optical flow, to capture global and local motion descriptors. Early recognition methods Ryoo et al. (2015b) inferred an ongoing activity at the early stages in first-person video obtained from a robot’s viewpoint. Histograms of time-series gradients were used to consider the event history in a video. Zaki et al. Zaki et al. (2017) modeled sub-event dynamics to track their relations over time. After the local temporal structure of sub-events was encoded, the global temporal structure of the video was encoded. Sudhakaran et al. Sudhakaran and Lanz (2017) used a pair of CNNs to capture the frame-wise features of two consecutive input images and convolutional LSTM to aggregate them.

Our proposed network differ from previous first-person interaction recognition methods in that, in addition to the appearance and the motion of the target, it uses the camera ego-motion a feature type. These three feature types are combined using our fusion method. The method considers the correlations between the three types of features and thus it is rendered more robust to camera ego-motion. We show that this is an important design choice for analyzing first-person interaction, where a large amount of movement is generated by the camera wearer.

3 Three-Stream Fusion Network

An overview of our three-stream fusion network is shown in Figure 2. Our proposed network consists of the three-stream architecture, TSCF, and an LSTM model. Our three-stream architecture captures features for the target’s appearance, motion, and the camera ego-motion. The output feature maps from the three streams are combined by the proposed TSCF. The LSTM model classifies the video considering the temporal dynamics of TSCF features.

Figure 2: Overview of the three-stream fusion network. Our proposed network is composed of three-stream architecture, the three-stream correlation fusion (TSCF), and a long short-term memory model. Each stream of the three-stream architecture respectively extracts appearance, motion, and ego-motion feature maps. Then, the proposed TSCF combines the output feature maps of the three streams. The LSTM model takes the fused features as an input value to classify the video class.

3.1 Three-Stream Architecture

The proposed three-stream architecture is composed of the target appearance stream, the target motion stream, and the ego-motion stream, which respectively consider each of the characteristics individually. Intuitively, the target appearance stream and the target motion stream capture the appearance and motion cues of the target, and the ego-motion stream captures the cues of the camera ego-motion.

The target appearance stream uses the target region of an input image. To focus on the appearance of the target, we exclude as much of the background region of the input image as possible. First, an input image captured at time is taken to detect the target using the Faster R-CNN Ren et al. (2017), which can capture the coordinates of the human region. Then, we crop the region where the target (human) is located from the input image using the coordinates of the human region. The cropped region is entered into the target appearance stream. The target motion and ego-motion streams are designed under the consideration that the movement of the camera wearer generates motion vectors in the background region of the optical flow. Thus, these two streams use different regions of the optical flow to distinguish the target motion and the camera ego-motion in video. First, we stack the optical flows extracted from consecutive video frames. The stack consists of 20 optical flows of two directions from time to . Next, we divide each optical flow with the target region and background region(non-target region). In this case, the coordinates of the human region detected from the video frames are utilized to divide the optical flow. The target motion stream uses the target region of the optical flow as an input value. The aim is exclude the vectors of the camera ego-motion as much as possible from the optical flow and focus on the target motion. To create an input value of the ego-motion stream using the non-target region of the optical flow, we fill the zero values at the target region of the optical flow to focus on the camera ego-motion.

When considerable camera ego-motion occurs and the target appearance is seriously distorted, the Faster R-CNN occasionally fails to detect the human region in video frames. In this case, we set the zero matrix size of as an input value of the target appearance stream and the target motion stream. In addition, the ego-motion stream uses the optical flow itself as an input value.

The CNN of each of the three streams generates deep feature maps of the target appearance, target motion, and camera ego-motion. Then, each of the final convolutional layers of the CNNs is extracted as a feature map. As a result, the target appearance stream generates the appearance feature maps with the target region of the input image, the target motion stream generates the motion feature maps with the target region of the optical flow, and the ego-motion stream generates the ego-motion feature maps with the non-target region of the optical flow. We denote the feature maps by .

3.2 Three-Stream Correlation Fusion

We propose a fusion method called TSCF to create robust features to mitigate effects of the camera ego-motion. Figure 3 shows an overview of TSCF. The appearance, motion, and ego-motion feature maps are generated from the three-stream architecture. First, we sum the appearance and motion feature maps to consider the correlation between the target appearance and the target motion. Then attentional pooling is applied to the appearance feature maps with the sum of the appearance and motion feature maps. This is aimed to complement the target appearance, which is distorted by the camera ego-motion. After the maximum values are taken from the combined attention, we generate a channel-wise max pooling vector, which is intended to create a discriminative feature even if the camera is shaken.

Figure 3: Overview of the three-stream correlation fusion (TSCF). TSCF fuses the output feature maps of the three-stream architecture, considering the correlations between the target appearance, target motion, and camera ego-motion.

Second, the sum of the motion and ego-motion feature maps is calculated to consider the correlation of the motion vectors resulting from the interaction of the target human and the camera wearer. By considering this correlation, we can determine whether the camera wearer’s movement is caused by the target. We take the average values from the sum of the motion and ego-motion feature maps and create the channel-wise average pooling vector, because the sum of the motion and ego-motion feature maps include a noise value when the camera shakes considerably. We also consider the correlation between the target’s action and the camera wearer’s movement. A correlation vector is generated from the sum of the channel-wise max pooling vector and the channel-wise average pooling vector. It considers the correlation between the target’s action and the camera wearer’s movement.

To fuse the three types of feature maps, we use the sum fusion Feichtenhofer et al. (2016b), which can combine two different kinds of feature maps , at the same channel and spatial location ,:

(1)

where , , . and refer to specific feature maps. First, the appearance and motion feature maps are combined to make , which contains the spatiotemporal cues of the target’s action. Then, the sum of the motion and ego-motion feature maps is calculated in the same manner to produce .

We also calculate the attentional pooling Girdhar and Ramanan (2017) to compensate for the target appearance and target motion, which are distorted by the camera ego-motion. Because the attentional pooling method can render feature maps more related to certain tasks, we utilize it to make the sum of the appearance and motion feature maps more robust to the camera ego-motion. The feature map is taken from the at channel , and the matrix multiplication of is performed as follows:

(2)

where . Then, we calculate a matrix by performing the rank-1 approximation from . In this case, the singular value decomposition (SVD) function in the numpy library 1 was used to perform the -rank approximation. We perform the matrix multiplication with the of and the appearance feature map at channel to make a top-down attention . In addition, the of is used to perform the matrix multiplication with the appearance feature map to make a bottom-up attention . A combined attention is generated by performing element-wise multiplication on and :

(3)

where and .

The maximum value is taken from , and the values for all channels are concatenated to generate a vector . Channel-wise average values are taken from , and concatenated to generate a vector . We perform the element-wise sum on these two vectors as follows:

(4)

where denotes a vector. The element-wise fused features for all channels are concatenated to generate a correlation vector . The correlation vector contains correlation cues between the target’s action and the camera ego-motion. As a result, the proposed TSCF generates features that are robust against the camera movement by considering the correlations of the three different types of features.

3.3 Long Short-Term Memory for Classification

To consider the time-series dynamics of videos, we model short-term intervals of videos rather than a single image. To generate feature vectors of the short-term intervals of a video, we first concatenate correlation vectors from times to as follows:

(5)

where is a concatenated vector with times. Then fast fourier transform (FFT) is applied to as follows:

(6)

The vector is used as an input value of the LSTM to track the sequence of the short-term intervals. Then, the LSTM classifies the interaction in the video. As a result, we can calculate the correlation vector encoding on the short-term intervals through the FFT. We can also track the time-series dynamics of the video using the LSTM model.

4 Experiments

4.1 Datasets

The proposed three-stream fusion network and TSCF were evaluated on two public first-person video datasets: the UTKinect-FirstPerson dataset Xia et al. (2015) and the JPL First-Person Interaction dataset Ryoo and Matthies (2013). Both datasets are proposed for the purpose of first-person activity recognition. The recognition performances were evaluated on each dataset.

UTKinect-FirstPerson Dataset (humanoid). In this dataset, eight subjects interact with a humanoid robot on which a Kinect sensor is mounted. The human performs friendly, hostile, and normal behaviors toward the robot in a few different background settings. Figure 4 shows an example frame of each of nine classes: hand shake, hug, stand up, wave, point, punch, throw, run, and reach. For the experiments, we used only the RGB frames to allow fair comparisons with the RGB-based competing activity recognition methods. According to the segmented labels, we composed 174 clips from continuous sequences. Half of the clips were chosen as the training data, and the remaining half was used as the testing data. The resolution of the video is .

Figure 4: Sample frames from the UTKinect-FirstPerson (humanoid) dataset.

JPL First-Person Interaction Dataset. The dataset was captured using a GoPro2 camera mounted on the head of a humanoid model. The dataset consists of 84 videos with 7 classes: hand shake, hug, pet, wave, point-converse, punch, and throw. Figure 5 shows an example frame of each class. The actions were performed by eight different subjects in various indoor environments and lighting conditions and were recorded at resolution with 30 fps. The dataset was composed of four positive (hand shake, hug, pet, and wave), one normal (point-converse), and two negative (throw and punch) behavior video clips. For the experiments, we selected half of the videos as the training data and the remaining half as the testing data.

Figure 5: Sample frames from the JPL First-Person Interaction dataset.

4.2 Implementation Details

In the three-stream architecture, we utilized the Wang et al.’s VGG-16 Wang et al. (2015) pre-trained on the UCF101 dataset Soomro et al. (2012). The pre-trained VGG-16 which contains 13 convolutional layers and 3 fully connected layers was fine-tuned with the UTKinect-FirstPerson dataset and JPL First-Person Interaction dataset such that features were more related to the first-person interaction video. The VGG-16 of the target appearance stream was fine-tuned with the target region of the input images. In this case, the target region of the input image was resized to . The VGG-16 of the target motion stream was fine-tuned with the target region of the optical flow. The non-target region of the optical flow was fed to the ego-motion stream for fine-tuning. The target region and non-target region of the optical flow were resized to . All the images of the training data were used in the training step, and 20 uniformly selected images were used as input images in the testing step. We used the RMSProp optimization algorithm Hinton et al. () with a learning rate of and batch size of 20. The VGG-16 of the target appearance stream was trained for 150 iterations. The VGG-16 of each target motion stream and ego-motion stream was trained for 200 iterations. The LSTM model had 700 units in the LSTM cell. The weights and biases of the LSTM were initialized with random values having normal distributions. It was trained for iterations with a 0.9 forget bias and learning rate. We used gradient descent algorithms to optimize the LSTM model. Our CNNs and LSTM were trained to minimize the cross-entropy loss. The implementation of the proposed method was done using python. One NVIDIA GeForce GTX 1080Ti was used to run all the experiments.

Our final accuracies were the average values for 100 recognition performances. In the experiments, we used the RGB frame and difference of frames separately as an input value for the target appearance stream.

4.3 Performance Evaluations

UTKinect-FirstPerson dataset. Table 1 shows the recognition performance of the proposed method as compared with that of the competing activity recognition methods on the UTKinect-FirstPerson dataset. The results of long-term recurrent convolutional network (LRCN) Donahue et al. (2015) and Laptev et al. Laptev et al. (2008) are taken from Sudhakaran et al. Sudhakaran and Lanz (2017). For the accuracies of a kernelized pooling scheme based on feature subspaces (KRP FS) Cherian et al. (2018), we performed experiments on the UTKinect-FirstPerson dataset using the author’s code 2, except the optical flow. We further compared the proposed method with the Two-Stream ConvNet Simonyan and Zisserman (2014). To implement it, the VGGNet-16 pre-trained on the UCF101 dataset was fine-tuned with the UTKinect dataset.

Methods UTK (%)
M. S. Ryoo et al. Ryoo and Matthies (2013) 57.1
I. Laptev et al. Laptev et al. (2008) 48.4
Two-Stream ConvNet Simonyan and Zisserman (2014) 65.9
LRCN (RGB frame) Donahue et al. (2015) 72.6
LRCN (difference of frames) Donahue et al. (2015) 63.1
KRP FS (RGB frame) Cherian et al. (2018) 35.6
KRP FS (difference of frame) Cherian et al. (2018) 33.3
S. Sudhakaran et al. (RGB frame) Sudhakaran and Lanz (2017) 79.6
S. Sudhakaran et al. (difference of frames) Sudhakaran and Lanz (2017) 66.7
Ours (RGB frame) 83.1
Ours (difference of frames) 84.4
Table 1: Performance comparison of the proposed method with competing methods on the UTKinect-FirstPerson dataset.

In the UTKinect-FirstPerson dataset, the video clips contain numerous camera ego-motions that are not related to the target’s action. Therefore, the difference of frames can play the role of noise to analyze the target’s action in the UTKinect-FirstPerson dataset. In Table 1, the competing methods show a higher performance when the RGB frame instead of the difference of frames is used as the input value. However, our proposed method outperformed all the competing methods, whether the input value was the RGB frame or the difference of frames. This is because the TSCF of the proposed network generates robust features that mitigate the effects of camera ego-motion. In Figure 6, classes such as hand shake, hug, and punch, where a relatively large amount of camera ego-motion occurs, show a higher performance. These results show that our proposed method can recognize interactions in first-person video, although a large amount of camera ego-motion occurs. Figure 6 also shows that the recognition of the point and reach classes remain difficult. The point, wave, reach, and throw classes are similar, because the target is standing and stretching his arm. These actions should be more elaborately considered, reflecting the shape of the target’s hand or the motion of the arm.

As compared to the previous competing first-person interaction recognition methods Ryoo and Matthies (2013); Sudhakaran and Lanz (2017), our method shows significantly improved accuracy on the UTKinect-FirstPerson dataset. Furthermore, we compared three recent activity recognition methods Simonyan and Zisserman (2014); Donahue et al. (2015); Cherian et al. (2018) that do not consider the camera ego-motion, because they were developed based on third-person video. The results show that the proposed method outperformed the competing methods by considering the camera ego-motion for the first-person interaction recognition.

Figure 6: Confusion matrix of the proposed three-stream fusion network on the UTKinect-FirstPerson dataset.

JPL First-Person Interaction dataset. The recognition accuracies of the proposed method and competing activity recognition methods on the JPL First-Person Interaction dataset are shown in Table 2. The LRCN results Donahue et al. (2015) are taken from Sudhakaran et al. Sudhakaran and Lanz (2017). The results of KRP FS Cherian et al. (2018) are reproduced by using the author’s code 2. To obtain the results of two-stream ConvNet Simonyan and Zisserman (2014), we fine-tuned the VGGNet-16 which was pre-trained on the UCF101 dataset.

Methods JPL (%)
M. S. Ryoo et al. Ryoo and Matthies (2013) 89.6
Boosted MKL Özkan et al. (2017) 87.4
Two-Stream ConvNet Simonyan and Zisserman (2014) 54.2
LRCN (RGB frame) Donahue et al. (2015) 59.5
LRCN (difference of frames) Donahue et al. (2015) 89.0
KRP FS (RGB frame) Cherian et al. (2018) 73.8
KRP FS (difference of frames) Cherian et al. (2018) 85.7
SeDyn Zaki et al. (2017) + FTP Wang et al. (2014) 92.9
S. Sudhakaran et al. (RGB frame) Sudhakaran and Lanz (2017) 70.6
S. Sudhakaran et al. (difference of frames) Sudhakaran and Lanz (2017) 91.0
Ours (RGB frame) 88.0
Ours (difference of frames) 94.4
Table 2: Performance comparison of proposed method with competing methods on the JPL First-Person Interaction dataset.

As shown in Table 2, Sudhakaran et al. Sudhakaran and Lanz (2017), LRCN Donahue et al. (2015), KRP FS Cherian et al. (2018), and our proposed method showed a higher performance when the difference of frames was used instead of the RGB frame as the input value. We note that the recognition performance with the difference of frames is much higher than with the RGB frame on the JPL First-Person Interaction dataset than on the UTKinect-FirstPerson dataset. This is because the JPL First-Person Interaction dataset consists of camera ego-motion related to the target’s actions. The videos were captured while the camera was fixed to the wearer, and the camera wearer rarely moved by himself. Owing to the relatively stable camera movement, we believe that the difference of frames is a better representation of the motion pattern for two consecutive frames. As a result, the proposed method with the difference of frames outperformed other competing methods in the experiments. In addition, Figure 7 illustrates that our proposed method performed well for all action classes. These results show that our method is effective regardless of the existence of camera movement. However, some wave class videos, where the target waves his hand close to the camera wearer, were recognized as the pet class, as seen in Figure 7. We assume that these results occurred because a large number of motion vectors caused by a waving hand appeared in the target regions of the optical flow as in the pet class.

Figure 7: Confusion matrix of the proposed three-stream fusion network on the JPL First-Person Interaction dataset.

4.4 Ego-Motion Stream Evaluations

To validate the contribution of using camera ego-motion, we compared the proposed method and the two-stream (target appearance stream and target motion stream) method, excluding only the ego-motion stream in the implementation of the proposed method. The two-stream method was fused by using Eqs. (1), (2), and (3) in order. In Table 3, it can be seen that the proposed method that considers the camera ego-motion obtains the best performance. However, the two-stream (difference of frames) method on the UTKinect dataset obtains a 14.9% accuracy performance. In first-person video, the appearance of the target is easily distorted when the camera shakes a considerable amount. Because the camera shake is greater in the UTKinect dataset than in the JPL dataset, the difference of frames on the UTKinect dataset can act as noise. These results show that it is difficult to recognize interactions using only target appearance and target motion features. In addition, the effect of the ego-motion stream is more pronounced in the UTKinect dataset, where the camera shakes considerably. The results in Table 3 show that it is important to consider the camera ego-motion in first-person interaction recognition.

Stream UTK JPL
Two-stream (RGB frames) 67.1 68.9
Two-stream (Difference of frames) 14.9 74.0
Three-stream fusion network (RGB frame) 83.1 88.0
Three-stream fusion network (Difference of frames) 84.4 94.4
Table 3: Accuracies on the two-stream and the three-stream fusion networks.

4.5 Three-Stream Correlation Fusion Evaluations

To validate the proposed fusion method called TSCF, we compared the TSCF and frequently used fusion methods. Table 4 shows the performances when the three-stream fusion network used common fusion methods instead of the TSCF. In Eq. (4) , we used the channel-wise maximum values of the feature maps combined by the common fusion method. The table shows that the TSCF is the most suitable method for fusing the features of three-stream architecture.

Fusion methods UTK JPL
Sum Feichtenhofer et al. (2016b) (RGB frames) 81.7 86.3
Sum Feichtenhofer et al. (2016b) (Difference of frames) 83.7 92.5
Max Feichtenhofer et al. (2016b) (RGB frames) 82.4 87.3
Max Feichtenhofer et al. (2016b) (Difference of frames) 81.8 85.8
Bilinear Lin et al. (2015) (RGB frames) 74.4 77.8
Bilinear Lin et al. (2015) (Difference of frames) 74.7 77.6
TSCF(RGB frames) 83.1 88.0
TSCF(Difference of frames) 84.4 94.4
Table 4: Accuracies on various fusion methods and the three-stream correlation fusion.

Further, we demonstrated the effectiveness of the proposed TSCF for the first-person interaction recognition as compared to other features. The t-SNE technique Maaten and Hinton (2008) was used to visualize the feature vectors of the target appearance stream, the target motion stream, the ego-motion stream, and our TSCF method. To obtain the results for our TSCF, the t-SNE was applied to the input value of the LSTM. For the three streams, channel-wise average values were taken from each feature map and concatenated as a vector. Then, we applied the aforementioned Eqs. (5) and (6) to the concatenated vector, and obtained the t-SNE results.

Figure 8: t-SNE results for the UTKinect-FirstPerson (humanoid) dataset. (a), (b), and (c) represent the feature vectors of the target appearance stream, the target motion stream, and the ego-motion stream, respectively. (d) represents the feature vectors of the three-stream correlation fusion. The color list for the nine classes is as follows: hand shake(red), hug(green), stand up(blue), wave(pink), point(purple), punch(yellow), throw(orange), run(brown), and reach(gray).
Stream of the three-stream architecture UTK (%)
Target appearance stream 60.5
Target motion stream 78.3
Ego-motion stream 75.6
TSCF 84.4
Table 5: Performance comparison of each target appearance stream, target motion stream, ego-motion stream, and three-stream correlation fusion on the UTKinect-FirstPerson dataset.

Figure 8 shows the t-SNE results on the UTKinect-FirstPerson dataset. As compared to Figure 8(a), Figure 8(d) shows that the feature vectors of nine classes are less discriminative. Compared to Figure 8(b), Figure 8(d) is a better representation for the hug and punch classes, where a large amount of camera ego-motion occurs, because the target directly interacts with the camera wearer at a close distance. As compared to Figure 8(c), Figure 8(d) has more discriminative feature vectors for the stand up, wave, point, and throw classes, where some camera ego-motion occurs, because the target acts at a long distance from the camera wearer. In Table 5, we additionally compare the performance of our fusion method with individual single streams of the three-stream architecture. Our fusion method shows a higher performance than the other features.

Figure 9: t-SNE results for the JPL First-Person Interaction dataset. The results in (a), (b), and (c) represent the feature vectors of the target appearance stream, target motion stream, and ego-motion stream, resepectively. (d) represents the feature vectors of the three-stream correlation fusion. The color list for seven classes is as follows: hand shake(red), hug(green), pet(blue), wave(pink), point-converse(purple), punch(yellow), and throw(orange).
Stream of the three-stream architecture JPL (%)
Target appearance stream 68.7
Target motion stream 92.8
Ego-motion stream 88.1
TSCF 94.4
Table 6: Performance comparison for each target appearance stream, target motion stream, ego-motion stream, and the proposed fusion method on the JPL First-Person Interaction dataset.

We also evaluated our TSCF on the JPL First-Person Interaction dataset, the results of which are shown in Figure 9. The feature vectors of Figure 9(a) are less discriminative for all seven classes than those in Figure 9(d). Compared to Figure 9(b), Figure 9(d) has more discriminative feature vectors for the hug and pet classes where the target appearance is similar and a large amount of camera ego-motion occurs. As compared to Figure 9(c), Figure 9(d) is a better representation for the wave, point-converse, and throw classes, where little camera ego-motion occurs. In Table 6, it can be seen that our fusion method shows a higher performance than a method using any one of the three streams alone.

In all of the aforementioned datasets, our TSCF obtained the highest performance. Interestingly, the results of the ego-motion stream show that the camera ego-motion can be an important clue for analyzing first-person interaction. These results show that consideration of the correlations of the different three types of features can improve the discriminative power of the feature vectors. Although we fuse three different types of features, our TSCF rearranges these three feature types and generates more discriminative features to handle first-person interaction video. In other words, our TSCF can compensate for target appearance, target motion, and camera ego-motion features by considering the correlations of these three types of features, which have their own characteristics.

4.6 Correlation Evaluations

The proposed TSCF considers the correlation between a human’s appearance and motion by generating the sum of the appearance and motion feature maps. It also considers the correlation between the target’s motion and the camera wearer’s motion by summing the motion and ego-motion feature maps. Finally, we consider the correlation between the target’s behavior and the camera wearer’s movement by generating the correlation vector. To validate that it is important to consider these correlations for first-person interaction recognition, we compared the proposed TSCF and the three-stream deep features Kim et al. (2018). After the channel-wise maximum value and the channel-wise average value are obtained from the feature maps, the three-stream deep features technique Kim et al. (2018) sums all these values to make features for first-person interaction recognition.

Table 7. shows the recognition accuracy of the three-stream deep features Kim et al. (2018) on the UTKinect-FirstPerson dataset and the JPL First-Person Interaction dataset. We performed experiments to obtain the results of the three-stream deep features Kim et al. (2018) on the JPL First-Person Interaction dataset. As shown in Table 7, the three-stream deep features Kim et al. (2018) obtains similar accuracies when the raw frame and the difference of frames are used by considering the features of target appearance, target motion, and camera ego-motion. However, three-stream deep features shows lower accuracies than the proposed method. The results show that it is important to consider the correlations in first-person interaction recognition.

Fusion methods UTK JPL
Three-stream deep fusion Kim et al. (2018) (RGB frames) 80.37 81.1
Three-stream deep fusion Kim et al. (2018) (Difference of frames) 86.5 86.4
TSCF(RGB frames) 83.1 88.0
TSCF(Difference of frames) 84.4 94.4
Table 7: Accuracies of the three-stream deep fusion and the three-stream correlation fusion.

5 Conclusion

We proposed a three-stream fusion network for interaction recognition in first-person videos where camera ego-motion occurs. The TSCF was introduced to consider the correlations of target appearance, target motion, and camera ego-motion features. The proposed three-stream fusion network with the TSCF successfully classified the first-person interaction video clips by means of robust video feature vectors that mitigate the effects of the camera’s movement. The proposed method showed a state-of-the-art performance on the UTKinect-FirstPerson dataset and the JPL First-Person Interaction dataset. Performance comparisons with third-person video-based approaches showed that consideration of the camera ego-motion is an important aspect of first-person interaction recognition. In addition, comparisons with other fusion methods showed the effectiveness of the proposed TSCF which considers the correlations between target appearance, target motion, and camera ego-motion. Furthermore, the three-stream fusion network can recognize first-person interactions when the the camera movement occurs by showing a higher accuracy when the difference of frames is used as the input value. In the future, we will focus on the recognition of interaction classes where the target performs certain actions, such as point and wave, at a distance from the camera wearer.

Acknowledgment

This work was supported by Institute for Information & Communcations Technology Planning & Evaluation(IITP) grant funded by the Korea goverment(MSIT) [No. 2019-0-00079, Department of Artificial Intelligence, Korea University] and [No. 2014-0-00059, Development of Predictive Visual Intelligence Technology].

Footnotes

  1. journal: Pattern Recognition

References

  1. Note: \urlhttps://docs.scipy.org Cited by: §3.2.
  2. Note: \urlhttps://github.com/anoopcherian/kernelized_rank_pool Cited by: §4.3, §4.3.
  3. A long short-term memory convolutional neural network for first-person vision activity recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1339–1346. Cited by: §2.
  4. Non-linear temporal subspace representations for activity recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2197–2206. Cited by: §4.3, §4.3, §4.3, §4.3, Table 1, Table 2.
  5. Histograms of oriented gradients for human detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 886–893. Cited by: §2.
  6. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2625–2634. Cited by: §2, §4.3, §4.3, §4.3, §4.3, Table 1, Table 2.
  7. Robust and discriminative labeling for multi-label active learning based on maximum correntropy criterion. IEEE Transactions on Image Processing 26 (4), pp. 1694–1707. Cited by: §2.
  8. Stacked convolutional denoising auto-encoders for feature representation. IEEE Transactions on Cybernetics 47 (4), pp. 1017–1027. Cited by: §2.
  9. RPAN: an end-to-end recurrent pose-attention network for action recognition in videos. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3725–3734. Cited by: §2.
  10. Spatio-temporal vector of locally max pooled features for action recognition in videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3205–3214. Cited by: §1, §2.
  11. Spatiotemporal residual networks for video action recognition. In Advances in Neural Information Processing Systems, pp. 3468–3476. Cited by: §1, §2.
  12. Convolutional two-stream network fusion for video action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1933–1941. Cited by: §1, §2, §3.2, Table 4.
  13. Attentional pooling for action recognition. In Advances in Neural Information Processing Systems, pp. 34–45. Cited by: §3.2.
  14. G. Hinton, N. Srivastava and K. Swersky Lecture 6a overview of mini-batch gradient descent (2012). Coursera Lecture slides https://class. coursera. org/neuralnets-2012-001/lecture. Cited by: §4.2.
  15. First-person animal activity recognition from egocentric videos. In Proceedings of the IEEE International Conference on Pattern Recognition, pp. 4310–4315. Cited by: §2.
  16. First-person activity recognition based on three-stream deep features. In International Conference on Control, Automation and Systems, pp. 297–299. Cited by: §4.6, §4.6, Table 7.
  17. Fast unsupervised ego-action learning for first-person sports videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3241–3248. Cited by: §1.
  18. First person action recognition via two-stream convnet with long-term fusion pooling. Pattern Recognition Letters 112, pp. 161–167. Cited by: §1, §2.
  19. Learning realistic human actions from movies. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1–8. Cited by: §2, §4.3, Table 1.
  20. Delving into egocentric actions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 287–295. Cited by: §2.
  21. Bilinear cnn models for fine-grained visual recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1449–1457. Cited by: Table 4.
  22. Action recognition with coarse-to-fine deep feature integration and asynchronous fusion. In AAAI, Cited by: §2.
  23. Going deeper into first-person activity recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1894–1903. Cited by: §1, §2.
  24. Visualizing data using t-SNE. Journal of machine learning research 9, pp. 2579–2605. Cited by: §4.5.
  25. Boosted multiple kernel learning for first-person activity recognition. In European Signal Processing Conference, pp. 1050–1054. Cited by: §2, Table 2.
  26. Qualitative estimation of camera motion parameters from the linear composition of optical flow. Pattern Recognition 37 (4), pp. 767–779. Cited by: §2.
  27. Egocentric activity recognition on a budget. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5967–5976. Cited by: §2.
  28. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (6), pp. 1137–1149. Cited by: §3.1.
  29. Multiple people tracking using an appearance model based on temporal color. In Proceedings of the International Workshop on Biologically Motivated Computer Vision, pp. 369–378. Cited by: §2.
  30. View-independent human action recognition with volume motion template on single stereo camera. Pattern Recognition Letters 31 (7), pp. 639–647. Cited by: §2.
  31. First-person activity recognition: what are they doing to me?. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2730–2737. Cited by: §1, §2, §4.1, §4.3, Table 1, Table 2.
  32. First-person activity recognition: feature, temporal structure, and prediction. International Journal of Computer Vision 119 (3), pp. 307–328. Cited by: §2.
  33. Pooled motion features for first-person videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 896–904. Cited by: §2.
  34. Robot-centric activity prediction from first-person videos: what will they do to me?. In Proceedings of the IEEE International Conference on Human-Robot Interaction, pp. 295–302. Cited by: §2.
  35. Recognizing human actions: a local SVM approach. In Proceedings of the IEEE International Conference on Pattern Recognition, pp. 32–36. Cited by: §2.
  36. Egocentric activity prediction via event modulated attention. In Proceedings of the European Conference on Computer Vision, pp. 197–212. Cited by: §2.
  37. Learning and refining of privileged information-based rnns for action recognition from depth sequences. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3461–3470. Cited by: §2.
  38. Two-stream convolutional networks for action recognition in videos. In Advances in Neural Information Processing Systems, pp. 568–576. Cited by: §1, §2, §4.3, §4.3, §4.3, Table 1, Table 2.
  39. First person action recognition using deep learned descriptors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2620–2628. Cited by: §2.
  40. Trajectory aligned features for first person action recognition. Pattern Recognition 62, pp. 45–55. Cited by: §2.
  41. UCF101: a dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402. Cited by: §4.2.
  42. Convolutional long short-term memory networks for recognizing first person interactions. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2339–2346. Cited by: §2, §4.3, §4.3, §4.3, §4.3, Table 1, Table 2.
  43. A network of dynamic probabilistic models for human interaction analysis. IEEE Transactions on Circuits and Systems for Video Technology 21 (7), pp. 932–945. Cited by: §2.
  44. Recognizing hand gestures using dynamic bayesian network. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition, pp. 1–6. Cited by: §2.
  45. Human action recognition using factorized spatio-temporal convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4597–4605. Cited by: §2.
  46. Learning actionlet ensemble for 3D human action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 36 (5), pp. 914–927. Cited by: Table 2.
  47. Towards good practices for very deep two-stream convnets. arXiv preprint arXiv:1507.02159. Cited by: §4.2.
  48. Spatiotemporal pyramid network for video action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1529–1538. Cited by: §1.
  49. Facial component extraction and face recognition with support vector machines. In Proceedings of the IEEE International Conference on Automatic Face Gesture Recognition, pp. 83–88. Cited by: §2.
  50. Robot-centric activity recognition from first-person rgb-d videos. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision, pp. 357–364. Cited by: §4.1.
  51. Spectral–spatial unified networks for hyperspectral image classification. IEEE Transactions on Geoscience and Remote Sensing 56 (10), pp. 5893–5909. Cited by: §2.
  52. Egocentric daily activity recognition via multitask clustering. IEEE Transactions on Image Processing 24 (10), pp. 2984–2995. Cited by: §2.
  53. Recognizing micro-actions and reactions from paired egocentric videos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2629–2638. Cited by: §2.
  54. Modeling sub-event dynamics in first-person action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7253–7262. Cited by: §1, §2, Table 2.
  55. Cascaded interactional targeting network for egocentric video analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1904–1913. Cited by: §2.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
408732
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description