Symbiotic Graph Neural Networks
for 3D Skeleton-based Human Action Recognition and Motion Prediction
3D skeleton-based action recognition and motion prediction are two essential problems of human activity understanding. In many previous works: 1) they studied two tasks separately, neglecting internal correlations; 2) they did not capture sufficient relations inside the body. To address these issues, we propose a symbiotic model to handle two tasks jointly; and we propose two scales of graphs to explicitly capture relations among body-joints and body-parts. Together, we propose symbiotic graph neural networks, which contains a backbone, an action-recognition head, and a motion-prediction head. Two heads are trained jointly and enhance each other. For the backbone, we propose multi-branch multi-scale graph convolution networks to extract spatial and temporal features. The multi-scale graph convolution networks are based on joint-scale and part-scale graphs. The joint-scale graphs contain actional graphs, capturing action-based relations, and structural graphs, capturing physical constraints. The part-scale graphs integrate body-joints to form specific parts, representing high-level relations. Moreover, dual bone-based graphs and networks are proposed to learn complementary features. We conduct extensive experiments for skeleton-based action recognition and motion prediction with four datasets, NTU-RGB+D, Kinetics, Human3.6M, and CMU Mocap. Experiments show that our symbiotic graph neural networks achieve better performances on both tasks compared to the state-of-the-art methods. The code is relased at github.com/limaosen0/Sym-GNN
Human action recognition and motion prediction are crucial problems in computer vision, being widely applicable to surveillance , pedestrian tracking , and human-machine interaction . Respectively, action recognition aims to accurately classify the categories of query actions ; and motion prediction forecasts the future movements based on observations .
The data of actions can be represented with various formats, including RGB videos  and 3D skeleton data . Notably, 3D skeleton data, locating 3D body-joints, is shown to be effective in action representation, efficient in computation, as well as robust against environmental noise . In this work, we focus on action recognition and motion prediction based on the 3D skeleton data.
In most previous studies, 3D skeleton-based action recognition and motion prediction are treated separately as the former needs to discriminate the classes; while the latter generates the future poses. For action recognition, methods employed full action sequences for pattern learning [10, 11, 12, 13, 14]; however, with the long-term inputs, these methods failed in some real-time applications due to the hysteretic discrimination, while the model should response as early as possible. As for motion prediction, previous works built generative models [15, 16, 17, 18]; they learned motion dynamics, but often ignored semantics. Actually, there are mutual promotions between the tasks of action recognition and motion prediction, while previous works rarely explored them. For example, the classifier provides the action categories as the auxiliary information to guide prediction, as well as the predictor preserves more detailed information for accurate recognition via self-supervision. Considering to exploit mutual promotions, we aim to develop a symbiotic method to enable action recognition and motion prediction simultaneously.
For both 3D skeleton-based action recognition and motion prediction, the key is to effectively capture the motion patterns of various actions. A lot of efforts have been made to push towards this direction. Concretely, some traditional attempts often vectorized all the joints to a pose vector and built hand-crafted models for feature learning [19, 20, 10, 15, 21, 22, 23]. Recently, some deep models based on either convolutional neural networks (CNN) or recurrent neural networks (RNN) learned high-level features from data [11, 24, 12, 13, 16, 17, 25, 18, 26, 27]; however, these methods rarely investigated the joint relations, missing crucial activity dynamics. To capture richer features, several works exploited joint relations from various aspects.  proposed skeleton-graphs with nodes as joints and edges as bones. [28, 29, 16] built the relations between body-parts, such as limbs.  merged individual part features.  leveraged spatial convolutions on pose vectors, but it varied on joint permutation. These works aggregated information from local or coarse neighborhoods. Notably, some relations may exist among action-related joints, such as hands and feet moving collaboratively during walking. Moreover, some methods of motion prediction fed ground-truth action categories to enhance performance in both training and testing phases, but the true labels are hard to obtain during the real-world scenarios. To solve those issues, we construct graphs to model both local and long-range body relations and use graph convolutions to capture informative spatial features.
In this paper, we propose a novel model called symbiotic graph neural network (Sym-GNN), which handles 3D skeleton-based action recognition and motion prediction simultaneously and uses graph-based operations to capture spatial features. As basic operators of Sym-GNN, we propose the joint-scale graph convolution (JGC) and part-scale graph convolution (PGC) operators to extract multi-scale spatial information. JGC is based on two types of graphs: actional graphs and structural graphs. The actional graphs are learned from 3D skeleton data by an actional graph inference module (AGIM), capturing action-based relations; the structural graphs are built by extending the skeleton graphs, capturing physical constraints. PGC is based on a part-scale graph, whose nodes are integrated body-part features and edges are based on body-part connections. We also propose a difference operator to extract multiple orders of motion differences, reflecting positions, velocities, and accelerations of body-joints.
The proposed Sym-GNN consists of a backbone, called multi-branch multi-scale graph convolutional network (multi-branch multi-scale GCN), an action-recognition head and a motion-prediction head; see Fig. 1. The backbone uses joint-scale and part-scale graphs for spatial relations presentation and high-level feature extraction; two heads work on two separated tasks. Moreover, there are task promotions, i.e. the action-recognition head determines action categories, which is used to enhance prediction performance; meanwhile, the motion-prediction head predicts poses and improves recognition by promoting self-supervision and preserving detailed features. The model is trained through a multitasking paradigm. Additionally, we build a dual bone-based network which treats bones as graph nodes and learns bone features to obtain complementary semantics for more effective classification and prediction.
To validate the Sym-GNN, we conduct extensive experiments on four large-scale datasets: NTU-RGB+D , Kinetics , Human 3.6M , and CMU Mocap111http://mocap.cs.cmu.edu/. The results show that 1) Sym-GNN outperforms the state-of-the-art methods in both action recognition and motion prediction; 2) Using the symbiotic model to train the two tasks simultaneously produces better performance than using individual models; and 3) the multi-scale graphs model complicated relations between body-joints and body-parts, and the proposed JGC extract informative spatial features.
Overall, the main contributions in this paper are summarized as follows:
Multitasking framework. We propose novel symbiotic graph neural networks (Sym-GNN) to achieve 3D skeleton-based action recognition and motion prediction in a multitasking framework. Sym-GNN contains a backbone, an action-recognition head, and a motion-prediction head. We exploit the mutual promotion between two heads, leading to improvements in both tasks; see Section 5
Basic operators. We propose novel operators to extract information from 3D skeleton data: 1) a joint-scale graph convolution operator is proposed to extract joint-level spatial features based on both actional and structral graphs; see Section 4.1.4; 2) a part-scale graph convolution operator is proposed to extract part-level spatial features based on part-scale graphs; see Section 4.2.1; 3) a pair of bidirectional fusion operators is proposed to fuse information across two scales; see Section 5.1.2 and 4) a difference operator is proposed to extract temporal features; see Section 4.3; and
Experimental findings. We conduct extensive experiments for both tasks of 3D skeleton-based action recognition and motion prediction. The results show that Sym-GNN outperforms the state-of-the-art methods in both tasks; see Section 6
2 Related Works
3D skeleton-based action recognition. Numerous methods are proposed for 3D skeleton-based action recognition. Conventionally, some models learned semantics based on hand-crafted features and physical intuitions [19, 20, 10]. In the deep learning era, models automatically learn features from data. Some recurrent-neural-network-based (RNN-based) models captured the temporal dependencies between consecutive frames [12, 34]. Moreover, convolutional neural networks (CNN) also achieve remarkable results [24, 13]. Recently, the graph-based approaches drew many attentions [14, 28, 29, 35, 36, 1, 37, 38]. In this work, we adopt the graph-based approach. We construct multi-scale graphs adaptively from data, capturing useful and comprehensive information about actions.
3D skeleton-based motion prediction. In earlier studies, state models were considered to predict future motions [15, 21, 39]. Recently, deep learning technique plays increasingly important roles. Some RNN-based methods learned the dynamics from sequences [16, 17, 40, 41, 42, 27]. Moreover, adversarial mechanics and geodesic loss could further improve predictions . As for our method, we use graph structures to explicitly model the relations between body-joints and body-parts, guiding the networks to learn local and non-local motion patterns.
Graph deep learning. Graphs, focused on by many recent studies, are effective to express data associated with non-grid structures [14, 43, 44, 45, 46, 47, 48, 28]. Given the fixed topologies, previous works explored to propagate node features based on the spectral domain [46, 47] or the vertex domain . [14, 1, 29, 36, 35] leveraged graph convolution for 3D skeleton-based action recognition.  also considered the skeleton-based relations for motion prediction. In this paper, we propose multi-scale graphs to represent multiple relations: joint-scale and part-scale relations. Then, we propose novel graph convolution operators to extract deep features for action recognition and motion prediction. Different from  obtaining multiple actional graphs with complicated inference processes, our method employs more efficient graph learning operations.
3 Problem Formulation
In this paper, we study 3D skeleton-based action recognition and motion prediction jointly. We here use the 3D joint positions along the time to represent the action sequences. Mathematically, let the action pose at time stamp be , where indicates the future frames, otherwise the observed frames; notably, denotes the current frame. is the number of joints and reflects the 3D joint positions. The action pose is essentially associated with a skeleton graph, which represents the pairwise bone connectivity. We can represent a skeleton graph by a binary adjacent matrix; that is, , where the th elements when the th and the th body-joints are connected with bones, and , otherwise. Note that includes self-loops.
For an action sequence belonging to one class, we have , where denotes the previous motion tensor; denotes the future motion tensor; and are the frame numbers of previous and future motions, respectively; and one-hot vector denotes the class-label in possible classes. Let be the overall model. The discriminated class category and the predicted motion are formulated as
where , and denote trainable parameters of the backbone, the action-recognition head and the motion-prediction head, respectively.
4 Basic Components
In this section, we propose some novel components in our model. We first propose some joint-scale graph operators, extracting features among body-joints; we next propose the part-scale graph operators, extracting features among body-parts; finally, we propose a difference operator to provide richer motion priors.
4.1 Joint-Scale Graph Operators
To model the joint relations, we build joint-scale graphs including actional graphs, capturing moving interactions between joints even without bone-connection, and structural graphs, extending the skeleton structures to represent physical constraints. Fig. 2 sketches some examples. Plot (a) shows a skeleton graph with local neighborhood; plot (b) shows an actional graph, which captures action-based dependencies, e.g. ‘Left Hand’ is linked with ‘Right Hand’ and feet during walking; plot (c) shows a structural graph, which allows ‘Left Hand’ to link with entire arm.
As follows, we propose the construction of joint-scale actional and structural graphs. And, we present the joint-scale graph and temporal convolution (J-GTC) block to learn spatial and temporal features of sequential actions.
4.1.1 Actional Graph Convolution
For different movements, some structurally distant joints may interact, leading to action-based relations. For example, a walking person moves hands and feet collaboratively. To represent actional relations, we employ an actional graph: , where is the joint set and is the adjacency matrix that reveals the pairwise joint-scale actional relations. To obtain this topology, we propose a data-adaptive module, called actional graphs inference module (AGIM), to learn purely from observations without knowing action categories.
To utilize the body dynamics, we let the vector representation of the th joint positions across all observed frames be , which includes the previous positions. To learn all relations, we propagate pose information between body-joints and possible edges. We first initialize , where is a multilayer perceptron (MLP) that maps the raw joint moving data to joint features . In the th iteration, the features are propagated as follows:
where , are the feature vectors of the th joints and the edge connecting the th and th joints at the th iteration; and are two MLP-formed feature extractors; is the concatenation; and denote the dimensions of edge and joint features, respectively. (1a) maps a pair of joint features to the in-between edge features; (1b) aggregates all edge features associated with the same joint and maps to the corresponding joint features. After iterations, information are fully propagated between joints and edges; in other words, each joint feature obtained has aggregated the integrated information in a long-range.
Given any joint feature after iterations, , we compute the relation strength between each pair of joints, leading to an actional graph. We build two individual embedding networks, and , to further learn the high-level representations of joints. The th element of the adjacent matrix of actional graph is formulated as
where and are the two different embeddings of joint . Notably, , indicating incoming and outgoing relations between joints. (2) uses the softmax to normalize the edge weights and promote a few large ones.
The structure of AGIM is illustrated in Fig. 3, where the information is propagated for times between joints and any possible edges and two embedded joint features are used to calculate actional graphs in the end.
Given the joint-scale actional graphs with adjacent matrix , we aggregate the joint information along the action-based relations. We design an actional graph convolution (AGC) to capture the actional features. Mathematically, let the input features at frame be , the output features be , the AGC works as
where is a trainable weight. Therefore, the model aggregates the action-based information from collaboratively moving joints even in the distance.
4.1.2 Structural Graph Convolution
Intuitively, the joint dynamics is limited due to physical constraints, namely bone connections. To capture these relations, we develop a structural graph, . Let is the adjacency matrix of the skeleton graph (see Section 3), the normalized adjacency matrix be
We note that only describes the 1-hop neighborhood on body; that is, the bone-connected joints. To represent long-range relations, we use the high-order polynomial of . Let the -order polynomial of be , which could be directly computed from the skeleton structure; indicates the relations between each joint and its -hop neighbors on skeleton. Given the high-order topologies, we introduce several individual edge-weight matrices corresponding to , where each element is trainable to reflect the relation strength. We finally obtain the -order structural graph, whose weighted adjacency matrices is
where denotes the element-wise multiplication. In this way, we are able to model the structure-based relations between one joint and others in relatively longer range. Practically, we consider the order , thus we have multiple structural graphs for one body. See plot (c) in Fig. 2, the hand is correlated with the entire arm.
Given the structural graphs , we propose the structural graph convolution (SGC) operator. Let the input feature at frame be , the output feature be , the SGC operator is formulated as
where is the trainable model parameters. Notably, the multiple structural graphs have different corresponding weights, which help to extract richer features.
4.1.3 Joint-Scale Graph Convolution
To extract both spatial and temporal features of actions, we now propose the joint-scale graph and temporal convolution block. Based on AGC (3) and SGC (4), we present the joint-scale graph convolution (JGC) to capture comprehensive joint-scale spatial features. Mathematically, let the input joint features at frame be , the output features be , the JGC is formulated as
where is a hyper-parameter to trade off the contribution between actional and structural features. Some non-linear activation functions can be applied to it. In this way, the joint features are effectively aggregated to update each center joint according to the joint-scale graphs.
We further show the stability of the proposed activated joint-scale graph convolution layer; that is, when input 3D skeleton data is disturbed, the distortion of the output features is upper bounded.
(Stability) Let two joint-scale feature matrices be and associated with a skeleton graph , where and (). Let and . Let and be the joint-scale actional graph inferred from and , respectively, where
with the amplify factor and some constant. Let , and , where , and . Then,
Note that denotes Frobenius norm and is zero norm. denotes ReLU-activation on each element of the data. denotes the effects that rely on ‘’.
See the proof in Appendix. Theorem 4.1.3 only shows the joint-scale graph convolution at the first layer, but the bound can be extended to the subsequent layers. We make this distinction because the actional graph only depends on the input data. Theorem 4.1.3 shows that 1) the outputs of JGC followed by the activation function can be upper bounded, reflecting its robustness against perturbation of inputs; and 2) given a fixed model, the bound is mainly related to the amplify factor , reflecting how much the actional graph would amplify the perturbation. In the experiments, we show that is around 1. We also test JGC’s robustness against the input perturbation, ensuring that Sym-GNN has stable performance given small noises.
4.1.4 Joint-Scale Graph and Temporal Convolution Block
While the JGC operator leverages the joint spatial relations and extracts rich features, we should consider modeling the temporal dependencies among consecutive frames. We develop the temporal convolution operator (TC); that is, a convolution along time to learn the movements. Stacking JGC and TC, we build the joint-scale graph and temporal convolution block (J-GTC block), which learn the spatial and temporal features in tandem. Mathematically, let be an input tensor, each J-GTC block works as
where represents a nonlinear ReLU function, is a standard 1D convolution along the time axis, whose temporal kernel size is ; is the convolution stride along time to shrink the temporal dimension; is the time stamp. In each J-GTC block, (6a) extracts spatial features using the multiple spatial relations between joints; and (6b) extracts temporal features by aggregating the information in several consecutive frames. Our J-GTC also includes batch normalizations and dropout operations. Moreover, there is a residual connection preserving the input features. The architecture of one J-GTC block is illustrated in Fig. 4.
By stacking several J-GTC blocks in a hierarchy, we could gradually convert the motion dynamics from the sample space to the feature space; that is we capture the high-level semantic information of input sequence for action recognition and motion prediction downstream.
4.2 Part-Scale Graph Operators
The joint-scale graphs treat body-joints as nodes and model their relations, but some action patterns depend on more abstract movements of body-parts. For example, ‘hand waving’ shows a rising arm, but the finger and wrist are less important. To model the part dynamics, we propose a part-scale graph and the part-scale graph and temporal convolution (P-GTC) block to extract part-scale features.
4.2.1 Part-Scale Graph Convolution
For a part-scale graph, we define body-parts as graph nodes: ‘head’, ‘torso’, pairs of ‘upper arms’, ‘forearms’, ‘thighs’ and ‘crura’, which integrates the covered joints on joint-scale body. And we build the edges according to body nature. The right plot of Fig. 5 shows an example of part-scale graph, whose vertices are parts and edges are based on nature. The self-looped binary adjacent matrix of part-scale graph is , where if the th and th parts are connected. We normalize by
where is the diagonal degree matrix of ; is a trainable weight matrix and is the element-wise multiplication.
Similarly to the JGC operator (5), we propose the part-scale graph convolution (PGC) for spatial feature learning. Let the part features at time be , the output features be , the PGC works as
where is the trainable parameters. With (7), we propagate information between body-parts on the part-scale graph, leading to abstract spatial patterns. Notably, We do not need a part-scale actional graph, because the part-scale graph includes some integrated relations internally, as well as it has a shorter distance to build long-range links.
4.2.2 Part-Scale Graph and Temporal Convolution Block
Considering the temporal evolution, we use the same temporal convolution as in J-GTC block to form the part-scale graph and temporal convolution block (P-GTC block). Let the input part feature tensor be , we have
where denotes the time stamp and is the temporal convolution stride. Comparing to the J-GTC block, the P-GTC block extracts the spatial and temporal features of actions in a higher scale.
4.3 Difference Operator
Intuitively, the states of motion, such as velocity and acceleration, carry important dynamics information and make it easier to extract spatial-temporal features. To achieve this, we employ a difference operator to preprocess the input sequences. The idea is to compute high-order differences of the pose sequences, guiding the model to learn motion information more easily. The zero-order difference is , where is the pose at the time , and the -order difference () of the pose is
where denotes the th-order difference operator. We use zero paddings to handle boundary conditions. We take the first three orders () to our model, reflecting positions, velocities, and accelerations. In the model, the three differences can be efficiently computed in parallel.
5 Symbiotic Graph Neural Networks
To construct the multitasking model, we need a deep backbone for high-level action pattern extraction as well as two task-specific modules. In this section, we present the architecture of our Symbiotic Graph Neural Networks (Sym-GNN). First, we present the deep backbone network, which uses multi-scale graphs for feature learning; We then present the action-recognition head and the motion-prediction head with an effective multitasking scheme. Finally, we present a dual network, which learns features from body bones, instead of body-joints, and provides complementary information for downstream tasks.
5.1 Backbone: Multi-Branch Multi-Scale Graph Convolution Networks
To learn the high-level action pattern, the proposed Sym-GNN consists of a deep backbone called multi-branch multi-scale graph convolution network (multi-branch multi-scale GCN). It employs parallel multi-scale GCN branches to treat high-order action differences for rich dynamics learning and also considers multi-scale graphs for spatial feature extraction. Fig. 6 shows the backbone, where the left plot is the backbone framework including three branches of multi-scale GCNs; the right plot is the structure of each branch of multi-scale GCN. As follows, we propose the backbone architecture in detail.
5.1.1 Multiple Branches
The backbone has three branches of multi-scale GCN. Each branch uses a distinct order of action differences as input, treating the motion states for dynamics learning (see Fig. 6). Calculating the differences by difference operators, we first obtain the proxies of ‘positions’, ‘velocities’ and ‘accelerations’ as the input of the network; see (9). The three branches have identical network architectures. We obtain semantics of high-order differences and concatenate them together for action recognition and motion prediction.
5.1.2 Multi-Scale GCN
To learn the detailed and general action features comprehensively, each branch of the backbone is a multi-scale GCN based on two scales of graphs: joint-scale and part-scale graphs. For each scale, we use the corresponding operators, i.e. J-GTC blocks (see section 4.1.4) and P-GTC block (see section 4.2.1) to extract spatial and temporal features.
Concretely, in the joint scale, the body is modeled by joint-scale graphs, where both the actional graph and structural graphs are used to capture body-joint correlations. This joint scale uses a cascade of J-GTC blocks based on the learned actional-structural graphs. In the part scale, we use part-scale graphs whose nodes are body-parts to represent high-level body instances, and we stack multiple P-GTC blocks for feature capturing. To be aware of the multi-scale immediate representations and learn rich and consistent patterns, we introduce a fusion mechanism between the hidden layers of two scales; called bidirectional fusion.
Joint2part pooling. For the joint scale, we use pooling to average the joint features on the same part to represent a super node. Then, we concatenate the pooling result to the corresponding part feature in the part scale. As shown in Fig. 5, we average torso joints to obtain a node in the part-scale graph and concatenate it to the original part-scale features.
Part2joint matching. For the part scale, the part features are copied for several times to match the number of corresponding joints, as well as we concatenate the copied parts to the joints. As shown in Fig. 5, we copy the thigh twice and concatenate them to the hip and knee in the joint scale.
Fig. 6 (right plot) shows the internal operations and one bidirectional fusion in multi-scale GCN. Given the joint-scale input attributes, we first use a J-GTC block to extract the initial joint-scale features, and a joint2part pooling is applied on the joint-scale features to compute the initial part-scale features. We next feed them into two parallel J-GTC and P-GTC blocks. Then we concatenate the responses mapped by joint2part pooling and part2joint matching to the features in opposite scales. Therefore, both scales have good adaptability to multi-scale information. After multiple interactive J-GTC and P-GTC blocks in the multi-scale GCN, we fuse the outputs of two scales through summation, followed by the average pooling to remove the temporal dimension, and obtain the high-level features.
Finally, we concatenate the outputs from three branches together and use them as the comprehensive semantics for action recognition and motion prediction.
5.2 Multitasking I: Action Recognition
For action recognition, Sym-GNN can be represented as , where is the recognition sub-model of entire Sym-GNN, ; that is, the recognition performance depends on the backbone network and a recognition module constructed following the backbone. Given the high-level features extracted by three branches of backbone, , and , we concatenate them and employ an MLP to produce the fused feature:
where denotes the fusing network of recognition task and is the concatenation operator of three matrices along feature dimension. To integrate the joint dynamics, we apply the global averaging pooling on the joints of and obtain a feature vector that represents the whole body. To generate the recognition results, we finally feed the vector into a 1-layer network with a softmax classifier, obtaining .
5.3 Multitasking II: Motion Prediction
For motion preidction, our Sym-GNN works as , using the backbone and a prediction module, where is the prediction sub-model of Sym-GNN. Therefore, we additionally build a motion-prediction head, whose functionality is to sequentially predict the future poses. Fig. 7 shows the overall structure. We adopt the self-regressive mechanics and identical connection in the motion-prediction head, which utilizes gated recurrent unit (GRU) to model the temporal evolution.
Concretely, an MLP is first employed to embed the features of three action differences:
where denotes the fusing network of prediction task. Let be the initial states of GRU-based predictor and be the pose in the current time stamp. To produce the th pose , the motion-prediction head works as
where , and represent JGC operator (5), GRU cell and output MLP, respectively. The following s are the predictions obtained sequentially and used recylingly. In Step (10a), we apply the JGC to update the hidden states; in Step (10b), we feed the updated hidden states, current pose and classified labels into the GRU cell to produce the features that reflect future displacement; In Step (10c), we add the predicted displacement to the previous pose to predict the next frame.
The motion-prediction head has three advantages: (i) we use JGC to update hidden features, capturing more complicated motion patterns; (ii) we input multiple orders of poses differences and classified labels to the GRU, providing explicit motion priors; and (iii) Connected by the residual, the GRU and MLP predict the displacement for each frame; this makes predictions precise and robust.
5.4 Multi-Objective Optimization
To train action recognition and motion prediction simultaneously, we consider a multi-objective scheme.
To recognize actions, we minimize the cross entropy between the ground-truth categorical labels and the inferred ones. Let the true label of the th sample be and the corresponding classification results be . For training samples in one mini-batch, the action recognition loss is formulated as
where denotes the transpose operation.
For motion prediction, we minimize the distance between the target motions and the predicted clips. Let the th target and predictions be and , for samples in one mini-batch, the prediction loss is
where denotes the norm. According to our experiments, the norm leads to more precise predictions compared to the common norm.
where trade-offs the importances of two tasks. To balance action recognition and motion prediction, instead of using a fixed selected by hand, we employ the ‘multiple-gradient descent algorithm’ (MGDA) to obtain the proper coefficient for multitasking loss terms. Following , the parameter is adaptively adjusted during training by matching one of Karush-Kuhn-Tucker (KKT) conditions of the optimization problem. Therefore, the optimized allocates weights for the two tasks adaptively, leading to better performances for both tasks. In our training scheme, all the model parameters are trained end-to-end with the stochastic gradient descent algorithm ; see more details in Appendix.
5.5 Bone-based Dual Graph Neural Networks
While the joints contain some information of action representation from the joint aspect, the attributes of bones, such as lengths and orientations, are crucial to provide some complementary information. In this section, we construct a bone-based dual graph against original joint-scale graph, whose vertices are bones and edges link bones.
To represent the feature of each bone, we compute the subtraction of two endpoint joints coordinates, which includes information of bone lengths and orientations. The subtraction order is from the centrifugal joint to the centripetal . Let the joint locations along time be , the bone attribute is . Then, we construct the bone-based dual actional and structural graphs to model the bone relations; and we also build the part-scale dual graph. The dual actional graph is learned from bone features by AGIM (see section 4.1.1); for dual structural graph, the 1-hop edges are linked when two bones with articulated joints and the high-hop edges are extended from the 1-hop edges; the part-scale attributes are obtained by integrating bone attributes and the part-scale graph is built according to body nature; The bone-based graphs are dual of joint-scale graphs, which are employed to extract complementary bone features.
Given the bone-based graphs, we train a bone-based graph neural network. We input bone attributes; then everything else follows the joint-based network. Finally, we fuse the joint-based and bone-based recognition outputs before softmax functions by a weighted summation to calculate the classification results, which tend to be more accurate and improve motion prediction effectively.
6 Experiments and Analysis
In this section, we evaluate the proposed Sym-GNN. First, we introduce the datasets and model settings in detail; then, the performance comparisons between Sym-GNN and other state-of-the-art methods are presented; and we finally show the ablation studies of our model.
6.1 Datasets and Model Setting
NTU-RGB+D: NTU-RGB+D, containing skeleton action sequences completed by one or two performers and categorized into classes, is one of the largest datasets for 3D skeleton-based action recognition. It provides the 3D spatial coordinates of joints for each subject in an action. For method evaluation, two protocols are recommended: ‘Cross-Subject’ (CS) and ‘Cross-View’ (CV). In CS, samples performed by subjects are separated into the training set, and the rest belong to the test set. CV assigns data according to camera views, where training and test set have and samples, respectively.
Kinetics: Kinetics is a large dataset for human action analysis, containing over video clips. There are classes of actions. Due to only RGB videos, we obtain skeleton data by estimating joint locations on pixels with OpenPose toolbox . The toolbox generates 2D pixel coordinates and confidence score for totally joints. We represent each joint as a three-element feature vector: . For the multiple-person cases, we select the body with the highest average joint confidence in each sequence. Therefore, one clip with frames is transformed into a skeleton sequence with the dimension of .
Human 3.6M: Human 3.6M (H3.6M) is a large motion capture dataset and also receives increasing popularity. Seven subjects are performing classes of actions, where each subject has joints. We downsample all sequences by two. The models are trained on six subjects and tested on the specific clips of the th subject. Notably, the dataset provides the joint locations in angle space, and we transform them into exponential maps and only use the joints with non-zero values (actually joints).
CMU Mocap: CMU Mocap includes five major action categories, and each subject in CMU Mocap has joints, which are presented by angle positions. We use the same strategy presented in  to select the actions. Thus we choose eight actions: ‘Basketball’, ‘Basketball Signal’, ‘Directing Traffic’, ‘Jumping’, ‘Running’, ‘Soccer’, ‘Walking’ and ‘Washing Window’. We preprocess the data and compute the corresponding exponential maps with the same approach as we do for Human 3.6M dataset.
6.1.2 Model Setting and Implementation Details
The models are implemented with PyTorch 0.4.1. Since different datasets have distinctive patterns and complexities, we employ specific configurations of Sym-GNN networks on corresponding datasets for features learning.
For NTU-RGB+D and Kinetics, the backbone network of Sym-GNN contains J-GTC blocks and P-GTC blocks. In each three J-GTC and P-GTC blocks, the feature dimensions are respectively , and . The kernel size of TC is and it shrinks the temporal dimension with stride after the rd and th blocks, where we use bidirectional fusion mechanisms. . The action-recognition head is a -layer MLP, whose hidden dimension is . For the motion-prediction head, the hidden dimensions of GRU and output MLP are . For the actional graph inference module (AGIM), we use -layer -D MLPs with ReLU, batch normalization and dropout in each iteration. We use SGD algorithm to train Sym-GNN, where the learning rate is initially and decays by every epochs. The model is trained with batch size for epochs on GTX-1080Ti GPUs. For both NTU-RGB+D and Kinetics, the last 10 frames are used for motion prediction and other previous frames are fed into Sym-GNN for action recognition.
As for Human 3.6M and CMU Mocap, due to the simpler dynamics and fewer categories, we propose a light version of Sym-GNN, which extracts meaningful features with more shallow networks, improving efficiency for motion prediction. In the backbone, we use J-GTC blocks and P-GTC blocks, whose feature dimensions are , , and ; the temporal convolution strides in blocks are: , , , , respectively. We apply bidirectional fusions at the last 3 layers. . The recognition and motion-prediction heads, as well as AGIM, leverage the same architecture as we set for NTU-RGB+D. We train the model using Adam optimizer with the learning rate and batch size for iterations on one GTX-1080Ti GPU. All the hyper-parameters are selected using a validation set.
6.2 Comparison with State-of-the-Arts
On the three large-scale skeleton-formed datasets, we compare the proposed Sym-GNN with state-of-the-art methods for human action recognition and motion prediction.
6.2.1 3D Skeleton-based Action Recognition
For action recognition, we first show the classification accuracies of Sym-GNN and baselines on two recommended benchmarks of NTU-RGB+D, i.e. Cross-Subject and Cross-View . The state-of-the-art models are based on manifold analysis , recurrent neural networks [12, 32, 28], convolution networks [24, 13, 9], and graph networks [14, 54, 28, 35, 29, 36, 37, 1, 38]. Moreover, to investigate different components of Sym-GNN, such as multiple graphs and multitasking, we test several model variants, including Sym-GNN using only joint-scale structural graphs (Only J-S), only joint-scale actional graphs (Only J-A), only-part scale graph (Only P), no bone-based dual graphs (No bone), no prediction for multitasking (No pred) and complete model. Table I presents recognition accuracies of methods.
|Lie Group ||50.1%||52.8%|
|Deep LSTM ||60.7%||67.3%|
|Temporal Conv ||74.3%||83.1%|
|Visualize CNN ||76.0%||82.6%|
|Sym-GNN (Only J-S)||88.3%||94.5%|
|Sym-GNN (Only J-A)||85.7%||93.7%|
|Sym-GNN (Only P)||86.5%||87.3%|
|Sym-GNN (No bone)||87.1%||93.8%|
|Sym-GNN (No pred)||89.0%||95.7%|
|Feature Encoding ||14.9%||25.8%|
|Deep LSTM ||16.4%||35.3%|
|Temporal Conv ||20.3%||40.0%|
|Sym-GNN (No pred)||36.4%||57.4%|
|Human 3.6M||CMU Mocap|
|Sym-GNN (Only J)||55.6%||93.9%||96.5%||99.4%|
|Sym-GNN (Only P)||54.3%||93.1%||94.9%||98.0%|
|Sym-GNN (No bone)||53.5%||93.2%||93.5%||95.8%|
|Sym-GNN (No pred)||55.2%||94.1%||96.6%||99.4%|
We see that the complete Sym-GNN, which utilizes joint-scale and part-scale graphs, motion motion-prediction head and dual bone-based network, outperforms the baselines on both benchmarks. The results reveal that richer joint relations promote to capture more useful patterns, and additional motion prediction and complementary bone-based features improve the discrimination.
|Sym-GNN (No recg)||98.74||97.07||94.95||93.94||93.48||91.79||90.69||89.27||87.87||86.01||92.34|
|Sym-GNN (No recg)||98.99||96.79||95.55||94.68||93.03||92.13||90.88||89.69||88.70||87.57||92.79|
|Sym-GNN (Only J-A)||0.19||0.35||0.54||0.63||0.18||0.34||0.54||0.66||0.23||0.43||0.84||0.82||0.26||0.62||0.81||0.87|
|Sym-GNN (Only J-S)||0.19||0.33||0.54||0.69||0.17||0.32||0.52||0.66||0.21||0.41||0.83||0.82||0.24||0.64||0.93||1.01|
|Sym-GNN (No recg)||0.18||0.31||0.50||0.59||0.16||0.29||0.49||0.61||0.21||0.40||0.80||0.80||0.22||0.57||0.85||0.93|
Then, we evaluate our model for action recognition on Kinetics and compare it with six previous models, including a hand-crafted based method, Feature Encoding , two deep models, Deep LTSM  and Temporal ConvNet , and three graph-based methods, ST-GCN , 2s-AGCN , and DGNN . Table II shows the top-1 and top-5 classification results, and ‘no pred’ denotes the Sym-GNN variant without motion-prediction head and multitasking framework. We see that Sym-GNN outperforms other methods on top-1 recognition accuracy and achieves competitive results on top-5 recognition accuracy.
Additionally, we evaluate our model for action recognition on Human 3.6M and CMU Mocap. Table III presents the top-1 and top-5 classification accuracies for both two datasets. Here we compare Sym-GNN with a few recently proposed methods: ST-GCN , HCN , and 2s-AGCN . We also show the effectiveness of our model.
|Sym-GNN (No recg)||0.24||0.45||0.61||0.67||0.36||0.61||0.98||1.17||0.50||0.86||1.29||1.43||0.18||0.44||0.99||1.22||0.40||0.62||1.00||1.08||0.23||0.41||0.80||0.97|
|Motion||Sitting Down||Taking Photo||Waiting||Walking Dog||Walking Together||Average|
|Sym-GNN (No recg)||0.30||0.62||0.91||1.03||0.16||0.34||0.55||0.66||0.22||0.49||0.89||1.09||0.42||0.74||1.09||1.25||0.17||0.34||0.52||0.57||0.26||0.50||0.82||0.94|
|Motion||Basketball||Basketball Signal||Directing Traffic||Jumping|
|Sym-GNN (No recg)||0.33||0.48||0.95||1.09||1.47||0.15||0.26||0.47||0.56||1.04||0.20||0.41||0.77||0.89||1.95||0.32||0.55||1.40||1.60||1.87|
|Sym-GNN (No recg)||0.21||0.33||0.53||0.56||0.66||0.22||0.38||0.72||0.83||1.38||0.26||0.32||0.38||0.41||0.54||0.22||0.33||0.62||0.83||1.07|
Notably, for Human 3.6M, there is a relatively large gap between the top-1 and top-5 accuracies, because the input motions are some fragmentary clips of long sequences with incomplete semantics and activities have subtle differences (e.g. ‘Eating’ and ‘Smoking’ are similar). In other words, Sym-GNN learns the common features and provides reasonable discrimination, resulting in high top-5 accuracy; but it confuses in non-semantic variances, causing not high top-1 accuracy. However, CMU Mocap has more distinctive actions, where we obtain high classification accuracies.
6.2.2 3D Skeleton-based Motion Prediction
To validate the model for predicting future motions, we train the Sym-GNN on NTU-RGB+D, Human 3.6M, and CMU Mocap. There are two specific tasks: short-term and long-term motion prediction. Concretely, the target of short-term prediction is commonly to predict poses within 400 milliseconds, while the long-term prediction aims to predict poses in 1000 ms or longer. To reveal the effectiveness of Sym-GNN, we introduce many state-of-the-art methods, which learned dynamics from pose vectors [42, 17, 56, 18, 27] or separate body-parts [16, 31, 30]. We also introduce a naive baseline, named ZeroV , which sets all predictions to be the last observed frame.
Short-term motion prediction: We validate Sym-GNN on two datasets: NTU-RGB+D and Human 3.6M. For NTU-RGB+D, we train Sym-GNN to generate future 10 frames for each input sequence. We compare our Sym-GNN with several previous methods and the Sym-GNN variant which abandons auxiliary action-recognition head (No recg). As the metric, We use the percentage of correct points within a normalized region 0.05 (PCK@0.05), that is, a joint is counted as correctly predicted if the normalized distance between the predicted location and ground-truth is less than 0.05. The PCK@0.05 of different models are presented in Table IV. We see: 1) our model extremely outperforms the baselines with a large margin especially for the longer term; 2) Using action recognition and motion prediction together obtains the highest PCK@0.05 along time, demonstrating the enhancements from recognition task for dynamics learning.
Then, we compare Sym-GNN to baselines for short-term prediction on Human 3.6M, where the models generate poses up to the future 400 ms. We analyze several variants of Sym-GNN with different components, including using only joint-scale actional graphs (Only J-A) or joint-scale structural graphs (Only J-S), as well as no recognition task (No recg). As another metric, the mean angle errors (MAE) between the predictions and the ground truths are computed, representing the errors from predicted poses to targets in angle space. We first test representative actions: ‘Walking’, ‘Eating’, ‘Smoking’ and ‘Discussion’. Table V shows MAEs of different methods that predict motions up to 400 ms. As we see, when Sym-GNN simultaneously employs multiple graphs and multitasking, our method outperforms all the baselines and its own ablations.
We also test Sym-GNN on the remaining actions in Human 3.6M, where the MAEs of some recent methods are shown in Table VI. Sym-GNN also achieves the best performance on most actions and the lowest average MAE on motions. Although the mentioned top-1 classification accuracy on this dataset is not very high (see Table III), we note that the estimated soft labels cover the common motion factors, resulting in high top-5 recognition accuracy. For example, people walk in ‘Walking’, ‘Walking Dog’ and ‘Walking Together’, and we need the walking factors instead of the specific labels for motion generation. Given the soft labels, the model tends to obtain precise predictions.
Long-term motion prediction: For long-term prediction, the Sym-GNN is tested on Human 3.6M and CMU Mocap. We predict the future poses up to 1000 millisecond. It is challenging due to action variation and non-linearity . Table VIII presents the MAEs of various models for predicting the 4 motions in Human 3.6M at the future 560 ms and 1000 ms.
We see that Sym-GNN outperforms the competitors on ‘Eating’, ‘Smoking’ and ‘Discussion’, and obtain competitive results on ‘Walking’.
To further evaluate Sym-GNN, we conduct long-term prediction on eight classes of actions in CMU Mocap. We present the MAEs of Sym-GNN with or without using the action-recognition head. Table VII shows the predicting MAEs ranging from future 80 ms to 1000 ms. We note that we train the model for long-term prediction, where the ‘short-term’ MAEs are the intermediate results during predicting up to 1000 ms. We see that Sym-GNN significantly outperforms the state-of-the-art methods on actions ‘Basketball’, ‘Basketball Signal’ and ‘Washing Window’, and obtains competitive performance on ‘Jumping’ and ‘Running’.
Effectiveness-efficiency tradeoff: We also compare the prediction errors and efficiency of various models, because the high response speed and precise generation are both essential for real-time motion prediction. Notably, the AGIM propagates the features between joints and edges iteratively. The iteration times trades off between effectiveness and speed; i.e. larger leads to a lower MAE but slower speed. To represent the running speed, we use the generated frame numbers in each ms (frame period) when we predict up to ms. We tune and compare Sym-GNN to other methods on Human 3.6M and show the running speeds and MAEs for prediction in 400 ms.
Fig. 8 shows the effectiveness-efficiency tradeoff, where the -axis is the generated frame numbers in ms (reflecting prediction speed) and the axis is the MAE. Different red circles denote different numbers of iterations in AGIM, i.e. from the rightmost circle to the leftmost one, . We see that the proposed Sym-GNN is both faster and more precise compared to its competitors.
6.3 Ablation Studies
6.3.1 Symbiosis of Recognition and Prediction
To analyze the mutual effects of action recognition and motion prediction, we conduct several experiments.
|Noise ratio||80 ms||160 ms||320 ms||400 ms|
We first study the effects on action recognition from motion prediction. We use accurate class labels but noisy future poses to train the multitasking Sym-GNN for action recognition. To represent noisy supervisions, we randomly shuffle a percentage of targets motions among training data. Table IX presents the recognition accuracies with various ratios of noisy prediction targets on two benchmarks of NTU-RGB+D. We also show the recognition results of the model without motion-prediction head. We see that 1) the predicted head benefits the action-recognition head. Introducing a motion-prediction head is beneficial even when the noise ratio is around ; 2) when the noise ratio exceeds , the recognition performance tends to be slightly worse than that of the model without the motion-prediction head, reflecting that the action recognition is robust against the deflected motion prediction. Consequently, we show that motion prediction strengthens action recognition.
On the other hand, we test how confused recognition results affect motion prediction by using noisy action categories. Following Table IX, we shuffle training labels to represent categorical noise. Table X presents the average MAEs for short-term prediction with noisy action labels on Human 3.6M. We demonstrate that an accurate action-recognition head helps effective motion prediction.
We finally test the promotion on recognition when the observed data is limited, where we intercept the early motions by a ratio (e.g. ) for action recognition. There are three models with various prediction strategies: 1) predicting the future 10 frames (‘Pred 10 frames’); 2) predicting all future frames (‘Pred all frames’); 3) no prediction (‘No pred’). Fig. 9 illustrates the recognition accuracies of three models on different observation ratios. As we see, when the observation ratio is low, ‘Pred all frames’ can be aware of the entire action sequences and capture richer dynamics, showing the best performance; when the observation ratio is high, predicting 10 or all frames are similar because the inputs carry sufficient patterns, but they outperform ‘No pred’ as they preserve information, showing enhancement. By introducing the motion-prediction head, our Sym-GNN has the potential for action classification in the early period.
6.3.2 Effects of Graphs
In this section, we study the abilities of various graphs, namely, only joint-scale structural graphs (Only J-S), only joint-scale actional graph (Only J-A), only part-scale graph (Only P), and combining them (full).
For action recognition, we train Sym-GNN on NTU-RGB+D, Cross-Subject and investigate different graph configurations. While involving joint-scale structural graph, we respectively set the number of hop in the joint-scale structural graphs (JS-Hop) to be . Note that when we use only joint-scale structural graph with , the corresponding graph is exactly the skeleton itself. Table XI presents the results of Sym-GNN with different graph components for action recognition.
|JS-Hop ()||Only J-S||Only J-A||Only P||full|
We see that 1) representing long-range structural relations, higher leads to more effective action recognition; 2) combining the multiple graphs introduced from different perspectives improves the action recognition performance significantly.
|JS-Hop ()||Only J-S||Only J-A||Only P||full|
We see that the effects multiple relations for promoting motion prediction are demonstrated. We note that too large introduces redundancy and confusing relations, enlarging the prediction error.
6.3.3 Balance Joint-Scale Actional and Structural Graphs
In our model, we present that the power of joint-scale actional and structural graphs in JGC operator are traded off by a hyper-parameter (see (5)). Here we analyze how affects the model performances.
For action recognition, we test our model on NTU-RGB+D, Cross-Subject, and present the classification accuracies with different ; for motion prediction, we show the average MAEs for short-term prediction. Fig. 10 illustrates the model performances for both tasks.
We see: 1) when , we obtain the highest recognition accuracies, showing large improvements than cases with other ; 2) for motion prediction, the performance is robust against different , where the MAEs fluctuate around , but and lead to the lowest errors.
6.3.4 High-order Difference
Here we study the high-order differences of the input actions for action recognition and motion prediction, which help to capture richer motion dynamics. In our model, the difference orders are considered to be , reflecting positions, velocities, and accelerations. We present the results of action recognition on NTU-RGB+D and the results of motion prediction on Human 3.6M.
For action recognition, we test Sym-GNN with , and on the two benchmarks (CS & CV) of NTU-RGB+D dataset. Table XIII presents the average recognition accuracies of Sym-GNN with three difference configurations. We see that leads to the highest classification accuracies on both benchmarks, indicating that positions and velocities capture comprehensive movement patterns to provide rich semantics, meanwhile the model complexity is not very high.
For motion prediction, we feed the model with various action differences of Human 3.6M to generate future poses in within 400 ms. We obtain the predicting average MAEs on the timestamps of 80, 160, 320 and 400 ms. The results are presented in Table XIV. We see that Sym-GNN forecasts the future poses with the lowest prediction errors when we feed the action differences with , showing the effectiveness of combining high-order motion states.
|Joint & Bone|
6.3.5 Bone-based Dual Graph Neural Networks
We validate the effectiveness of using dual networks which take joint and bone features as inputs for action recognition, respectively. Table XV presents the recognition accuracies for different combinations of joint-based and bone-based dual networks on two benchmarks of NTU-RGB+D dataset. We see that only using joint features or bone features for action recognition cannot obtain the most accurate recognition, but combining joint and bone features could improve the classification performances with a large margin, indicating the complementary information carried by the two networks.
In this section, we visualize some representations of Sym-GNN, including the learned joint-scale actional graphs and their low dimensional manifolds. Moreover, we show some predicted motions to evaluate model qualitatively.
6.4.1 Joint-Scale Actional Graphs
We first show the learned joint-scale actional graphs on four motions in Human 3.6M. Fig. 11 illustrates the edges which have the top-15 largest weights in each graph, indicating the 15 strongest action-based relations associated with different motions.
We see: 1) The joint-scale actional graphs capture some action-based long-range relations beyond direct bone-connections; 2) Some reasonable relations are captured, e.g. for ‘Directions’, the stretched arms are correlated to other joints; 3) for motions with the same category, we tend to obtain the similar graphs; see two plots of ‘Walking’, while different classes of motions have distinct actional graphs; see ‘Walking’ and the other motions, where the model learns the discriminative patterns from data.
6.4.2 Manifolds of Joint-Scale Actional Graphs
To verify how discriminative the patterns embedded in the joint-scale actional graphs, we visualize the low-dimension manifolds of different joint-scale actional graphs. We select 8 representative classes of actions in Human 3.6M and sample more clips from long test motion sequences. Here we treat all the joint-scale actional graphs as vectors and obtain their 2D T-SNE map; see Fig. 12. We see that ‘Walking’, ‘Walking Dog’ and ‘Walking Together’, which have the common walking dynamics, are distributed closely, as well as ‘Sitting’ and ‘Sitting Down’ are clustered; however, walking-related actions and sitting-related actions are separated with a large margin; as for ‘Eating’, ‘Smoking’ and ‘Taking Photo’, they have similar movements on arms, showing a new cluster.
6.4.3 Predicted Sequences
Finally, we compare the generated samples of Sym-GNN to those of Res-sup , AGED , and CSM  on Human 3.6M and CMU Mocap. Fig. 13 illustrates the future poses of ‘Eating’ and ‘Running’ in 1000 ms with the frame interval of 80 ms, where plot (a) shows the predictions of ‘Eating’ in Human 3.6M and plot (b) visualize ‘Running’ in CMU Mocap. Comparing to baselines, we see that Sym-GNN provides significantly better predictions. The poses generated by Res-sup has large errors after the 600th ms (two orange boxes); AGED produces over movements for the downward hand in long-term (red box in plot (a)); CSM gives tortile poses in long-term (red box in plot (b)). But Sym-GNN completes the action accurately and reasonably.
6.5 Stability Analysis: Robustness against Input Perturbation
According to Theorem 4.1.3, we present that Sym-GNN is robust against perturbation on inputs, where we calculate an upper bound of output deviation. To verify the stability, we add Gaussian noises sampled from on input actions. We show the recognition accuracies on NTU-RGB+D (Cross-Subject) and short-term prediction MAEs on Human 3.6M with standard deviation varied from to . The recognition/prediction performances with different are illustrated in Fig. 14. We see: 1) for action recognition, Sym-GNN stays a high accuracy when the noise has , but it tends to deteriorate due to severe perturbation when ; 2) for motion prediction, Sym-GNN produces precise poses when the noise has , but the prediction performance is degraded for larger . In all, Sym-GNN is robust against small perturbation.
Given two inputs and , which satisfy , we validate the assumption of claimed in Theorem 4.1.3. We calculate the ratio between the perturbations of responses and inputs; that is,
Similar to Fig. 14, we tune the standard deviations of input noises, obtaining the corresponding and calculate the perturbation ratios. Fig. 15 illustrated the with different noise standard deviations. We see that the is steady at around for adjusted from to , which indicates the amplify factor and the responses and inputs are mostly linearly correlated. In other words, the actional graph inference module does not amplify the perturbation and the stability of JGC is still preserved.
In this paper, we propose a novel symbiotic graph neural network (Sym-GNN), which handles action recognition and motion prediction jointly and use graph-based operations to capture action patterns. Our model consists of a backbone, an action-recognition head, and a motion-prediction head, where the two heads enhance each other. As building components in the backbone and the motion-prediction head, graph convolution operators based on learnable joint-scale and part-scale graphs are used to extract spatial information. We conduct extensive experiments for action recognition and motion prediction with four datasets, NTU-RGB+D, Kinetics, Human 3.6M, and CMU Mocap. Experiments show that our model achieves consistently improvements compared to the previous methods.
-  M. Li, S. Chen, X. Chen, Y. Zhang, Y. Wang, and Q. Tian, “Actional-structural graph convolutional networks for skeleton-based action recognition,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
-  U. Gaur, Y. Zhu, B. Song, and A. Roy-Chowdhury, “A âstring of feature graphsâ model for recognition of complex activities in natural videos,” in The IEEE International Conference on Computer Vision (ICCV), November 2011, pp. 2595–2602.
-  D. Huang and K. Kitani, “Action-reaction: Forecasting the dynamics of human interaction,” in The European Conference on Computer Vision (ECCV), July 2014, pp. 489–504.
-  L. Gui, K. Zhang, Y. Wang, X. Liang, J. Moura, and M. Veloso, “Teaching robots to predict human motion,” in IEEE International Conference on Intelligent Robots and Systems (IROS), October 2018.
-  S. Mathe and C. Sminchisescu, “Actions in the eye: Dynamic gaze datasets and learnt saliency models for visual recognition,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 37, no. 7, pp. 1408–1424, July 2015.
-  Y. Shi, B. Fernando, and R. Hartley, “Action anticipation with rbf kernelized feature mapping rnn,” in The European Conference on Computer Vision (ECCV), September 2018, pp. 301–317.
-  D. Wang, W. Ouyang, W. Li, and D. Xu, “Dividing and aggregating network for multi-view action recognition,” in The European Conference on Computer Vision (ECCV), September 2018, pp. 451–476.
-  X. Liang, K. Gong, X. Shen, and L. Lin, “Look into person: Joint body parsing pose estimation network and a new benchmark,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 4, pp. 871–885, April 2019.
-  C. Li, Q. Zhong, D. Xie, and S. Pu, “Co-occurrence feature learning from skeleton data for action recognition and detection with hierarchical aggregation,” in IJCAI, July 2018, pp. 786–792.
-  R. Vemulapalli, F. Arrate, and R. Chellappa, “Human action recognition by representing 3d skeletons as points in a lie group,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2014, pp. 588–595.
-  B. Fernando, E. Gavves, J. M. Oramas, A. Ghodrati, and T. Tuytelaars, “Modeling video evolution for action recognition,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015, pp. 5378–5387.
-  Y. Du, W. Wang, and L. Wang, “Hierarchical recurrent neural network for skeleton based action recognition,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2015, pp. 1110–1118.
-  M. Liu, H. Liu, and C. Chen, “Enhanced skeleton visualization for view invariant human action recognition,” in Pattern Recognition, vol. 68, August 2017, pp. 346–362.
-  S. Yan, Y. Xiong, and D. Lin, “Spatial temporal graph convolutional networks for skeleton-based action recognition,” in AAAI Conference on Artificial Intelligence, February 2018, pp. 7444–7452.
-  A. Lehrmann, P. Gehler, and S. Nowozin, “Efficient nonlinear markov models for human motion,” in The IEEE Conf