Multi-task Deep Learning for Real-Time 3D Human Pose Estimation and Action Recognition

The problem of human pose estimation has been intensively studied in the last years, from Pictorial Structures [1, 18, 44] to more recent CNN based approaches [41, 30, 45, 23, 48, 62, 6, 58, 59, 43]. We can identify from the literature two distinct families of methods for pose estimation: detection and regression based methods. Recent detection methods handle pose estimation as a heat map prediction problem, where each pixel in a heat map represents the detection score of a given body joint being localized at this pixel [7, 20]. Exploring the concepts of stacked architectures, residual connections, and multiscale processing, Newell \etal [40] proposed the Stacked Hourglass networks (SHG), which improved scores on 2D pose estimation challenges significantly. Since then, methods in the state of the art are frequently proposing complex variations of the SHG architecture. For example, Chu \etal [17] proposed an attention model based on conditional random field (CRF) and Yang \etal [64] replaced the residual unit from SHG by the Pyramid Residual Module (PRM). Very recently, [53] proposed a high-resolution network that keeps a high-resolution flow, resulting in more precise predictions. With the emergence of Generative Adversarial Networks (GANs) [21], Chou \etal [15] proposed to use a discriminative network to distinguish between estimated and target heat maps. This process could increase the quality of predictions, since the generator is stimulated to produce more plausible predictions. Another application of GANs in that sense is to enforce the structural representation of the human body [12].

However, all the previous mentioned detection based approaches do not provide body joint coordinates directly. To recover the body joints in $(x,y)$ coordinates, predicted heat maps have to be converted to joint positions, generally using the argument of the maximum a posteriori probability (MAP), called $argmax$. On the other hand, regression based approaches use a nonlinear function to project the input image directly to the desired output, which can be the joint coordinates. Following this paradigm, Toshev and Szegedy [59] proposed a holistic solution based on cascade regression for body part regression and Carreira \etal [9] proposed the Iterative Error Feedback. The limitation of current regression methods is that the regression function is frequently sub-optimal. In order to tackle this weakness, the soft-argmax function [36] has been proposed to compute body joint coordinates from heat maps in a differentiable way.

Recently, deep architectures have been used to learn 3D representations from RGB images [69, 57, 37, 56, 38, 46] thanks to the availability of high precise 3D data [24], and are now able to surpass depth-sensors [39]. Chen and Ramanan [11] divided the problem of 3D pose estimation into two parts. First, they target 2D pose estimation considering the camera coordinates and second, the 2D estimated poses are matched to 3D representations by means of a nonparametric shape model. However, this is an ill-defined problem, since two different 3D poses could have the same 2D projection. Other methods propose to regress the 3D relative position of joints, which usually presents a lower variance than the absolute position. For example, Sun \etal [54] proposed a bone representation of the human body. However, since the errors are accumulative, such a structural transformation might effect tasks that depend on the extremities of the human body, like action recognition.

Pavlakos \etal [42] proposed the volumetric stacked hourglass architecture, but the method suffers from significant increase in the number of parameters and from the required memory to store all the gradients. A similar technique is used in [55], but instead of using argmax for coordinate estimation, the authors use a numerical integral regression, which is similar to the soft-argmax operation [34]. More recently, Yang \etal [65] proposed to use adversarial networks to distinguish between generated and ground truth poses, improving predictions on uncontrolled environments. Differently form our previous work in [34], we show that a volumetric representation is not required for 3D prediction. Similarly to methods on hand pose estimation [26] and on 3D human pose estimation [39], we predict 2D depth maps which encode the relative depth of each body joint.

In this section we revisited some methods that exploit pose information for action recognition. For example, classical methods for feature extraction have been used in [63, 27], where the key idea is to use body joint locations to select visual features in space and time. 3D convolutions have been stated as the best option to handle the temporal dimension of images sequences [8, 10, 60], but they involve a high number of parameters and cannot efficiently benefit from the abundant still images during training. Another option to integrate the temporal aspect is by analysing motion from image sequences [13, 19], but these methods require the difficult estimation of optical flow. Unconstrained temporal and spatial analysis are also promising approaches to tackle action recognition, since it is very likely that, in a sequence of frames, some very specific regions in a few frames are more relevant than the remaining parts. Inspired on this observation, Baradel \etal [4] proposed an attention model called Glimpse Clouds, which learns to focus on specific image patches in space and time, aggregating the patterns and soft-assigning each feature to workers that contribute to the final action decision. The influence of occlusions could be alleviated by multi-view videos [61] and inaccurate pose sequences could be replaced by heat maps for better accuracy [33]. However, this improvement is not observed when pose predictions are sufficiently precise.

2D action recognition methods usually use the body joint information only to extract localized visual features [63, 13], as an attention mechanism. Methods that directly explore the body joints usually do not generate it [27] or present lower precision with estimated poses [8]. Our approach removes these limitations by performing pose estimation together with action recognition. As such, our model only needs the input RGB frames while still performing discriminative visual recognition guided by the estimated body joints.

Differently from video based action recognition, 3D action recognition is mostly based on skeleton data as the primary information [35, 47]. With depth sensors such as the Microsoft Kinect, it is possible to capture 3D skeletal data without a complex installation procedure frequently required for motion capture systems (MoCap). However, due to the required infrared projector, depth sensors are limited to indoor environments, have a low range of operation, and are not robust to occlusions, frequently resulting in noisy skeletons. To cope with the noisy skeletons, Spatio-Temporal LSTM networks [31] have been widely used to learn the reliability of skeleton sequences or as an attention mechanism [32, 52]. In addition to the skeleton data, multimodal approaches can also benefit from visual cues [51]. In that direction, pose-conditioned attention mechanisms have been proposed [5] to focus on image patches centered around the hands.

Since our architecture predicts precise 3D poses from RGB frames, we do not have to cope with the noisy skeletons from Kinect. Moreover, we show in the experiments that, despite being based on temporal convolution instead of the more common LSTM, our system is able to reach state of the art performance on 3D action recognition, indicating that action recognition does not necessarily require long term memory.

For pose estimation, prediction blocks take as input the single frame features $X{}_t^{p-1,l}$ from the previous pyramid and the features $X{}_t^{p,l\mp 1}$ from lower or higher levels, respectively for downscaling and upscaling pyramids. A similar propagation of previous features $Y{}^{p-1,l}$ and $Y{}^{p,l\mp 1}$ happens for action. Note that both $X{}_t^{p,l}$ and $Y{}^{p,l}$ feature maps are three-dimensional tensors (2D maps plus channels) that can be easily handled by 2D convolutions.

Given a 2D input signal, the main idea is to consider that the argument of the maximum (argmax) can be approximated by the expectation of the input signal after being normalized to have the properties of a distribution. Indeed, for a sufficiently pointy (Leptokurtic) distribution, the expectation should be close to the maximum a posteriori (MAP) estimation. For a 2D heat map as input, the normalized exponential function (softmax) can be used, since it alleviates the undesirable influences of values below the maximum and increases the “pointiness” of the resulting distribution, producing a probability map, as defined in Equation 6.

Let’s define a single probability map for the $j$th joint as $h{}^j$, in such a way that $h\equiv [h{}^1,\dots ,h{}^{N_j}]$. Then, the expected coordinates $(x^j,y^j)$ are given by the function $\Psi$:

where $H_h\times W_h$ is the size of the input probability map, and $l$ and $c$ are line and column indexes of $h$. According to Equation 8, the coordinates $(x^j,y^j)$ are constrained between the interval $[0,1]$, which corresponds to the normalized limits of the input image.

Differently from our previous work [34], where volumetric heat maps were required to estimate the third dimension of body joints, here we use a similar apprach to [26], where specialized depth maps $d$ are used to encode the depth information. Similarly to the probability maps decomposition from section 3.2.1, here we define $d{}^j$ as a depth map for the $j$th body joint. Thus, the regressed depth coordinate $z^j$ is defined by:

Since $h{}^j$ is a normalized unitary and positive probability map, Equation 9 represents a spatially weighted pooling of depth map $d{}^j$ based on the 2D body joint location.

The probability of a certain body joint being present (even if occluded) in the image is computed by the maximum value in the corresponding probability map. Considering a pose layout with $N_j$ body joints, the estimated joint confidence vector is represented by $\hat{c}\in R^{N_j\times 1}$. If the probability map is very pointy, this score is close to 1. On the other hand, if the probability map is uniform or has more than one region with high response, the confidence score drops.

As systematically noted in recent works [7, 20, 40, 42], predictions re-injection is a very efficient way to improve precision on estimated poses. Differently from all previous methods based on direct heat map regression, our approach can benefit from prediction re-injection at different resolutions, since our pose regression method is invariant to the feature map resolution. Specifically, in each PB at different pyramid and different level, we compute a new set of features $X{}_t^{p,l}$ based on features from previous blocks and on the current prediction, as follows:

where ${\text{W}}_r^{p,l}$ and ${\text{W}}_s^{p,l}$ are weight matrices related to the re-injection of 2D pose and depth information, respectively. With this approach, further PB at different pyramids and levels are able to refine predictions, considering different sets of features at different resolutions.

In order to explore the rich information encoded with body joint positions, we convert a sequence of $T$ poses with $N_j$ joints each into an image-like representation. Similar representations were previously used in [5, 28]. We choose to encode the temporal dimension as the vertical axis, the joints as the horizontal axis, and the coordinates of each point ($(x,y)$ for 2D, $(x,y,z)$ for 3D) as the channels. With this approach, we can use classical 2D convolutions to extract patterns directly from the temporal sequence of body joints. The predicted coordinates of each body joints are pondered by their confidence scores, thus points that are not present in the image (and consequently cannot be correctly predicted) have less influence on action recognition. A graphical representation of pose features is presented in Fig. 4(a).

In addition to the pose information, visual cues are very important to action recognition, since they bring contextual information. In our method, localized visual information is encoded as appearance features, which are extracted in a similar process to the one of pose features, with the difference that the first relies on local visual information instead of joint coordinates. In order to extract localized appearance features, we multiply each channel from the tensor of multi-task features $Z{}_t^{p,l}\in R^{H_f\times W_f\times N_f}$ by each channel from the probability maps $h{}_t\in R^{H_f\times W_f\times N_j}$ (outer product of $N_f$ and $N_j$), which is learned as a byproduct of the pose estimation process. Then, the spatial dimensions are collapsed by a sum, resulting in the appearance features for time $t$ of size $R^{N_j\times N_f}$. For a sequence of frames, we concatenate each appearance feature map for $t=\{1,2,\dots ,T\}$ resulting in the video clip appearance features $V\in R^{T\times N_j\times N_f}$. To clarify this process, a graphical representation is shown in Fig. 4(b).

We argue that our multi-task framework has two benefits for the appearance based part: First, it is computationally very efficient since most part of the computations are shared. Second, the extracted visual features are more robust since they are trained simultaneously for different but related tasks and on different datasets.

Some actions are hard to be distinguished from others only by the high level pose representation. For example, the actions drink water and make a phone call are very similar if we take into account only the body joints, but are easily separated if we have the visual information corresponding to the objects cup and phone. On the other hand, other actions are not directly related to visual information but with body movements, like salute and touch chest, and in this case the pose information can provide complementary information. In our method, we combine visual cues and body movements by aggregating pose and appearance features. This aggregation is a straightforward process, since both feature types have the same spacial dimensions.

Similarly to the single frame features re-injection mechanism discussed in section 3.2.4, our approach also allows action features re-injection, as detailed in the action prediction part in Fig. 4. We demonstrate in the experiments that this technique also improves action recognition results with no additional parameters.

Since the multi-task architecture is trained simultaneously on pose estimation and on action recognition, we may have an effect of competing gradients from poses and actions, specially in the predicted poses, which are used as the output for the first task and as the input for the second task. To mitigate that influence, late in the training process, we propose to decouple estimated poses (used to compute pose scores) from action poses (used by the action recognition part) as illustrated in Fig. 6.

Specifically, we first train the network on pose estimation for about one half of the full training iterations, then we replicate only the last layers that project the multi-task feature map $Z$ to heat maps and depth maps (parameters ${\text{W}}_h$ and ${\text{W}}_d$), resulting in a “copy” of probability maps $h{}^{\prime }$ and depth maps $d{}^{\prime }$. Note that this replica corresponds to a simple $1\times 1$ convolution from the feature space to the number of joints, which is almost insignificant in terms of parameters and computations. The “copy” of this layer is a new convolutional layer with its weights ${\text{W}}^{\prime }$ initialized with W. Finally, for the remaining training, the action recognition part propagates its loss through the replica poses. This process allows the original pose predictions to stay specialized on the first task, while the replicated poses absorb partially the action gradients and are optimized accordingly to the action recognition task. Despite the replicated poses not being directly supervised in the final training stage (which corresponds to a few more epochs), we show in our experiments that they still remain coherent with supervised estimated poses.

On 2D pose estimation, we evaluate our method on the MPII validation set composed of 3K images, using the probability of correct keypoints measure with respect to the head size (PCKh) [2]. On 3D pose estimation, we evaluate our method on Human3.6M by measuring the mean per joint position error (MPJPE) after alignment of the root joint. We follow the most common evaluation protocol [65, 54, 37, 38, 42] by taking five subjects for training (S1, S5, S6, S7, S8) and evaluating on two subjects (S9, S11) on one every 64 frames. We use ground truth person bounding boxes for a fair comparison with previous methods on single person pose estimation. We report results using a single cropped bounding box per sample.

On action recognition, we report results using the percentage of correct action classification score. We use the proposed evaluation protocol for Penn Action [63], splitting the data as 50/50 for training/testing, and the more realistic cross-subject scenario for NTU, on which 20 subjects are used for training, and the remaining are used for testing. Our method is evaluated on single-clip and/or multi-clip. In the first case, we crop a single clip with $T$ frames in the middle of the video. In the second case, we crop multiple video clips temporally spaced of $T/2$ frames one from another, and the final predicted action is the average decision among all clips from one video.

In our experiments, we consider two scenarios: A) 2D pose estimation and action recognition, on which we use respectively MPII and Penn Action datasets, and B) 3D pose estimation and action recognition, using MPII, Human3.6M, and NTU datasets.

For the pose estimation task, we train the network using the elastic net loss [71] function on predicted poses:

where ${\hat{p}}^j$ and $p{}^j$ are respectively the estimated and the ground truth positions of the j$th$ body joint. The same loss is used for both 2D and 3D cases, but only available values ($(x,y)$ for 2D and $(x,y,z)$ for 3D) are taken into account for backpropagation, depending on the dataset. We use poses in the camera coordinate system, with $(x,y)$ laying on the image plane and $z$ corresponding to the depth distance, normalized in the interval $[0,1]$, where the top-left image corner corresponds to $(0,0)$, and the bottom-right image corner corresponds to $(1,1)$. For depth normalization, the root joint is assumed to have $z=0.5$, and a range of 2 meters is used to represent the remaining joints. If a given body joint falls outside the cropped bounding box on training, we set the ground truth confidence flag $c{}^j$ to zero, otherwise we set it to one. The ground truth confidence information is used to supervise predicted joint confidence scores $\hat{c}$ with the binary cross entropy loss. Despite giving an additional information, the supervision on confidence scores has negligible influence on the precision of estimated poses. For the action recognition part, we use categorical cross entropy loss on predicted actions.

Since the pose estimation part is the most computationally expensive, we chose to use separable convolutions with kernel size equals to $5\times 5$ for single frame layers and standard convolutions with kernel size equals to $3\times 3$ for video clip processing layers (action recognition layers). We performed experiments with the network architecture using 4 levels and up to 8 pyramids ($L=4$ and $P=8$). No further significant improvement was noticed on pose estimation by using more than 8 pyramids. On action recognition, this limit was observed at 4 pyramids. For that reason, when using the full model with 8 pyramids, the action recognition part starts only at the 5$th$ pyramid, reducing the computational load. In our experiments, we used normalized RGB images of size $256\times 256\times 3$ as input, which are reduced to a feature map of size $32\times 32\times 288$ by the entry flow network, corresponding to level $l=1$. At each level, the spatial resolution is reduced by a factor of 2 and the size of features is arithmetically increased by $96$. For action recognition, we used $N_v=160$ and $N_v=192$ features for Penn Action and NTU, respectively.

For all the experiments, we first initialize the network by training pose estimation only, for about 32k iterations with mini batches of 32 images (equivalent to 40 epochs on MPII). Then, all the weights related to pose estimation are fixed and only the action recognition part is trained for 2 and 50 epochs, respectively for Penn Action and NTU datasets. Finally, the full network is trained in a multi-task scenario, simultaneously for pose estimation and action recognition, until the validation scores plateau. Training the network on pose estimation for a few epochs provides a good general initialization and a better convergence of the action recognition part. The intermediate training stage of action recognition has two objectives: first, it is useful to allow a good initialization of the action part, since it is built on top of the pre-initialized pose estimator; and second, it is about 3 times faster than performing multi-task training directly while resulting in similar scores. This process is specially useful for NTU, due to the large amount of training data. The training procedure takes about one day for the pose estimation initialization, then two/three days for the remaining process for Penn Action/NTU, using a desktop GeForce GTX 1080Ti GPU.

For initialization on pose estimation, the network was optimized with RMSprop and initial learning rate of 0.001. For action and multi-task training, we use RMSprop for Penn Action with learning rate reduced by a factor of 0.1 after 15 and 25 epochs, and, for NTU, a vanilla SGD with Nesterov momentum of 0.9 and initial learning rate of 0.01, reduced by a factor of 0.1 after 50 and 55 epochs. We weight the loss on body joint confidence scores and action estimations by a factor of 0.01, since the gradients from the cross entropy loss are much stronger than the gradients from the elastic net loss on pose estimation. This parameter was empirically chosen and we did not observe a significant variation in the results with slightly different values (e.g., with 0.02). Each iteration is performed on 4 batches of 8 frames, composed of random images for pose estimation and video clips for action. We train the model by alternating one batch containing pose estimation samples only and another batch containing action samples only. This strategy resulted in slightly better results compared to batches composed of mixed pose and action samples. We augment training data by performing random rotations from $-{40}^{\circ }$ to $+{40}^{\circ }$, scaling from $0.7$ to $1.3$, video temporal subsampling by a factor from 3 to 10, random horizontal flipping, and random color shifting. On evaluation, we also subsampled Penn Action/NTU videos by a factor of 6/8, respectively.

We performed several experiments on the proposed network architecture in order to identify its best arrangement for solving both tasks with the best performance vs computational cost trade-off. In Table 4, we show the results on 2D pose estimation and on action recognition considering different network layouts. For example, in the first line, a single PB is used at pyramid 1 and level 2. In the second line, a pair of full downscaling and upscaling pyramids are used, but with supervision only at the last PB. This results in 97.5% of accuracy on action recognition and 84.2% on PCKh for pose estimation. An equivalent network is used in the third line, but then with supervision on all PB blocks, which brings an improvement of 0.9% on pose and 0.6% on action, with the same number of parameters. Note that the networks from the second and third lines are exactly the same, but in the first case, only the last PB is supervised, while in the latter all PB receive supervision. Finally, the last line shows results with the full network, reaching 88.3% on MPII and 98.2% on Penn Action (single-clip), with a single multi-task model.

The proposed method benefits from both pose and appearance features, which are complementary to the action recognition task. Additionally, the confidence score $\hat{c}$ is also complementary to pose itself and leads to marginal action recognition gains if used to weight pose predictions. Similar results are achieved if confidence scores are concatenated to poses. In Table 5, we present results on pose estimation and on action recognition for different features extraction strategies. Considering pose features or appearance features alone, the results on Penn Action are respectively 97.4% and 97.9%, respectively 0.7% and 0.2% lower than combined features. We also show in the last row the influence of decoupled action poses, resulting in a small gain of 0.1% on action scores and 0.3% on pose estimation, which shows that decoupling action poses brings additional improvements, specially for pose estimation. When not considering decoupled poses, note that the best score on pose estimation happens when poses are not directly used for action, which also supports the evidence of competing losses.

Additionally, we can observe that decoupled action poses remain coherent with supervised poses, as shown in Fig. 7, which suggests that the initial pose supervision is a good initialization overall. Nonetheless, in some cases, decoupled probability maps can drift to regions in the image more relevant for action recognition, as illustrated in Fig. 8. For example, feet heat maps can drift to objects in the hands, since the last is more informative with respect to the performed action.

In this part we compare the results on human action recognition considering single-task and multi-task training protocols. In Table 6, in the first row, are shown results on PennAction and NTU datasets considering training with action supervision only, \ie, with the full network architecture (including pose estimation layers) but without pose supervision. In the second row we show the results when using the manually annotated poses from PennAction for pose supervision. We did not use NTU (Kinect) poses for supervision since they are very noisy. From this, we can notice an improvement of almost 10% on PennAction, only by adding pose supervision. When mixing with MPII data, it further increases 0.8%. On NTU, multi-tasking improves a significant 1.9%. We believe that the improvement of multi-tasking on PennAction is much more evident because this is a small dataset, therefore it is difficult to learn good representations for complex actions without explicit pose information. On the contrary, NTU is a large scale dataset, more suitable for learning approaches. As a consequence, the gap between single and multi-task on NTU is smaller, but still relevant.

Once the network is trained, it can be easily cut to perform faster inferences. For instance, the full model with 8 pyramids can be cut at the 4th or 2nd pyramids, which generally degrades the performance, but allows faster predictions. To show the trade-off between precision and speed, we cut the trained multi-task model at different prediction blocks and estimate the throughput in frames per second (FPS), evaluating pose estimation precision and action recognition classification accuracy. We consider mini batches with 16 images for pose estimation and single video clips of 8 frames for action. The results are shown in Fig. 9. For both 2D and 3D scenarios, the best predictions are at more than 90 FPS. For the 3D scenario, pose estimation on Human3.6M can be performed at more than 180 FPS and still reach a competitive result of 57.3 millimeters error, while for action recognition on NTU, at the same speed, we still obtain state of the art results with 87.7% of correctly classified actions, or even comparable results with recent approaches at more than 240 FPS. Finally, we show our results for both 2D and 3D scenarios compared to previous methods in Table 7, considering different inference speed. Note that our method is the only to perform both pose and action estimation in a single prediction, while achieving state-of-the-art results at a very high speed.