Learnable Triangulation of Human Pose
Abstract
We present two novel solutions for multiview 3D human pose estimation based on new learnable triangulation methods that combine 3D information from multiple 2D views. The first (baseline) solution is a basic differentiable algebraic triangulation with an addition of confidence weights estimated from the input images. The second solution is based on a novel method of volumetric aggregation from intermediate 2D backbone feature maps. The aggregated volume is then refined via 3D convolutions that produce final 3D joint heatmaps and allow modelling a human pose prior. Crucially, both approaches are endtoend differentiable, which allows us to directly optimize the target metric. We demonstrate transferability of the solutions across datasets and considerably improve the multiview state of the art on the Human3.6M dataset. Video demonstration, annotations and additional materials will be posted on our project page^{1}^{1}1https://saicviolet.github.io/learnabletriangulation.
1 Introduction
3D human pose estimation is one of the fundamental problems in computer vision, with applications in sports, action recognition, computerassisted living, humancomputer interfaces, special effects, and telepresence. To date, most of the efforts in the community are focused on monocular 3D pose estimation. Despite a lot of recent progress, the problem of inthewild monocular 3D human pose estimation is far from being solved. Here, we consider a simpler yet still challenging problem of multiview 3D human pose estimation.
There are at least two reasons why multiview human pose estimation is interesting. First, multiview pose estimation is arguably the best way to obtain ground truth for monocular 3D pose estimation [5, 22] inthewild. This is because the competing techniques such as markerbased motion capture [10] and visualinertial methods [19] have certain limitations such as inability to capture rich pose representations (e.g. to estimate hands pose and face pose alongside limb pose) as well as various clothing limitations. The downside is, previous works that used multiview triangulation for constructing datasets relied on excessive, almost impractical number of views to get the 3D ground truth of sufficient quality [5, 22]. This makes the collection of new inthewild datasets for 3D pose estimation very challenging and calls for the reduction of the number of views needed for accurate triangulation.
The second motivation to study multiview human pose estimation is that, in some cases, it can be used directly to track human pose in realtime for the practical end purpose. This is because multicamera setups are becoming progressively available in the context of various applications, such as sports or computerassisted living. Such practical multiview setups rarely go beyond having just a few views. At the same time, in such a regime, modern multiview methods have accuracy comparable to welldeveloped monocular methods [18, 12, 6, 16, 13]. Thus, improving the accuracy of multiview pose estimation from few views is an important challenge with direct practical applications.
In this work, we argue that given its importance, the task of multiview pose estimation has received disproportionately little attention. We propose and investigate two simple and related methods for multiview human pose estimation. Behind both of them lies the idea of learnable triangulation, which allows us to dramatically reduce the number of views needed for accurate estimation of 3D pose. During learning, we either use marker based motion capture ground truth or “meta”ground truth obtained from the excessive number of views. The methods themselves are as follows: (1) a simpler approach based on algebraic triangulation with learnable camerajoint confidence weights, and (2) a more complex volumetric triangulation approach based on dense geometric aggregation of information from different views that allows modelling a human pose prior. Crucially, both of the proposed solutions are fully differentiable, which permits endtoend training.
Below, we review related work in monocular and multiview human pose estimation, and then discuss the details of the new learnable triangulation methods. In the experimental section, we perform an evaluation on the popular Human3.6M [3] and CMU Panoptic [5] datasets, demonstrating stateoftheart accuracy of the proposed methods and and their ability of crossdataset generalization.
2 Related work
Single view 3D pose estimation.
Current stateoftheart solutions for the monocular 3D pose estimation can be divided into two subcategories. The first category is using high quality 2D pose estimation engines with subsequent separate lifting of the 2D coordinates to 3D via deep neural networks (either fullyconnected, convolutional or recurrent). This idea was popularized in [8] and offers several advantages: it is simple, fast, can be trained on motion capture data (with skeleton/view augmentations) and allows switching 2D backbones after training. Despite known ambiguities inherent to this family of methods (i.e. orientation of arms’ joints in current skeleton models), this paradigm is adopted in the current multiframe state of the art [13] on the Human3.6M benchmark [3].
Multiview view 3D pose estimation.
Studies of multiview 3D human pose estimation are generally aimed at getting the groundtruth annotations for the monocular 3D human pose estimation [14, 5]. The work [6] proposed concatenating joints’ 2D coordinates from all views into a single batch as an input to a fully connected network that is trained to predict the global 3D joint coordinates. This approach can efficiently use the information from different views and can be trained on motion capture data. However, the method is by design unable to transfer the trained models to new camera setups, while the authors show that the approach is prone to strong overfitting.
Few works used volumetric pose representation in multiview setups[12, 5]. Specifically, [5] utilized unprojection of 2D keypoint probability heatmaps (obtained from a pretrained 2D keypoint detector) to volume with subsequent nonlearnable aggregation. Our work differs in two ways. First, we process information inside the volume in a learnable way. Second, we train the network endtoend, thus adjusting the 2D backbone and alleviating the need for interpretability of the 2D heatmaps. This allows to transfer several selfconsistent pose hypotheses from 2D detectors to the volumetric aggregation stage (which was not possible with previous designs).
The work [18] used a multistage approach with an external 3D pose prior [17] to infer the 3D pose from 2D joints’ coordinates. During the first stage, images from all views were passed through the backbone convolutional neural network to obtain 2D joints’ heatmaps. The positions of maxima in the heatmaps were jointly used to infer the 3D pose via optimizing latent coordinates in 3D pose prior space. In each of the subsequent stages, 3D pose was reprojected back to all camera views and fused with predictions from the previous layer (via a convolutional network). Next, the 3D pose was reestimated from the positions of heatmap maxima, and the process repeated. Such procedure allowed correcting the predictions of 2D joint heatmaps via indirect holistic reasoning on a human pose. In contrast to our approach, in [18] there is no gradient flow from the 3D predictions to 2D heatmaps and thus no direct signal to correct the prediction of 3D coordinates.
3 Method
Our approach assumes we have synchronized video streams from cameras with known projection matrices capturing performance of a single person in the scene. We aim at estimating the global 3D positions of a fixed set of human joints with indices at timestamp . For each timestamp the frames are processed independently (i.e. without using temporal information), thus we omit the index for clarity.
For each frame, we crop the images using the bounding boxes either estimated by available offtheshelf 2D human detectors or from ground truth (if provided). Then we feed the cropped images into a deep convolutional neural network backbone based on the ”simple baselines” architecture [21].
The convolutional neural network backbone with learnable weights consists of a ResNet152 network (output denoted by ), followed by a series of transposed convolutions that produce intermediate heatmaps (the output denoted by ) and a  kernel convolutional neural network that transforms the intermediate heatmaps to interpretable joint heatmaps (output denoted by ; the number of output channels is equal to the number of joints ). In the two following sections we describe two different methods to infer joints’ 3D coordinates by aggregating information from multiple views.
Algebraic triangulation approach.
In the algebraic triangulation baseline we process each joint independently of each other. The approach is built upon triangulating the 2D positions obtained from the joint’s backbone heatmaps from different views: (Figure 1). To estimate the 2D positions we first compute the softmax across the spatial axes:
(1) 
where parameter is discussed below. Then we calculate the 2D positions of the joints as the center of mass of the corresponding heatmaps (socalled softargmax operation):
(2) 
An important feature of softargmax is that rather than getting the index of the maximum, it allows the gradients to flow back to heatmaps from the output 2D position of the joints . Since the backbone was pretrained using a loss other than softargmax (MSE over heatmaps without softmax [16]), we adjust the heatmaps via multiplying them by an ’inverse temperature’ parameter in (1), so at the start of the training the softargmax gives an output close to the positions of the maximum.
To infer the 3D positions of the joints from their 2D estimates we use a linear algebraic triangulation approach [1]. The method reduces the finding of the 3D coordinates of a joint to solving the overdetermined system of equations on homogeneous 3D coordinate vector of the joint :
(3) 
where is a matrix composed of the components from the full projection matrices and (see [1] for full details).
A naïve triangulation algorithm assumes that the joint coordinates from each view are independent of each other and thus all make comparable contributions to the triangulation. However, on some views the 2D position of the joints cannot be estimated reliably (e.g. due to joint occlusions), leading to unnecessary degradation of the final triangulation result. This greatly exacerbates the tendency of methods that optimize algebraic reprojection error to pay uneven attention to different views. The problem can be dealt with by applying RANSAC together with the Huber loss (used to score reprojection errors corresponding to inliers). However, this has its own drawbacks. E.g. using RANSAC may completely cut off the gradient flow to the excluded cameras.
To address the aforementioned problems, we add learnable weights to the coefficients of the matrix corresponding to different views:
(4) 
where ; denotes the Hadamard product (i.e. th row of matrix is multiplied by th element of vector ). The weights are estimated by a convolutional network with learnable parameters (comprised of two convolutional layers, global average pooling and three fullyconnected layers), applied to the intermediate output of the backbone:
(5) 
This allows the contribution of the each camera view to be controlled by the neural network branch that is learned jointly with the backbone joint detector.
The equation (4) is solved via differentiable Singular Value Decomposition of the matrix , from which is set as the last column of . The final nonhomogeneous value of is obtained by dividing the homogeneous 3D coordinate vector by its fourth coordinate: .
Volumetric triangulation approach.
The main drawback of the baseline algebraic triangulation approach is that the images from different cameras are processed independently from each other, so there is no easy way to add a 3D human pose prior and no way to filter out the cameras with wrong projection matrices.
To solve this problem we propose to use a more complex and powerful triangulation procedure. We unproject the feature maps produced by the 2D backbone into 3D volumes (see Figure 2). This is done by filling a 3D cube around the person via projecting output of the 2D network along projection rays inside the 3D cube. The cubes obtained from multiple views are then aggregated together and processed. For such volumetric triangulation approach, the 2D output does not have to be interpretable as joint heatmaps, thus, instead of unprojecting themselves, we use the output of a trainable single layer convolutional neural network with kernel and output channels (the weights of this layer are denoted by ) applied to the input from the backbone intermediate heatmaps :
(6) 
To create the volumetric grid, we place a  sized 3D bound box in the global space around the human pelvis (the position of the pelvis is estimated by the algebraic triangulation baseline described above, denotes the size of the box in meters) with the Yaxis perpendicular to the ground and a random orientation of the Xaxis. We discretize the bounding box by a volumetric cube , filling it with the global coordinates of the center of each voxel (in a similar way to [5]).
For each view, we then project the 3D coordinates in to the plane: (note that ) and fill a cube by bilinear sampling [4] from the maps of the corresponding camera view using 2D coordinates in :
(7) 
where denotes bilinear sampling. We then aggregate the volumetric maps from all views to form an input to the further processing that does not depend on the number of camera views. We study three diverse methods for the aggregation:

Raw summation of the voxel data:
(8) 
Summation of the voxel data with normalized confidence multipliers (obtained similarly to using a branch attached to backbone):
(9) 
Calculating a relaxed version of maximum. Here, we first compute the softmax for each individual voxel across all cameras, producing the volumetric coefficient distribution with the role similar to scalars :
(10) Then, the voxel maps from each view are summed with the volumetric coefficients :
(11)
Aggregated volumetric maps are then fed into a learnable volumetric convolutional neural network (with weights denoted by ), with architecture similar to V2V [11], producing the interpretable 3Dheatmaps of the output joints:
(12) 
Next, we compute softmax of across the spatial axes (similar to (1)):
(13) 
and estimate the center of mass for each of the volumetric joint heatmaps to infer the positions of the joints in 3D:
(14) 
Lifting to 3D allows getting more robust results, as the wrong predictions are spatially isolated from the correct ones inside the cube, so they can be discarded by convolutional operations. The network also naturally incorporates the camera parameters (uses them as an input), allows modelling the human pose prior and can reasonably handle multimodality in 2D detections.
Losses.
For both of the methods described above, the gradients pass from the output prediction of 3D joints’ coordinates to the input RGBimages making the pipeline trainable endtoend. For the case of algebraic triangulation, we apply a soft version of perjoint Mean Square Error (MSE) loss to make the training more robust to outliers. This variant leads to better results compared to raw MSE or L1 (mean absolute error):
(15) 
Here, denotes the threshold for the loss, which is set to in the experiments. The final loss is the average over all valid joints and all scenes in the batch.
For the case of volumetric triangulation, we use the L1 loss with a weak heatmap regularizer, which maximizes the prediction for the voxel that has inside of it the groundtruth joint:
(16) 
Without the second term, for some of the joints (especially, pelvis) the produced output volumetric heatmaps are not interpretable, probably due to insufficient size of the training datasets [16]. Setting the to a small value () makes them interpretable, as the produced heatmaps always have prominent maxima close to the prediction. At the same time, such small does not seem to have any effect on the final metrics, so its use can be avoided if interpretability is not needed. We have also tried the loss (15) from the algebraic triangulation instead of L1, but it performed worse in our experiments.
4 Experiments
Human3.6M dataset.
The Human3.6M [3] is currently one of the largest 3D human pose benchmarks with many reported results both for monocular and multiview setups. The full dataset consist of 3.6 million frames from 4 synchronized 50 Hz digital cameras along with the 3D pose annotations (collected using a markerbased MoCap system comprised of 10 separate IRcameras). The dataset has 11 human subjects (5 females and 6 males) split into train, validation and test (only train and validation have the groundtruth annotations).
The 2D backbone for Human3.6M was pretrained on the COCO dataset [7] and finetuned jointly on MPII and Human3.6M for 10 epochs using the Adam optimizer with learning rate. We use the 3D groundtruth and camera parameters provided by Martinez et al. [8]. We undistort the images by applying gridsampling on the video frames. If not mentioned explicitly, all networks are trained using four cameras and evaluated using all available cameras (either three or four, as one of the subjects lacks data from one camera). All algorithms use the 2D bounding boxes annotations provided with the dataset. The networks are trained for 6 epochs with learning rate for the 2D backbone and a separate learning rate for the volumetric backbone.
Note that the volumetric triangulation approach uses predictions obtained from the algebraic triangulation (the whole system, however, potentially can be finetuned endtoend). The size of volumetric cube was set to 2.5 m, which can enclose all subjects even if there is a few tens of centimeter error in pelvis prediction (which is much higher than the average error by the algebraic triangulation baseline). The number of output channels from the 2D backbone was set to . We did not apply any augmentations during the training, other than rotating the orientation of the cube in volumetric triangulation around the vertical axis.
Protocol 1 (relative to pelvis)  Dir.  Disc.  Eat  Greet  Phone  Photo  Pose  Purch.  Sit  SitD.  Smoke  Wait  WalkD.  Walk  WalkT.  Avg 
Monocular methods (MPJPE relative to pelvis, mm)  
Martinez et al. [8]  51.8  56.2  58.1  59.0  69.5  78.4  55.2  58.1  74.0  94.6  62.3  59.1  65.1  49.5  52.4  62.9 
Sun et al. [16]                                49.6 
Pavllo et al. [13]  45.2  46.7  43.3  45.6  48.1  55.1  44.6  44.3  57.3  65.8  47.1  44.0  49.0  32.8  33.9  46.8 
Hossain & Little [2]  48.4  50.7  57.2  55.2  63.1  72.6  53.0  51.7  66.1  80.9  59.0  57.3  62.4  46.6  49.6  58.3 
Ours, volumetric single view ()  41.9  49.2  46.9  47.6  50.7  57.9  41.2  50.9  57.3  74.9  48.6  44.3  41.3  52.8  42.7  49.9 
Multiview methods (MPJPE relative to pelvis, mm)  
MultiView Martinez [18]  46.5  48.6  54.0  51.5  67.5  70.7  48.5  49.1  69.8  79.4  57.8  53.1  56.7  42.2  45.4  57.0 
Pavlakos et al. [12]  41.2  49.2  42.8  43.4  55.6  46.9  40.3  63.7  97.6  119.0  52.1  42.7  51.9  41.8  39.4  56.9 
Tome et al. [18]  43.3  49.6  42.0  48.8  51.1  64.3  40.3  43.3  66.0  95.2  50.2  52.2  51.1  43.9  45.3  52.8 
Kadkhodamohammadi & Padoy [6]  39.4  46.9  41.0  42.7  53.6  54.8  41.4  50.0  59.9  78.8  49.8  46.2  51.1  40.5  41.0  49.1 
RANSAC (our implementation)  24.1  26.1  24.0  24.6  27.0  25.0  23.3  26.8  31.4  49.5  27.8  25.4  24.0  27.4  24.1  27.4 
Ours, algebraic (w/o conf)  22.9  25.3  23.7  23.0  29.2  25.1  21.0  26.2  34.1  41.9  29.2  23.3  22.3  26.6  23.3  26.9 
Ours, algebraic  20.4  22.6  20.5  19.7  22.1  20.6  19.5  23.0  25.8  33.0  23.0  21.6  20.7  23.7  21.3  22.6 
Ours, volumetric (softmax aggregation)  18.8  20.0  19.3  18.7  20.2  19.3  18.7  22.3  23.3  29.1  21.2  20.3  19.3  21.6  19.8  20.8 
Ours, volumetric (sum aggregation)  19.3  20.5  20.1  19.3  20.6  19.8  19.0  22.9  23.5  29.8  22.0  21.4  19.8  22.1  20.3  21.3 
Ours, volumetric (conf aggregation)  19.9  20.0  18.9  18.5  20.5  19.4  18.4  22.1  22.5  28.7  21.2  20.8  19.7  22.1  20.2  20.8 
Protocol 1, absolute positions, filtered validation  Dir.  Disc.  Eat  Greet  Phone  Photo  Pose  Purch.  Sit  SitD.  Smoke  Wait  WalkD.  Walk  WalkT.  Avg 

Multiview methods (absolute MPJPE, mm)  
RANSAC  21.6  22.9  20.9  21.0  23.1  23.0  20.8  22.0  26.4  26.6  24.0  21.5  21.0  23.9  20.8  22.8 
Ours, algebraic (w/o conf)  21.7  23.7  22.2  20.4  26.7  24.2  19.9  22.6  31.2  35.6  26.8  21.2  20.9  24.6  21.1  24.5 
Ours, algebraic  18.1  20.0  17.6  17.0  18.9  19.3  17.4  19.2  21.9  23.2  19.5  18.0  18.3  20.5  17.9  19.2 
Ours, volumetric (softmax aggregation)  16.9  18.1  16.6  16.0  17.1  17.9  16.5  18.5  19.6  20.1  18.2  16.8  17.2  19.0  16.6  17.7 
Ours, volumetric (sum aggregation)  17.7  18.5  17.2  16.5  17.8  18.4  17.0  18.9  19.8  20.9  18.9  17.8  17.8  19.2  17.3  18.3 
Ours, volumetric (conf aggregation)  18.0  18.3  16.5  16.1  17.4  18.2  16.5  18.5  19.4  20.1  18.2  17.4  17.2  19.2  16.6  17.9 
For Human3.6M we follow the most popular protocol with 17joint subset and testing on the validation. We used the MPJPE (Mean Per Joint Position Error) metric, which is L2 distance between the groundtruth and predicted positions of the joints (in most cases, measured with respect to pelvis). We use every fifth frame for the evaluation. As a baseline, we implemented a simple triangulation method with RANSAC and Huber loss, which is the defacto standard for solving robust estimation problems. The baseline uses the same pretrained 2D backbone. The results of the standard protocol are summarized in Table 1.
Our implementations surpass the previous art by a large margin, even for the simplest RANSAC baseline. The introduced volumetric methods performs the best, providing about additional reduction in the error to the RANSAC, which is significant.
While most of the works on monocular 3D human pose evaluate the positions of the joints relative to the pelvis (in order to avoid the estimation of global coordinates, which is problematic for monocular algorithms), evaluating with respect to the global coordinates is more reasonable for multiview setups. This, however, is not straightforward due to 3D pose annotation errors in the Human3.6M  the problem is that some scenes of the ’S9’ validation actor (parts of ’Greeting’, ’SittingDown’ and ’Waiting’, see our project page) have the ground truth erroneously shifted in 3D compared to the actual position. Interestingly, the error is nullified when the pelvis is subtracted (as done for monocular methods), however, to make the results for the multiview setup interpretable we must exclude these scenes from the evaluation. The results for the absolute MPJPE for our methods with these scenes excluded are presented in Table 2, giving a better sense of the magnitude of errors. Interestingly, the average MPJPE for volumetric is much smaller than the size of the voxel (3.9 cm), showing the importance of the subpixel resolving softargmax.
Our volumetric multiview methods can be generalized to the case of a single camera view, naturally taking into account the camera intrinsics. To check the monocular performance we have done a separate experiment with training using a random number of cameras from 1 to 4. For the case of a single camera we used the L1 loss on joint positions relative to the pelvis (as is usually done for the monocular methods). Without any tuning this resulted in 49.9 mm error, which is close to the current state of the art. Note that the position of the cube center was estimated by triangulating the pelvis from all 4 cameras, so the performance of the methods might be somewhat overoptimistic. On the other hand, the average relative positions error of the monocular method lies within 46 cm, which corresponds to a negligible shift for the volumetric cube position (less than 2 voxels).
Model  MPJPE, mm 

RANSAC  39.5 
Ours, algebraic (w/o conf)  33.4 
Ours, algebraic  21.3 
Ours, volumetric (softmax aggregation)  13.7 
Ours, volumetric (sum aggregation)  13.7 
Ours, volumetric (conf aggregation)  14.0 
CMU Panoptic dataset.
The CMU panoptic is a new multicamera dataset maintained by the Carnegie Mellon University [5, 20, 15]. The dataset provides 30 Hz FullHD videostreams of 40 subjects from up to 31 synchronized cameras.
The dataset is provided with annotations of the FullHD cameras acquired via triangulation using all camera views. Since there are no other published results on the quality of multiview pose estimation on CMU, we use our own protocol. For the tests we use the 17subset of the 19joint annotation format in the dataset, which coincides with the popular COCO format [7]. We used the same train/val split as in [20], which only has scenes with a single person at each point of time. Additionally, we split the dataset by camera views (4 cameras in val, up to 27 cameras in train), so there is no overlap between the test and validation both in terms of subjects and camera views. To get the human bounding boxes we use Mask RCNN 2D detector with ResNet152 backbone [9]). Note that we process the frames without taking into account the lens distortion which leads to some loss of accuracy.
The networks are trained using the Adam optimizer similarly to Human3.6M as described in the previous section. The number of views during training was selected randomly from 2 to 5.
The comparison between our multiview methods is presented in Table 3 with absolute MPJPE used as the main metric. Here, the volumetric approach has a dramatic advantage over the algebraic one, and its superiority is far more evident than on Human3.6M. We believe that the main point in which CMU differs from Human3.6M is that in CMU most cameras do not have the full view of a person the whole time, leading to strong occlusions and missing parts. This suggests the importance of the 3D prior that can be learnt only by volumetric models. It seems that RANSAC performs worse compared to algebraic triangulation without confidences because it is not finetuned on the CMU data. The difference between the methods’ prediction is illustrated in Figure 3. We have not observed any significant difference between the volumetric aggregation methods.
In Figure 4 we present a plot for the error versus the numbers of used cameras. The plot demonstrates that the proposed volumetric triangulation methods allow drastically reducing the number of cameras in reallife setups: the accuracy on ”meta”groundtruth for RANSAC approach with 28 cameras is surpassed by the volumetric approach with just four cameras.
We also conducted experiments to demonstrate that the learnt model indeed generalizes to new setups. For that we applied a CMUtrained model to Human3.6M validation scenes. The qualitative results for the learnable triangulation are presented in Figure 5. Please see videos for all of the methods on our project page. To provide a quantitative measure of the generalizing ability, we have measured the MPJPE for the set of joints which seem to have the most similar semantics (namely, ’elbows’, ’wrists’ and ’knees’). The measured MPJPE is 36 mm for learnable triangulation and 34 mm for the volumetric approach, which seems reasonable when compared to the results of the methods trained on Human3.6M (1618 mm, depending on the triangulation method).
5 Conclusion
We have presented two novel methods for the multiview 3D human pose estimation based on learnable triangulation that achieve stateoftheart performance on the Human3.6M dataset. The proposed solutions drastically reduce the number of views needed to achieve high accuracy, and produce smooth pose sequences on the CMU Panoptic dataset without any temporal processing (see our project page for demonstration), pointing that it can potentially improve the ground truth annotation of the dataset. An ability to transfer the trained method between setups is demonstrated for the CMU Panoptic Human3.6M pair. The volumetric triangulation strongly outperformed all other approaches both on CMU Panoptic and Human3.6M datasets. We speculate that due to its ability to learn a human pose prior this method is robust to occlusions and partial views of a person. Another important advantage of this method is that it explicitly takes the camera parameters as independent input. Finally, volumetric triangulation also generalizes to monocular images if human’s approximate position is known, producing results close to state of the art.
One of the major limitations of our approach is that it supports only a single person in the scene. This problem can be mitigated by applying available ReID solutions to the 2D detections of humans, however there might be more seamless solutions. Another major limitation of the volumetric triangulation approach is that it relies on the predictions of the algebraic triangulation. This leads to the need for having at least two camera views that observe the pelvis, which might be a problem for some applications. The performance of our method can also potentially be further improved by adding multistage refinement in a way similar to [18].
References
 [1] R. Hartley and A. Zisserman. Multiple view geometry in computer vision. Cambridge university press, 2003.
 [2] M. R. I. Hossain and J. J. Little. Exploiting temporal information for 3d human pose estimation. In European Conference on Computer Vision, pages 69–86. Springer, 2018.
 [3] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325–1339, jul 2014.
 [4] M. Jaderberg, K. Simonyan, A. Zisserman, et al. Spatial transformer networks. In Advances in neural information processing systems, pages 2017–2025, 2015.
 [5] H. Joo, T. Simon, X. Li, H. Liu, L. Tan, L. Gui, S. Banerjee, T. S. Godisart, B. Nabbe, I. Matthews, T. Kanade, S. Nobuhara, and Y. Sheikh. Panoptic studio: A massively multiview system for social interaction capture. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017.
 [6] A. Kadkhodamohammadi and N. Padoy. A generalizable approach for multiview 3D human pose regression. apr 2018.
 [7] T.Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740–755. Springer, 2014.
 [8] J. Martinez, R. Hossain, J. Romero, and J. J. Little. A simple yet effective baseline for 3d human pose estimation. In Proceedings of the IEEE International Conference on Computer Vision, pages 2640–2649, 2017.
 [9] F. Massa and R. Girshick. maskrcnnbenchmark: Fast, modular reference implementation of Instance Segmentation and Object Detection algorithms in PyTorch. https://github.com/facebookresearch/maskrcnnbenchmark, 2018. Accessed: 01 Feb 2019.
 [10] T. B. Moeslund, A. Hilton, and V. Krüger. A survey of advances in visionbased human motion capture and analysis. Computer vision and image understanding, 104(23):90–126, 2006.
 [11] G. Moon, J. Chang, and K. M. Lee. V2vposenet: Voxeltovoxel prediction network for accurate 3d hand and human pose estimation from a single depth map. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
 [12] G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis. Harvesting multiple views for markerless 3D human pose annotations. Proceedings  30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017Janua:1253–1262, 2017.
 [13] D. Pavllo, C. Feichtenhofer, D. Grangier, and M. Auli. 3d human pose estimation in video with temporal convolutions and semisupervised training. arXiv, abs/1811.11742, 2018.
 [14] H. Rhodin, F. Meyer, J. Sporri, E. Muller, V. Constantin, P. Fua, I. Katircioglu, and M. Salzmann. Learning Monocular 3D Human Pose Estimation from Multiview Images. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8437–8446. IEEE, jun 2018.
 [15] T. Simon, H. Joo, and Y. Sheikh. Hand keypoint detection in single images using multiview bootstrapping. CVPR, 2017.
 [16] X. Sun, B. Xiao, F. Wei, S. Liang, and Y. Wei. Integral human pose regression. In Proceedings of the European Conference on Computer Vision (ECCV), pages 529–545, 2018.
 [17] D. Tome, C. Russell, and L. Agapito. Lifting from the deep: Convolutional 3d pose estimation from a single image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2500–2509, 2017.
 [18] D. Tome, M. Toso, L. Agapito, and C. Russell. Rethinking Pose in 3D: Multistage Refinement and Recovery for Markerless Motion Capture. In 2018 International Conference on 3D Vision (3DV), pages 474–483. IEEE, sep 2018.
 [19] T. Von Marcard, G. PonsMoll, and B. Rosenhahn. Human pose estimation from video and imus. IEEE transactions on pattern analysis and machine intelligence, 38(8):1533–1547, 2016.
 [20] D. Xiang, H. Joo, and Y. Sheikh. Monocular total capture: Posing face, body, and hands in the wild. arXiv preprint arXiv:1812.01598, 2018.
 [21] B. Xiao, H. Wu, and Y. Wei. Simple baselines for human pose estimation and tracking. In European Conference on Computer Vision (ECCV), 2018.
 [22] Z. Yu, J. S. Yoon, P. Venkatesh, J. Park, J. Yu, and H. S. Park. Humbi 1.0: Human multiview behavioral imaging dataset. arXiv preprint arXiv:1812.00281, 2018.