T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensor
Recent findings indicate that over-parametrization, while crucial for successfully training deep neural networks, also introduces large amounts of redundancy. Tensor methods have the potential to efficiently parametrize over-complete representations by leveraging this redundancy. In this paper, we propose to fully parametrize Convolutional Neural Networks (CNNs) with a single high-order, low-rank tensor. Previous works on network tensorization have focused on parametrizing individual layers (convolutional or fully connected) only, and perform the tensorization layer-by-layer separately. In contrast, we propose to jointly capture the full structure of a neural network by parametrizing it with a single high-order tensor, the modes of which represent each of the architectural design parameters of the network (e.g. number of convolutional blocks, depth, number of stacks, input features, etc). This parametrization allows to regularize the whole network and drastically reduce the number of parameters. Our model is end-to-end trainable and the low-rank structure imposed on the weight tensor acts as an implicit regularization. We study the case of networks with rich structure, namely Fully Convolutional Networks (FCNs), which we propose to parametrize with a single \myth–order tensor. We show that our approach can achieve superior performance with small compression rates, and attain high compression rates with negligible drop in accuracy for the challenging task of human pose estimation.
For a wide range of challenging tasks, including recognition [22, 34, 13], detection , semantic segmentation [25, 12] and human pose estimation , the state-of-the-art is currently attained with Convolutional Neural Networks (CNNs). There is evidence that a key feature behind the success of these methods is over-parametrization, which helps find good local minima [11, 35]. However, at the same time, over-parametrization leads to a great amount of redundancy, and from a statistical perspective, it might hinder generalization because it excessively increases the number of parameters. Furthermore, models with an excessive number of parameters have increased storage and computation requirements, rendering them problematic for deployment on devices with limited computational resources. This paper focuses on a novel way of leveraging the redundancy in the parameters of CNNs by jointly parametrizing the whole network using tensor methods.
There is a significant amount of recent work on using tensors to reduce the redundancy and improve the efficiency of CNNs, mostly focusing on re-parametrizing individual layers. For example, [36, 17] treat a convolutional layer as a 4D tensor and then compute a decomposition of the 4D tensor into a sum of a small number of low-rank tensors. Similarly,  proposes tensorizing the fully-connected layers. The bulk of these methods focus on tensorizing individual layers only, and perform the tensorization layer-by-layer disjointly, usually by applying a tensor decomposition to the pre-trained weights and then fine-tuning to compensate for the performance loss. For example,  tensorizes the second convolutional layer of AlexNet .
Our paper primarily departs from prior work by using a single high-order tensor to parametrize the whole CNN as opposed to using different tensors to parametrize the individual layers. In particular, we propose to parametrize the network with a single high-order tensor, each dimension of which represents a different architectural design parameter of the network. For the case of Fully Convolutional Networks (FCNs) with an encoder-decoder structure considered herein (see also Fig. 1), each dimension of the dimensional tensor represents a different architectural design parameter of the network such as the number of (stacked) FCNs used, the depth of each network, the number of input and output features for each convolutional block and the spatial dimensions of each of the convolutional kernels. By modelling the whole FCN with a single tensor, our approach allows for learning correlations between the different tensor dimensions and hence to fully capture the structure of the network. Moreover, this parametrization implicitly regularizes the whole network and drastically reduces the number of parameters by imposing a low-rank structure on that tensor. Owing to these properties, our framework is much more general and flexible compared to prior work offering increased accuracy and high compression rates. In summary, the contributions of this work are:
We propose using a single high-order tensor for whole network tensorization and applying it for capturing the rich structure of Fully Convolutional Networks. Our end-to-end trainable approach allows for a wide spectrum of network decompositions and compression rates which can be chosen and optimized for a particular application.
We show that for a large range of compression rates (both high and low), our method preserves high accuracy. Compared to prior work based on tensorizing individual convolutional layers, our method consistently achieves higher accuracy, especially for the case of high compression rates. In addition, we show that, for lower compression rates, our method outperforms the original uncompressed network.
We illustrate the favorable properties of our method by performing a large number of experiments and ablation studies for the challenging task of human pose estimation. The experiments shed light on several interesting aspects of our method including the effect of varying the rank for each mode of the tensor, as well as the decomposition method used. We further validate our conclusions on a different dense prediction task, namely semantic facial part segmentation.
2 Related Work
In this section, we review related work, both for tensor methods and human pose estimation.
offer a natural extension of traditional algebraic methods to higher orders. For instance, Tucker decomposition can be seen as a generalization of PCA to higher dimensions . Tensor decompositions have wide-reaching applications, including learning a wide range of probabilistic latent-variable models . Tensor methods have been recently applied to deep learning, for instance, to provide a theoretical analysis of deep neural nets . New layers were also proposed, leveraging tensor methods.  proposes tensor contraction layers to reduce the dimensionality of activation tensors while preserving their multi-linear structure. Tensor regression layers  express outputs through a low-rank multi-linear mapping from a high-order activation tensor to an output tensor of arbitrary order.
A lot of existing work has been dedicated to leveraging tensor decompositions in order to re-parametrizing existing layers, either to speed up computation or to reduce the number of parameters. Separable convolutions, for instance, can be obtained from existing ones by applying CP decomposition to their kernel. The authors in  propose such parametrization of individual convolutional layers using CP decomposition with the goal of speeding them up. Specifically, each of the 4D tensors parametrizing the convolutional layers of a pre-trained network are decomposed into a sum of rank– tensors using CP decomposition. The resulting factors are used to replace each existing convolution with a series of convolutional layers with smaller kernels. The network is then fine-tuned to restore performance.  proposes a similar approach but uses Tucker decomposition instead of CP to decompose the convolutional layers of a pre-trained network, before fine-tuning to restore the performance. Specifically, Tucker decomposition is applied to each convolutional kernel of a pre-trained network, on two of the modes (input and output channel modes). The resulting network is fine-tuned to compensate for the drop in performance induced by the compression.
In , the layers of deep convolutional neural networks are also re-parametrized using CP decomposition, optimized using the tensor power method. The network is then iteratively fine-tuned to restore performance. Similarly,  proposes to use tensor decomposition to remove redundancy in convolutional layers and express these as the composition of two convolutional layers with less parameters. Each 2D filter is approximated by a sum of rank– matrices. Thanks to this restricted setting, a closed-form solution can be readily obtained with SVD. This is done for each convolutional layer with a kernel of size larger than 1. While all these focus of convolutional layers, other types of layers can also be parametrized. For instance,  uses the Tensor-Train (TT) format  to impose a low-rank tensor structure on the weights of the fully-connected layers. Tensorization of generative adversarial networks  and sequence models  have been also proposed.
The work of  proposes a new residual block, the so-called collective residual unit (CRU), which is obtained by applying a generalized block term decomposition to the last two modes of a 4\myth–order tensor obtained by stacking the convolutional kernels of several residual units. Similarly to existing works, each of the CRUs is parametrized individually.  leverages tensor decomposition for multi-task learning to allow for weight sharing between the fully-connected and convolutional layers of two or more deep neural networks.
Overall, to the best of our knowledge, our work is the first to propose an end-to-end trainable architecture, fully parametrized by a single high order low-rank tensor.
Other methods for network decomposition.
There are also other methods, besides tensor-based ones, for reducing the redundancy and number of parameters in neural networks. A popular approach is quantization which is concerned with quantizing the weights and/or the features of a network [41, 43, 46, 37, 10, 31]. Quantization methods should be considered orthogonal to tensor methods as one could apply them to the output of tensor decompositions, too. Similarly, complementary to our work should be considered methods for improving the efficiency of neural networks using weight pruning [24, 14].
More related to our work are hand-crafted decomposition methods such as MobileNet  and Xception  which decompose convolutions using efficient depth-wise and point-wise convolutions. We compare our methods with MobileNet, the method of choice for improving the efficiency of CNNs, and show that our approach outperforms it by large margin.
Human pose estimation.
CNN–based methods have recently produced results of remarkable accuracy for the task of human pose estimation, outperforming traditional methods by large margin [40, 39, 30, 4, 26, 42]. Arguably, one of the most widely used architectures for this task is the stacked HourGlass (HG) network proposed by . An HG is an encoder-decoder network with skip connections between the encoder and the decoder, suitable for making predictions at a pixel level in a fully convolutional manner.  uses a stack of 8 of these networks to achieve state-of-the-art performance on the MPII dataset . The architecture is shown in Fig. 1. In this work, we choose tensorizing the HG network primarily because of its rich structure which makes it suitable to model it with a high-order tensor. We note that the aim of this work is not to produce state-of-the-art results for the task of human pose estimation but to show the benefits of modelling a state-of-the-art architecture with a single high-order tensor.
3 Mathematical background
In this section we first introduce some mathematical background regarding the notation and tensor methods used in this paper.
We denote vectors (1\myst–order tensors) as , matrices (2\mynd–order tensors) as , and tensors of order 3 or greater as . We denote element of a tensor as . A colon is used to denote all elements of a mode, e.g. the mode–1 fibers of are denoted as . Finally, for any denotes the set of integers .
of a tensor , is a matrix , with , defined by the mapping from element to , with .
For a tensor and a matrix , the n-mode product of a tensor is a tensor of size and can be expressed using the unfolding of and the classical dot product as
While –order tensors can easily be depicted as rectangles and –order tensors as cubes, it is impractical to represent high-order tensors in such way. We instead use tensor diagrams, which are undirected graphs where the vertices represent tensors. The degree of each vertex (i.e. the number of edges originating from this circle) specifies the order of the corresponding tensor. Tensor contraction over two modes is then represented by simply linking together the two edges corresponding to these two modes. Fig. 2 depicts the Tucker decomposition (i.e. contraction of a core tensor with factor matrices along each mode) of an 8\myth–order tensor with tensor diagrams.
4 T-Net: Fully-tensorized FCN architecture
In this section, we introduce our fully-tensorized method by first introducing the architecture before detailing the structure of the parametrization weights.
4.1 FCN tensorization
In this section, we describe how to tensorize the stacked HourGlass (HG) architecture of . The HG has a number of design parameters namely the number of (stacked) HGs, the depth of each HG, the three signal pathways of each HG (skip, downsample and upsample), the number of convolutional layers in each residual block (\myiethe depth of each block), the number of input and output features of each block and finally, the spatial dimensions of each of the convolutional kernels.
To facilitate the tensorization of the whole network, we used a modified HG architecture in which we replaced all the residual modules with the basic block introduced by , maintaining the same number of input and output channels throughout the network. We made the encoder and the decoder symmetric, with 4 residual modules each. Figure 1 illustrates the modified HG architecture. We note that from an accuracy perspective, this modification performs (almost) the same as the original HG proposed in .
From the network described above, we derive the high-order tensor for the proposed Tensorized-Network (T-Net) as follows: all weights of the network are parametrized by a single –order tensor , the modes of which correspond to the number of HGs ( #hg), the depth of each HG ( hg\mysubdepth), the three signal pathways ( hg\mysubsubnet), the number of convolutional layers per block ( b\mysubdepth), the number of input features ( f\mysubin), the number of output features ( f\mysubout), and finally the height ( h) and width ( w) of each of the convolutional kernels.
4.2 T-Net variants
Based on the previous parametrization of the network, we can add various low-rank constraints on the weight tensor, leading to variants of our method.
The Tucker form of our model expresses the constructed 8\myth–order tensor as a rank– Tucker tensor, composed of a low rank core along with projection factors , with . This allows us to write the network’s weight tensor in a decomposed form as:
See also Fig. 2 for a tensor diagram of the Tucker form of the weight tensor. Note that the CP decomposition is the special case of the Tucker decomposition, where the core is super-diagonal.
The Matrix-Product-State (MPS) form (also known as tensor-train ) of our model expresses the constructed 8\myth–order weight tensor as a series of third-order tensors (the cores) and allows for especially large space-savings. In our case, given , we can decompose it into a rank –MPS as a series of third-order cores . The boundary conditions dictate . In terms of individual elements, we can then write, for any :
4.3 Parameter analysis
This section compares the number of parameters of our model which parameterizes the whole weight tensor with a single high-order tensor with methods based on layer-wise decomposition (e.g. [17, 23]). Considering a Tucker rank– of the weight tensor parametrizing the whole network, the resulting number of parameters is:
Compressing each of the convolutional layer separately , with a rank and for the number of input and output features, respectively, and writing , we obtain the total number of parameters:
In comparison, our model, with the same ranks and imposed on the number of features, would only have parameters. In other words, our model has parameters less than a corresponding layer-wise decomposition.
Speeding up the convolutions.
When parametrized using a CP or Tucker decomposition, a convolutional layer can be efficiently replaced by a series of convolutions with smaller kernels [23, 17], thus allowing for large computational speedups. This efficient re-parametrization also applies to our model. To see this, given the weight tensor of our Tucker T-Net, we have
For any , let us denote , corresponding to one of the convolutional kernels of the T-Net. By re-arranging the terms, and considering the partially contracted core, we can write:
This gives us an effective way of approximating each convolution by three smaller convolutions . While getting the full speedup would require the writing of specialized CUDA kernels, some timings results with a naive implementation using PyTorch are shown in Table 1, for a single convolutional layer with a kernel tensor of size compressed using Tucker decomposition.
|3.79 ms.||4.36 ms.||2.72 ms.||2.45 ms.|
5 Experimental Setup
The bulk of our experiments were conducted for the task of human pose estimation. We also validated some of our conclusions by conducting experiments for a different dense prediction task, namely facial part segmentation.
Human pose estimation.
Semantic facial part segmentation.
We used a stacked HG architecture with the following architectural parameters: #hg = 4, hg\mysubdepth = 4, hg\mysubsubnet=3, b\mysubdepth = 2, f\mysubin=128, f\mysubout=128, and h = w = 3. This resulted in a –order tensor of size .
For the uncompressed baseline network, we reduced the number of its parameters by simply decreasing the number of channels in each residual block, varying it from 128 to 64. By doing so, (as opposed to reducing the number of stacks), we maintain all the architectural advantages offered by the stacked HG architecture and ensure a fair comparison with the proposed tensorized network.
All models were trained for 110 epochs using RMSprop . The learning rate was varied from to using a Multi-Step fixed scheduler. During training, we randomly augmented the data using: rotation ( to for human pose and to for face part segmentation), scale distortion ( to ), horizontal flipping and color jittering.
|Uncompressed Baseline||full, f\mysubin=f\mysubout=128||1x||87%|
|||rank–(128, 128, 2, 2)||1.4x||84.9%|
|||rank–(96, 96, 3, 3)||1.3x||86.8%|
|||rank–(64, 64, 3, 3)||2.3x||86.4%|
|||rank–(32, 32, 3, 3)||4.7x||85.3%|
|||rank–(16, 16, 3, 3)||6.9x||83.7%|
|Tucker T-Net [Ours]||rank–||1.7x||87.5%|
|Tucker T-Net [Ours]||rank–||1.8x||87.4%|
|Tucker T-Net [Ours]||rank–||3.7x||87.1%|
|Tucker T-Net [Ours]||rank–||3.4x||86.7%|
|Tucker T-Net [Ours]||rank–||4.2x||86.3%|
|Tucker T-Net [Ours]||rank–||5.2x||86.0%|
|MPS T-Net [Ours]||rank–||7.4x||85.5%|
For the human pose estimation experiments, we report accuracy in terms of PCKh . For facial part segmentation, we report segmentation accuracy using the mean accuracy and mIOU metrics . Finally, we measure the parameter savings using the , defined as the total number of parameters of the uncompressed network divided by the number of parameters of the compressed network.
This section offers an in-depth analysis of the performance and accuracy of the proposed T-Net. Our main results are that the proposed approach: i) outperforms the layer-wise decomposition of  and , which are the most closely related works to our method; ii) outperforms the uncompressed, original network for low compression rates; iii) achieves consistent compression ratios across arbitrary dimensions and iv) outperforms MobileNet  by large margin. Finally, we further validate some of these results for the task of semantic facial part segmentation.
All results reported were obtained by fine-tuning our networks in an end-to-end manner from a pre-trained uncompressed original network. We were able to reach the same level of accuracy when training from scratch, though this required training for more iterations. In contrast, we found that when trained from scratch, the layer-wise method of  reaches sub-par performance, as also reported in their paper.
|Uncompressed baseline||full, f\mysubin=f\mysubout=128||1x||76.02%||97.31%|
6.1 Redundancy of the weight tensor
In order to better understand the compressibility of each mode of the weight tensor, we first investigate the redundancy of each of the modes of the tensor by compressing only one of the modes at a time. Table 2 shows the accuracy (PCKh) as well as the compression ratio obtained by compressing one of the modes, corresponding respectively to the number of HGs (#hg), the depth of each HG (hg\mysubdepth), the three pathways of each HG (hg\mysubsubnet), the number of convolutional layers per blocks (b\mysubdepth) and, finally, the number of input features (f\mysubin), output features (f\mysubout), height (h) and the width (w) of each of the convolutional kernels. The results are shown along with the performance of the original uncompressed network. We observe that by taking advantage of the redundancy at network-level (as opposed to [23, 17] which compress individual layers), the proposed approach is able to effectively compress across arbitrary dimensions for large compression ratios while maintaining similar, or even in some cases higher, accuracy than that of the original uncompressed network.
6.2 Performance of the T-Net
Based on the insights gained from the previous experiment, we selected promising configurations and compressed over multiple dimensions simultaneously. We then compared these configurations with baseline and state-of-the-art methods. The results can be seen in Table 3.
Compression vs. trimming. The obvious comparison is between T-Net and the original baseline network, “compressed” by trimming it, reducing the number of parameters to match the compression ratio achieved by T-Net.
Comparison with efficient architectures. A natural question is whether T-Net performs favourably when compared to architectures designed for efficiency. To answer this, we performed a comparison with MobileNet , for which we adjusted the number of channels of the convolutional layers in order to vary the number of parameters and obtain comparable compression ratios.
Comparison with the state-of-the-art. We also compared with the layer-wise decomposition method of .
We firstly observe that by just reducing the number of channels in the original network, a significant drop in performance can be noticed. Secondly, our method consistently outperforms  across the whole spectrum of compression ratios. This can be seen by comparing the accuracy provided for any compression ratio for  with the accuracy of the closest but higher compression ratio for our method (for example, compare x for  with x for our method). Our method always achieves higher accuracy even though the compression ratio is also higher. In addition, unlike to  which does not seem to work well when the size of the convolutional kernel is compressed from to , our method is able to compress that dimension too while maintaining similar level of accuracy. Finally, our method outperforms MobileNet  by a large margin.
In the same table, we also report the performance of a variant of our method, using an MPS decomposition on the weights rather than a Tucker one. This result shows that our method works effectively with other decomposition methods as well. Nevertheless, we focused mainly on Tucker as it is the most flexible compression method, allowing us to control the rank of each mode of the weight tensor.
Results on face segmentation. Finally, we selected two of our best performing models and retrained them for the task of semantic facial part segmentation. Our method offers significant compression ratios (up to 7x) with virtually no loss in accuracy (see Table 4). These results further confirm that our method is task-independent.
We proposed an end-to-end trainable method to jointly capture the full structure of a fully-convolutional neural network, by parametrizing it with a single, high-order low-rank tensor. The modes of this tensor represent each of the architectural design parameters of the network (e.g. number of convolutional blocks, depth, number of stacks, input features, etc). This parametrization allows for a joint regularization of the whole network. The number of parameters can be drastically reduced by imposing a low-rank structure on the parameter tensor. We show that our approach can achieve superior performance with low compression rates, and attain high compression rates with negligible drop in accuracy, on both the challenging task of human pose estimation and semantic face segmentation.
-  Animashree Anandkumar, Rong Ge, Daniel Hsu, Sham M. Kakade, and Matus Telgarsky. Tensor decompositions for learning latent variable models. JMLR, 15(1):2773–2832, jan 2014.
-  Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d human pose estimation: New benchmark and state of the art analysis. In CVPR, 2014.
-  Marcella Astrid and Seung-Ik Lee. Cp-decomposition with tensor power method for convolutional neural networks compression. CoRR, abs/1701.07148, 2017.
-  Adrian Bulat and Georgios Tzimiropoulos. Human pose estimation via convolutional part heatmap regression. In ECCV, 2016.
-  Adrian Bulat and Georgios Tzimiropoulos. Binarized convolutional landmark localizers for human pose estimation and face alignment with limited resources. In ICCV, 2017.
-  Xingwei Cao and Qibin Zhao. Tensorizing generative adversarial nets. CoRR, abs/1710.10772, 2017.
-  Yunpeng Chen, Xiaojie Jin, Bingyi Kang, Jiashi Feng, and Shuicheng Yan. Sharing residual units through collective tensor factorization to improve deep neural networks. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 635–641. International Joint Conferences on Artificial Intelligence Organization, 7 2018.
-  François Chollet. Xception: Deep learning with depthwise separable convolutions. arXiv preprint, pages 1610–02357, 2017.
-  Nadav Cohen, Or Sharir, and Amnon Shashua. On the expressive power of deep learning: A tensor analysis. CoRR, abs/1509.05009, 2015.
-  Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In NIPS, 2015.
-  Simon S Du and Jason D Lee. On the power of over-parametrization in neural networks with quadratic activation. In ICML, 2018.
-  Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
-  Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
-  Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In ICCV, 2017.
-  Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861, 2017.
-  Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. CoRR, abs/1704.04861, 2017.
-  Yong-Deok Kim, Eunhyeok Park, Sungjoo Yoo, Taelim Choi, Lu Yang, and Dongjun Shin. Compression of deep convolutional neural networks for fast and low power mobile applications. CoRR, 05 2016.
-  Tamara G. Kolda and Brett W. Bader. Tensor decompositions and applications. SIAM REVIEW, 51(3):455–500, 2009.
-  Jean Kossaifi, Aran Khanna, Zachary Lipton, Tommaso Furlanello, and Anima Anandkumar. Tensor contraction layers for parsimonious deep nets. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on, pages 1940–1946. IEEE, 2017.
-  Jean Kossaifi, Zachary C. Lipton, Aran Khanna, Tommaso Furlanello, and Anima Anandkumar. Tensor regression networks. CoRR, abs/1707.08308, 2018.
-  Jean Kossaifi, Yannis Panagakis, Anima Anandkumar, and Maja Pantic. Tensorly: Tensor learning in python. Journal of Machine Learning Research, 20(26):1–6, 2019.
-  Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012.
-  Vadim Lebedev, Yaroslav Ganin, Maksim Rakhuba, Ivan V. Oseledets, and Victor S. Lempitsky. Speeding-up convolutional neural networks using fine-tuned cp-decomposition. CoRR, abs/1412.6553, 2014.
-  Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. In ICLR, 2017.
-  Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In CVPR, 2015.
-  Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In ECCV, 2016.
-  Alexander Novikov, Dmitry Podoprikhin, Anton Osokin, and Dmitry Vetrov. Tensorizing neural networks. In Proceedings of the 28th International Conference on Neural Information Processing Systems, NIPS’15, pages 442–450, 2015.
-  I. V. Oseledets. Tensor-train decomposition. SIAM J. Sci. Comput., 33(5):2295–2317, Sept. 2011.
-  Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017.
-  Tomas Pfister, James Charles, and Andrew Zisserman. Flowing convnets for human pose estimation in videos. In ICCV, 2015.
-  Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In ECCV, 2016.
-  Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015.
-  Christos Sagonas, Georgios Tzimiropoulos, Stefanos Zafeiriou, and Maja Pantic. 300 faces in-the-wild challenge: The first facial landmark localization challenge. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 397–403, 2013.
-  Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv, 2014.
-  Mahdi Soltanolkotabi, Adel Javanmard, and Jason D Lee. Theoretical insights into the optimization landscape of over-parameterized shallow neural networks. IEEE Transactions on Information Theory, 2018.
-  Cheng Tai, Tong Xiao, Xiaogang Wang, and Weinan E. Convolutional neural networks with low-rank regularization. CoRR, abs/1511.06067, 2015.
-  Zhiqiang Tang, Xi Peng, Shijie Geng, Lingfei Wu, Shaoting Zhang, and Dimitris Metaxas. Quantized densely connected u-nets for efficient landmark localization. In ECCV, 2018.
-  Tijmen Tieleman and Geoffrey Hinton. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2), 2012.
-  Jonathan J Tompson, Arjun Jain, Yann LeCun, and Christoph Bregler. Joint training of a convolutional network and a graphical model for human pose estimation. In NIPS, 2014.
-  Alexander Toshev and Christian Szegedy. Deeppose: Human pose estimation via deep neural networks. In CVPR, 2014.
-  Frederick Tung and Greg Mori. Clip-q: Deep network compression learning by in-parallel pruning-quantization. In CVPR, 2018.
-  Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In CVPR, 2016.
-  Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In CVPR, 2016.
-  Yongxin Yang and Timothy M. Hospedales. Deep multi-task representation learning: A tensor factorisation approach. CoRR, abs/1605.06391, 2016.
-  Rose Yu, Stephan Zheng, Anima Anandkumar, and Yisong Yue. Long-term forecasting using tensor-train rnns. CoRR, abs/1711.00073, 2017.
-  Aojun Zhou, Anbang Yao, Kuan Wang, and Yurong Chen. Explicit loss-error-aware quantization for low-bit deep neural networks. In CVPR, 2018.