ComputationPerformance Optimization of Convolutional Neural Networks with Redundant Kernel Removal
Abstract
Deep Convolutional Neural Networks (CNNs) are widely employed in modern computer vision algorithms, where the input image is convolved iteratively by many kernels to extract the knowledge behind it. However, with the depth of convolutional layers getting deeper and deeper in recent years, the enormous computational complexity makes it difficult to be deployed on embedded systems with limited hardware resources. In this paper, we propose two computationperformance optimization methods to reduce the redundant convolution kernels of a CNN with performance and architecture constraints, and apply it to a network for super resolution (SR). Using PSNR drop compared to the original network as the performance criterion, our method can get the optimal PSNR under a certain computation budget constraint. On the other hand, our method is also capable of minimizing the computation required under a given PSNR drop.
I Introduction
In recent years, deep neural network has been widely employed in the stateoftheart works in many fields like computer vision [1], natural language processing [2], and speech recognition [3]. Convolutional Neural Networks (CNNs) are getting a great success especially in many visual tasks, including image classification [4], object detection [5], image style transfer [6], super resolution [7], etc. Though these CNNs are powerful, they often consume substantial storage and computational resources. Commonly, the training step of CNNs can be carried out offline on high performance CPU/GPU clusters; nevertheless, for the inference stage, we often prefer local computation on embedded systems rather than cloudbased solutions owing to the privacy, latency, transmission bandwidth, and power consumption constraints [8]. Therefore, reducing the parameters in CNNs in order to avoid huge storage and computation complexity has become critical.
Several works have been proposed to reduce the redundancy of neural networks. Network pruning based on the weight magnitude can remove insignificant connections between neurons [9, 10]. [11] shows that the redundancies in CNN convolutional layers can be exploited by approximating the filter bank as combinations of a rank1 filter bias. Several works optimize neural networks by quantizing the weights and intermediate results of CNNs [12, 13, 14]. Vector quantization is also been used in compressing the weighting parameters as well as reducing the computational complexity [15, 16]. However, when deploying these redundancy removal methods on existing embedded devices, the performance improvement highly depends on the processor architecture and the effort to optimize the implementation accordingly. A kernel pruning approach [8] removes kernels with high sparsity to reduce the computation. One advantage of this approach is that it is easy to be deployed on all kinds of computing architectures.
In this paper, two computationperformance optimization (CPO) methods are proposed for constrained computing environment, such as embedded systems. Based on [8], redundant kernels are removed according to the computation resource limitation and performance (quality or accuracy) requirements. The experiment model is a network for super resolution (SR) in [7], and the performance benchmark is the PSNR drop compared to the original SR network. In the first method, layerwise polishment (LWP), under a specified computational budget, the minimal PSNR drop is achieved, where we sort the kernels by its sparsity and modify the removal percentage of every layer. Second, under a given PSNR drop requirement, the minimized computation is achieved by gradient optimization (GO), where a regression model is trained for optimizing the kernel removal percentage. Specifically, our contributions are:

Applying the kernel redundancy removal method on a super resolution (SR) network,

Adjusting the reducing factor of each layer by kernel sparsity, optimizing the PSNR drop under a computation budget and

Generating an appropriate reducing factor to optimize the network with our trained regression model under given PSNR drop.
Ii Proposed Method
The opreations of a CNN layer involve convolving a 3D tensor (input, ) with 4D tensors (kernels, ) to extract different features and then generating a 3D tensor (output, ), where are channel, height and width of input tensor, are channel, height and width of one convolution kernel, and is the number of convolution kernels. are slightly different from owing to the boundary of convolution. The output tensor is also the input tensor of the next layer. In this paper, we focus on the 4D convolutional kernels and propose two methods to prune the redundant kernels layer by layer, achieving computationperformance optimization (CPO).
Iia Definition of Redundancy
The criterion of redundancy is defined layerbylayer according to the weights in a layer. For a specified layer , We use to represent the mean value of all absoluted kernel weights at that layer:
(1) 
where the are the indices of the 4D tensor. Then the sparsity of the th kernel at layer can be written as:
(2) 
In other words, if a kernel has several coefficients which are less than the mean value at that layer, is close to , which means this kernel is more redundant than others. Based on this representation, our two proposed CPO methods, LayerWise Polishment (LWP) and Gradient Optimization (GO), are described as follows.
IiB LayerWise Polishment (LWP)
Given a computation budget, LWP can help find the optimized reducing factor for removing redundant kernels.
After calculating the sparsity of all convolution kernels with (2), we sort sparsity values of the same layer to form the sparsity list in descending order. We start to remove the kernels from those with maximum sparsity values, and the number of kernels to be removed of each layer is based on a reducing factor . The element means the proportion of kernels to be removed at the th layer, and is the total number of convolution layers. As illustrated in Figure 1, if =10, =0.3 and given the sorted kernels, we will prune = redundant kernels at the th layer, and use the remaining kernels to convolve the input feature maps. After pruning the redundant kernels, we will finetune the CNN models to reinforce all kernels left; hence, we can still retain a great performance.
Different from Chen’s work [8], LWP can determine the reducing factor under an expected computation budget. This procedure is split into two steps. To begin with, we uniformly increase the elements in and use it to prune the kernels, for example from to , then count the remaining parameters. After finetuning every models with different , we choose one = with the greatest performance under our computation budget, and then go forward to Step two, which is a key process that can further improve the performance.
To calculate the parameters remained of all the kernel weights, we can use the following formula.
(3) 
The vector means the proportion of kernels remained at every layers. is a diagonal matrix with , which is the product of the four dimensions in the 4D kernel tensor at the th layer. is a vector concatenate and the slice of elements in .
Back to Step two, we split the CNN model into three segments, front, middle and end segments, then separately adjust the reducing factor of each part: , and . We explore the idea that assigning the same reducing factors for all layers is not the best way to prune the model; therefore, we can find out the characteristic of a model through experiments and know which segment is more redundant than others. In this paper, we reveal that we can increse the reducing factor in the front segment, and decrease those in both middle and end segments, while maintaining the computation budget almost the same as before, as follows.
(4)  
where = , = and = . is the changing vector, which is decided empirically, and is the number of layers of the end segment.
Again, by finetuning the model with our adjusted reducing factor, the performance will be further improved compared to that with uniform . The objective of LWP is trying to keep an expected number of parameters, or we call it computation budget, then regulate the reducing factors of different layers to achieve the best performance.
IiC Gradient Optimization (GO)
From another aspect, given a target performance of the network, GO can help derive the optimized reducing factor .
This method is based on a learnable regression model, where the input vector is , and the output is the performance criterion, whis is PSNR drop in our experiment. We generate some training data with randomly assigned and collect the output performance with finetuned CNN model. With the training data, we construct a simple regression model, , with neural netwok as shown in Figure 2. After training, we fix the weights inside the model, and use this trained model to start the iteration loops by giving an initial point .
The iterations operate with a goal to minimize the difference between and , where is an expected performance we can claim. After calculating the gradients, they will be backpropagated to the input , and will be updated by gradient descent as the following equation for every iteration:
(5) 
where is the learning rate, is the iteration index and denotes the difference between and . These operations are also illustrated in Figure 2. Finally, after some iterations, we can minimize the distance between and within an acceptable margin, and the optimal reducing factor for the CNN models is obtained.
IiD Architecture specified tuning
Those two methods can both prune the kernels and reduce the computation time. We can either choose one of them to optimize a convolutional neural network according to different application scenarios. However, owing to the architecture (instruction set, cache, etc.) of each processor, we further take the architecture characteristics into consideration. For example, in Intel processors, the computation time of CNN inference step is especially fast when the kernels of CNN are certain numbers such as 32. The final flows are shown in Procedures LWP and GO as follows.
Iii Experiment Result
We take the residual CNN model in Very Deep Super Resolution [7] (VDSR) for our experiment. This model is constructed only with convolutional layers; therefore, the model size and the computation time will not be influenced by the fullyconnected layers. Figure 3 is the VDSR model [7]. The input image is an interpolated lowresolution picture with one channel (Y channel), and the output is the derived highresolution one. The structure is a 20layer residual CNN model. Each of the first 19 layers has 64 kernels, and one kernel at the last layer to generate the residual part, which is added by the low resolution image to generate the highresolution one.
Iiia Experiment of LayerWise Polishment (LWP)
According to the procedure of LWP, we choose seven different for experiments as shown in Table I. After pruning, we finetune the VDSR model with only three epochs, which is much lower than the training step of the original VDSR network, which is about 80 epochs. Our testing set are Set5 and Set14 of SR, and the performance criterion we choose is the PSNR drop compared to those with the original VDSR network. Parameters remained are calculated by (3) and the experimental results are shown in Table I. We then choose = to be modified later because almost parameters are removed while the performance is still acceptable, where the PSNR drop is only dB.







64  100% 



56  76.6%  0.16 / 0.20  
52  66.1%  0.37 / 0.29  
48  56.3%  0.29 / 0.27  
44  47.4%  0.42 / 0.40  
40  39.2%  0.49 / 0.47  
36  31.7%  0.50 / 0.48  
32  25.1%  0.65 / 0.56 
We then split the 20layers CNN model into three segments with 6, 7 and 7 layers respectively, and make an experiment for increasing and decreasing the reducing factor for each segment and try to find out the characteristic of the deep CNN model. As Table II suggests, under almost the same parameters remained, we can improve the performance: for Set5, the PSNR drop is decreased from to when pruning more kernels at the front segment () and retaining more at the middle and end segments.






48, 48, 48  56.3%  0.29 / 0.27  
36, 52, 52  55.4%  0.27 / 0.26  
36, 56, 48  56.4%  0.24 / 0.26  
56, 52, 36  58.6%  0.34 / 0.33  
52, 52, 40  57.7%  0.38 / 0.33  
48, 36, 60  55.9%  0.37 / 0.38  
52, 36, 56  55.5%  0.41 / 0.40 
IiiB Experiment of Gradient Optimization (GO)
In the procedure of GO, we first need to collect enough training data pairs (, ), where we use here is the PSNR drop when testing on Set14 with randomly assigned . Then we construct a linear regression model whose input vector is and output is . The training criterion is meansquareerror (MSE) loss, and the optimizer is Adam Optimizer [17]. After some iterations, we can get the welltrained .
The trained regression model is then employed to optimize the input vector . With this , GO can automatically decide the appropriate reducing factor to prune our CNN model under the expected performance. We use as an example, as shown in Table III. The modified will be used to prune our model, and Set5 and Set14 are used as the testing sets. We first give an initial point and use the absolute difference as the loss . Between every iteration, we calculate the gradient of loss and backpropagate it to optimize input with Gradient Descent (learning rate ). After minimizing within a margin, we stop the iteration and get the optimized , and then start to adjust it with considering the architecture characteristics. The result of the modified reducing factor and the performance after finetuning the CNN model are also shown on Table III. We can clearly see that the final PSNR drop results (0.24dB/0.25dB) are successfully close to the goal we set.




Reducing Factor  
[0.30, 0.07, 0.12, 0.07, 0.11,, 0.27, 0.08, 0.08, 0.18, 0]  

Reducing Factor  
[0.32, 0.06, 0.12, 0.06, 0.12,, 0.25, 0.12, 0.12, 0.18, 0]  
# Kernel Remained (20 layers)  
(44, 60, 56, 60, 56,, 48, 56, 56, 52, 1)  


Iv Conclusion
In this paper, we propose computationperformance optimization (CPO) methods with removing the redundant kernels of a deep convolutional neural network (CNN) to make it more feasible on embedded systems. Two CPO methods, LWP and GO, are proposed. The first achieves about size reduction but only causes about 0.2dB in performance drop. The other can automatically derive a removing policy under our performance goal. Compared to the previous works about kernel pruning, our work is more flexible than that with a fixed threshold and can be applied to more complex network models. In the future, we expect to profile more networks and apply it to even more architectures, such as ASIC or FPGA platforms.
Acknowledgment
This research was supported in part by the Ministry of Science and Technology of Taiwan (MOST 1072633E002001), National Taiwan University, Intel Corporation, and Delta Electronics.
References
 [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Proc. Advances in neural information processing systems, 2012, pp. 1097–1105.
 [2] R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, “Natural language processing (almost) from scratch,” Journal of Machine Learning Research, vol. 12, pp. 2493–2537, 2011.
 [3] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath et al., “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82–97, 2012.
 [4] K. Simonyan and A. Zisserman, “Very deep convolutional networks for largescale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
 [5] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proc. IEEE conference on computer vision and pattern recognition, 2014, pp. 580–587.
 [6] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2414–2423.
 [7] J. Kim, J. Kwon Lee, and K. Mu Lee, “Accurate image superresolution using very deep convolutional networks,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1646–1654.
 [8] C.F. Chen, G. G. Lee, V. Sritapan, and C.Y. Lin, “Deep convolutional neural network on iOS mobile devices,” in Proc. 2016 IEEE International Workshop on Signal Processing Systems (SiPS). IEEE, 2016, pp. 130–135.
 [9] S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” in Proc. Advances in Neural Information Processing Systems, 2015, pp. 1135–1143.
 [10] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149, 2015.
 [11] M. Jaderberg, A. Vedaldi, and A. Zisserman, “Speeding up convolutional neural networks with low rank expansions,” arXiv preprint arXiv:1405.3866, 2014.
 [12] V. Vanhoucke, A. Senior, and M. Z. Mao, “Improving the speed of neural networks on CPUs,” in Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop, vol. 1. Citeseer, 2011, p. 4.
 [13] P.H. Hung, C.H. Lee, S.W. Yang, V. S. Somayazulu, Y.K. Chen, , and S.Y. Chien, “Bridge deep learning to the physical world: An efficient method to quantize network,” in Proc. 2015 IEEE International Workshop on Signal Processing Systems (SiPS). IEEE, 2015.
 [14] S. Anwar, K. Hwang, and W. Sung, “Fixed point optimization of deep convolutional neural networks for object recognition,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2015, p. 1131â1135.
 [15] Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing deep convolutional networks using vector quantization,” arXiv preprint arXiv:1412.6115, 2014.
 [16] J. Wu, C. Leng, Y. Wang, Q. Hu, and J. Cheng, “Quantized convolutional neural networks for mobile devices,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4820–4828.
 [17] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR, vol. abs/1412.6980, 2014. [Online]. Available: http://arxiv.org/abs/1412.6980