Computation-Performance Optimization of Convolutional Neural Networks with Redundant Kernel Removal

Computation-Performance Optimization of Convolutional Neural Networks with Redundant Kernel Removal

Chih-Ting Liu, Yi-Heng Wu, Yu-Sheng Lin, and Shao-Yi Chien Media IC and System Lab
Graduate Institute of Electronic Engineering and Department of Electrical Engineering
National Taiwan University
No. 1, Sec. 4, Roosevelt Road, Taipei, 10617 Taiwan
{jackieliu, jasonwu, johnjohnlys, sychien}

Deep Convolutional Neural Networks (CNNs) are widely employed in modern computer vision algorithms, where the input image is convolved iteratively by many kernels to extract the knowledge behind it. However, with the depth of convolutional layers getting deeper and deeper in recent years, the enormous computational complexity makes it difficult to be deployed on embedded systems with limited hardware resources. In this paper, we propose two computation-performance optimization methods to reduce the redundant convolution kernels of a CNN with performance and architecture constraints, and apply it to a network for super resolution (SR). Using PSNR drop compared to the original network as the performance criterion, our method can get the optimal PSNR under a certain computation budget constraint. On the other hand, our method is also capable of minimizing the computation required under a given PSNR drop.

I Introduction

In recent years, deep neural network has been widely employed in the state-of-the-art works in many fields like computer vision [1], natural language processing [2], and speech recognition [3]. Convolutional Neural Networks (CNNs) are getting a great success especially in many visual tasks, including image classification [4], object detection [5], image style transfer [6], super resolution [7], etc. Though these CNNs are powerful, they often consume substantial storage and computational resources. Commonly, the training step of CNNs can be carried out offline on high performance CPU/GPU clusters; nevertheless, for the inference stage, we often prefer local computation on embedded systems rather than cloud-based solutions owing to the privacy, latency, transmission bandwidth, and power consumption constraints [8]. Therefore, reducing the parameters in CNNs in order to avoid huge storage and computation complexity has become critical.

Several works have been proposed to reduce the redundancy of neural networks. Network pruning based on the weight magnitude can remove insignificant connections between neurons [9, 10]. [11] shows that the redundancies in CNN convolutional layers can be exploited by approximating the filter bank as combinations of a rank-1 filter bias. Several works optimize neural networks by quantizing the weights and intermediate results of CNNs [12, 13, 14]. Vector quantization is also been used in compressing the weighting parameters as well as reducing the computational complexity [15, 16]. However, when deploying these redundancy removal methods on existing embedded devices, the performance improvement highly depends on the processor architecture and the effort to optimize the implementation accordingly. A kernel pruning approach [8] removes kernels with high sparsity to reduce the computation. One advantage of this approach is that it is easy to be deployed on all kinds of computing architectures.

In this paper, two computation-performance optimization (CPO) methods are proposed for constrained computing environment, such as embedded systems. Based on [8], redundant kernels are removed according to the computation resource limitation and performance (quality or accuracy) requirements. The experiment model is a network for super resolution (SR) in [7], and the performance benchmark is the PSNR drop compared to the original SR network. In the first method, layer-wise polishment (LWP), under a specified computational budget, the minimal PSNR drop is achieved, where we sort the kernels by its sparsity and modify the removal percentage of every layer. Second, under a given PSNR drop requirement, the minimized computation is achieved by gradient optimization (GO), where a regression model is trained for optimizing the kernel removal percentage. Specifically, our contributions are:

  1. Applying the kernel redundancy removal method on a super resolution (SR) network,

  2. Adjusting the reducing factor of each layer by kernel sparsity, optimizing the PSNR drop under a computation budget and

  3. Generating an appropriate reducing factor to optimize the network with our trained regression model under given PSNR drop.

Ii Proposed Method

The opreations of a CNN layer involve convolving a 3-D tensor (input, ) with 4-D tensors (kernels, ) to extract different features and then generating a 3-D tensor (output, ), where are channel, height and width of input tensor, are channel, height and width of one convolution kernel, and is the number of convolution kernels. are slightly different from owing to the boundary of convolution. The output tensor is also the input tensor of the next layer. In this paper, we focus on the 4-D convolutional kernels and propose two methods to prune the redundant kernels layer by layer, achieving computation-performance optimization (CPO).

Ii-a Definition of Redundancy

The criterion of redundancy is defined layer-by-layer according to the weights in a layer. For a specified layer , We use to represent the mean value of all absoluted kernel weights at that layer:


where the are the indices of the 4-D tensor. Then the sparsity of the -th kernel at layer can be written as:


In other words, if a kernel has several coefficients which are less than the mean value at that layer, is close to , which means this kernel is more redundant than others. Based on this representation, our two proposed CPO methods, Layer-Wise Polishment (LWP) and Gradient Optimization (GO), are described as follows.

Fig. 1: Illustration of pruning redundant kernels at -th layer with sorted list of their sparsity. If = and = for example, we will remove = kernels with the highest three sparsity values (0.95, 0.90, 0.85), and the -th output will left = channels.

Ii-B Layer-Wise Polishment (LWP)

Given a computation budget, LWP can help find the optimized reducing factor for removing redundant kernels.

After calculating the sparsity of all convolution kernels with (2), we sort sparsity values of the same layer to form the sparsity list in descending order. We start to remove the kernels from those with maximum sparsity values, and the number of kernels to be removed of each layer is based on a reducing factor . The element means the proportion of kernels to be removed at the -th layer, and is the total number of convolution layers. As illustrated in Figure 1, if =10, =0.3 and given the sorted kernels, we will prune = redundant kernels at the -th layer, and use the remaining kernels to convolve the input feature maps. After pruning the redundant kernels, we will fine-tune the CNN models to reinforce all kernels left; hence, we can still retain a great performance.

Different from Chen’s work [8], LWP can determine the reducing factor under an expected computation budget. This procedure is split into two steps. To begin with, we uniformly increase the elements in and use it to prune the kernels, for example from to , then count the remaining parameters. After fine-tuning every models with different , we choose one = with the greatest performance under our computation budget, and then go forward to Step two, which is a key process that can further improve the performance.

To calculate the parameters remained of all the kernel weights, we can use the following formula.


The vector means the proportion of kernels remained at every layers. is a diagonal matrix with , which is the product of the four dimensions in the 4-D kernel tensor at the -th layer. is a vector concatenate and the slice of elements in .

Back to Step two, we split the CNN model into three segments, front, middle and end segments, then separately adjust the reducing factor of each part: , and . We explore the idea that assigning the same reducing factors for all layers is not the best way to prune the model; therefore, we can find out the characteristic of a model through experiments and know which segment is more redundant than others. In this paper, we reveal that we can increse the reducing factor in the front segment, and decrease those in both middle and end segments, while maintaining the computation budget almost the same as before, as follows.


where = , = and = . is the changing vector, which is decided empirically, and is the number of layers of the end segment.

Again, by fine-tuning the model with our adjusted reducing factor, the performance will be further improved compared to that with uniform . The objective of LWP is trying to keep an expected number of parameters, or we call it computation budget, then regulate the reducing factors of different layers to achieve the best performance.

Fig. 2: Steps of optimizing the reducing factor between every iterations in Gradient Optimization (GO). Step 1 is feeding the to the model. Step 2 is calculating the difference and gradients, then backpropagate. Step 3 is updating the input vector .

Ii-C Gradient Optimization (GO)

From another aspect, given a target performance of the network, GO can help derive the optimized reducing factor .

This method is based on a learnable regression model, where the input vector is , and the output is the performance criterion, whis is PSNR drop in our experiment. We generate some training data with randomly assigned and collect the output performance with fine-tuned CNN model. With the training data, we construct a simple regression model, , with neural netwok as shown in Figure 2. After training, we fix the weights inside the model, and use this trained model to start the iteration loops by giving an initial point .

The iterations operate with a goal to minimize the difference between and , where is an expected performance we can claim. After calculating the gradients, they will be backpropagated to the input , and will be updated by gradient descent as the following equation for every iteration:


where is the learning rate, is the iteration index and denotes the difference between and . These operations are also illustrated in Figure 2. Finally, after some iterations, we can minimize the distance between and within an acceptable margin, and the optimal reducing factor for the CNN models is obtained.

Ii-D Architecture specified tuning

Those two methods can both prune the kernels and reduce the computation time. We can either choose one of them to optimize a convolutional neural network according to different application scenarios. However, owing to the architecture (instruction set, cache, etc.) of each processor, we further take the architecture characteristics into consideration. For example, in Intel processors, the computation time of CNN inference step is especially fast when the kernels of CNN are certain numbers such as 32. The final flows are shown in Procedures LWP and GO as follows.

1:Choose a trained CNN models and zero the .
3:     Uniformly increase the elements of under
4:     architecture limitation.
5:     Prune and Fine-tune the model.
6:     Calculate the parameters remained.
7:     Testing the performance compared to the original CNN a . .models.
8:until achieve the computation budget with great performance.
9:Save the we’ve choosed.
10:Split into three parts, and modify them with considering architecture characteristics.
11:At the end, fine-tune the model with new .
Procedure Modified Flow for LWP
1:Choose a trained CNN models.
3:     Randomly generate .
4:     Prune and Fine-tune the model, testing theperformance .
5:     Collect training pair (,).
6:until Enough training data.
7:Train a regression model whose input vector is , output number is .
8:Fix weights inside .
9:Claim an expected , and initiate the start point .
11:     Minimizing the difference by updating .
12:until  less than a margin at the iteration.
13:Slightly modified with considering architecture characteristics.
14:At the end, fine-tune the model with new .
Procedure Modified Flow for GO

Iii Experiment Result

We take the residual CNN model in Very Deep Super Resolution [7] (VDSR) for our experiment. This model is constructed only with convolutional layers; therefore, the model size and the computation time will not be influenced by the fully-connected layers. Figure 3 is the VDSR model [7]. The input image is an interpolated low-resolution picture with one channel (Y channel), and the output is the derived high-resolution one. The structure is a 20-layer residual CNN model. Each of the first 19 layers has 64 kernels, and one kernel at the last layer to generate the residual part, which is added by the low resolution image to generate the high-resolution one.

Iii-a Experiment of Layer-Wise Polishment (LWP)

According to the procedure of LWP, we choose seven different for experiments as shown in Table I. After pruning, we fine-tune the VDSR model with only three epochs, which is much lower than the training step of the original VDSR network, which is about 80 epochs. Our testing set are Set5 and Set14 of SR, and the performance criterion we choose is the PSNR drop compared to those with the original VDSR network. Parameters remained are calculated by (3) and the experimental results are shown in Table I. We then choose = to be modified later because almost parameters are removed while the performance is still acceptable, where the PSNR drop is only dB.

Fig. 3: Network structure of VDSR. It cascades a pair of layers (convolutional and nonlinear) repeatedly for 20 times. Input is the Y channel of an Interpolated Low-Resolution (ILR) image, and then transforms into the Y channel of the High-Resolution (HR) one. The network predicts a residual image, and the addition of the input and the residual gives the desired output.
per layer
remained (%)
PSNR drop
Set5 /Set14
64 100%
(PSNR: 37.50 / 33.08)
56 76.6% 0.16 / 0.20
52 66.1% 0.37 / 0.29
48 56.3% 0.29 / 0.27
44 47.4% 0.42 / 0.40
40 39.2% 0.49 / 0.47
36 31.7% 0.50 / 0.48
32 25.1% 0.65 / 0.56
TABLE I: LWP: Fine-tune Result (3 epochs) of Different Reducing Factor.                                                                                                                                    (The reduce factor number is chosen by architecture characteristic.)

We then split the 20-layers CNN model into three segments with 6, 7 and 7 layers respectively, and make an experiment for increasing and decreasing the reducing factor for each segment and try to find out the characteristic of the deep CNN model. As Table II suggests, under almost the same parameters remained, we can improve the performance: for Set5, the PSNR drop is decreased from to when pruning more kernels at the front segment () and retaining more at the middle and end segments.

per part
remained (%)
PSNR drop
Set5 / Set14
48, 48, 48 56.3% 0.29 / 0.27
36, 52, 52 55.4% 0.27 / 0.26
36, 56, 48 56.4% 0.24 / 0.26
56, 52, 36 58.6% 0.34 / 0.33
52, 52, 40 57.7% 0.38 / 0.33
48, 36, 60 55.9% 0.37 / 0.38
52, 36, 56 55.5% 0.41 / 0.40
TABLE II: LWP: Experiment of Different Reducing Factor at Different Fragments. (F,M,E) Means Front, Middle and End Segments.

Iii-B Experiment of Gradient Optimization (GO)

In the procedure of GO, we first need to collect enough training data pairs (, ), where we use here is the PSNR drop when testing on Set14 with randomly assigned . Then we construct a linear regression model whose input vector is and output is . The training criterion is mean-square-error (MSE) loss, and the optimizer is Adam Optimizer [17]. After some iterations, we can get the well-trained .

The trained regression model is then employed to optimize the input vector . With this , GO can automatically decide the appropriate reducing factor to prune our CNN model under the expected performance. We use as an example, as shown in Table III. The modified will be used to prune our model, and Set5 and Set14 are used as the testing sets. We first give an initial point and use the absolute difference as the loss . Between every iteration, we calculate the gradient of loss and backpropagate it to optimize input with Gradient Descent (learning rate ). After minimizing within a margin, we stop the iteration and get the optimized , and then start to adjust it with considering the architecture characteristics. The result of the modified reducing factor and the performance after fine-tuning the CNN model are also shown on Table III. We can clearly see that the final PSNR drop results (0.24dB/0.25dB) are successfully close to the goal we set.

Goal: 0.29 PSNR drop
Optimized by
Gradient Descent
Reducing Factor
[0.30, 0.07, 0.12, 0.07, 0.11,, 0.27, 0.08, 0.08, 0.18, 0]
Adjusted by
Reducing Factor
[0.32, 0.06, 0.12, 0.06, 0.12,, 0.25, 0.12, 0.12, 0.18, 0]
# Kernel Remained (20 layers)
(44, 60, 56, 60, 56,, 48, 56, 56, 52, 1)
(PSNR drop)
Set5() / Set14()
0.29 / 0.26
TABLE III: GO: Experiment Result of Goal = 0.29 PSNR drop.

Iv Conclusion

In this paper, we propose computation-performance optimization (CPO) methods with removing the redundant kernels of a deep convolutional neural network (CNN) to make it more feasible on embedded systems. Two CPO methods, LWP and GO, are proposed. The first achieves about size reduction but only causes about 0.2dB in performance drop. The other can automatically derive a removing policy under our performance goal. Compared to the previous works about kernel pruning, our work is more flexible than that with a fixed threshold and can be applied to more complex network models. In the future, we expect to profile more networks and apply it to even more architectures, such as ASIC or FPGA platforms.


This research was supported in part by the Ministry of Science and Technology of Taiwan (MOST 107-2633-E-002-001), National Taiwan University, Intel Corporation, and Delta Electronics.


  • [1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Proc. Advances in neural information processing systems, 2012, pp. 1097–1105.
  • [2] R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa, “Natural language processing (almost) from scratch,” Journal of Machine Learning Research, vol. 12, pp. 2493–2537, 2011.
  • [3] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath et al., “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 82–97, 2012.
  • [4] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  • [5] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proc. IEEE conference on computer vision and pattern recognition, 2014, pp. 580–587.
  • [6] L. A. Gatys, A. S. Ecker, and M. Bethge, “Image style transfer using convolutional neural networks,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2414–2423.
  • [7] J. Kim, J. Kwon Lee, and K. Mu Lee, “Accurate image super-resolution using very deep convolutional networks,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 1646–1654.
  • [8] C.-F. Chen, G. G. Lee, V. Sritapan, and C.-Y. Lin, “Deep convolutional neural network on iOS mobile devices,” in Proc. 2016 IEEE International Workshop on Signal Processing Systems (SiPS).   IEEE, 2016, pp. 130–135.
  • [9] S. Han, J. Pool, J. Tran, and W. Dally, “Learning both weights and connections for efficient neural network,” in Proc. Advances in Neural Information Processing Systems, 2015, pp. 1135–1143.
  • [10] S. Han, H. Mao, and W. J. Dally, “Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding,” arXiv preprint arXiv:1510.00149, 2015.
  • [11] M. Jaderberg, A. Vedaldi, and A. Zisserman, “Speeding up convolutional neural networks with low rank expansions,” arXiv preprint arXiv:1405.3866, 2014.
  • [12] V. Vanhoucke, A. Senior, and M. Z. Mao, “Improving the speed of neural networks on CPUs,” in Proc. Deep Learning and Unsupervised Feature Learning NIPS Workshop, vol. 1.   Citeseer, 2011, p. 4.
  • [13] P.-H. Hung, C.-H. Lee, S.-W. Yang, V. S. Somayazulu, Y.-K. Chen, , and S.-Y. Chien, “Bridge deep learning to the physical world: An efficient method to quantize network,” in Proc. 2015 IEEE International Workshop on Signal Processing Systems (SiPS).   IEEE, 2015.
  • [14] S. Anwar, K. Hwang, and W. Sung, “Fixed point optimization of deep convolutional neural networks for object recognition,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).   IEEE, 2015, p. 1131–1135.
  • [15] Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing deep convolutional networks using vector quantization,” arXiv preprint arXiv:1412.6115, 2014.
  • [16] J. Wu, C. Leng, Y. Wang, Q. Hu, and J. Cheng, “Quantized convolutional neural networks for mobile devices,” in Proc. IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4820–4828.
  • [17] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR, vol. abs/1412.6980, 2014. [Online]. Available:
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description