Deep Learning Markov Random Field
for Semantic Segmentation
Abstract
Semantic segmentation tasks can be well modeled by Markov Random Field (MRF). This paper addresses semantic segmentation by incorporating highorder relations and mixture of label contexts into MRF. Unlike previous works that optimized MRFs using iterative algorithm, we solve MRF by proposing a Convolutional Neural Network (CNN), namely Deep Parsing Network (DPN), which enables deterministic endtoend computation in a single forward pass. Specifically, DPN extends a contemporary CNN to model unary terms and additional layers are devised to approximate the mean field (MF) algorithm for pairwise terms. It has several appealing properties. First, different from the recent works that required many iterations of MF during backpropagation, DPN is able to achieve high performance by approximating one iteration of MF. Second, DPN represents various types of pairwise terms, making many existing models as its special cases. Furthermore, pairwise terms in DPN provide a unified framework to encode rich contextual information in highdimensional data, such as images and videos. Third, DPN makes MF easier to be parallelized and speeded up, thus enabling efficient inference. DPN is thoroughly evaluated on standard semantic image/video segmentation benchmarks, where a single DPN model yields stateoftheart segmentation accuracies on PASCAL VOC 2012, Cityscapes dataset and CamVid dataset.
1 Introduction
Semantic segmentation is a fundamental and longstanding problem in computer vision. It is defined as a multilabel classification problem, aiming to assign each pixel with a category label. There are two widely adopted research realms, including semantic image segmentation [1, 2, 3, 4, 5, 6, 7] and semantic video segmentation [8, 9, 10, 11, 12, 13, 14]. The former employs a static image as input, while the latter employs a video sequence. The obtained perpixel segmentation results are extremely useful for several applications like smart editing [15], scene understanding [4] and automated driving [16].
Since pixels in natural images or videos generally exhibit strong correlation, jointly modeling label distribution in all locations is desirable. To capture these contextual information, Markov random field (MRF) and conditional random field (CRF) [17] are commonly used as classic frameworks for semantic segmentation. They model the joint distribution of labels by defining both unary term and pairwise terms. Unary term reflects the perpixel confidence of assigning labels while pairwise terms capture the interpixel constraints.
Most previous studies focus on designing pairwise terms that possess strong expressive power. For example, Krähenbühl et al. [18] attained accurate segmentation boundary by inferring on a fullyconnected graph. Vineet et al. [19] extended [18] by defining both highorder and longrange terms between pixels. Global or local semantic contexts between labels were also investigated by [20]. However, their performance are limited by the relatively shallow models (e.g. SVM or Adaboost) used as unary term. As deep learning gradually takes over in many image recognition fields [21], researchers have also explored the possibility of designing effective deep architecture for semantic segmentation. For instance, Long et al. [6] transformed fullyconnected layers of CNN into convolutional layers, making accurate perpixel classification possible using contemporary CNN architectures that were pretrained on ImageNet [22]. Chen et al. [23] improved [6] by feeding the outputs of CNN into a MRF with simple pairwise potentials, but it treated CNN and MRF as separate components. A recent advance was made in joint training CNN and MRF by passing the error of MRF inference backward into CNN [24]. Nonetheless, an iterative inference of MRF such as the mean field algorithm (MF) [25] is required for each training image during the backpropagation (BP). Zheng et al. [26] further showed that the procedure of MF inference can be represented as a Recurrent Neural Network (RNN), but their computational costs are similar to that of [24].
We observed that a direct combination of CNN and MRF as above is inefficient, posing challenges on both optimization difficulty and inference speed. Since CNN typically has millions of parameters and MRF typically has thousands of latent variables, they are cumbersome to jointly optimize and infer. Even worse, incorporating complex pairwise terms into a MRF becomes impractical, limiting the performance of the entire system. In this study, we propose a novel Deep Parsing Network (DPN), which is an endtoend system enabling jointly training of CNN and complex pairwise terms. DPN has several appealing properties:
(1) DPN solves MRF with a single feedforward pass, reducing computational cost and meanwhile maintaining high performance. Specifically, DPN models unary terms by extending the VGG16 network (VGG) [21] pretrained on ImageNet, while additional layers are carefully designed to model complex pairwise terms. The learning of these terms is transformed into deterministic endtoend computation by BP, instead of embedding MF into BP as [24, 27] did. Although MF can be represented by a RNN [26], it needs to recurrently compute the forward pass so as to achieve good performance and thus the process is timeconsuming, e.g. each forward pass contains hundred thousand of weights. DPN approximates MF by using only one iteration of inference. This is made possible by joint learning strong unary terms and rich pairwise information.
(2) Pairwise terms determine the graphical structure. In previous studies, if the former is changed, so is the latter as well as its inference procedure. But with DPN, modifying the complexity of pairwise terms, e.g. range of pixels and contexts, is as simple as modifying the receptive fields of convolutions, without varying BP. Furthermore, DPN is capable of representing multiple types of pairwise terms, making many previous works [23, 26, 24] as its special cases.
(3) DPN approximates MF with convolutional and pooling operations, which can be speeded up by lowrank approximation [28] and easily parallelized [29] in a Graphical Processing Unit (GPU).
Our contributions are summarized as below. (1) We propose the novel DPN to jointly train VGG for unary terms with rich pairwise information, i.e. mixture of label contexts and highorder relations. In comparison to existing deep models, DPN approximates MF with only one iteration of inference, reducing computational cost but still maintaining high performance. (2) We show that multiple types of MRFs can be represented in DPN, making many previous works such as RNN [26] and DeepLab [23] as its special cases. (3) We conduct extensive experiments to investigate which component of DPN is crucial to achieve high performance. We demonstrate the generalizability of DPN model by showing its stateoftheart performance on several standard semantic image/video segmentation benchmarks, including PASCAL VOC 2012 [30], CityScapes dataset [16] and CamVid dataset [9].
In comparison to our earlier version of this work [7], we propose a generic deep learning framework, Deep Parsing Network (DPN) to model and solve Dimension (D) highorder Markov Random Field (MRF). Our previous study [7] only shows the possibility on 2Dimension image segmentation problem. Specifically, we employ dynamic node linking to construct graph in D space, which results in a model of D highorder MRF. To solve this highdimensional and highorder MRF, we reformulate the mean field (MF) update process into a feedforward pass of Convolutional Neural Network (CNN). D local and global convolutional layers are designed to approximate different terms in a MF solver. Apart from the methodology, the paper was also substantially improved by providing more technical details and more extensive experimental evaluations.
2 Related Work
Existing studies [1, 31, 2, 32, 33, 34, 4, 5, 6, 35, 36] on semantic segmentation focus on either constructing specific graph structure so that contextual information and longterm dependencies can be captured, or designing suitable network architecture to leverage the power of deep learning. In the following, we summarize recent research advances with respect to these two aspects.
Markov Random Field. Markov Random Field (MRF) or Conditional Random Field (CRF) has achieved great successes in semantic image segmentation, which is one of the most challenging problems in computer vision. Researchers improved labeling accuracy by exploring rich information to define the pairwise functions, including longrange dependencies [18, 37], highorder potentials [19, 38], and semantic label contexts [3, 39, 20]. For example, Krähenbühl et al. [18] attained accurate segmentation boundary by inferring on a fullyconnected graph. Vineet et al. [19] extended [18] by defining both highorder and longrange terms between pixels. Global or local semantic contexts between labels were also investigated by [20]. Although they accomplished promising results, they modeled the unary terms as SVM or Adaboost, whose learning capacity becomes a bottleneck. The learning and inference of complex pairwise terms are often expensive.
MRF and CRF have also been utilized in semantic video segmentation by extending their graph structure to spatiotemporal domain. For example, Wang et al. [40] unified foreground object segmentation, tracking and occlusion reasoning into a carefully designed MRF model. Optical flow based longterm trajectories [10] were also exploited to discover moving objects. Liu et al. [12] employed fullyconnected CRF augmented with object potentials for efficient multiclass inference. However, these methods are based on handcrafted features, thus lacking sufficient learning capacity.
Convolutional Neural Network. More recently, Convolutional Neural Network (CNN) has been leveraged as a strong unary classifier. With deep models, existing works [41, 42, 5, 6, 23, 43, 26, 24, 27] demonstrated encouraging segmentation results through using just simple definition of the pairwise function or even neglecting it. For instance, Long et al. [6] transformed fullyconnected layers of CNN into convolutional layers, making accurate perpixel classification possible using the contemporary CNN architectures that were pretrained on ImageNet [22]. Chen et al. [23] improved [6] by feeding the outputs of CNN into a MRF with simple pairwise potentials, but it treated CNN and MRF as separated components. A recent advance was obtained by [24], which jointly trained CNN and MRF by passing the error of MRF inference backward into CNN, but iterative inference of MRF such as the mean field algorithm (MF) [25] is required for each training image during the backpropagation (BP). Zheng et al. [26] further showed that the procedure of MF inference can be represented as a Recurrent Neural Network (RNN), but their computational costs are similar to that of [24].
Little attempts have been made to develop unified deep learning framework for semantic video segmentation. Recent efforts in this direction include SegNet [44], which adopted an encoderdecoder architecture but did not take temporal relationships into consideration. Here we extend DPN to further include temporal voxels into the joint learning and inference process, which results in an endtoend trainable system with rich spatiotemporal information encoded.
3 Our Approach
We develop a unified framework, DPN, for modeling and solving highorder MRF. The architecture of DPN is shown in Fig. 1(a). Our model imposes no restrictions on the dimension of MRF. For example, it can be either Dimension (2D) for images segmentation, Dimension (3D) for video segmentation, or Dimension (ND) for sensor data. DPN learns MRF by extending VGG to model unary terms and additional layers are carefully designed to model pairwise terms. In the following, we describe the formulation of DPN in terms of D MRF. Note that it can be easily resorted to D cases or extended to D cases by removing or adding relationships between nodes.
Markov Random Field. MRF [45] is an undirected graph where each node represents a voxel in a video, , and each edge represents relation between voxels, as shown in Fig. 1(b). Each node is associated with a binary latent variable, , indicating whether a voxel has label . Here, indicates a voxel’s spatial index with respect to an image, and is its temporal index with respect to a sequence. We have , representing a set of labels. The energy function of MRF is written as
(1) 
where , , and denote a set of latent variables, nodes, and edges, respectively. is the unary term, measuring the cost of assigning label to voxel . For instance, if voxel belongs to the first category other than the second one at time , we should have . Moreover, is the pairwise term that measures the penalty of assigning labels to a pair of voxel respectively.
Dynamic Node Linking. Traditional approaches [13, 46, 47] usually define the edges on rectangular grid in D space. However, when large motion exists, the actual temporal trajectory for certain pixel will not reside inside a rigid cube, which means the rectangular grid assumption does not hold. To better preserve the contextual information in a spatiotemporal space, we employ dynamic node linking to construct edges in DPN. Specifically, we keep the 2D structure in the spatial domain, as illustrated in Fig. 1(c). In the temporal domain, the neighboring voxels and are defined as those lie on the same temporal trajectories . This trajectory can be estimated by standard optical flow techniques [48]. The formulation of edges in the temporal domain is
(2) 
In this setting, adjacent nodes in the temporal space would be more likely to belong to the same category, making the label contexts more easily to be captured.
Unary and Pairwise Terms. Intuitively, the unary terms represent pervoxel classifications, while the pairwise terms represent a set of smoothness constraints. The unary term in Eqn. (1) is typically defined as
(3) 
where indicates the probability of the presence of label at voxel , modeling by VGG. To simplify discussions, we abbreviate it as . The smoothness term can be formulated as
(4) 
where the first term learns the penalty of global cooccurrence between any pair of labels. For example, the output value of is large if and should not coexist. In Eqn.(4), the second term calculates the distances between voxels. We have
(5) 
where indicates a feature vector such as RGB values extracted from the input video for voxel , denote coordinates of voxels’ positions, and are the constant weights. Eqn. (4) implies that if two voxels are close and look similar, they are encouraged to have labels that are compatible. This formulation has been adopted by most of the recent deep models [23, 26, 24] for semantic image segmentation.
However, Eqn. (4) has two main drawbacks. First, its first term captures the cooccurrence frequency of two labels in the training data, but neglects the spatial context between objects. For example, ‘person’ may appear beside ‘table’, but not at its bottom. This spatial context is a mixture of patterns, as different object configurations may appear in different images, such as ‘person’ standing beside ‘table’ and ‘person’ sitting behind ‘table’. Second, it defines only the pairwise relations between pixels, missing their highorder interactions.
To resolve these issues, we define the smoothness term by leveraging rich information between voxels, which is one of the advantages of DPN over existing deep models. We have
(6) 
The first term in Eqn. (6) learns a mixture of local label contexts, penalizing label assignment in a local cube, where is the number of components in the mixture and is an indicator, determining which component is activated. We define and . An intuitive illustration is given in Fig. 2(a), where the dots in red and blue represent a center voxel and its neighboring voxels , i.e. , and indicates assigning label to voxel . Here, outputs labeling cost between and with respect to their relative positions. For instance, if represent ‘person’ and ‘table’, the learned penalties of positions that are at the bottom of voxel should be large. The second term of Eqn. (6) basically models a triple penalty, which involves voxels , , and ’s neighbors, implying that if and are compatible, then should be also compatible with ’s nearby pixels , , as shown by the purple arrows in Fig. 2(b).
Learning parameters (i.e. weights of VGG and costs of label contexts) in Eqn. (1) requires us to minimize the distances between groundtruth label map and the predicted label , which needs to be inferred subject to the smoothness constraints.
Inference Overview. Inference of Eqn. (1) can be obtained by the mean field (MF) algorithm [25], which estimates the joint distribution of MRF
(7) 
by using a fullyfactorized proposal distribution
(8) 
where each is a variable we need to estimate, indicating the predicted probability of assigning label to voxel . The KullbackLeibler divergence between them is then calculated as
(9) 
Since is a constant, minimizing the KullbackLeibler divergence between and is equivalent to minimizing the former terms in Eqn. (9), which is denoted as free energy [17]. To simplify the discussion, we denote and as and , respectively. And we can further substitute Eqn.(1) into . We have
(10) 
Specifically, the first term in Eqn. (10) characterizes the cost of each voxel’s predictions, while the second term characterizes the consistencies of predictions between voxels. The last term denotes the entropy, measuring the confidences of predictions. Then, a constrained optimization problem regarding could be formulated as
(11)  
subject to 
To solve this minimization problem, we define by introducing Lagrange multipliers . The final closedform solution can be obtained by differentiating w.r.t. to and equating the resulting expression to zero
(12) 
such that the predictions for each voxel is independently attained by repeatedly calculating Eqn. (12), which implies whether voxel have label is proportional to the estimated probabilities of all its neighboring voxels, weighted by their corresponding smoothness penalties. Substituting Eqn. (6) into Eqn. (12), we have
where each is initialized by the corresponding in Eqn. (3), which is the unary prediction of VGG. Eqn. (3) satisfies the smoothness constraints.
In the following, DPN approximates one iteration of Eqn. (3) by decomposing it into two steps. Let be a predicted label map of the th category. In the first step as shown in Fig. 2(c), we calculate the triple penalty term in Eqn. (3) by applying a filter on each position , where each element of this filter equals , resulting in . indicates the time span. Apparently, this step smoothes the prediction of voxel with respect to the distances between it and its neighborhood. In the second step as illustrated in Fig. 2(d), the labeling contexts can be obtained by convolving with a filter, each element of which equals , penalizing the triple relations as shown in Fig. 2(a).
Fig. 3 depicts the semantic meaning of the triple penalty and the mixture of local label contexts term in the spatialtemporal domain. From Fig. 3(a) we can see that the triple penalty term tracks the movement of local pixels, such as ‘pole’, ‘car’ and ‘road’. These temporal trajectories combined with local regions are subsequently used to smooth original predictions. Fig. 3(b) demonstrates that a mixture of local label contexts term not only captures label cooccurrence in a single image, but also encodes the change of label configurations along time. For example, as the observer vehicle drives, ‘tree’ will move backward and get nearer to ‘road’.
4 Deep Parsing Network
(a) VGG: 2242243 input image; 11000 output labels  
1  2  3  4  5  6  7  8  9  10  11  12  














(b) DPN: 5125123 input image; 512512 output label maps  
1  2  3  4  5  6  7  8  9  10  11  12  13  14  15  

















As shown in (a) and (b) respectively. Each table contains five rows, ‘layer’, ‘fi.st.’, ‘#ch.’, ‘act.’, and ‘size’ represent the ‘name of layer’, ‘receptive field of filter’‘stride’, ‘number of output feature maps’, ‘activation function’, and ‘size of output feature maps’, respectively. Furthermore, ‘conv’, ‘lconv3D’, ‘conv3D’, ‘max’, ‘bmin’, ‘fc’, and ‘sum’ represent the convolution, 3D convolution, 3D local convolution, max pooling, block min pooling, fully connection, and summation, respectively. Moreover, ‘relu’, ‘idn’, ‘soft’, ‘sigm’, and ‘lin’ represent the activation functions, including rectified linear unit [49], identity, softmax, sigmoid, and linear, respectively. , , and represent the length of frames, number of categories, and number of mixture filters.
This section describes the implementation of Eqn. (3) in a Deep Parsing Network (DPN). DPN extends VGG to model the unary term and with additional layers to approximate one iteration of MF inference as the pairwise term. The hyperparameters of VGG and DPN are compared in Table I.
As listed in Table I, the first row represents the name of layer and ‘’ in the second row represents the size of the receptive field and the stride of convolution, respectively. For instance, ‘31’ in the convolutional layer implies that the receptive field of each filter is 33 and it is applied on every single pixel of an input feature map, while ‘22’ in the maxpooling layer indicates each feature map is pooled over every other pixel within a 22 local region. On the other hand, ‘3501’ implies a 3D convolution of size 50503, where ‘3’ suggests that this 3D filter is applied on three consecutive frames. The last three rows show the number of the output feature maps, activation functions, and the size of output feature maps, respectively. represents the number of frames in the underlying video sequence.
As summarized in Table I(a), VGG contains thirteen convolutional layers, five maxpooling layers, and three fullyconnected layers. These layers can be partitioned into twelve groups, each of which covers one or more homogeneous layers. For example, the first group comprises two convolutional layers with 33 receptive field and 64 output feature maps, each of which is 224224.
4.1 Modeling Unary Terms
To make full use of VGG, which is pretrained by ImageNet, we adopt all its parameters to initialize the filters of the first ten groups of DPN. To simplify the discussions, we take PASCAL VOC 2012 (VOC12) [30] as an example. Note that DPN can be easily adapted to any other semantic image segmentation dataset by modifying its hyperparameters. VOC12 contains 21 categories and each image is rescaled to 512512 in training. Therefore, DPN needs to predict a total of 51251221 labels, i.e. one label for each pixel. To this end, we extends VGG in two aspects.
In particular, let a and b denote the th group in Table I(a) and (b), respectively. First, we increase the resolution of VGG by removing its max pooling layers at a8 and a10, since most of the information is lost after pooling, e.g. a10 reduces the input size by 32 times, i.e. from 224224 to 77. As a result, the smallest size of feature map in DPN is 6464, keeping much more information compared with VGG. Note that the filters of b8 are initialized as the filters of a9, but the 33 receptive field is padded into 55 as shown in Fig. 4(a), where the cells in white are the original values of the a9’s filter and the cells in gray are zeros. This step is performed because a8 is not presented in DPN, therefore each filter in a9 should be convolved on every other pixel of a7. To maintain the convolution with one stride, we pad the filters with zeros. Furthermore, the feature maps in b11 are upsampled to 512512 by bilinear interpolation. Since DPN is trained with label maps of the entire images, the missing information in the preceding layers of b11 can be recovered by BP. The supervision signals in the interpolated pixels will guide the feature learning for fullresolution images.
Second, two fullyconnected layers at a11 are transformed into two convolutional layers at b9 and b10, respectively. As shown in Table I(a), the first ‘fc’ layer learns 775124096 parameters, which can be altered to 4096 filters in b9, each of which is 2525512. Since a8 and a10 have been removed, the 77 receptive field is padded into 2525 similar as above and shown in Fig.4 (b). The second ‘fc’ layer learns a 40964096 weight matrix, corresponding to 4096 filters in b10. Each filter is 114096.
Overall, b11 generates the unary labeling results, producing twentyone 512512 feature maps, each of which represents the probabilistic label map of each category.
4.2 Modeling Smoothness Terms
The last four layers of DPN, i.e. from b12 to b15, are carefully designed to smooth the unary labeling results.
b12. As listed in Table I (b), ‘lconv’ in b12 indicates a 3D locally convolutional layer. A counterpart of it (i.e. 2D locally convolutional layer) is widely used in face recognition [50, 51, 52] to capture different information from different facial positions. Similarly, distinct spatial positions of b12 have different filters, and each filter is shared across 21 input channels, as shown in Fig. 4(c). It can be formulated as
(14) 
where representing the linear activation function, ‘’ is the convolutional operator, and is a 505031 filter at position of channel . The choice and effect of filter size will be discussed in the experiments (Sec.5.1). We have shared across 21 channels. indicates a local cube in b11, while is the corresponding output of b12. Since b12 has a stride of one, the result of is scalar. In summary, b12 has 512512 different filters and produces 21 output feature maps.
Eqn. (14) implements the triple penalty of Eqn. (3). Recall that each output feature map of b11 indicates a probabilistic label map of a specific object appearing in the frame. As a result, Eqn. (14) suggests that the probability of object presented at position is updated by weighted averaging over the probabilities at its nearby positions. Thus, as shown in Fig. 2(c), corresponds to a cube of centered at , which has values , . Similarly, is initialized by , implying each filter captures dissimilarities between positions. These filters remain fixed during BP, other than learned as in conventional CNN^{1}^{1}1Each filter in b12 actually represents a distance metric between pixels in a specific region. In VOC12, the patterns of all the training images in a specific region are heterogeneous, because of various object shapes. Therefore, we initialize each filter with Euclidean distance. Nevertheless, Eqn. (14) is a more general form than the triple penalty in Eqn. (3), i.e. filters in Eqn. (14) can be automatically learned from data, if the patterns in a specific region are homogeneous, such as face or human images, which have more regular shapes than images in VOC12..
b13. As shown in Table I(b) and Fig. 5(a), b13 is a 3D global convolutional layer that generates 105 feature maps by using 105 filters of size 99321. For example, the value of is attained by applying a 99321 filter at positions . In other words, b13 learns a filter for each category to penalize the probabilistic label maps of b12, corresponding to the local label contexts in Eqn.(3) by assuming and , as shown in Fig.2 (d).
b14. As illustrated in Table I and Fig. 5(b), b14 is a block min pooling layer that pools over every 11 region with one stride across every 5 input channels, leading to 21 output channels, i.e. 105521. Layer b14 activates the contextual pattern with the smallest penalty.
b15. This layer combines both the unary and smoothness terms by summing the outputs of b11 and b14 in an elementwise manner similar to Eqn. (3),
(15) 
where probability of assigning label to voxel is normalized over all the labels.
Relation to Previous Deep Models. Many existing deep models such as [26, 23, 24] employed Eqn. (4) as the pairwise terms, which are the special cases of Eqn. (3). To see this, let , and omit (i.e. ), the right hand side of Eqn. (3) reduces to
(16)  
where and represent the global label cooccurrence and pairwise pixel similarity of Eqn. (4), respectively. This is because is a constant, , and . Eqn. (16) is the corresponding MF update equation of (4).
4.3 Learning Algorithms
We describe the training strategy of DPN and also its time complexity and efficient implementation.
Learning. The first ten groups of DPN are initialized by VGG^{2}^{2}2We use the released VGG model, which is publicly available at http://www.robots.ox.ac.uk/~vgg/research/very_deep/, while the last four groups can be initialized randomly. DPN is then finetuned in an incremental manner with four stages. During finetuning, all these stages solve the pixelwise softmax loss [6], but updating different sets of parameters.
First, we add a loss function to b11 and finetune the weights from b1 to b11 without the last four groups, in order to learn the unary terms. Second, to learn the triple relations, we stack b12 on top of b11 and update its parameters (i.e. in the distance measure), but the weights of the preceding groups (i.e. b1b11) are fixed. Third, b13 and b14 are stacked onto b12 and similarly, their weights are updated with all the preceding parameters fixed, so as to learn the local label contexts. Finally, all the parameters are jointly finetuned.
Complexity. DPN transforms Eqn. (3) into convolutions and poolings in the groups from b12 to b15, such that filtering at each pixel can be performed in a parallel manner. Assume we have input and output feature maps, pixels, filters with receptive field, and a minibatch with samples. b12 takes a total operations, b13 takes operations, while both b14 and b15 require operations. For example, when 10 as in our experiment, we have 2151250101.310 operations in b12, which has the highest complexity in DPN. If we parallelize these operations using matrix multiplication on GPU as [29] did, the operation in b12 can be computed within 30ms. The total runtime of the last four layers of DPN is 75ms. Note that convolutions in DPN can be further speeded up by lowrank decompositions [28] of the filters and model compressions [53].
In contrast, existing works [23, 26] employ fast Gaussian filtering [54] to accelerate the direct calculation of Eqn. (3). For a minibatch of ten 512512 images, a recently optimized implementation [18] needs 3.510 operations and takes 114ms on GPU to compute one iteration of (3). In DPN, simultaneous message passing is enabled over every training image, in the sense that parallelization is obtained in image level. Therefore, DPN makes (3) easier to be parallelized and speeded up.
Efficient Implementation. As mentioned in Eqn. (4), the local filters in b12 are computed by the distances between RGB values of the pixels. XY coordinates are omitted here because they could be precomputed. To accelerate the computation of local convolution, the lookup tablebased filtering approach is employed. Specifically, we first construct a lookup table storing distances between any two pixel intensities (ranging from 0 to 255), which results in a matrix. Then when we perform local convolution, the kernels’ coefficients can be obtained efficiently by just looking up the table.
5 Experiments
In this section, we demonstrate the effectiveness of DPN and benchmark it against other stateoftheart semantic segmentation methods. Below, we give an overview of the dataset and evaluation metrics used in these experiments. Representative methods are also introduced. In the following experiments, we denote D DPN as DPN and D DPN as spatialtemporal DPN.
Dataset. We compare DPN with the stateoftheart methods on PASCAL VOC 2012 (VOC12) [30] , Cityscapes [16] and CamVid [55] datasets. VOC12 is a wellknown benchmark for generic image segmentation and Cityscapes dataset focuses on parsing urban street scenes. We choose those two benchmarks to evaluate the original DPN. On the other hand, CamVid dataset is composed of several video sequences, which is suitable for the evaluation of spatialtemporal DPN. We summarize the information of all datasets we used in Table II.
name  training  validation  testing  video data 

VOC12  10582  1449  1456  no 
Cityscapes  2975  500  1525  no 
CamVid  367    233  yes 
Evaluation Metrics. All existing works employed mean pixelwise intersectionoverunion (denoted as mIoU) [6] to evaluate their performance. To fully examine the effectiveness of DPN, we introduce another three metrics, including tagging accuracy (TA), localization accuracy (LA), and boundary accuracy (BA). (1) TA compares the predicted imagelevel tags with the ground truth tags, calculating the accuracy of multiclass image classification. (2) LA evaluates the IoU between the predicted object bounding boxes^{3}^{3}3They are the bounding boxes of the predicted segmentation regions. and the ground truth bounding boxes (denoted as bIoU), measuring the precision of object localization. (3) For those objects that have been correctly localized, we compare the predicted object boundary with the ground truth boundary, measuring the precision of semantic boundary similar to [56].
Comparisons. DPN is compared with the stateoftheart segmentation methods, including FCN [6], Zoomout [5], DeepLab [23], WSSL [43], BoxSup [57], Piecewise [27], RNN [26], SuperParsing [58], Dilation10 [59], ALE [60], Multiclass [12], and SegNet [44]. All these methods are based on CNNs or MRFs. They can be grouped according to different aspects: (1) jointtrain: Piecewise and RNN; (2) w/o jointtrain: DeepLab, WSSL, FCN, and BoxSup; (3) pretrain on COCO: RNN, WSSL, and BoxSup. The first and the second groups are the methods with and without joint training CNNs and MRFs, respectively. Methods in the last group also employed MSCOCO [61] to pretrain deep models. To conduct a comprehensive comparison, the performance of DPN are reported on both settings, i.e. , with and without pretraining on COCO.
In the following, Sec. 5.1 investigates the effectiveness of different components of DPN on the VOC12. Sec. 5.2 evaluates the spatialtemporal DPN on the CamVid. Sec. 5.3 provides detailed analysis of the DPN system and its performance. Sec. 5.4 compares DPN with the stateoftheart methods on the benchmarks.
5.1 Effectiveness of DPN
All the models evaluated in this section are trained on the training set and tested on the validation set of VOC12.
Triple Penalty. The receptive field of b12 indicates the range of triple relations for each pixel. We examine different settings of the receptive fields, including ‘1010’, ‘5050’, and ‘100100’, as shown in Table III(a), where ‘5050’ achieves the best mIoU, which is sightly better than ‘100100’. For a 512512 image, this result implies that 5050 neighborhood is sufficient to capture relations between pixels, while smaller or larger regions tend to underfit or overfit the training data. Moreover, all models of triple relations outperform the ‘baseline’ method that models dense pairwise relations, i.e. VGG+denseCRF [18].


Label Contexts. Receptive field of b13 indicates the range of local label context. To evaluate its effectiveness, we fix the receptive field of b12 as 5050. As summarized in Table III(b), ‘99 mixtures’ improves preceding settings by 1.7, 0.5, and 0.2 percent respectively. We observe large gaps exist between ‘11’ and ‘55’. Note that the 11 receptive field of b13 corresponds to learning a global label cooccurrence without considering local spatial contexts. Table III(c) shows that the pairwise terms of DPN are more effective than DSN and DeepLab^{4}^{4}4The other deep models such as RNN and Piecewise did not report the exact improvements after combining unary and pairwise terms..
More importantly, mIoU of all the categories can be improved through increasing the size of receptive field and learning a mixture. Specifically, for each category, the improvements of the last three settings in Table III(b) over the first one are 1.20.2, 1.50.2, and 1.70.3, respectively.
We also visualize the learned label compatibilities and contexts in Fig. 6(a) and (b), respectively. Fig. 6(a) is obtained by summing each filter in b13 over a 99 region, indicating how likely a column object would present when a row object is presented. Blue color represents high favorability. It is worth pointing out that Fig. 6(a) is nonsymmetry. For example, when a ‘horse’ is presented, a ‘person’ is more likely to present than the other objects. On the other hand, when a ‘person’ presents in an image, it is more likely to find a ‘bike’ (than a ‘horse’). The ‘bkg’ (background) is compatible with all the objects. Fig. 6(b) visualizes some contextual patterns, where ‘A:B’ indicates that when ‘A’ is presented, where ‘B’ is more likely to present. For example, ‘bkg’ is around ‘train’, ‘motor bike’ is below ‘person’, and ‘person’ is sitting on ‘chair’.
2D  3D  

Unary Term (mIoU)  59.67  59.67 
+ Triple Penalty  59.93  60.16 
+ Label Contexts  60.06  60.25 
5.2 Effectiveness of Spatialtemporal DPN
In this section, we quantitatively investigate the effectiveness of the pairwise term in spatialtemporal DPN. CamVid train are used for training and performance are reported on CamVid test. Images in CamVid are captured from a moving vehicle, thus exhibiting certain temporal regularization.
3D Convolution. In spatialtemporal DPN, the b12 is a 3D local convolutional layer and b13 is a 3D global convolutional layer. In Table IV, we evaluate the performance gain of each stage on DPN. According to Table III(a), we set the receptive field of b12 as 5050 and b13 as 99 in 2D setting. In 3D pairwise term, the receptive field of b12 is 50503 and b13 is 773. We observe that applying 2D and 3D pairwise terms on unary term both improves the performance. Since 3D pairwise terms capture the information between successive frames, it performs slightly better than 2D pairwise terms.
Temporal Regularization. In Fig. 7, we visualize the 3D learned label compatibilities and contexts across frames.
Fig. 7(a) depicts the possibility of column object would present in the  frame when a row object is presented in the  frame. Blue color indicates high favorability. For instance, when ‘pavement’ is presented in the  frame, ‘road’ is more likely to present around in the  frame. However, if ‘sky’ is presented in the  frame, the pairwise term will penalize the probability which ‘road’ is presented in the  frame.
Fig. 7(b) visualizes the spatialtemporal contextual patterns. In each row, ‘A:B’ indicates how ‘A’ influences ‘B’.The figures suggest that when ‘A’ is presented in the  or the  frame, where ‘B’ is more likely to present in the  frame. In the first row, (1) represents that when ‘sky’ is presented in the  frame, its neighborhood is more likely to be ‘sky’. (2) means when ‘sky’ is presented in the  frame, it is more likely presented above this position in the  frame. As shown in Fig. 8, (a) and (b) are neighboring video frames, when the vehicle moves forward from (a) to (b), ‘sky’ region (yellow) shifts upward in the scene. So compared with the favor region in Fig. 7(b)(1), the favor region also shifts upward in (2). In the second row of Fig. 7(b), (3) and (4) represent that when ‘pedestrian’ is presented in the  or the  frame, ‘building’ is more likely presented above in the  frame. We observe that the shifting from (3) to (4) is similar with the shifting of ‘pedestrian’(orange) in Fig. 8 from (a) to (b).
5.3 Further Analysis
All the analysis in this section are conducted on the validation set of VOC12.
Incremental Learning. As discussed in Sec. 4.3, DPN is trained in an incremental manner. The right hand side of Table V(a) demonstrates that each stage leads to performance gain compared to its previous stage. For instance, ‘triple penalty’ improves ‘unary term’ by 2.3 percent, while ‘label contexts’ improves ‘triple penalty’ by 1.8 percent. More importantly, joint finetuning all the components (i.e. unary terms and pairwise terms) in DPN achieves another gain of 1.3 percent. A stepbystep visualization is provided in Fig. 9.
We also compare ‘incremental learning’ with ‘joint learning’, which finetunes all the components of DPN at the same time. The training curves of them are plotted in Fig.10 (a), showing that the former leads to higher and more stable accuracies with respect to different iterations, while the latter may get stuck at local minima. This difference is easy to understand – incremental learning only introduces new parameters until all existing parameters have been finetuned.
Oneiteration MF. DPN approximates one iteration of MF. Fig. 10(b) illustrates that DPN reaches a good accuracy with one MF iteration. A CRF [18] with dense pairwise edges needs more than 5 iterations to converge. It also has a large gap compared to DPN. Note that the existing deep models such as [23, 26, 24] required 510 iterations to converge as well.


coarse  depth  sub  road  swalk  build.  wall  fence  pole  tlight  sign  veg.  terrain  sky  person  rider  car  truck  bus  train  mbike  bike  mIoU  
Dilation10 [59]  no  no  no  97.6  79.2  89.9  37.3  47.6  53.2  58.6  65.2  91.8  69.4  93.7  78.9  55  93.3  45.5  53.4  47.7  52.2  66  67.1 
Piecewise [27]  no  no  no  97.3  78.5  88.4  44.5  48.3  34.1  55.5  61.7  90.1  69.5  92.2  72.5  52.3  91  54.6  61.6  51.6  55  63.1  66.4 
SiCNN+CRF  no  yes  no  96.3  76.8  88.8  40  45.4  50.1  63.3  69.6  90.6  67.1  92.2  77.6  55.9  90.1  39.2  51.3  44.4  54.4  66.1  66.3 
FCN [6]  no  no  no  97.4  78.4  89.2  34.9  44.2  47.4  60.1  65  91.4  69.3  93.9  77.1  51.4  92.6  35.3  48.6  46.5  51.6  66.8  65.3 
WSSL [43]  yes  no  2  97.4  78.3  88.1  47.5  44.2  29.5  44.4  55.4  89.4  67.3  92.8  71  49.3  91.4  55.9  66.6  56.7  48.1  58.1  64.8 
DeepLab [23]  no  no  2  97.3  77.7  87.7  43.6  40.5  29.7  44.5  55.4  89.4  67  92.7  71.2  49.4  91.4  48.7  56.7  49.1  47.9  58.6  63.1 
RNN [26]  no  no  2  96.3  73.9  88.2  47.6  41.3  35.2  49.5  59.7  90.6  66.1  93.5  70.4  34.7  90.1  39.2  57.5  55.4  43.9  54.6  62.5 
DPN  no  no  no  97.5  78.5  89.5  40.4  45.9  51.1  56.8  65.3  91.5  69.4  94.5  77.5  54.2  92.5  44.5  53.4  49.9  52.1  64.8  66.8 
build.  tree  sky  car  sign  road  person  fence  pole  pave.  cyclist  mIoU  
ALE [60]  73.4  70.2  91.1  64.24  24.4  91.1  29.1  31  13.6  72.4  28.6  53.59 
SuperParsing [58]  70.4  54.8  83.5  43.3  25.4  83.4  11.6  18.3  5.2  57.4  8.9  42.03 
Tripathi et al. [13]  74.2  67.9  91  66.5  23.6  90.7  26.2  28.5  16.3  71.9  28.2  53.18 
Liu and He [12]  66.8  66.6  90.1  62.9  21.4  85.8  28  17.8  8.3  63.5  8.5  47.2 
SegNet [44]  68.7  52  87  58.5  13.4  86.2  25.3  17.9  16.0  60.5  24.8  46.4 
DPN  80.6  72.6  91.2  77.8  40  90.7  43.9  28.7  15.9  71.4  47.9  60.06 
Spatialtemporal DPN  80.6  73.1  91.4  77.9  40  90.8  43.9  29.2  16  71.9  47.9  60.25 
Perstage Analysis. We further evaluate DPN using three metrics. The results are given in Fig. 11. For example, (a) illustrates that the tagging accuracy can be improved in the third stage, as it captures label cooccurrence with a mixture of contextual patterns. However, TA decreases a little after the final stage. Since joint tuning maximizes segmentation accuracies by optimizing all components together, extremely small objects, which rarely occur in VOC training set, are discarded. As shown in (b), accuracies of object localization are significantly improved in the second and the final stages. This is intuitive because the unary prediction can be refined by longrange and highorder pixel relations, and joint training further improves results. (c) discloses that the second stage also captures object boundary, since it measures dissimilarities between pixels.
Perclass Analysis. Table V(a) reports the perclass accuracies of four evaluation metrics, where the first four rows represent the mIoU of four stages, while the last three rows represent TA, LA, and BA, respectively. We have several valuable observations, which motivate future researches. (1) Joint training benefits most of the categories, except animals such as ‘bird’, ‘cat’, and ‘cow’. Some instances of these categories are extremely small so that joint training discards them for smoother results. (2) Training DPN with pixelwise label maps implicitly models imagelevel tags, since it achieves a high averaged TA of 96.4%. (3) Object localization always helps. However, for objects with complex boundary such as ‘bike’, its mIoU is low even it can be localized, e.g. ‘bike’ has high LA but low BA and mIoU. (4) Failures of different categories have different factors. With these three metrics, they can be easily identified. For example, the failures of ‘chair’, ‘table’, and ‘plant’ are caused by the difficulties to accurately capture their bounding boxes and boundaries. Although ‘bottle’ and ‘tv’ are also difficult to localize, they achieve moderate mIoU because of their regular shapes. In other words, mIoU of ‘bottle’ and ‘tv’ can be significantly improved if they can be accurately localized.
5.4 Benchmarks
We evaluate the performance of DPN and spatiotemporal DPN on several standard semantic segmentation benchmarks.
5.4.1 Pascal VOC12
The VOC12 dataset is one of the most popular benchmarks for semantic image segmentation. This dataset contains 20 indoor and outdoor object categories and one background category. As previously mentioned, we employ images for training, images for validation, and images for testing. Results are given Table V (b), we compare DPN with the bestperforming methods^{5}^{5}5The results of these methods were presented in either the published papers or arXiv preprints. on VOC12 test set based on two settings, i.e. with and without pretraining on COCO. The approaches pretrained on COCO are marked with ‘’. We evaluate DPN on several scales of the images and then average the results following [23, 27].
DPN outperforms several existing methods that were trained on VOC12, but DPN needs only one MF iteration to solve MRF, other than 10 iterations of RNN, DeepLab, and Piecewise. By averaging the results of two DPNs, we achieve 74.1% accuracy on VOC12 without outside training data. As discussed in Sec.4.3, MF iteration is the most complex step even when it is implemented as convolutions. Therefore, DPN at least reduces 10 runtime compared to previous works.
Following [26, 57], we pretrain DPN with COCO, where 20 object categories that are also presented in VOC12 are selected for training. A single DPN has achieved 77.5% mIoU on VOC12 test set. The system in [27] has incorporated multiscale pyramid training and midlevel feature refinement, which we believe is complementary to our contribution and can boost the performance of DPN as well. As shown in Table V (b), we observe that DPN achieves competitive performances among multiple object classes.
5.4.2 Cityscapes
Cityscapes dataset focuses on street scenes segmentation. All images are captured from a moving vehicle in various seasons and cities. This dataset is highly different from VOC12. Because of the great depth of images, the street scenes have large scale variations even in a single image. It defines 19 object categories for evaluation. There are 2975 training, 500 validation and 1525 testing images with fine pixellevel annotations and extra 20000 images with cores annotations. To conduct a fair comparison, we only use the images with fine pixellevel annotations in our experiment.
As shown in Table VI, DPN achieves 66.8% on Cityscapes dataset, which is the second best method, and it is close to the first place 67.1% [59]. [59] takes a multiscale aggregation strategy, which can be easily integrated into DPN to further boost the performance. Our method shows advantages on objects with arbitrary shape (i.e. ‘road’, ‘building’, ‘vegetation, ‘terrain’ and ‘sky’) which can be easily confused with other objects. Thanks to our highorder and longrange pairwise term, DPN captures their appearance and segments them from other objects accurately.
5.4.3 CamVid
CamVid dataset consists of 367 training and 233 testing images with 11 classes annotations. All images are extracted from three video sequences at 1Hz. Similar to Cityscapes dataset, these video sequences are also captured from a moving vehicle. As shown in Table VII, DPN outperforms all existing methods, and spatialtemporal pairwise term further improves the performance to 60.25%. Considering the relatively sparse sampling rate (one frame per second) and strong unary term used, our 3D MRF can indeed leverage temporal contextual information for joint inference.
We can observe that DPN achieves much better performance than other methods, especially on narrow and small objects (i.e. ‘pole’ and ‘sign’), which are very difficult in segmentation task. Since our triple penalty captures the appearance of pixels, we can predict more accurate boundary on those objects. The spatialtemporal pairwise term further improves the performance. Spatialtemporal DPN encodes the relationship between successive frames, which improve the performance on some specific categories like ‘tree’, ‘sky’ and ‘road’, because they have flexible shape and are continuous in successive frames. It is also worthwhile to note that our spatialtemporal DPN achieves best performances on 7 object categories out of 11.
5.5 Visual Quality Comparisons
In the following, we inspect visual quality of obtained label maps. Fig. 12 demonstrates the comparisons of DPN with FCN [6] and DeepLab [23]. We use the publicly released model^{6}^{6}6http://dl.caffe.berkeleyvision.org/fcn8spascal.caffemodel to regenerate label maps of FCN while the results of DeepLab are extracted from their published paper. DPN generally makes more accurate predictions in both imagelevel and instancelevel. For example, in Fig. 12 (row 2), DPN is able to discover all persons available while still recovering sharp boundaries of the aeroplane. Even in the challenging case of Fig. 12 (row 4) due to complex reflectance, the integrity of the bus is maintained by DPN. More examples of DPN label maps are shown in Fig. 14. We observe that learning local label contexts helps differentiate confusing objects and learning triple penalty facilitates the capturing of intrinsic object boundaries.
We also include some typical failure modes of DPN in Fig. 13. In the first case, the object with atypical pose is hard to be found. Further considering welltrained object detector as additional unary potential might be a potential solution to this challenge. The second case suggests that scale and illumination are also important factors that influence the performance. This problem can be partially alleviated by augmenting or adding the variance of training data.
6 Conclusion
We proposed Deep Parsing Network (DPN) to address semantic image/video segmentation. DPN has several appealing properties. First, DPN unifies the inference and learning of unary term and pairwise terms in a single convolutional network. No iterative inference is required during backpropagation. Second, highorder relations and mixtures of label contexts are incorporated to its pairwise terms modeling, making existing works as special cases. Third, DPN is built upon conventional operations of CNN, thus easy to be parallelized and speeded up.
DPN achieves stateoftheart performance on VOC12, Cityscapes and CamVid datasets. Multiple valuable facts about semantic segmentation are revealed through extensive experiments, such as the interplay between tagging, localization and boundary accuracies along different processing stages. Future directions include investigating the generalizability of DPN to more challenging scenarios with large number of object classes and substantial appearance/scale variations.
Acknowledgment
This work is supported by SenseTime Group Limited and the General Research Fund sponsored by the Research Grants Council of the Hong Kong SAR (CUHK 416713, 14241716, 14224316. 14209217).
References
 [1] J. Shi and J. Malik, “Normalized cuts and image segmentation,” TPAMI, vol. 22, no. 8, 2000.
 [2] P. F. Felzenszwalb and D. P. Huttenlocher, “Efficient belief propagation for early vision,” IJCV, vol. 70, no. 1, 2006.
 [3] C. Liu, J. Yuen, and A. Torralba, “Nonparametric scene parsing via label transfer,” TPAMI, vol. 33, no. 12, 2011.
 [4] C. Farabet, C. Couprie, L. Najman, and Y. LeCun, “Learning hierarchical features for scene labeling,” TPAMI, vol. 35, no. 8, 2013.
 [5] M. Mostajabi, P. Yadollahpour, and G. Shakhnarovich, “Feedforward semantic segmentation with zoomout features,” in CVPR, 2015.
 [6] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in CVPR, 2015.
 [7] Z. Liu, X. Li, P. Luo, C.C. Loy, and X. Tang, “Semantic image segmentation via deep parsing network,” in ICCV, 2015.
 [8] C. Gu and M.C. Lee, “Semiautomatic segmentation and tracking of semantic video objects,” TCSVT, vol. 8, no. 5, 1998.
 [9] G. J. Brostow, J. Fauqueur, and R. Cipolla, “Semantic object classes in video: A highdefinition ground truth database,” Pattern Recognition Letters, 2008.
 [10] J. Lezama, K. Alahari, J. Sivic, and I. Laptev, “Track to the future: Spatiotemporal video segmentation with longrange motion cues,” in CVPR, 2011.
 [11] A. Jain, S. Chatterjee, and R. Vidal, “Coarsetofine semantic video segmentation using supervoxel trees,” in ICCV, 2013.
 [12] B. Liu and X. He, “Multiclass semantic video segmentation with objectlevel active inference,” in CVPR, 2015.
 [13] S. Tripathi, S. Belongie, Y. Hwang, and T. Nguyen, “Semantic video segmentation: Exploring inference efficiency,” in ISOCC, 2015.
 [14] A. Kundu, V. Vineet, and V. Koltun, “Feature space optimization for semantic video segmentation,” in CVPR, 2016.
 [15] P. Isola and C. Liu, “Scene collaging: Analysis and synthesis of natural images with semantic layers,” in ICCV, 2013.
 [16] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele, “The cityscapes dataset for semantic urban scene understanding,” in CVPR, 2016.
 [17] M. I. Jordan, Z. Ghahramani, T. S. Jaakkola, and L. K. Saul, “An introduction to variational methods for graphical models,” Machine learning, vol. 37, no. 2, 1999.
 [18] P. Krähenbühl and V. Koltun, “Efficient inference in fully connected crfs with gaussian edge potentials,” in NIPS, 2011.
 [19] V. Vineet, J. Warrell, and P. H. Torr, “Filterbased meanfield inference for random fields with higherorder terms and product labelspaces,” in ECCV, 2012.
 [20] J. Yang, B. Price, S. Cohen, and M.H. Yang, “Context driven scene parsing with attention to rare classes,” in CVPR, 2014.
 [21] K. Simonyan and A. Zisserman, “Very deep convolutional networks for largescale image recognition,” in ICLR, 2015.
 [22] J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, and L. FeiFei, “Imagenet: A largescale hierarchical image database,” in CVPR, 2009.
 [23] L.C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Semantic image segmentation with deep convolutional nets and fully connected crfs,” in ICLR, 2015.
 [24] A. G. Schwing and R. Urtasun, “Fully connected deep structured networks,” arXiv:1503.02351v1, 2015.
 [25] M. Opper, O. Winther et al., “From naive mean field theory to the tap equations,” 2001.
 [26] S. Zheng, S. Jayasumana, B. RomeraParedes, V. Vineet, Z. Su, D. Du, C. Huang, and P. Torr, “Conditional random fields as recurrent neural networks,” in ICCV, 2015.
 [27] G. Lin, C. Shen, I. Reid, and A. Hengel, “Efficient piecewise training of deep structured models for semantic segmentation,” in CVPR, 2016.
 [28] M. Jaderberg, A. Vedaldi, and A. Zisserman, “Speeding up convolutional neural networks with low rank expansions,” in BMVC, 2014.
 [29] S. Chetlur, C. Woolley, P. Vandermersch, J. Cohen, J. Tran, B. Catanzaro, and E. Shelhamer, “cudnn: Efficient primitives for deep learning,” in NIPS Deep Learning Workshop, 2014.
 [30] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” IJCV, vol. 88, no. 2, 2010.
 [31] X. Ren and J. Malik, “Learning a classification model for segmentation,” in ICCV, 2003.
 [32] M. Szummer, P. Kohli, and D. Hoiem, “Learning crfs using graph cuts,” in ECCV, 2008.
 [33] B. Fulkerson, A. Vedaldi, and S. Soatto, “Class segmentation and object localization with superpixel neighborhoods,” in ICCV, 2009.
 [34] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour detection and hierarchical image segmentation,” TPAMI, vol. 33, no. 5, 2011.
 [35] G. Wang, P. Luo, L. Lin, and X. Wang, “Learning object interactions and descriptions for semantic image segmentation,” in CVPR, 2017.
 [36] P. Luo, G. Wang, L. Lin, and X. Wang, “Deep dual learning for semantic image segmentation,” in ICCV, 2017.
 [37] P. Krähenbühl and V. Koltun, “Parameter learning and convergent inference for dense random fields,” in ICML, 2013.
 [38] V. Vineet, G. Sheasby, J. Warrell, and P. H. Torr, “Posefield: An efficient meanfield based method for joint estimation of human pose, segmentation, and depth,” in Energy Minimization Methods in Computer Vision and Pattern Recognition, 2013.
 [39] R. Mottaghi, X. Chen, X. Liu, N.G. Cho, S.W. Lee, S. Fidler, R. Urtasun, and A. Yuille, “The role of context for object detection and semantic segmentation in the wild,” in CVPR, 2014.
 [40] C. Wang, M. de La Gorce, and N. Paragios, “Segmentation, ordering and multiobject tracking using graphical models.” in ICCV, 2009.
 [41] P. Luo, X. Wang, and X. Tang, “Hierarchical face parsing via deep learning,” in CVPR, 2012.
 [42] P. Luo, X. Wang, and X. Tang, “Pedestrian parsing via deep decompositional network,” in ICCV, 2013.
 [43] G. Papandreou, L.C. Chen, K. Murphy, and A. L. Yuille, “Weaklyand semisupervised learning of a dcnn for semantic image segmentation,” in ICCV, 2015.
 [44] V. Badrinarayanan, A. Kendall, and R. Cipolla, “Segnet: A deep convolutional encoderdecoder architecture for image segmentation,” arXiv preprint arXiv:1511.00561, 2015.
 [45] W. T. Freeman, E. C. Pasztor, and O. T. Carmichael, “Learning lowlevel vision,” IJCV, vol. 40, no. 1, 2000.
 [46] Z. Liu, L. Yuan, X. Tang, M. Uyttendaele, and J. Sun, “Fast burst images denoising,” TOG, vol. 33, no. 6, 2014.
 [47] Z. Liu, R. A. Yeh, X. Tang, Y. Liu, and A. Agarwala, “Video frame synthesis using deep voxel flow,” in ICCV, 2017.
 [48] C. Liu, J. Yuen, and A. Torralba, “Sift flow: Dense correspondence across scenes and its applications,” TPAMI, vol. 33, no. 5, 2011.
 [49] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in NIPS, 2012.
 [50] Y. Sun, X. Wang, and X. Tang, “Deep learning face representation by joint identificationverification,” in NIPS, 2014.
 [51] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf, “Deepface: Closing the gap to humanlevel performance in face verification,” in CVPR, 2014.
 [52] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in ICCV, 2015.
 [53] G. E. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” in NIPS Deep Learning Workshop, 2014.
 [54] A. Adams, J. Baek, and M. A. Davis, “Fast highdimensional filtering using the permutohedral lattice,” in Eurographics, 2010.
 [55] G. J. Brostow, J. Shotton, J. Fauqueur, and R. Cipolla, “Segmentation and recognition using structure from motion point clouds,” in ECCV, 2008.
 [56] B. Hariharan, P. Arbeláez, L. Bourdev, S. Maji, and J. Malik, “Semantic contours from inverse detectors,” in ICCV, 2011.
 [57] J. Dai, K. He, and J. Sun, “Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation,” in ICCV, 2015.
 [58] J. Tighe and S. Lazebnik, “Superparsing: scalable nonparametric image parsing with superpixels,” in ECCV, 2010.
 [59] F. Yu and V. Koltun, “Multiscale context aggregation by dilated convolutions,” in ICLR, 2016.
 [60] C. Russell, P. Kohli, P. H. Torr et al., “Associative hierarchical crfs for object class image segmentation,” in ICCV, 2009.
 [61] T.Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in ECCV, 2014.
Ziwei Liu received his B.E. degree in Department of Electronics and Information Engineering in 2013, from Huazhong University of Science and Technology, Wuhan, China. He is currently a PhD student at the Department of Information Engineering, The Chinese University of Hong Kong. His research interests include computer vision, machine learning and computational photography, with focus on humancentric visual understanding and deep learning. 
Xiaoxiao Li received his B.E. degree in Department of Computer Science and Technology from Tsinghua University in 2014. He is currently a PhD student at the Department of Information Engineering, The Chinese University of Hong Kong. His research interests include computer vision and machine learning, especially image segmentation and deep learning. 
Ping Luo received his PhD degree in 2014 in Information Engineering, Chinese University of Hong Kong (CUHK). He is currently a Research Assistant Professor in Electronic Engineering, CUHK. His research interests focus on deep learning and computer vision, including optimization, face recognition, webscale image and video understanding. He has published 40+ peerreviewed articles in toptier conferences such as CVPR, ICML, NIPS and journals such as TPAMI and IJCV. He received a number of awards for his academic contribution, such as Microsoft Research Fellow Award in 2013 and Hong Kong PhD Fellow in 2011. He is a member of IEEE. 
Chen Change Loy (S’06M’10SM’17) received the PhD degree in computer science from the Queen Mary University of London in 2010. He is currently a research assistant professor in the Department of Information Engineering, Chinese University of Hong Kong. Previously, he was a postdoctoral researcher at Queen Mary University of London and Vision Semantics Ltd. His research interests include computer vision and pattern recognition, with focus on face analysis, deep learning, and visual surveillance. He serves as an Associate Editor of IET Computer Vision Journal. He is a senior member of the IEEE. 
Xiaoou Tang (S’93M’96SM’02F’09) received the B.S. degree from the University of Science and Technology of China, Hefei, in 1990, and the M.S. degree from the University of Rochester, Rochester, NY, in 1991. He received the Ph.D. degree from the Massachusetts Institute of Technology, Cambridge, in 1996. He is a Professor and the Chairman of the Department of Information Engineering, Chinese University of Hong Kong. He worked as the group manager of the Visual Computing Group at the Microsoft Research Asia from 2005 to 2008. His research interests include computer vision, pattern recognition, and video processing. He received the Best Paper Award at the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2009 and Outstanding Student Paper Award at the AAAI 2015. He was a program chair of the IEEE International Conference on Computer Vision (ICCV) 2009 and served as an associate editor of the IEEE Transactions on Pattern Analysis and Machine Intelligence and International Journal of Computer Vision. He is now the EditorinChief of the International Journal of Computer Vision. He is a fellow of the IEEE. 