Domain Adaptation for Semantic Segmentation with Maximum Squares Loss

In unsupervised domain adaptation (UDA), the labeled source domain is denoted as $D_S=\left\{\right(x_s,y_s)|x_s\in R^{H\times W\times 3},y_s\in R^{H\times W}\}$, and the unlabeled target domain is denoted as $D_T=\{x_t|x_t\in R^{H\times W\times 3}\}$. The general objective function of UDA for semantic segmentation can be formulated as follows:

where $L_{CE}$ is the cross entropy loss of source samples, $n$ represents a pixel point in the $H\times W$ space and $N=HW$ is the total number of pixels in a picture . $p_s^{n,c}$ is the model prediction probability of the class $c$ at point $n$ for sample $x_s$. $L_T\left(x_t\right)$ is the loss part for target samples.

Entropy Minimization. In the  [31], they try to minimize the Shannon entropy of the target sample prediction. Thus, their objective function for target samples is:

For the sake of simplicity, we consider the binary classification case. Then the entropy formula and the gradient function of the entropy can be written as follows:

After plotting the gradient function image on Fig. 1, we can see that the gradient of the high probability point is much larger than the mediate point. As a result, the key principle behind the entropy minimization method is that the training of target samples is guided by the high probability area, which is assumed to be more accurate.

Probability Imbalance Problem. The probability of different classes varies widely. Classes with high accuracy always have higher prediction probabilities (Fig. 2). However, the gradient growth (Eq. 5) of the high probability point is approximated as $|\log p|(p\to 0)$, which will grow to infinity. Then the simple class will produce a much larger gradient on each pixel than the difficult class, resulting in the probability imbalance problem mentioned in Section 1. To remedy this problem, we define the maximum squares loss as:

As Fig. 4 demonstrates, classes with higher accuracy always have more pixels on the label map, which leads to an imbalance in quantity. The regular method to balance the number of classes is to introduce weighting factor ${\alpha }_c$, which is usually set as the inverse class frequency [18]. However, in the UDA task, there is no class label to calculate the class frequency. It is also not appropriate to replace the target class statistics with the class statistics on the source dataset, because there is no guarantee that the target domain will have the same class frequency as the source domain.

Instead of using the class frequency of the entire target dataset, we calculate them on each target image:

In Eq. 6, we divide the sum by N to average the loss on the target image. Instead, we average the loss based on the number of classes $N^c$. Due to inaccurate predictions, interpolation between these two numbers is more stable:

where $\alpha$ is treated as a hyper-parameter to be selected by cross-validation.

As mentioned in [28], adapting low-level feature can enhance the final performance. We extract the feature maps from the conv4 layer of ResNet [12] and add an ASPP module to it as the low-level output. Then we extend the objective function of target samples as:

where $L_T^{final}\left(x_t\right)$ denotes the loss function of model final prediction for the target sample, \eg, the maximum squares loss (Eq. 6). Because the high-level output is more accurate than the low-level output, it is more reasonable to use the high-level output to guide the training of low-level features. As a result, we adopt the idea of the self-produced guidance learning [34] in weakly-supervised learning. We first get the ensemble output $P_{ens}$ by averaging the output map of different levels, \ie, $P_{final}$ and $P_{low}$. Then we generate the self-produced guidance ${\overline{y}}_t^{n,c^{\ast }}$ by:

where the choice of $\delta$ dose not effect the experimental result and we set $\delta =0.95$. We use this high-qualify guidance to guide the low-level training:

In the experiment, we fix ${\lambda }_{low}=0.1$, the same as  [28].

Classification. Office-31 [27] is the most commonly used dataset for unsupervised domain adaptation, which contains 4,652 images and 13 categories collected from three domains: Amazon (A), Webcam (W) and DSLR (D). We evaluate all methods across six domain adaptation tasks A $\to$ W, D $\to$ W, W $\to$ D, A $\to$ D, D $\to$ A and W $\to$ A.

Semantic Segmentation. As for the transfer from synthetic datasets to real-world datasets, we consider Cityscapes [6] as the target domain, and set GTA5 [25] or SYNTHIA [26] dataset as the source domain, which is same as the setting in previous works [28, 36]. Cityscapes dataset contains 5,000 annotated images with $2048\times 1024$ resolution taken from real urban street scenes. GTA5 dataset [25] contains 24,966 annotated images with $1914\times 1052$ resolution taken from the the GTA5 game. For SYNTHIA dataset, we use the SYNTHIA-RAND-CITYSCAPES subset consisting of 9,400 $1280\times 760$ synthetic images. During training, we use the labeled training sets of GTA5 or SYNTHIA as the source domain and the 2,975 images from Cityscapes training set without annotation as the target domain. We evaluate all methods on the 500 images from Cityscapes validation set.

In the evaluation, we adopt the Intersection-over-Union (IoU) of each class and the mean-Intersection-over-Union (mIoU) as performance metrics. We consider the IoU and mIoU of all 19 classes in the GTA5-to-Cityscapes case. While SYNTHIA only shares 16 classes with Cityscapes, we consider the IoU and mIoU of 16-class and 13-class in the SYNTHIA-to-Cityscapes case.

As for cross-city adaptation, we choose the training set of Cityscapes as the source domain and NTHU dataset [5] as the target domain. The NTHU dataset consists of images with $2048\times 1024$ resolution from four different cities: Rio, Rome, Tokyo, and Taipei. For each city, we use 3200 images without annotations as the target domain for training and 100 images labeled with 13 classes for evaluation. We consider the shared 13-class IoU and mIoU for evaluation.

Classification. We applied entropy minimization and maximum square loss to ResNet-50 [12]. We adopt the model pre-trained on ImageNet [7], except the final classifier layer. We train the model using stochastic gradient descent (SGD) with momentum of 0.9. Following learning rate annealing strategy in  [10], the learning rate is adjusted by ${\eta }_p=\frac{{\eta }_0}{(1+\alpha p{)}^{\beta }}$, where p is the training progress linearly changing from 0 to 1, ${\eta }_0=0.01$, $\alpha =10$, $\beta =0.75$. We set the batch size to 128, half of which is source samples and half is target samples. We set ${\lambda }_T=0.3$ for maximum square loss and ${\lambda }_T=0.03$ for entropy minimization.

Semantic Segmentation. As argued in [28], it is important to adopt a stronger baseline model to understand the effect of different adaption approaches and enhance the performance for the practical application. Therefore, in all experiment, we use Deeplabv2 [2] with ResNet-101 [12] backbones pre-trained on ImageNet [7] as our base model, which is the same as other works [28, 31].

Before the adaptation, we pre-train the network on the source domain for $70k$ steps to get a high-quality source trained network. We implement the algorithms using PyTorch [23] on a single NVIDIA $1080$Ti GPU. Due to memory limitations, we train the model with batch size $2$ (one from the source domain and one from the target domain).

Following [28], we train the model with Stochastic Gradient Descent (SGD) optimizer with learning rate $2.5\times {10}^{-4}$ , momentum $0.9$ and weight decay $5\times {10}^{-4}$. We schedule the learning rate using “poly” policy: the learning rate is multiplied by $(1-\frac{iter}{max\_iter}{)}^{0.9}$ [2]. We employ the random mirror and gaussian blur to augment data, the same as [35].

As for the selection of hyper-parameters, we set ${\lambda }_T=0.1$ in all experiments. In the experiments related to the image-wise weighting factor (Eq. 13), we fix $\alpha =0.2$.

Results Tab. 4 shows comparison results on office-31. Although the results are uncompetitive with state-of-the-art methods, the maximum square loss (MaxSquare) exceeds the entropy minimization (EntMin) and DANN [10] by a large margin. Because the semantic segmentation task is much harder than the classification, this difference will be more apparent in the following semantic segmentation experiments.

Verification of Maximum Square Loss. As shown in Section  3.2, the maximum squares loss can make difficult samples be trained more efficiently than the entropy minimization. We use A$\to$W task to verify this conclusion experimentally. We first train the model on the source domain and mark the 30% most confident samples in the test set as “top set” and the 30% least confident samples as “bottom set”. Then we fine-tune the model with EntMin or MaxSquare and record the accuracy on the test set, “top set” and “bottom set”. As Fig. 5 shows, there is no difference between the accuracy of two methods on the “top set”. However, the accuracy of MaxSquare on the “bottom set” is much higher than EntMin. These results imply that the main improvement of MaxSquare to EntMin comes from the improvement of difficult samples.

Following the evaluation protocol of other works [31, 36], we evaluate the IoU and mIoU of the shared 16 classes between two datasets and the 13 classes excluding the classes with ${}^{\ast }$. As Table 3 shows, our methods achieve competitive results to other methods. “MaxSquare+IW” surpasses “MaxSquare” method on the several small object classes, \eg, traffic light, traffic sign, and motorbike.

To show the efficiency of our methods for smaller domain shift, we conduct our experiment on the NTHU dataset with ResNet-101 backbone. We consider the IoU and mIoU of shared 13 classes for evaluation. Table 6 shows the results of transferring from Cityscapes to the four cities in the NTHU dataset. In all four adaptation experiments, our “MaxSquare+IW” outperforms the other most advanced methods by about 1 point. These excellent results demonstrate the effectiveness of our maximum squares loss and our image-wise weighting factor. Moreover, unlike self-training [36], our approach does not assume that source and target domains share the same spatial priors. Therefore, our method is robust to various transfer settings.

This work was supported in part by the National Nature Science Foundation of China (Grant Nos: 61751307) and the National Youth Top-notch Talent Support Program.