U-Net Based Multi-instance Video Object Segmentation

The State-of-the-Art image semantic segmentation solution typically takes a encoder-decoder architecture, as popularized in Long, et al’s work  [10]. Briefly, the encoder is a pre-trained classification network like VGG  [5]/ ResNet [7]; and decoder’s task is to semantically project the discriminative features (lower resolution) learned by the encoder onto the pixel space (higher resolution) to get a dense classification.

Few different decoding mechanism has been proposed over the years. When the first breakthrough of semantic segmentation arrives with Long, et al [10] they only employed a coarse up-sampling for the decoder architecture. Although a good start, the heatmap produced using this technique is quite coarse. Later, SegNet [1] introduced transposed convolutional layers into the decoder work. Built on similar idea, U-Net [15] further introduced skip connection to improve.

Another different approach is Region-Based Semantic Segmentation, where a detection is performed first followed by segmentation. It often relies on a two-stage processing: with Region of Interest (ROI) proposal followed by a heavy-lifting working network. Earlier effort on the first stage has been focusing on ”segment candidate proposal”, e.g. DeepMask [13]. But it turns out that these methods are slow and less accurate because segmentation precedes recognition. Most recently, He, et al proposed an elegant solution Mask R-CNN  [4]. It extends Faster R-CNN by adding a branch for predicting segmentation masks in parallel with the existing branch for classification and bounding box recognition. Mask R-CNN is an end-to-end model, outperforms all existing, single-model entries on the architecture for Pascal VOC [3] and MSCOCO  [9] challenges.

There are three leading approaches to Video Object Segmentation, OSVOS [2], MaskTrack [11] and Recurrent mask propagation  [17].

The current non-temporal State-of-the-Art is OSVOS. OSVOS approach first converts a pre-trained image classification CNNs, eg VGG-16, to a fully convolutional network by removing the FC layer and insert new loss. It then train this fully convolutional network on the DAVIS dataset. By fine-tuning this network with the first frame of the video, OSVOS generates a one-time model and test it on that entire sequence, using its new weight. This approach shows that we can achieve great result without using temporal information of the video. The model we developed is on top of this approach.

MaskTrack, on the other hand, feed the predicted mask of the previous frame as additional input to the network, to make the input 4 channels. Then train a CNN eg VGG-16, from scratch on a combination of semantic segmentation and image saliency datasets. Finally it adds an identical second stream network, based on optical flow input. Variation approaches like Online Adaptation [16] and re-identification [8] has also gain great result.

A new released Recurrent Mask Propagation  [17] approach formulates a deep recurrent network that is capable of segmenting and tracking objects in video simultaneously by their temporal continuity, so it’s able to re-identify them when they re-appear after a prolonged occlusion and achieve great result. This approach uses temporal info, which is not the focus area of this paper.

For this paper, we propose an U-Net based fully convolutional networks architecture, as shown in Figure 2.

DAVIS 2017 Challenge [14] on Video Object Segmentation was used for this project. DAVIS dataset spans multiple occurrences of common video object segmentation challenges, such as occlusions, motion blur and appearance changes. On top of DAVIS 2016 dataset  [12], the 2017 dataset added multi-instance challenge. This requires semantic/instance-level segmentation. A detailed summary of the dataset is shown in Table 1.

We used official evaluation code from DAVIS-2017 to report our model performance. Specifically, we evaluated the following:

• Region Similarity: the intersection-over union between the predicted mask M and ground-truth G. Note the similarity between this metric and the Dice coefficient loss we introduced earlier.

  $$J=\frac{|M\cap G|}{|M\cup G|}$$

• Contour Accuracy: the Harmonic mean of contour’s precision and recall

  $$F=\frac{P_c\ast R_c}{P_c+R_c}$$

The model was built on tensorflow 1.8 framework and implemented using Python 3.6. The code is open-sourced and can be found at $github.com/hyuna915/vide{o}_{s}egmentation$. In addition to original code to pre-process image, isolate objects, construct U-Net, and SegNet graph, we used the davis-2017 code repository for model evaluation and OSVOS code repository for constructing OSVOS layer.

Our model was trained on N1-HighMem-8 instance, with v8CPU, 52 GB memory, on Google Cloud Compute Platform. We’ve also attached NVIDIA Tesla P100 GPU with 16GB memory to the virtual machine. Due to the sheer amount of model structure/parameters we have experimented upon, in total 5 GPUs are used for this project.

We experimented a few different architectures. The results of the two most promising architecture, SegNet and U-Net, are summarized on Table 2. It is observed that best U-Net has a J-mean comparable with the State-of-the-Art OSVOS, in spite of a slightly worse F-mean. Figure 3 shows the convergence pattern of the two architectures on an example training image.

Furthermore, a sample annotation comparison between the two is shown on Figure 4. From Figure 4 (c), we noticed U-Net tends to produce a more complete instance with a better coverage and smoother contour in exchange of contour accuracy (i.e. lower F value). As a comparison, OSVOS tends to have less coverage and a rigid contour. We believe our model works better for recall-focus scenario, like pedestrian segmentation.

SegNet, on the other hand, has a much coarse contour with lower precision. Its F-mean is much lower compared with U-Net. The reason is likely to be the absence of merging step, which connects the high resolution features from contracting path with the up-sampling features from the expanding path. Without this connection, SegNet tends to ”forget” high-resolution details by the end of the contracting process. We can see this clearly in Figure 4 (d).

For U-Net, we carried many hyper-parameters tuning experiments, including tuning on number of filters in each U-Net layer, batch size and learning rate. The experiment results are summarized in Table 3.

Figure 5 shows the train loss (first row) and test loss (second row) over 10K iterations for two of our hyper-parameters tuning experiment. the periodic up and downs in training loss is because we were not able to do shuffling on training dataset, due to GPU memory limitation. Although learning rates were different, qualitatively they showed similar trend. The training loss keeps going down while the test loss reaches the best point and goes up afterwards.

Over simplified U-Net with 700K parameters wasn’t able to achieve good performance (table 3). The best U-Net Model has convolutional filter number of 64, 128, 256, 512, contains 31M trainable parameters, with learning rate 4e-5 and batch size 8. This model is not over-fitting because both training and test loss is around 1.02. Due to the fact that U-Net has 31M+ parameters in the graph, the batch size can’t go beyond 8 to prevent GPU out of memory. The training of the best U-Net takes around 1 hour per epoch and 8 hours to fully converge.

We list 4 representative test annotations produced by our best U-Net model, as shown in Figure 6. Based on the case study, we have the following observations:

1. The model can handle intensive motion and appearance change gracefully.

   In Figure 6(a) drift chicane sequence, the annotation starts with a small and far car object, which makes multiple turns accompanied by drastic appearance change, distance change, and background fogs. The model keeps good and sharp track on the moving vehicle.

2. The model can handle multi-instances with similar motion very well, even with overlapping.

   In Figure 6(b) Horse Jump sequence, the rider and the horse has similar motion. The model can keep track of the two objects and draw relatively clear boundaries.

3. The model doesn’t handle multiple object collusion very well.

   In Figure 6(c) Camel sequence, the annotation is very accurate in the first few frames, but when the target camel bypass another camel, the model starts tracking both camels afterwards.

4. The model lost track when object goes beyond image boundary and goes back.

   In Figure 6(d) Motocross jump, the annotation is very accurate in the first few frames, but when the rider moves out of the image boundary, the model lost track of the rider and start tracking only the motorcycle afterwards.

For this paper, we also explored few other approaches, which did not achieve a good result. We still want to report them and hope it can help with future research.

• Mask R-CNN: We noticed Mask R-CNN didn’t produce reasonable result for semi-supervised video segmentation problem as in the case of DAVIS-2017 dataset, for two reasons. First, being a instance segmentation algorithm, Mask R-CNN requires recognition for segmentation. This caused difficulty when DAVIS dataset tracks object that Mask R-CNN model has not seen before. Figure 7 is a classic example where Mask R-CNN, trained with COCO dataset that did not include camel, was used to segment camel sequence. Consequently, Mask R-CNN mistakingly predict the camel as an horse with 0.998 confidence and further tried to correct the shape to be a horse, hurting the annotation accuracy.

  

  Another failure reason on vanilla Mask R-CNN is it was not able to incorporate the semi-supervised information provided on the first image. This often lead to unnecessary objects being segmented in the following frames, as well as wrong segmentation orders for the original targets. However, this problem may be mitigated by training the Mask R-CNN backbone on the first image of each video sequence, similar as OSVOS’s approach.

• Unweighted Cross Entropy as Loss Function: For DAVIS dataset, since most of the instance is very small compared with the background, when we tried to use unweighted cross entropy as loss function, almost all the pixels are predicted as background.

• Direct Feed Multi-instance Image: We also tried to feed multi-instance image to the networks directly and transform this problem into a multi-class classification problem. Because convolutional networks is not able to project the value of 1, 2, 3 to layer 1, 2, 3. The result produced by this approach is also very poor.