Learning to Count Objects
with Few Exemplar Annotations

predicts instance count directly based on global regressors with image features [2, 3, 4, 5]. For example, in [2], the video sequence is first segmented into different components of homogeneous motion, and a Gaussian process regression is learned for each segmented region to count the number of instances. Cumulative attribute representation [3] was proposed and used to learn the regressor to handle the imbalanced training data. Inter-dependent features are used to mine the spatially importance among different region to learn the number of count [4]. Feature normalization is taken into account to deal with perspective projections in [5]. With the advantage of the deep learning,  [16] estimate the number of cars using CNNs. These regression-based approaches provide no clue of individual location of each object, which limits its potential applications.

first maps the image to a density map, such that the integral over any sub region gives the count of objects within that region [17, 7, 8, 6]. In [17], the pixel-level density is learned by minimizing a regularized risk quadratic cost function. Based on the density map, an interactive counting system is introduced in [7] to incorporate the relevance feedback. Instead of hand-crafted feature, [6] focuses on the CNN-based density map and instance count estimation in cross-scene scenario. While the density map provides certain clue of the crowdedness, it still lacks the exact position of each instance.

gives the total count by localizing each object instance. Such approaches can be considered as the application of object detection and thus benefit a lot from the improvement of object detection [12, 13, 14, 15]. However, directly applying object detection for object counting requires special focus on small-scale objects and extraordinary effort on massive instance labeling. In [18], a hierarchical part-template matching approach is proposed to detection humans, which requires careful feature and template design, while in [9], the neural network learning is applied to detect and count car instances by incorporating layout information. With its large potential in applications demanding location information, we mainly work on the detection-based approach with special focus on incompletely-supervised learning.

The differences with other supervised learning settings in the context of object detection are also shown in Fig. 1. Fully-supervised learning requires that all the bounding boxes are labeled. This represents the upper bound of detection-based object counting performance, but is costly to label every instance. Weakly-supervised learning assumes that only image-level category information is available. That means, we know there are some object instances in one image, but we have no information of their locations. Another problem setting is low-shot learning, where the number of training images is small, but each training image has full annotations. This is more like the fully-supervised learning but with only a few training images.

Fig. 2 illustrates the intuition of the label propagation by PFOD. The plus symbol denotes a positive sample while the minus symbol denotes negative. The sample in red is known (labeled) while the sample in blue is unknown. The line with the dotted line is the decision boundary. If all the positive and negative samples are known, we can easily figure out the decision boundary to separate true positive and true negative samples, as shown in Fig. 2(a). When only one positive sample is known and all the rest are unknown, as shown in Fig. 2(b), if we simply treat all the unknowns as negative sample, the decision boundary could mis-classify the unlabeled positive samples as being negative.

To compensate the lack of negative samples, we can introduce some images as extra negative data, as shown in green from Fig. 2(c) to Fig. 2(e). For object detection, it is indeed easy to find images without any target objects in the same domain. For example, we can use the PASCAL VOC [24] 2007 data set as extra background images for the problem of car counting on the CARPK [9] dataset.

In Fig. 2(c) we show how to propagate the labels. We first treat all the unlabeled samples (shown as both blue plus and minus symbols in Fig. 2(c)) as negative samples to train the detector, and the learned decision boundary will be pushed close to the only positive sample (red plus symbol) as depicted with the red dotted line. Next, we ignore the unknown samples (note that we still have the extra negative samples shown as green minus symbols) and gradually update the learned classifier to classify the labeled positive sample and the extra negative samples. As the unknown positive samples (shown as blue plus symbols) are not taken as negative samples, they no longer push the decision boundary. As a result, the decision boundary (as shown as black dotted line) will be moved a bit further from the known positive sample and classify a few more unknown positive samples as positive samples.

If the propagation is carefully controlled, we can treat the newly classified positive samples as known labeled data for the next stage, as shown in Fig. 2(d). We repeat the process above to iteratively learn the boundary and propagate the label set. Each iteration here is called a stage, and we use $L^s=\{x_i|i=1,\dots M_s\}$ to denote the label set in the $s$-th stage, where $M_s$ is the number of expanded bounding boxes in one image. Finally, we combine all the expanded positive samples and take all the others as background to learn the final decision boundary.

Fig. 3 shows the framework of the proposed strategy to learn the object detector. At the initial state of training, the bounding boxes we have are the labeled set, i.e. $L^1=\{x_i|i=1,\dots ,M_1\}$ for each training image. Based on $L^s$, we train a positiveness-focused object detector (PFOD), which can be based on any object detector [19, 12, 13, 14, 15]. In this work, we choose YoloV2 [15] for its simplicity. The network first processes each training image in a batch manner by a fully convolutional neural network, and then outputs three components at each spatial position: bounding box coordinates, objectiveness to tell how confident the bounding box contains an object, and classification scores to tell which category the bounding box contains. Here we assume the number of categories is 1 and remove the classification module. The Euclidean loss is used for bounding box coordinate regression and objectiveness confidence regression.

With the model trained by PFOD, we run the prediction over all the training images from target domain. The predicted bounding boxes with high probability scores (0.9 in experiments) will be merged into the original label set $L^s$ to form $L^{s+1}$. We also discard any predicted bounding box if it has a high overlap (Intersection-over-Union $>$ 0.2 in experiments) with any of the original bounding boxes in $L^s$.

After $S$ stages, we feed the training images and the expanded bounding box set $L^{S+1}$ into a normal object detector training pipeline, in which the bounding boxes in $L^{S+1}$ are positive while all the rest are negative. We still apply the YoloV2 algorithm here for training and testing. Ideally, if all the unlabeled object instances could be propagated, the trained model should be able to achieve an accuracy on par with that trained from the full annotations.

We mainly evaluate the approaches on the CARPK [9] dataset, which contains 989 training images and 459 testing images. The task is to detect and count the car instances in the image. Each training image has $42.7\pm 15.7$ annotated cars, while each testing image has $103.5\pm 39.1$ cars. The images are collected by a drone on top of car parks. An example image is shown in the first row of Fig. 1

Another interesting application is to count the number of drinks or products on retail store shelves. To demonstrate the effectiveness of the incompletely-supervised learning algorithm on other domains, we collect a small dataset Drink35, which contains 10 images as the training set and 25 images as the testing set. The task is to detect and count all the product instances. One example image is shown in the second row of Fig. 1.

To simulate the incompletely-supervised settings, we randomly select $C_i$ training images, and for each image we randomly select at most $C_a$ annotated cars. All the other unselected images are discarded during training. This training set is denoted as $\text{CARPK}\_C_i\_C_a$ or $\text{Drink35}\_C_i\_C_a$ for CARPK and Drink35 datasets, respectively. For example, CARPK_5_5 means the training set with 5 images and each with at most 5 annotated bounding boxes. Similarly, the suffix of $\_C_i\_\text{ALL}$ denotes the training set of $C_i$ images with all the annotated boxes, and $\_\text{ALL}\_C_a$ denotes all training images with at most $C_a$ annotated bounding boxes in each image. The test set is not altered for consistent evaluation.

We use the PASCAL VOC [24] 2007 trainval set (5011 images) as the extra background images with all the original bounding box labels removed. Note that the images in PASCAL VOC 2007 contains the object of cars and drinks. We keep these images as negative samples because the cars and drinks in VOC 2007 are generally of difference appearances or views compared with the object instances in CARPK and Drink35.

Since we focus on the detection-based approaches, we adopt the widely-used [12, 13, 14, 15] mAP@0.5 as one of the metrics, which measures the mean average precision (mAP) using 0.5 as the interaction-over-union (IoU) threshold.

Following [17, 9], we also use the Mean Absolute Error (MAE) and the Root Mean Squared Error (RMSE) to evaluate the accuracy of the counting results. MAE is defined as ${\sum }_i|n_i-n_i^{\prime }|/N$, while RMSE as ${\sum }_i\sqrt{(n_i-n_i^{\prime }{)}^2}/N$, where $n_i$ is the number of objects predicted by the model for the $i$-th testing image, $n_i^{\prime }$ is the ground-truth number of objects, and $N$ is the total number of testing images.

The data augmentation is of great importance for the network learning due to the small training set with incomplete labels. Motivated by the implementation of Yolo [15] and SSD [14], we incorporate the random scaling, random aspect ratio distortion, and color jittering. Random rotation is also implemented for the car counting problem by multiple data samplers (motivated by SSD [14]), so that non-rotated images are preferred than rotated images. That is, images with 0, 90, 180, and 270 degrees’ rotations are preferred than images with arbitrary rotations, and images with less than 10 degrees’ rotation are preferred than images with other arbitrary rotations. Specifically, for each training image, we have 25% chance to select non-rotated image; 25% chance to rotate images by $90$x degrees; 25% chance to rotate images by less than $\pm 10$ degrees plus a random $90$x rotation; and the last 25% chance to rotate images for any angle.

Another important parameter is that we use a large input resolution for both training and testing due to large number of objects and small object sizes in each image. During training, we resize the image so that the longer side randomly ranges from $832$ to $1664$, and then crop a subregion of $416\times 416$ as the network input. During testing, we resize the input image so that the whole area is close to $1248\times 1248$ while its aspect ratio is kept. Non-Maximum Compression (NMS) is used to filter the bounding box and the IoU threshold is 0.2.

The network backbone is Darknet19, which is the same as in YoloV2 [15]. We use 9 stages of PFOD to propagate the labels. In each stage of PFOD, we train the network with 10K iterations. The learning rate is 0.0001 for the first 100 iterations, 0.001 for the next 4900 iterations, 0.0001 for 4000 iterations and 0.00001 for the last 1000 iterations¹¹1The total number iterations could be greatly reduced as the number of training images is small. However, as the training time is not a concern in this work, we leave the training optimization as a future work.. The batch size is set to 64, and the weight decay is 0.0005. The last detector training shares the same parameters. The training takes 1.25 hours on 4 NVidia P100 GPUs to finish the 10K iterations. The implementation is based on Caffe [25] under the environment of CUDA 8.0 and CUDNN 5.1.

We also report the accuracy without any label propagation, and directly train the object detector on the provided label set. This straightforward approach is denoted as OD as a naive baseline, and all the data augmentation and learning strategy parameters are the same with a single stage of PFOD.

In the incomplete annotation settings, we enable the extra background samples by replacing 16 images in each batch of 64 with the extra background images.

In the fully supervised setting ($C_i=\text{ALL}$, $C_a=\text{ALL}$), our detector (OD) could achieve 96.67% mAP@0.5, 2.94 MAE and 3.94 RMSE. This significantly outperforms the state-of-the-art method of LPN [9], whose MAE is 23.80 and RMSE is 36.79 (mAP is not reported). Other baseline approaches are not shown here because the accuracy is lower than LPN. This demonstrates a strong object detector towards the counting problem.

We apply the OD on $C_i=5$ images with all the labeled annotations ($C_a=\text{ALL}$), and the detector can still achieves 91.97 mAP@0.5, 3.00 MAE, and 5.58 RSME without extra background images. Compared with $C_i=\text{ALL}$, $C_a=\text{ALL}$ which uses 989 training images, this result is very encouraging. It clearly indicates that to develop a car counting algorithm, 5 or a few more images might be enough rather than several hundreds. The 5 image IDs are 20160524_GF2_00133, 20161030_GF1_00153, 20161030_GF1_00036, 20160331_NTU_00007, and
20161030_GF2_00071 for reproducing the result.

When the number of bounding boxes is decreased to 5, the mAP significantly drops to 4.40. To identify the reason, we evaluate the trained model against the training images and show two examples in Fig 4. The predicted bounding boxes are drawn on the image with confidence scores around each box. Only the boxes with confidence scores higher than 0.05 are displayed. The selected 5 labeled boxes are located at the same position with the predicted boxes and the probability is close to 1. Since the threshold here is 0.05, all the other regions including the unlabeled cars are classified as background with high confidence. This shows that the model overfits the training data severely, which degrades the accuracy.

We apply the proposed PFOD on CARPK_5_5, and surprisingly the accuracy is boosted to 72.44 mAP, 23.63 MAE and 26.36 RSME. In terms of MAE and RSME, the accuracy has surpassed the LRN [9] trained on the full training set of 989 images.

In Fig. 5, we illustrate the label propagation process by PFOD on CARPK_5_5. Each column corresponds to one training image. The two numbers below each images are the number of the expanded boxes (initially provided + propagated), and the number of correct boxes among those boxes. A box is correct if its IoU is larger than 0.3 with at least one bounding box in CARPK_5_ALL. From the figure, the correct bounding boxes used for training could be gradually populated. Taking the leftmost image as an example, the number of correct boxes is increased from the initial number 5 to 22. The rightmost one can have 33 correct boxes.

Meanwhile, we observe that the propagation is still not perfect - it introduces several false bounding boxes while missing a few cars. This is the reason why there is still a gap between this setting and CARPK_5_ALL, and will motivate us to continue investigating the problem.

By increasing the number of labeled boxes per training image from 5 to 50, the accuracy can be smoothly increased for both OD and PFOD. With less than 25 boxes in each image, the accuracy of PFOD is consistently higher than OD, while with 50 boxes, the accuracy is lower. The reason is that under the setting of 50 boxes, most of the true boxes are included, while the label propagation introduced some false boxes. That is, under the almost full annotations, it is enough to apply the OD instead of propagating the boxes.

Fig. 6 visualizes the detection and counting results on three testing images based on the model with OD trained on CARPK_5_5, OD trained on CARPK_5_ALL, and the model with PFOD on CARPK_5_5. The yellow boxes are the correct bounding boxes while the red one is the incorrect one. A predicted box is correct if it has an IoU larger than 0.3 with one of the ground truth bounding boxes. The two numbers under the image are the number of instances predicted by the model and the ground-truth number of instances, respectively.