AU-AIR: A Multi-modal Unmanned Aerial Vehicle Dataset for Low Altitude Traffic Surveillance

Unmanned aerial vehicles (UAVs) are extensively used as flying platforms of sensors for different domains such as traffic surveillance [20], managing the urban environment [11], package delivery [18] or aerial cinematography [2]. For these applications, UAVs are equipped with mounted cameras and mainly gather visual data of the environment. Then, computer vision algorithms are applied to aerial visual data to extract high-level information regarding the environment.

Object detection is one of the most studied problems in computer vision. The recent advances in deep learning (variants of convolutional neural networks (CNNs) mainly) have led to breakthrough object detection performances with the availability of large datasets and computing power. Since these methods require a large number of training samples, several datasets (e.g., COCO [15], Pascal VOC [8]) have been introduced for benchmarking for the object detection task. The samples in these datasets consist of natural images that are mainly captured by handheld cameras. The significant differences between natural and aerial images (such as object layouts and sizes) cause these object detectors to have trouble to find objects in aerial images. Therefore, several datasets (e.g., [29, 7, 13, 24, 19, 6, 14]) have been introduced in recent years as a benchmark for object detection in aerial images.

Besides visual data gathered by a camera, the data from other sensors might give crucial information about the environment. The use of UAVs as only flying cameras cut off the potential advance in multi-modal object detection algorithms for aerial applications. For instance, the recent advances in perception for autonomous driving have brought new datasets such as [12, 4, 5] including multi-modal data (e.g., RGB images, Global Positioning System (GPS) coordinates, inertial measurement unit (IMU) data). Although the data fusion for object detection is still open research topic [9], these multi-modal datasets allow a benchmark for further research. However, to the best of our knowledge, there is no such multi-modal dataset collected in a real-world outdoor environment for UAVs.

In this work, we present a multi-modal UAV dataset (The AU-AIR dataset) in order to push forward the development of computer vision and robotic algorithms targeted at autonomous aerial surveillance. The AU-AIR dataset meets vision and robotics for UAVs having the multi-modal data from different on-board sensors. The dataset consists of 8 video streams (over 2 hours in total) for traffic surveillance. The videos mainly are recorded at Skejby Nordlandsvej and P.O Pedersensvej roads (Aarhus, Denmark). The dataset includes aerial videos, time, GPS coordinates and the altitude of the UAV, IMU data, and the velocity. The videos are recorded at different flight altitudes from 5 meters to 30 meters and in different camera angles from 45 degrees to 90 degrees (i.e., complete bird-view images that the camera is perpendicular to the Earth). Instances belonging to different object categories related to the traffic surveillance context are annotated with bounding boxes in video frames. Moreover, each extracted video frame is labeled with the flight data (See Fig. 1).

The whole dataset includes 32,823 labeled video frames with object annotations and the corresponding flight data. Eight object categories are annotated including person, car, van, truck, motorbike, bike, bus, trailer. The total number of annotated instances is 132,034. The dataset is split into 30,000 training-validation samples and 2,823 test samples.

In this work, we emphasize differences between aerial and natural images in the context of object detection tasks. To this end, we compare image samples and object instances between the AU-AIR dataset and the COCO dataset [15]. In our experiments, we train and evaluate two mobile object detectors (including YOLOv3-tiny [22] and MobileNetv2-SSD Lite [25] on the AU-AIR dataset. We form a baseline, including mobile object detectors since we focus on real-time performance and the applicability of object detection task onboard computers mounted on UAV.

The availability of large amounts of data and processing power enables deep neural networks to achieve state-of-the-art results for object detection. Currently, deep learning-based object detectors are separated into two groups. The first group consists of region-based CNNs that ascend on image classifiers. Region-based CNNs propose image regions that are likely to contain an object and classify the region into a predefined object category. The second group has only one stage converting to the object detection problem into the bounding box prediction for objects, without re-purposing image classifiers. Faster-R-CNN [23] is one of the well-known models belonging to the first group, YOLO [21] and SSD [16] are the popular object detectors that belong to the second group.

Deep learning-based object detectors have trained and performed on large datasets such as COCO [15] and PASCAL [8]. These datasets include natural images that contain a single object or multi objects in their natural environments. Most of the images in these datasets are captured by humans using a handheld camera so that the vast majority of images have side-view. There are challenges of the object detection in natural images such as occlusion, illumination changes, rotation, low resolution, crowd existence of instances.

Aerial images have different characteristics from natural images due to having a bird’s-eye view. First of all, objects in natural images are much larger than their counterparts in aerial images. For example, an object category such as humans may occupy a large number of pixels in natural images. However, it may have a few numbers of pixels in an aerial image that is quite challenging to detect for object detectors (See Fig. 2). Moreover, aerial images can be fed to a network with higher dimensions that increases computational cost in order to prevent the diminishing of pixels belonging to small objects.

Secondly, an occlusion is observed in different conditions for natural and aerial images. In natural images, an object instance may be occluded by another foreground object instance (e.g., a human in front of a car). However, objects in aerial images are less likely to be occluded by other foreground objects (especially, bird-view images captured by a camera that is perpendicular to the Earth). (See Fig. 3.

Thirdly, the perspective in aerial images makes appearances of objects short and squat. This fact diminishes the information regarding an object height (See Fig. 4). Moreover, although aerial images can supply more contextual information about an environment by a broader view angle, the object instances may be amid cluttered.

Lastly, having a drone to capture aerial images, the altitude changes during the flight can cause varieties in object size and appearance in aerial images. Therefore, a recording of aerial videos at different altitudes may change the levels of challenges mentioned above.

To address the challenges mentioned in Section II, we propose a multi-modal drone dataset (AU-AIR) including videos, object annotations in the extracted frames and sensor data for the corresponding frames. The data are captured by low-level flight (max. 30 meters) and for the scenario of a traffic surveillance. The AU-AIR dataset consists of video clips, sensor data, and object bounding box annotations for video frames.

We train and evaluate mobile object detectors with our dataset. During the evaluation, we consider real-time performance rather than achieving a state-of-the-art accuracy for the sake of the applicability. Therefore, we choose two mobile object detectors (YOLOv3-Tiny [22] and MobileNetv2-SSDLite [25]), which have a reasonable trade-off between the detection accuracy and the inference time.

Since the number of instances of each object category is imbalanced in the AU-AIR dataset (Fig. 5), we consider several methods to solve the imbalanced class problem in the next version of the dataset. As a first step, we will try to collect more data to balance the number of instances. Besides, we may consider adding synthetic data (i.e., changing the brightness of images, translation, rotation) to increase the number of object categories which has a low number of samples in the current version.

We use AMT to annotate objects in images. Although three different people annotate one image and the annotations are manually checked by ourselves, there might be still overlooked samples that have weak annotations (e.g., unlabelled instances, loose bounding box drawings). Therefore, we consider using a three-step workflow proposed by Su et al. [26]. In this workflow, the first worker draws a bounding box around an instance, the second worker verifies whether the bounding box is correctly drawn, and the third worker checks whether all object instances are annotated.

Unlike other UAV object detection datasets, ours includes sensor data corresponding to each frame. In this work, we give a baseline only for object annotations and visual data. As future work, more baselines may be added to encourage research using sensor data (e.g., navigation and control of a UAV, object detection using multi-modal data). Also, we can add more visual sensors, such as multi-spectral cameras.

We have used a ready-to-fly quadrotor (i.e., Parrot Bebop 2) to collect the whole dataset. We also consider collecting more samples from other platforms (e.g., different types of UAVs) using cameras that have different resolutions and frame rates.

In this dataset, traffic surveillance is the primary context. In future work, we consider increasing the number of environment contexts to increase diversity in the dataset.

In this work, we propose the AU-AIR dataset that is a multi-modal UAV dataset collected in an outdoor environment. Our aim is to fill the gap between computer vision and robotics having a diverse range of recorded data types for UAVs. Including visual data, object annotations, and flight data, it can be used for different research fields focused on data fusion.

We have emphasized the differences between natural images and aerial images affecting the object detection task. Moreover, since we consider real-time performance and applicability in real-world scenarios, we have created a baseline, including two mobile object detectors in the literature (i.e., YOLOv3-Tiny [22] and MobileNetv2-SSDLite [25]). In our experiments, we showed that mobile networks trained on natural images have trouble in detecting objects in aerial images.