The ActivityNet Large-Scale Activity Recognition Challenge 2018 Summary

The ActivityNet Large-Scale Activity Recognition Challenge 2018 Summary

Bernard Ghanem King Abdullah University of Science and Technology Juan Carlos Niebles Cees Snoek Universiteit van Amsterdam Fabian Caba Heilbron King Abdullah University of Science and Technology Humam Alwassel King Abdullah University of Science and Technology Victor Escorcia King Abdullah University of Science and Technology Ranjay Khrisna Stanford University Shyamal Buch Stanford University Cuong Duc Dao King Abdullah University of Science and Technology
Abstract

The 3rd annual installment of the ActivityNet Large-Scale Activity Recognition Challenge, held as a full-day workshop in CVPR 2018, focused on the recognition of daily life, high-level, goal-oriented activities from user-generated videos as those found in internet video portals. The 2018 challenge hosted six diverse tasks which aimed to push the limits of semantic visual understanding of videos as well as bridge visual content with human captions. Three out of the six tasks were based on the ActivityNet dataset, which was introduced in CVPR 2015 and organized hierarchically in a semantic taxonomy. These tasks focused on tracing evidence of activities in time in the form of proposals, class labels, and captions. In this installment of the challenge, we hosted three guest tasks to enrich the understanding of visual information in videos. The guest tasks focused on complementary aspects of the activity recognition problem at large scale and involved three challenging and recently compiled datasets: the Kinetics-600 dataset from Google DeepMind, the AVA dataset from Berkeley and Google, and the Moments in Time dataset from MIT and IBM Research.

1 Introduction

This challenge was the 3rd annual installment of the ActivityNet Large-Scale Activity Recognition Challenge held as a full-day workshop in CVPR 2018. It focused on the recognition of daily life, high-level, goal-oriented activities from user-generated videos as those found in internet video portals. The 2018 challenge hosted six diverse tasks which aimed to push the limits of semantic visual understanding of videos as well as bridge visual content with human captions. Three out of the six tasks were based on the ActivityNet dataset [1], which was introduced in CVPR 2015 and organized hierarchically in a semantic taxonomy. These tasks focused on tracing evidence of activities in time in the form of proposals, class labels, and captions [4]. In this installment of the challenge, we hosted three guest tasks to enrich the understanding of visual information in videos. The guest tasks focused on complementary aspects of the activity recognition problem at large scale and involved three challenging and recently compiled datasets: the Kinetics-600 dataset [3] from AVA dataset [2] from Berkeley and Google, and the Moments in Time dataset [5] from MIT and IBM Research.

2 Main Challenge Tasks

The challenge had three main tasks: Temporal Action Proposals (ActivityNet), Temporal Action Localization (ActivityNet), and Dense-Captioning Events in Videos (ActivityNet Captions). In the following subsections, we describe each task’s objective, dataset, and evaluation metric. We finally give the top- results on each task.

2.1 Task 1: Temporal Action Proposals

Description and Objective. In many large-scale video analysis scenarios, one is interested in localizing and recognizing human activities occurring in short temporal intervals within long untrimmed videos. Current approaches for activity detection still struggle to handle large-scale video collections and efficiently addressing this task remains elusive to our visual systems. This is in part due to the computational complexity of current action recognition approaches and the lack of methods that propose fewer intervals in the video, where activity processing can be focused. These set of candidate temporal segments are widely known as Action Proposals.

To be applicable at large-scales and in practical scenarios, a useful action proposal method is driven by two competing goals. (i) The proposal method must be computationally efficient in representing, encoding, and scoring a temporal segment. (ii) The proposal method must be discriminative of activities that we are interested in, so as to only retrieve temporal segments that contain visual information indicative of these activity classes. Thus, this task is intended to push the state-of-the-art in action proposal generation algorithms forward.

Dataset. This task is evaluated on the ActivityNet version 1.3 dataset [1]. The dataset consists of more than hours of untrimmed videos from a total of  K videos. It contains different daily activities such as: walking the dog, long jump, and vacuuming floor. The distribution among training, validation, and testing is roughly , , and of the total videos, respectively.

Evaluation Metric. We use the area under the Average Recall vs. Average Number of Proposals per Video (AR-AN) curve as the evaluation metric for this task. A proposal is a true positive if it has a temporal intersection over union (tIoU) with a ground-truth segment that is greater than or equal to a given threshold (e.g. tIoU ). AR is defined as the mean of all recall values using tIoU between and (inclusive) with a step size of . AN is defined as the total number of proposals divided by the number of videos in the testing subset. We consider bins for AN, centered at values between and (inclusive) with a step size of , when computing the values on the AR-AN curve.

Top Results. Table 1 shows the top-3 submissions. Please refer to challenge website for the full version of this summary document which includes a copy of all the papers submitted to the workshop.

Rank Organization AUC
1 Baidu Vis 71.00
2 Shanghai Jiao Tong University 69.30
3 YH Technologies 67.78
Table 1: The top-3 submissions for task 1.

2.2 Task 2: Temporal Action Localization

Description and Objective. Despite the recent advances in large-scale video analysis, temporal action localization remains as one of the most challenging unsolved problems in computer vision. This search problem hinders various real-world applications ranging from consumer video summarization to surveillance, crowd monitoring, and elderly care. Therefore, we are committed to push forward the development of efficient and accurate automated methods that can search and retrieve events and activities in video collections. This task is intended to encourage computer vision researchers to design high performance action localization systems.

Dataset. This task is evaluated on the ActivityNet version 1.3 dataset [1]. The dataset consists of more than hours of untrimmed videos from a total of  K videos. It contains different daily activities such as: walking the dog, long jump, and vacuuming floor. The distribution among training, validation, and testing is roughly , , and of the total videos, respectively.

Evaluation Metric. We use the Interpolated Average Precision (AP) to evaluate the results on each activity category. The performance on the dataset is measured by the mean AP (mAP) over all the activity categories. To determine if a detection is a true positive, we inspect the tIoU with a ground truth segment, and check whether it is greater or equal to a given threshold (e.g. tIoU ). The official metric used in this task is the average mAP, which is defined as the mean of all mAP values computed with tIoU thresholds between and (inclusive) with a step size of .

Top Results. Table 2 shows the top-3 submissions. Please refer to challenge website for the full version of this summary document which includes a copy of all the papers submitted to the workshop.

Rank Organization Average mAP
1 Shanghai Jiao Tong University 38.53
2 YH Technologies 35.49
3 Baidu Vis 35.27
Table 2: The top-3 submissions for task 2.

2.3 Task 3: Dense-Captioning Events in Videos

Description and Objective. Most natural videos contain numerous events. For example, in a video of a man playing a piano, the video might also contain another man dancing or a crowd clapping. This task aims to tackle the challenges of dense-captioning events, which involves both detecting and describing events in a video.

Dataset. This task is evaluated on the ActivityNet Captions dataset [4]. The dataset connects videos to a series of temporally annotated sentence descriptions. Each sentence covers a unique segment of the video, describing multiple events that occur. These events may occur over very long or short periods of time and are not limited in any capacity, allowing them to co-occur. On average, each of the K videos in ActivityNet Captions contains temporally localized sentences, resulting in a total of K sentences. The number of sentences per video follows a relatively normal distribution. Furthermore, as the video duration increases, the number of sentences also increases. Each sentence has an average length of words, which is also normally distributed.

Evaluation Metric. Inspired by the dense-image captioning metric, we use a similar metric to measure the joint ability of our model to both localize and caption events. This metric computes the average precision (AP) across tIoU thresholds of , , and , when captioning the top- proposals. We measure precision of captions using the traditional evaluation metrics: Bleu, METEOR and CIDEr.

Top Results. Table 3 shows the top-2 submissions. Please refer to challenge website for the full version of this summary document which includes a copy of all the papers submitted to the workshop.

Rank Organization Average Meteor
1 RUC and CMU 8.53
2 YH Technologies 8.13
Table 3: The top-2 submissions for task 3.

3 Hosted Challenge Tasks

In this installment of the challenge, we hosted three guest tasks to enrich the understanding of visual information in videos. These guest tasks focused on complementary aspects of the activity recognition problem at large scale and involved three challenging and recently compiled datasets: the Kinetics-600 dataset [3] from Google DeepMind, the AVA dataset [2] from Berkeley and Google, and the Moments in Time dataset [5] from MIT and IBM Research.

3.1 Task A: Trimmed Activity Recognition

Description and Objective. This task is intended to evaluate the ability of algorithms to recognize activities in trimmed video sequences. Here, videos contain a single activity, and all the clips have a standard duration.

Dataset. This task is evaluated on the Kinetics-600 dataset [3]. Kinetics is a large-scale, high-quality dataset of YouTube video URLs which include a diverse range of human focused actions. The dataset consists of approximately K video clips, and covers human action classes with at least video clips for each action class. Each clip lasts around 10s and is labeled with a single class. All of the clips have been through multiple rounds of human annotation, and each is taken from a unique YouTube video. The actions cover a broad range of classes including human-object interactions such as playing instruments, as well as human-human interactions such as shaking hands and hugging.

Evaluation Metric. We use the top- accuracy on the testing set as the official metrics for this task. For each video, an algorithm should produce labels , . The ground truth label for the video is . The error of the algorithm for that video would be: with if and otherwise. The overall error score for an algorithm is the average error over all videos. We will use and and the winner of the challenge is selected based on the average of these two errors.

Top Results. Table 4 shows the top-3 submissions. Please refer to challenge website for the full version of this summary document which includes a copy of all the papers submitted to the workshop.

Rank Organization Average Error
1 Baidu Vis 10.99
2 YH Technologies 11.69
3 QINIU and SARI 12.20
Table 4: The top-3 submissions for task A.

3.2 Task B: Spatio-temporal Action Localization

Description and Objective. This task is intended to evaluate the ability of algorithms to localize human actions in space and time. Each labeled video segment can contain multiple subjects, each performing potentially multiple actions. The goal is to identify these subjects and actions over continuous video clips. This task is divided into two tracks. Track #1 is strictly computer vision, i.e. participants are requested not to use signals derived from audio, metadata, etc. Track #2 lifts this restriction, allowing creative solutions that leverage any input modalities.

Dataset. This task is evaluated on the AVA Dataset version v2.1 [2]. The AVA dataset densely annotates atomic visual actions in -minute movie clips, where actions are localized in space and time, resulting in M action labels with multiple labels per human occurring frequently. Clips are drawn from contiguous segments of movies, to open the door for temporal reasoning about activities. The dataset is split into videos for training, videos for validation, and videos for test.

Evaluation Metric. We use the Frame-mAP at spatial IoU as the metric for evaluating algorithms on this task. Since action frequency in AVA follows the natural distribution, the metric is averaged across the top most common action classes in AVA.

Top Results. Tables 5 and 6 show the top-3 submissions for each track. Please refer to challenge website for the full version of this summary document which includes a copy of all the papers submitted to the workshop.

Rank Organization mAP@0.5IoU
1 Tsinghua University 21.08
2 Google DeepMind 21.03
3 YH Technologies 19.60
Table 5: The top-3 submissions for task B (computer vision only track).
Rank Organization mAP@0.5IoU
1 Tsinghua University 20.99
2 YH Technologies 19.60
3 UMD 16.76
Table 6: The top-3 submissions for task B (full track).

3.3 Task C: Trimmed Event Recognition

Description and Objective. This task is intended to evaluate the ability of algorithms to classify events in trimmed -second videos. Here, videos contain a single activity, and all clips have a standard duration of seconds. This task is divided into two tracks. The first track uses the Moments in Time dataset [5], a new large-scale dataset for video understanding, which has K videos in the training set. The second track use the Moments in Time Mini dataset, a subset of Moments in Time with k videos provided in the training set.

Dataset. This task is evaluated on the Moments in Time Dataset [5]. Moments in Time Dataset is a large-scale collection of M -second videos corresponding to spatial-audio-temporal events.

Evaluation Metric. We use the top- accuracy on the testing set as the official metrics for this task. For each video, an algorithm should produce labels , . The ground truth label for the video is . The error of the algorithm for that video would be: with if and otherwise. The overall error score for an algorithm is the average error over all videos. We will use and and the winner of the challenge is selected based on the average of these two errors.

Top Results. Tables 7 and 8 show the top-3 submissions for each track. Please refer to challenge website for the full version of this summary document which includes a copy of all the papers submitted to the workshop.

Rank Organization Average Accuracy
1 Hikvision 52.91
2 Megvii 51.26
3 Qiniu AtLab 50.06
Table 7: The top-3 submissions for task C (full track).
Rank Organization Average Accuracy
1 Sun Yat-Sen University 47.72
2 Beihang University 45.49
3 National Taiwan University 45.10
Table 8: The top-3 submissions for task C (mini track).

References

  • [1] F. Caba Heilbron, V. Escorcia, B. Ghanem, and J. Carlos Niebles. ActivityNet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
  • [2] C. Gu, C. Sun, S. Vijayanarasimhan, C. Pantofaru, D. A. Ross, G. Toderici, Y. Li, S. Ricco, R. Sukthankar, C. Schmid, et al. Ava: A video dataset of spatio-temporally localized atomic visual actions. CVPR, 2018.
  • [3] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman. The kinetics human action video dataset. CoRR, abs/1705.06950, 2017.
  • [4] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles. Dense-captioning events in videos. In International Conference on Computer Vision (ICCV), 2017.
  • [5] M. Monfort, B. Zhou, S. A. Bargal, T. Yan, A. Andonian, K. Ramakrishnan, L. Brown, Q. Fan, D. Gutfruend, C. Vondrick, et al. Moments in time dataset: one million videos for event understanding.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
254346
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description