Are All the Frames Equally Important?

It seems obvious that some fragments of a video are more important than others. Such fragments concentrate most of the viewer’s attention while others remain of no interest. The naïve examples are: a culmination scene in a movie, a screamer in a horror film, the moment of an explosion, or even a slight motion in very calm footage. We denote such fragments as groups of frames with high temporal saliency. Information about temporal saliency is an essential part of a video characterization which gives valuable insights about the video structure. Such information is directly applicable in video compression (frames which do not attract attention may be compressed more), video summarization (salient frames contain the most of perceived video content), indexing, memorability prediction, and others tasks. So, the reader may expect that there is a big number of algorithms and techniques aimed at measuring and predicting temporal saliency. However, this is not the case. The most, if not all, of the well-known works on video saliency are aimed at spatial saliency, i.e., a prediction of spatial distribution of the observer’s attention across the frame (in a similar way as if it was an individual image). We hypothesize that this is due to the absence of established methodology for measuring temporal saliency in the experiment, which is crucial for obtaining ground truth data. Conventionally, saliency data are collected using eye-tracking, which is a technique that produces a continuous temporal signal. In other words, it does not allow to differentiate between the frames as a whole, because each frame produces the same kind of output – a pair of gaze fixation coordinates with a rate defined by hardware.
In this work, we propose a new methodology for measuring temporal video saliency in the experiment – the first, to the best of our knowledge, method of this kind. For this, we develop a special interface based on mouse-contingent moving-window approach for measuring saliency maps of static images. We also show that it can simultaneously gather meaningful spatial information which can serve as an approximation of gaze fixations.
During the experiment, observers are presented with repeated blurry video-sequences which they can partially deblur using mouse click (Fig. 2). Users can deblur a circular region with a center at cursor location which approximates the confined area of focus in the human eye fovea surrounded by a blurred periphery [3]. Since the number of clicks is limited - observers are forced to use clicks only on most “interesting” frames which attract their attention. Statistical analysis of the collected clicks allows to assign the corresponding level of importance to each frame. This information can be applied directly in numerous tasks of video processing.
To summarize, unlike the conventional approaches which only try to understand where the observer looks, we also study when the observer pay the most attention.

The straightforward method of retrieving the information about attention is based on the utilization of commercial eye-trackers (e.g. EyeLink, Tobii). Hardware-based eye-tracking has been used widely in various studies on human-computer interaction [6][13]. A less accurate, but much more affordable, way of measuring saliency is based on measuring the mouse cursor position which was proven to correlate strongly with gaze fixations [4][5][15]. The most successful algorithms of this type utilize a moving-window paradigm, which masks information outside of the area adjacent to the cursor and requires a user to move the cursor (followed by a window around it) to make other regions visible. Such algorithms include Restricted Focus Viewer software by Jansen et al. [7] and more recent SALICON [8] and BubbleView [10]. These algorithms were also used in large online crowdsourcing experiments due to the native scalability of cursor-based approaches. However, they were studied only in the context of spatial saliency of static images. This is fair for static images, but for video-sequences, temporal information is commonly even more important than spatial regions. Furthermore, there are no well-known experimental datasets which can provide this kind of information¹ and be used for training of computational algorithms. For example, the popular video saliency datasets Hollywood-2 [16], UCF sports [12], SAVAM [2], DHF1K [17] only provide eye-tracking results which are constant in the temporal domain.

[208 pt] The proposed interface. A more representative video demonstration is available online: [link].

Our approach is inspired by moving-window gaze approximations methods for still images. In the proposed setup all video frames are blurred. Clicking the mouse deblurs a round window around the cursor. Users are demonstrated repeated video sequence during which they can click the mouse for short periods of time. The total number of times when the frame was deblurred defines temporal saliency score, while location of the cursor when the mouse button is pressed approximates gaze fixation location and allows to detect what caused the interest.

The experiments were performed offline using a special setup in the laboratory (Fig. 3.3) for the sake of fully-controlled conditions (in future we are also planning to run the experiment on Amazon Mechanical Turk for gathering larger database, which would be impossible to do with an eye-tracker). The display used is 24.1″ EIZO ColorEdge CG241W color-calibrated with X-Rite Eye-One Pro. The distance between the display and the observer was 50 cm.
The code is written in MatLab with Psychtoolbox-3 [11] and is publicly available by the link ².
Videos with ground-truth eye-tracking data were taken from SAVAM dataset [2] due to their high quality and diverse content. We used eight 10-seconds long HD videos including two test videos. The content of the videos is diverse and includes: a basketball game with a score moment, a calm shot of leaves in the wind, marine animals underwater, a cinematic scene of a child coming home, a surveillance camera footage of two men meeting, a suffocating diver emerging from the water.
Interface parameters: radius of a circular window – 200 px (${6.2}^{\circ }$ visual angle), blur kernel – Gaussian with standard deviation of 15, video duration – 10 s, limit of deblurred frames per one round – 4 s (100 frames), limit of deblurred frames at one click – 1 s (25 frames), number of repetitions – 5, frame-rate of the videos – 25 fps, video resolution – 1280 px $\times$ 720 px (${38.2}^{\circ }\times {22}^{\circ }$ visual angle), videos are silent.
The observers were invited from the University staff and students. 30 subjects in total, 15 women and 15 men. Age: 21-42 (mean 25.6).

The proposed interface allows measuring both temporal and spatial saliency at the same time, thus, we evaluate the accuracy of both these outputs.

In this work, we presented a novel mouse-contingent interface designed for measuring temporal and spatial video saliency. Temporal saliency is a novel concept which is studied incongruously less than it should in comparison to spatial saliency. Temporal video saliency allows identifying the important fragments of a video by assigning a saliency score to each frame. The analysis of the experimental study shows that the use of the proposed interface allows to accurately approximate the temporal saliency “map” as well as gaze-fixations of the observers at the same time.