Multimodal Matching Transformer for Live Commenting

Namely, the input representation of the Transformer network contains two parts: word embeddings $\{e_{w_1},e_{w_2},...,e_{w_n}\}$ and positional embeddings $\{e_{p_1},e_{p_2},...,e_{p_n}\}$, where $n$ is the length of the input sentence. The two parts are fused by an addition operation.

After obtaining outputs of previous layers, the Transformer network uses a multi-head attention mechanism to learn the context-aware representation for the sequence.

Figure 2 shows the architecture of the multi-head attention. $Q$, $K$ and $V$, three matrices, are inputs derived from previous layers and $H$ is the output. The Multi-head attention can be denoted as:

We formulate the automatic live commenting as a ranking problem. Formally, given a video $V$ and a time-stamp $t$, automatic live commenting aims to select a comment $y^{\ast }$ from a candidate set $Y$, which is most relevant to the video clip near the time-stamp based on the surrounding comments $C$, the visual part $F$ and the audio part $A$. Concretely, we extract $N_c$ comments near the time-stamp $t$ as $C=\{c_1,c_2,...,c_{N_c}\}$, where $c_i$ is a comment. For the vision part $F$, we sample $N_f$ video frames near the time-stamp $t$ as $F=\{f_1,f_2,...,f_{N_f}\}$, where $f_i$ is an image and the interval between two images is 1 second. We convert a 5-second audio clip surrounding the time-stamp $t$ to a log magnitude mel-frequency spectrogram, and it can be denoted as $A=\{a_1,a_2,...,a_{N_a}\}$, where $a_i$ is a vector and $N_a$ is the length of the audio clip. The candidate comment can be denoted as $y=\{y_1,y_2,...,y_k\}$, where $y_i$ is a word and $k$ is the number of words.

Figure 3 shows the architecture of the Matching Transformer. The model consists of three components: (1) an encoder layer converts different modalities of a video clip (including surrounding comments, the vision part of the video, the audio part of the video) and a candidate comment into vectors; (2) a matching layer iteratively learns the attention-aware representation for each modality; (3) a prediction layer outputs a score measuring the matching degree between a video clip and a comment.

As shown in part (a) of Figure 3, our model contains three kinds of encoder: a comment encoder, a vision encoder and an audio encoder. These encoders convert a comment, a vision clip and an audio clip into vectors respectively. In our model, the existing surrounding comments $C$ and the candidate comment $y$ share the same comment encoder.

Inspired by the recent successful deep learning frameworks [13, 20], we adopt a matching layer which consists of $N$ matching blocks to iteratively learn the attention-aware representation for each modality. The structure of a matching block is shown in part (b) of Figure 3. Each matching block is composed of four parts: a multi-head self-attention, a multi-head cross attention and two position-wise FNN. Compared to the basic block defined in [20], our matching block adds a multi-head cross attention and a position-wise FNN. We use these auxiliary mechanisms to learn attention-aware representation from other modalities.

For simplicity, we take the candidate comment as the example to illustrate the matching layer.

As described above, Eq. (9)-Eq. (16) illustrate how to compute the representation of candidate comment $H_y^t$. In implementation, we adopt the same way to compute the representations of surrounding comment $H_c^t$, vision clip $H_f^t$ and audio clip $H_a^t$.

We evaluate our model on a live commenting dataset¹ that is released by [16]. The live commenting dataset is a large-scale video-comment dataset. It contains 2,361 videos and 895,929 comments. The data is collected from a popular Chinese video streaming website, Bilibili. Therefore, it has strong authenticity and practicability. In our experiment, we use the same partition as in [16]. The detailed statistics of the dataset is shown in Table 1.

Following the previous work [8, 16], we adopt Recall@k, Mean Recall (MR) and Mean Reciprocal Rank (MRR) for automatic evaluation, which are standard evaluation metrics of the ranking task. For testing, we construct a candidate comments set in which each video clip contains 100 comments, which is exactly the same as the previous work [16] for fair comparison. The comments in the candidate comment set are comprised of three parts: (1) the ground-truth comments; (2) top 20 popular comments; (3) random selected comments. We evaluate our model on the testing set.

In addition, we also test the performance of our model by human evaluation. Following [16], we use the metrics of relevance (Rel) and correctness (Cor) to evaluate our model. Relevance measures the relevance between produced comments and videos and correctness measures the confidence that produced comments are made by human in the context of the video. We do not evaluate the fluency of the produced comments, because our model just selects a proper comment from a candidate comment set, which is naturally fluent. For both relevance and correctness, we use a score $s\in \{1,2,3,4,5\}$ to denote the degree, the higher the better. When testing, three human annotators are asked to give a score to evaluate the top one comment produced by our model and we use the average score as the final result.

In our experiments, the word embeddings and video frame vectors are in 512 dimensions while audio frame vectors are in 64 dimensions. The GRU for audio encoder is in 512 dimensions. For positional embedding, we use fixed sinusoidal positional embedding and set the dimension as 512. The word embeddings are randomly initialized and updated during training, while the video frame vectors and audio frame vectors are fixed. There are 6 matching blocks in the matching layer. In each matching block, the number of heads in the multi-head attention is 8 and the dimension of the position-wise FNN is 2,048. The margin $M$ is set to 0.1 in our experiment. We employ the Adam [14] for training, whose default hyper-parameters ${\beta }_1$ and ${\beta }_2$ are set to 0.9 and 0.999 for optimization respectively. The initial learning rate of Adam is set to 0.00009. The learning rate is halved when the accuracy on the development set drops. We also employ a dropout strategy [18] and layer normalization [4] to reduce the risk of over-fitting. The dropout rate is set to 0.2 and the batch size is 64.

For pre-processing, we use the Stanford tokenizer [17] to tokenize the comments and audio feature extractor² released by [12] to process the audio clip. During training, we draw 1 negative sample for each video clip.

• S2S [21] is a traditional sequence to sequence model without the attention mechanism. Specifically, the model uses two encoders to encode visual and textual information respectively. During decoding, the decoder uses the concatenation of the outputs from the two encoders as input.

• Fusional RNN [16] consists of three parts: a video encoder, a comment encoder and a comment decoder. The three parts are all RNN-based networks and they are related by an attention layer. This model uses the visual and textual context as input.

• Unified Transformer [16] is a transformer-based generative model. Similar to Fusional RNN, this model is comprised of three parts: a video encoder, a comment encoder and a comment decoder. The difference is that these three parts are all stacked attention-based transformer blocks.

Table 2 shows the automatic evaluation results of the baseline models and our proposed models. The baselines (S2S, Fusional RNN and Unified Transformer) only use the text and vision of the videos. Our matching transformer leverages three modalities (text, vision, and audio) and significantly outperforms the baselines in terms of all metrics. Moreover, we also report the results of our model with only one modality (less than baselines) and two modalities (equal to baselines). It shows that the matching transformer achieves comparable performance with the baselines using only one modality. Meanwhile, our model achieves better performance than baselines by using the same two modalities, which verifies the efficiency of the proposed model. Finally, the triple-modality model significantly outperforms the baselines, achieving +5.51 points on Recall@1, +8.87 points on Recall@5 and +8.46 points on Recall@10.

We also would like to know how different kinds of modalities contribute to our proposed model. Therefore, we conduct ablation experiments by removing different modalities from our model. Table 3 summarizes the results of the ablation experiments. Under the single-modality setting, it shows that the model with the modality of text achieves better performance over the other two modalities. Among the possible alternatives of double-modality, the combination of text and vision obtains the best performance. Finally, the model with triple-modalities get the highest scores in terms of all the automatic metrics. Besides, it is observed that more modalities always lead to better performance, which indicates that the proposed model can capture the semantic information of different modalities to help the live commenting task.

We randomly sample 100 video clips from the test set to evaluate our model in terms of the relevance and the correctness. For both metrics, we use a score range from 1 to 5 to denote the degree, where the higher the better. We have three human annotators to give a score that evaluates the top one comment produced by our model, and we use the average score as the reported result.

The result is shown in Table 4. It shows that the produced comments of our model are more relevant than those of the baseline models. Besides, our model can produce more correct and proper comments. The scores of both relevance and correctness degrees are also closer to that of the comments made by human. This result indicates that our model is able to produce the relevant comments to the videos by modeling the relevance in different modalities.

To further compare our model with the baselines, we provide an example for case study. This example is talking about a Chinese food called soup dumplings. As illustrated in Figure 4, this example consists of three frames, three surrounding comments and three target comments. Since the audio is not visible, we do not provide the audio part of the video. The surrounding comments are in the first row of the table below the three frames. The second row contains three target comments which are naturally made by human viewers and correspond to a specific time-stamp. We compare the produced comments of different models with the target comments. It shows that when we select the top one output as the produced comment, both unified transformer and matching transformer can produce a comment relevant to the target comments. However, the produced comments of both fusional RNN and S2S are of low relevance to the video. The output of S2S is talking about eggs and the output of fusional RNN is about dancing, both of which are far away from the video clip. Furthermore, we compare the produced comments between matching transformer and unified transformer. According to the case in Figure 4, it is obvious that the comment from the matching transformer is more relevant to the video clip. The matching transformer can make comments about the soup dumplings that are exactly the key point of the video clip, while the unified transformer can only produce comments about how to process the dirty soup dumplings that fell on the ground, which do not appear in the video. In conclusion, the comments made by our matching transformer are more relevant and correct than that of other baselines.