Multimodal Transformer with Multi-View
Visual Representation for Image Captioning
Image captioning aims to automatically generate a natural language description of a given image, and most state-of-the-art models have adopted an encoder-decoder framework. The framework consists of a convolution neural network (CNN)-based image encoder that extracts region-based visual features from the input image, and an recurrent neural network (RNN)-based caption decoder that generates the output caption words based on the visual features with the attention mechanism. Despite the success of existing studies, current methods only model the co-attention that characterizes the inter-modal interactions while neglecting the self-attention that characterizes the intra-modal interactions. Inspired by the success of the Transformer model in machine translation, here we extend it to a Multimodal Transformer (MT) model for image captioning. Compared to existing image captioning approaches, the MT model simultaneously captures intra- and inter-modal interactions in a unified attention block. Due to the in-depth modular composition of such attention blocks, the MT model can perform complex multimodal reasoning and output accurate captions. Moreover, to further improve the image captioning performance, multi-view visual features are seamlessly introduced into the MT model. We quantitatively and qualitatively evaluate our approach using the benchmark MSCOCO image captioning dataset and conduct extensive ablation studies to investigate the reasons behind its effectiveness. The experimental results show that our method significantly outperforms the previous state-of-the-art methods. With an ensemble of seven models, our solution ranks the 1st place on the real-time leaderboard of the MSCOCO image captioning challenge at the time of the writing of this paper.
Recent advances in deep learning have resulted in great progress in both the computer vision and natural language processing communities. These achievements make it possible to connect vision and language, and facilitate multimodal learning tasks such as image-text matching , visual question answering , visual grounding  and image captioning .
Image captioning aims to automatically describe an image’s content using a natural language sentence. The task is challenging since it requires one to recognize key objects in an image, and to understand their relationships with each other. Most successful image captioning approaches adopt the encoder-decoder framework, which is inspired by the sequence-to-sequence model for machine translation . The framework consists of a convolutional neural network (CNN)-based image encoder that extracts region-based visual features from an input image, and an recurrent neural network (RNN)-based caption decoder that iteratively generates the output caption words based on the visual features. The encoder-decoder model is usually trained in an end-to-end manner to minimize the cross-entropy loss. Based on the framework, plenty of improvements have been made by recent works to further improve image captioning performance further. For instance, to establish the fine-grained connections of caption words and their related image regions, an attention mechanism can be seamlessly inserted into the framework . To provide a better understanding of the objects in the image, region-based bottom-up-attention features can be extracted from a pre-trained object detector to replace the traditional CNN convolutional features . To address the exposure bias of generated captions by using the cross-entropy loss, reinforcement learning (RL)-based algorithms are designed to directly optimize the non-differentiable evaluation metrics (e.g., BLEU  and CIDEr ) .
Despite the success that existing approaches have achieved, they have the following limitations: 1) the current attention mechanism in image captioning only models the co-attention that characterizes inter-modal interactions (i.e., object-to-word) while neglecting the self-attention that characterizes intra-modal interactions (i.e., word-to-word and object-to-object); 2) current image captioning models are usually shallow and may fail to fully understand the complex relationships among visual objects; and 3) the region-based visual features may fail to cover all objects in the image, leading to insufficient visual representations for generating accurate captions.
To address the first and second limitations, we extend the Transformer model for machine translation  to a Multimodal Transformer (MT) model for image captioning. Different from the CNN-RNN captioning models, the MT model does not use RNN and instead relies entirely on an attention mechanism to assess the global dependencies between the input and output. By properly stacking such attention blocks in depth, MT forms a deep encoder-decoder model that simultaneously captures the self-attention within each modality and the co-attention across different modalities. To address the last limitation, we introduce multi-view feature learning into the MT model to adapt both the aligned and unaligned multi-view visual features.
To summarize, the main contributions of this study are three-fold:
The joint modeling of the self-attention and the co-attention interactions for image captioning is first proposed in the MT model. The MT model is capable of modeling three types of relationships using a modular attention block, i.e., word-to-word, object-to-object, and word-to-object. By stacking such attention blocks in depth, the deep MT model significantly outperforms the state-of-the-art models, thereby highlighting the importance of deep reasoning for image captioning.
Multi-view learning on the image is introduced in conjunction with the MT model to provide more diverse and discriminative visual representations. We introduce two alternative strategies to handle aligned and unaligned multi-view features, respectively.
Extensive experiments on the benchmark MSCOCO image captioning dataset are conducted to quantitatively and qualitatively prove the effectiveness of the proposed models. The experimental results show that the MT significantly outperforms previous state-of-the-art approaches with a single model. Furthermore, our solution ranks the 1st place on the real-time leaderboard of the MSCOCO image captioning challenge with an ensemble of MT models.
The rest of the paper is organized as follows: In section II, we review the related work of image captioning approaches, especially the ones introducing attention mechanisms. In section III, we revisit the basic Transformer model and then propose the Multimodal Transformer model for image captioning. In section IV, we introduce multi-view image representation into the MT model to increase the visual representation capacity, and the quality of the generated captions. In section V, we introduce our extensive experimental results for algorithm evaluation and use the benchmark MSCOCO image captioning dataset to evaluate our proposed approaches. Finally, we conclude this work in section VI.
Ii Related Work
In this section, we briefly review the most relevant research on image captioning, especially those studies that introduce attention models.
Ii-a Image Captioning
The research on image captioning can be categorized into the following three classes: template-based approaches , retrieval-based approaches , and generation-based approaches .
The template-based approaches address the task using a two-stage strategy: 1) align the sentence fragments (e.g., subject, object, and verb) with the predicted labels from the image; and 2) generate the sentence from the segments using pre-defined language templates. Kulkarni et al. use the conditional random field (CRF) model to predict labels based on the detected objects, attributes, and prepositions, and then generate caption sentences with a template by filling in the blanks with the most likely labels . Yang et al. employ the HMM model to select the best objects, verbs, and prepositions with respect to the log-likelihood for segments generation . Intuitively, the captions that are generated by the template-based approaches highly depend on the quality of the templates and usually follow the syntactical structures. However, the diversity of the generated captions is severely restricted.
To ease the diversity problem, retrieval-based approaches are proposed to search the most relevant captions from a large-scale caption database with respect to their cross-modal similarities to the given image. Karpathy et al. propose a deep fragment embedding approach to match the image-caption pairs based on the alignment of visual segments (the detected objects) and caption segments (subjects, objects, and verbs) . In the testing stage, the cross-modal matching over the whole caption database (usually the captions from the training set) is performed to generate the caption for one image. Other methods such as  use different metrics or loss functions to learn the cross-modal matching model. However, the retrieval efficiency becomes a bottleneck for these approaches when the caption database is large and restricting the size of the database may reduce the caption diversity. Moreover, retrieval-based approaches cannot generate novel captions beyond the database, which means the diversity problem has not been completely resolved.
Different from template-based and retrieval-based models, generation-based models aim to learn a language model that can generate novel captions with more flexible syntactical structures. With this purpose, recent works explore this direction by introducing the neural networks for image captioning. Vinyals et al. propose an encoder-decoder architecture by utilizing the GoogLeNet  and LSTM networks  as its backbones. Similar architectures are also proposed by Donahue et al.  and Karpathy et al. . Due to the flexibility and excellent performance, generation-based models have become the mainstream for image captioning.
Ii-B Attention Mechanism
Within the encoder-decoder framework, one of the most important improvements for generation-based models is the attention mechanism. Xu et al. introduce the soft and hard attention models to mimic the human eye focusing on different regions in an image when generating different caption words. The attention model is a pluggable module that can be seamlessly inserted into previous approaches to remarkably improve the caption quality. The attention model is further improved in . Anderson et al. introduce a bottom-up module, that uses a pre-trained object detector to extract region-based image features, and a top-down module that utilizes soft attention to dynamically attend to these object . Chen et al. propose a spatial- and channel-wise attention model to attend to visual features . Lu et al. present an adaptive attention encoder-decoder model for automatically deciding when to rely on visual or language signals . Rennie et al. design a FC model and an Att2in model that achieve good performance .
Beyond the image captioning tasks, attention mechanisms are widely used in other multi-modal learning tasks such as visual question answering (VQA). Lu et al. propose a co-attention learning framework to alternately learn the image attention and question attention . Yu et al. reduce the co-attention method into two steps, self-attention for a question embedding and the question-conditioned attention for a visual embedding . Nam et al. propose a multi-stage co-attention learning model to refine the attentions based on the memory of previous attentions . However, these co-attention models learn separate attention distributions for each modality (image or question) and neglect the dense interaction between each question word and each image region, which becomes a bottleneck for understanding the fine-grained relationships of multimodal features. To address this issue, dense co-attention models have been proposed, which establish the complete interaction between each question word and each image region . Compared to the previous co-attention models with coarse interactions, the dense co-attention models deliver significantly better VQA performance.
Iii Multimodal Transformer
In this section, we first briefly describe the preliminary knowledge of the Transformer model . Then, we introduce the proposed Multimodal Transformer (MT) framework for image captioning, which consists of an image encoder and a caption decoder. The image encoder learns the deep image representation in a self-attention manner, and then, the caption decoder uses the attended image representations to generate textual captions.
Iii-a The Transformer Model
The Transformer model  was first proposed for machine translation, and has been successfully applied to many natural language processing tasks. We first introduce the scaled dot-product attention, which is the core component of the Transformer.
The input of the scaled dot-product attention consists of a query , a set of keys and values , where is the number of key-value pairs and is the common dimensionality of all the inputs features. We calculate the dot products of query with all keys, divide each by and apply a softmax function to obtain the attention weights on the values. In practice, we pack all the keys and values into matrices and respectively. The attention function on a set of queries can be computed in parallel as follows:
where correspond to the attended features of the queries .
Instead of performing a single attention function for the queries, multi-head attention is introduced in  to allow the model to attend to diverse information from different representation subspaces. The multi-head attention contains parallel ‘heads’ with each head corresponding to an independent scaled dot-product attention function. The attended features of the multi-head attention functions is given as follows:
where are the projection matrices of the -th head. is the output projection matrix that aggregates the information from different heads. is the dimensionality of the output features of each head. To prevent the model from becoming too large, we set .
In addition to the multi-head attention (MHA), another basic component in the Transformer is the feed-forward networks (FFN). FFN takes the input from MHA and further transform it using two linear layers with the ReLU activation and dropout  in between as follows:
The Transformer is a deep end-to-end architecture that stacks attention blocks to form an encoder-decoder strategy (see Fig. 1). Both the encoder and the decoder consist of attention block, and each attention block contains the MHA and FFN modules. The MHA module learns the attended features that consider the pairwise interactions between two input features, and the FFN module further nonlinearly transforms the attended features. In the encoder, each attention block is self-attentional such that the queries, keys and values in Eq.(1) refer to the same input features. In contrast, the attention block in the decoder contains a self-attention layer and a guided-attention layer. It first models the self-attention of given input features and then takes the output features of the last encoder attention block to guide the attention learning. To simplify the optimization, shortcut connection  and layer normalization  are applied after all the MHA and FFN modules.
Iii-B Multimodal Transformer for Image Captioning
Based on the preliminary information about the Transformer above, we describe the Multimodal Transformer (MT) architecture for image captioning, which consists of an image encoder and a textual decoder. The image encoder takes an image as its input and uses a pre-trained Faster-RCNN model  to extract region-based visual features. The visual features are then fed into the encoder to obtain the attended visual representation with self-attention learning. The decoder takes the attended visual features and the previous word to predict the next word recursively. The flowchart of the MT architecture is shown in Fig. 2.
Image Encoder. The input image is represented as a group of visual features that are extracted from a pre-trained object detector . Specifically, the detector is a Faster-RCNN model  that is pre-trained on the Visual Genome dataset . We sort the detected objects w.r.t. their confidence scores in descending order and keep the top- objects. Each object is represented as a feature vector by mean-pooling the convolutional feature from its detected region. Finally, the image is represented as a feature matrix .
The visual features is first fed into a fully-connected layer to adapt the feature dimensionality to the encoder. The projected features (denote as ) are then fed into the encoder with attention blocks . The attention block takes the output features from the attention block, and output their attended features in a recursive manner.
where each consists of a MHA module and a FFN module with independent model weights (see Fig. 1).
Caption Decoder. Based on the visual representations from the encoder, the textual decoder generates captions for the image. The input caption is first tokenized into words and trimmed to a maximum length of words. Each word in the caption is first represented as a word vector by using the 300-D GloVe word embedding  pre-trained on a large-scale corpus. We use a feature matrix to represent a caption sentence. For the captions that are shorter than 16 words, we use zero-padding to fill them to the maximum size. To model the temporal information of the captions, the word embeddings are then pass through a one-layer LSTM network  with hidden units, resulting in caption representations .
In the training stage, the caption decoder takes the inputs from both the image encoder and caption representations. Given the attended image features and the caption input features , the caption decoder with attention blocks () learns to predict the attended word features in an analogous manner to the strategy in the encoder.
where each consists of two MHA modules and one FFN module (see Fig. 1). The first MHA module models the self-attentions on the caption words and the second MHA module learns the image-guided attention on the caption words. Note that the self-attention (i.e., the first MHA module) is only allowed to attend to earlier positions in the output sequence and is implemented by masking subsequent positions (setting them to -) before the softmax step in the self-attention calculation, thereby resulting in a triangular mask matrix . The output features are fed into a linear word embedding layer to transform the features to a -dimensional space, where is the vocabulary size. Subquently, softmax cross-entropy loss is performed on each word to predict the probability of its next word.
In the testing stage, the caption is generated word-by-word in a sequential manner. When generating the word, the input features are represented as , where corresponds a zero-padded feature. The input caption features along with the image features are fed forward the model to obtain the word with the largest probability among the whole word vocabulary. The predicted word is then integrated into the inputs to recursively generate the new inputs . To improve the diversity of generated captions, we also introduce the beam search strategy during the testing stage.
Iv Image Encoder with Multi-view Visual Representation
In this section, we introduce multi-view image representations and modify the the image encoder in section III-B to multi-view image encoder to facilitate the representation capacity of the MT model. Though it has been intensively investigated by previous works ,, existing multi-view learning approaches focus on integrating the global multi-view features (e.g., color histogram or GIST descriptor) from the whole image. The global multi-view features may fail to preserve the fine-grained semantics of the image, thus leading to incorrect caption. In contrast, we extract region-based local multi-view features to represent the image. Each object detector (i.e., pre-trained Faster R-CNN models with different backbones) is regarded as one single view.
Note that the objects extracted from different detectors are naturally unaligned, thereby making it challenging to learn the correspondence across different views. To address this problem, we extend the proposed image encoder model in section III-B, and introduce two multi-view image encoder models, namely, the Aligned Multi-View (AMV) image encoder and the Unaligned Multi-View (UMV) image encoder, respectively.
Iv-a Aligned Multi-View Image Encoder
The AMV model uses a simple strategy to obtain the aligned multi-view features from different object detectors. Rather than extracting the object bounding boxes and corresponding features for each view, we propose a two-stage feature extraction framework. Given pre-trained Faster R-CNN models, we first select one detector as the primary model to generate the unified bounding boxes for all views. The choices of different primary models has little influence on the quality of the generated features, and we simply choose the model with the highest detection performance. Subquently, the unified bounding boxes are used to extract features from different Faster R-CNN models. Specifically, the Faster R-CNN models degenerate to their Fast R-CNN versions  that take the pre-computed bounding boxes as inputs. The resulting multi-view features are aligned such that each paired multi-view features correspond to one object in the image.
Assuming that we generate unified bounding boxes, the extracted features from the -th view () can be represented as , where is the dimensionality of the features. By simply concatenating the features in columns, we obtain the multi-view features . These aligned multi-view features can replace the aforementioned single-view feature, and be seamlessly fed into the image encoder. The overall flowchart of the AMV model is shown in Fig. 3.
To align the multi-view features, the AMV model uses the unified bounding boxes. However, we argue that this strategy may harm the diversity of multi-view features, leading to a limited representation capacity of the encoded image features. Moreover, the AMV model implicitly constrains the object detector for each view to be a Faster R-CNN model, which can either take the pre-computed proposals as inputs or generate the proposals using the built-in Region Proposal Networks (RPN) . This constraint limits the usage of one-stage object detectors, e.g., RetinaNet  and YOLO .
Iv-B Unaligned Multi-View Image Encoder
To address the limitations of the AMV encoder model, we propose a more generalized unaligned multi-view (UMV) image encoder model that can directly integrate the unaligned multi-view features from different object detectors (see the flowchart in Fig. 4).
The extracted visual features for the -th view can be represented as , where the number of features and the feature dimensionality can be different across multiple views. The unaligned multi-view features are fed into an encoder to be aligned and fused simultaneously. Specifically, we choose one view as the primary view and use its features to guide the attention learning for other views. The attended features from other views are then integrated into the features in the primary view to output the output features.
Given the multi-view features , they are first linearly projected into a common -dimensional space to obtain their transformed representations . Assuming that corresponds to the features of the primary view, we have MHA modules in total to model the interactions between and with .
where is the attended output features for the -th view. The obtained features have the same shape as , and so they can be integrated with via an element-wise summation. The MHA modules here can be understood as learning the image-guided attention over the image features from other views.
Following the image encoder model in section III-B, the integrated features that are followed by layer normalization  are then fed forward through the FFN module to obtain the transformed representations. It is worth noting that the UMV model can also be stacked in depth to learn more accurate interactions across different views, thus resulting in more discriminative output visual features for generating captions.
In this section, we conduct experiments and evaluate the proposed MT models on MSCOCO 2015 image captioning dataset . Additionally, we use the Visual Genome dataset  to pre-train the object detectors that are further used to extract the bottom-up-attention visual features .
MSCOCO is a benchmark dataset for various computer vision tasks, including object detection, instance segmentation, and image captioning . It contains 83k training images, 40k validation images, and 81k test images. Each image is associated with five captions. Similar to , we use the Karpathy splits  that have been extensively used for reporting results in prior works. These splits merge the images from the original train and val splits, resulting in 121k images in total. After that, the 123k images are split into 113k/5k/5k images for training/validation/testing, respectively. The trained models are ensembled to obtain the predictions that are submitted to the official MSCOCO test server. To evaluate the caption quality, we use four automatic evaluation metrics, namely, BLEU , ROUGE-L , METEOR  and CIDEr .
Visual Genome is a large-scale dataset to evaluate the interactions between objects in the images. It contains 108k images with densely annotated objects, attributes, and relationships. Following the strategies in , we use the object and attribute annotations to pre-train the bottom-up-attention models. All the images are split into training (98k images), validation (5k images) and testing (5k images). Since part of images in Visual Genome are also found in the MSCOCO captioning dataset, we perform careful checking to avoid affecting the MSCOCO validation and testing splits. Similar to , we perform extensive cleaning and filtering of the training data to select 1,600 object classes and 400 attributes. This cleaned dataset is used for training our object detection models.
V-B Implementation Details
For the captions, we perform the pre-processing as follows. All the caption sentences are converted to lower case and tokenized into words with white space. The rare words that occur less than 5 times or do not exist in the pre-trained GloVe vocabulary  are discarded, resulting in a vocabulary of 9,343 words. Each word in the caption is represented as word embedding vector by looking-up the GloVe word vocabulary. The out-of-vocabulary words are represented as all-zero vectors.
For the images, we use the pre-trained bottom-up-attention models to detect the objects and extract visual features for the detected objects. For multi-view image representation, we trained up to three Faster R-CNN  models (i.e., number of views =3) with different backbones, namely ResNet-101 , ResNet-152  and ResNeXt-101 , respectively. For each model, we select the top-100 objects with the highest confidence scores to represent the image, where each object is represented as a vector by mean-pooling the last convolutional feature from its detected region.
The hyper-parameters of the MT models that are used in the experiments are listed as follows. The dimensionality of input image features , and the input caption features are 2048 and 512, respectively. According to the recommendation in , the latent dimensionality in the MHA module is 512, the number of heads is 8, and the latent dimensionality for each head . The number of attention blocks in the encoder and decoder ranges in .
To train the MT models, we use the Adam solver  with a batch size of 10. The base learning rate is set to , where is the current epoch number that starts at 1. After 6 epochs, the learning rate is decayed by 1/2 after every 3 epochs. All models are first trained for 15 epochs using the cross-entropy loss and then are further trained for additional 10 epochs using the self-critical loss to alleviate the exposure bias during cross-entropy optimization .
V-C Ablation Studies
We run a number of ablation experiments on MSCOCO image captioning dataset to explore the effectiveness of the single-view MT models (MT) with different hyper-parameters, as well as its multi-view variants with aligned multi-view image encoder MT and unaligned multi-view image encoder MT. The results shown in Table I are discussed in detail below.
Caption Representations: Table I(a) summarizes the ablation experiments on different caption representations for MT with the number of attention blocks =6. Compared with the reference model that uses randomly initialized word embeddings and positional encoding , we can see that using the word embeddings that are pre-trained by GloVe  brings significant improvements. In addition, introducing other tricks such as replacing PE with an LSTM network to model the temporal information, or fine-tuning the GloVe word embeddings along with the MT model can slightly improve the performance further. Note that the GloVe+LSTM model and the GloVe+LSTM model report the same performance in the cross-entropy loss stage, as the fine-tuning is performed only in the self-critical loss stage. Directly fine-tuning the GloVe embedding from scratch (i.e., from the cross-entropy loss) leads to inferior performance. This result can be explained as the word embeddings being sensitive to the captioning performance, and training from scratch may degenerate their representation capacity.
|Model||Backbone||Cross-Entropy Loss||Self-Critical Loss|
|RFNet ||R, D, I-v3, I-v4 and IR-v2||76.4||35.8||27.4||56.5||112.5||79.1||36.5||27.7||57.3||121.9|
|MT (ours)||R-101, R-152 and X-101*||77.3||37.4||28.7||57.4||119.6||81.9||40.7||29.5||59.7||134.1|
|Google NIC ||71.3||89.5||54.2||80.2||40.7||69.4||30.9||58.7||25.4||34.6||53.0||68.2||94.3||94.6|
Number of Attention Blocks: Table I(b) shows the performance of the MT models with different number of attention blocks . We can see that the model size grows linearly as increases. Regarding the performance, we have two observations as follows: 1) as increasing , the model’s performance gradually improves and is saturated at a certain number. This can be explained as a deeper model capturing more complex relationships among objects, providing a more accurate understanding of the image contents. In addition, a deeper model has a larger representation capacity and has a larger risk to overfit the training set, and 2) the optimal model is achieved at different that are trained with different losses, i.e., =4 for the cross-entropy loss and =6 for the self-critical loss. The reinforcement learning-based self-critical loss provides a more diverse exploration of the hypothesis space to avoid overfitting, and thus it can better utilize the potential of large models.
Single-view vs. Multi-view: Next, we compare the MT model with single-view or multi-view features in Table I(c). We use two Faster R-CNN models with different backbones (ResNet-101 or ResNet-152) to extract the multi-view features. For MT, the unified object boxes are extracted from the detector with the ResNet-152 backbone. From the results, we can see following that: 1) the representation capacity of the object detectors may slightly influence the image captioning performance. The MT model with the ResNet-152 backbone steadily outperforms the counterpart with the ResNet-101 backbone; and 2) introducing multi-view features significantly improves the captioning performance over the single-view models. MT slightly outperforms MT, thus highlighting the effect of using diverse multi-view features with unaligned objects.
Number of Views: In Table I(d), we show the performance of the MT models with different number of views . We can see that when , different backbone combinations have little influence on the captioning performance. Moreover, increasing the number of views from 2 to 3 results in a slight performance improvement for MT, thus indicating that the model is nearly saturated. Therefore, we do not further introduce more views to the image encoder.
V-D Comparison with the State-of-the-Art
By taking the ablation results into account, we compare our best single-view and multi-view MT models to the current state-of-the-art approaches.
Results on the Karpathy test split: In Table II, we report the comparative results of our approaches along with the SCST , ADP-ATT , LSTM-A , Up-Down  and GCN-LSTM  on the Karpathy test split. Note that all the compared methods use the same ResNet-101 backbone. With single-view features, the MT model outperforms most state-of-the-art methods, especially when it is optimized using the self-critical loss. When equipped with multi-view features, the MT model (trained with the self-critical loss) achieves the new state-of-the-art single-model performance for this split in terms of all evaluation metrics. Note that the RFNet  also incorporates multi-view features, and they introduce more views than our approach (4 vs. 2). However, its performance is inferior to MT, which suggests that the strategy to fuse multi-view features, rather than the number of views, is the key to the captioning performance.
Results on the official test server: We also submitted the results of seven MT model ensemble (the MT, MT and MT models with different random seeds) to the official MSCOCO test server.111https://competitions.codalab.org/competitions/3221#results Table III demonstrates the results of the comparison to the state-of-the-art solutions on the leaderboard including the published ones (in the first split) and the unpublished ones (in the second split). C5 (or c40) denotes the official test settings with 5 (or 40) ground-truth captions, respectively. Compared to all the top performing solutions on the leaderboard, our solution significantly outperforms all the other solutions in terms of all reported evaluation metrics at the time of submission (April 21st, 2019).
V-E Qualitative Analysis
To better understand the effectiveness of the proposed approach, we visualize the learned attentions of MT and MT in Fig. 5 and Fig. 6, respectively. Due to space limitations, we only show one example for each model and visualize the attention maps from typical attention blocks. From the demonstrated results, we have the following observations.
Attentions of the MT Encoder: The self-attentions (SA) of the 1st and 6th blocks in the image encoder that are in Fig. 5 reflect the pairwise similarity of the visual objects. From the results, we can see that the following: 1) in Enc(SA)-1, the largest attention values almost appear on the diagonal line, indicating that the pairwise interactions are not learned in the first block; and 2) the largest values in Enc(SA)-6 form vertical lines (e.g., the 4th, 9th and 13th columns), which correspond to the key objects of the image (e.g., the girl and the skateboard). This result reveals that all the attended features tend to use the features of these key objects for the representation.
Attentions of the MT Decoder: The self-attention the 1st and 6th blocks of the caption decoder that are shown in Fig. 5 reflects the similarity of paired words. The largest attention values in Dec(SA)-1 almost appear on the diagonal line, which is similar to those in the Enc(SA)-1. In Dec(SA)-6, the word importance and pairwise word similarities are simultaneously represented. For example, the columns of ‘woman’ and ‘riding’ obtain focused attention weights, and the relationship between ‘woman’ and ‘skateboard’ is highlighted.
The guided-attention (GA) reflects the multimodal relationships between word-object pairs. In Dec(GA)-1, the learned attentions are not concentrated, and some word-object similarities are incorrect (e.g., the 15th object is not related to the word ‘skateboard’). In contrast, the attention in Dec(GA)-6 has much clearer meanings. The co-attention of key objects along with their word-object relationships are highlighted accordingly.
Attentions of the MT Encoder: In Fig 6, we visualize the 1st, 3rd and 6th guided-attention (GA) blocks in the multi-view image encoder. In Enc(GA)-1, the unaligned objects from different views are adaptively aligned (e.g., the 5th object in R-101 and the 5-th object in R-152, and the 3rd object in R-101 and the 6th object in R-152). In Enc(GA)-3, the contextual relationships are also involved (e.g., the 5th object in R-152 has large attention values to the 1st and the 4th objects in R-101, which correspond to different parts of the woman’s body). In Enc(GA)-6, the modeled contextual relationships cover specific objects and contain background scenes (e.g., the 13th object in R-152 and the 10-th object in R-101). These observations reveal that the UMV image encoder learns to align the objects and explores more complex interactions across multi-view features to provide a fine-grained understanding of the image content.
Moreover, we show some predicted captioning examples in Fig 7. The first two rows show four examples where MT outperforms MT, and the third row shows two examples where MT outperforms MT. The last row shows two examples where both models generate incorrect captions. From the demonstrated results, we can see the following that: 1) although MT quantitatively outperforms MT, the performance gap is not qualitatively different and they have their own advantages in different cases. This results in a diverse ensemble when they are integrated together; 2) the incorrect captions are caused by small objects (e.g., the newspaper or the second person).
In this paper, we present a novel Multimodal Transformer (MT) framework for image captioning. The MT consists of an image encoder that generates visual representations via deep self-attention learning, and a caption decoder to transform the encoder’s visual features to textual captions. To further facilitate the capacity of visual features, we introduce multi-view learning into the image encoder and propose two MT variants, MT and MT, to model the aligned multi-view features and unaligned multi-view features, respectively. We quantitatively and qualitatively evaluate the proposed MT models on the benchmark MSCOCO image captioning dataset and conduct extensive ablation studies to explore the reasons behind the MT’ s effectiveness. Experimental results show that our method significantly outperforms existing approaches, and an ensemble of seven models achieves the best performance on the real-time leaderboard of the MSCOCO image captioning challenge.
-  Z. Yu, F. Wu, Y. Yang, Q. Tian, J. Luo, and Y. Zhuang, “Discriminative coupled dictionary hashing for fast cross-media retrieval,” in Association for Computing Machineryâs Special Interest Group on Information Retrieval (ACM SIGIR)), 2014, pp. 395–404.
-  A. Fukui, D. H. Park, D. Yang, A. Rohrbach, T. Darrell, and M. Rohrbach, “Multimodal compact bilinear pooling for visual question answering and visual grounding,” arXiv preprint arXiv:1606.01847, 2016.
-  J.-H. Kim, J. Jun, and B.-T. Zhang, “Bilinear attention networks,” Advances in Neural Information Processing Systems (NIPS), 2018.
-  Z. Yu, J. Yu, J. Fan, and D. Tao, “Multi-modal factorized bilinear pooling with co-attention learning for visual question answering,” IEEE International Conference on Computer Vision (ICCV), pp. 1839–1848, 2017.
-  Z. Yu, J. Yu, C. Xiang, Z. Zhao, Q. Tian, and D. Tao, “Rethinking diversified and discriminative proposal generation for visual grounding,” International Joint Conference on Artificial Intelligence (IJCAI), pp. 1114–1120, 2018.
-  P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang, “Bottom-up and top-down attention for image captioning and visual question answering,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
-  H. Xu and K. Saenko, “Ask, attend and answer: Exploring question-guided spatial attention for visual question answering,” in European Conference on Computer Vision (ECCV), 2016, pp. 451–466.
-  N. Xu, A.-A. Liu, Y. Wong, Y. Zhang, W. Nie, Y. Su, and M. Kankanhalli, “Dual-stream recurrent neural network for video captioning,” IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2018.
-  J. Lu, C. Xiong, D. Parikh, and R. Socher, “Knowing when to look: Adaptive attention via a visual sentinel for image captioning,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 375–383.
-  S. J. Rennie, E. Marcheret, Y. Mroueh, J. Ross, and V. Goel, “Self-critical sequence training for image captioning,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 7008–7024.
-  I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in Advances in Neural Information Processing Systems (NIPS), 2014, pp. 3104–3112.
-  K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Association for Computational Linguistics (ACL). Association for Computational Linguistics, 2002, pp. 311–318.
-  R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus-based image description evaluation,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 4566–4575.
-  A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems (NIPS), 2017, pp. 6000–6010.
-  G. Kulkarni, V. Premraj, V. Ordonez, S. Dhar, S. Li, Y. Choi, A. C. Berg, and T. L. Berg, “Babytalk: Understanding and generating simple image descriptions,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 35, no. 12, pp. 2891–2903, 2013.
-  M. Mitchell, X. Han, J. Dodge, A. Mensch, A. Goyal, A. Berg, K. Yamaguchi, T. Berg, K. Stratos, and H. Daumé III, “Midge: Generating image descriptions from computer vision detections,” in Association for Computational Linguistics (ACL). Association for Computational Linguistics, 2012, pp. 747–756.
-  Y. Yang, C. L. Teo, H. Daumé III, and Y. Aloimonos, “Corpus-guided sentence generation of natural images,” in Conference on Empirical Methods in Natural Language Processing (EMNLP), 2011, pp. 444–454.
-  A. Karpathy, A. Joulin, and L. F. Fei-Fei, “Deep fragment embeddings for bidirectional image sentence mapping,” in Advances in Neural Information Processing Systems (NIPS), 2014, pp. 1889–1897.
-  A. Farhadi, M. Hejrati, M. A. Sadeghi, P. Young, C. Rashtchian, J. Hockenmaier, and D. Forsyth, “Every picture tells a story: Generating sentences from images,” in European Conference on Computer Vision (ECCV). Springer, 2010, pp. 15–29.
-  J. Devlin, H. Cheng, H. Fang, S. Gupta, L. Deng, X. He, G. Zweig, and M. Mitchell, “Language models for image captioning: The quirks and what works,” arXiv preprint arXiv:1505.01809, 2015.
-  T. Yao, Y. Pan, Y. Li, Z. Qiu, and T. Mei, “Boosting image captioning with attributes,” in IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4894–4902.
-  T. Yao, Y. Pan, Y. Li, and T. Mei, “Exploring visual relationship for image captioning,” in European Conference on Computer Vision (ECCV), 2018, pp. 684–699.
-  C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1–9.
-  S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997.
-  J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venugopalan, K. Saenko, and T. Darrell, “Long-term recurrent convolutional networks for visual recognition and description,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 2625–2634.
-  A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 3128–3137.
-  L. Chen, H. Zhang, J. Xiao, L. Nie, J. Shao, W. Liu, and T.-S. Chua, “Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 5659–5667.
-  J. Lu, J. Yang, D. Batra, and D. Parikh, “Hierarchical question-image co-attention for visual question answering,” in Advances in neural information processing systems (NIPS), 2016, pp. 289–297.
-  Z. Yu, J. Yu, C. Xiang, J. Fan, and D. Tao, “Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering,” IEEE Transactions on Neural Networks and Learning Systems, vol. 29, no. 12, pp. 5947–5959, 2018.
-  H. Nam, J.-W. Ha, and J. Kim, “Dual attention networks for multimodal reasoning and matching,” arXiv preprint arXiv:1611.00471, 2016.
-  D.-K. Nguyen and T. Okatani, “Improved fusion of visual and language representations by dense symmetric co-attention for visual question answering,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6087–6096, 2018.
-  N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” The Journal of Machine Learning Research (JMLR), vol. 15, no. 1, pp. 1929–1958, 2014.
-  K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778.
-  J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer normalization,” arXiv preprint arXiv:1607.06450, 2016.
-  S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in Neural Information Processing Systems (NIPS), 2015, pp. 91–99.
-  R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma et al., “Visual genome: Connecting language and vision using crowdsourced dense image annotations,” International Journal of Computer Vision (IJCV), vol. 123, no. 1, pp. 32–73, 2017.
-  J. Pennington, R. Socher, and C. Manning, “Glove: Global vectors for word representation,” in Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014, pp. 1532–1543.
-  J. Yu, Y. Rui, Y. Y. Tang, and D. Tao, “High-order distance-based multiview stochastic learning in image classification,” IEEE Transactions On Cybernetics (CYB), vol. 44, no. 12, pp. 2431–2442, 2014.
-  Z. Tu, W. Xie, J. Dauwels, B. Li, and J. Yuan, “Semantic cues enhanced multi-modality multi-stream cnn for action recognition,” IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), 2018.
-  D. Tao, Y. Guo, B. Yu, J. Pang, and Z. Yu, “Deep multi-view feature learning for person re-identification,” IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), vol. 28, no. 10, pp. 2657–2666, 2018.
-  R. Girshick, “Fast r-cnn,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1440–1448.
-  T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in IEEE international conference on computer vision (ICCV), 2017, pp. 2980–2988.
-  J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779–788.
-  T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár, “Microsoft coco: Common objects in context,” arXiv preprint arXiv:1405.0312, 2014.
-  C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in The 42nd Annual Meeting of the Association for Computational Linguistics (ACL) Workshop, 2004, p. 10.
-  S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in The 43rd Annual Meeting of the Association for Computational Linguistics (ACL) Workshop, 2005, pp. 65–72.
-  S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1492–1500.
-  D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
-  W. Jiang, L. Ma, Y.-G. Jiang, W. Liu, and T. Zhang, “Recurrent fusion network for image captioning,” in European Conference on Computer Vision (ECCV), 2018, pp. 499–515.
-  O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 3156–3164.
-  J. Mao, W. Xu, Y. Yang, J. Wang, Z. Huang, and A. Yuille, “Deep captioning with multimodal recurrent neural networks (m-rnn),” in International Conference on Learning Representations (ICLR), 2015.
Jun Yu(M’13) received the B.Eng. and Ph.D. degrees from Zhejiang University, Zhejiang, China. He was an Associate Professor with the School of Information Science and Technology, Xiamen University, Xiamen, China. From 2009 to 2011, he was with Nanyang Technological University, Singapore. From 2012 to 2013, he was a Visiting Researcher at Microsoft Research Asia (MSRA). He is currently a Professor with the School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China. He has authored or coauthored more than 100 scientific articles. Over the past years, his research interests have included multimedia analysis, machine learning, and image processing. He is the associate editor of IEEE Trans. on CSVT and Pattern Recognition, and the reviewer of various international journals including IEEE Trans. on PAMI, IEEE Trans. on Image Processing, IEEE Trans. on Multimedia, etc. In 2017, he received the IEEE SPS Best Paper Award. Dr. Yu has (co-)chaired several special sessions, invited sessions, and workshops. He served as a program committee member or reviewer of top conferences and prestigious journals. He is a Professional Member of the Association for Computing Machinery and the China Computer Federation.
Jing Li received the B.Eng. degree from the School of Management, Hangzhou Dianzi University, Hangzhou, China, in 2017. He is currently pursuing the M.Eng. degree with the School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China. His current research interests include multimodal analysis, computer vision and machine learning.
Zhou Yu received the B.Eng. and Ph.D. degrees from Zhejiang University, Zhejiang, China, in 2010 and 2015, respectively. He is currently an Associate Professor with the School of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China. His research interests includes multimodal analysis, computer vision, machine learning and deep learning. He has served as reviewers or program committee members of prestigious journals and top conferences including IEEE Trans. on CSVT, IEEE Trans. on Multimedia, IEEE Trans. on Image Processing, IJCAI and AAAI, etc.
Qingming Huang(F’18) is a professor in the University of Chinese Academy of Sciences and an adjunct research professor in the Institute of Computing Technology, Chinese Academy of Sciences. He graduated with a Bachelor degree in Computer Science in 1988 and Ph.D. degree in Computer Engineering in 1994, both from Harbin Institute of Technology, China. His research areas include multimedia computing, image processing, computer vision and pattern recognition. He has authored or coauthored more than 400 academic papers in prestigious international journals and top-level international conferences. He is the associate editor of IEEE Trans. on CSVT and Acta Automatica Sinica, and the reviewer of various international journals including IEEE Trans. on PAMI, IEEE Trans. on Image Processing, IEEE Trans. on Multimedia, etc. He is a Fellow of IEEE and has served as general chair, program chair, track chair and TPC member for various conferences, including ACM Multimedia, CVPR, ICCV, ICME, ICMR, PCM, BigMM, PSIVT, etc.