Iterative Answer Prediction with Pointer-Augmented
Multimodal Transformers for TextVQA
Many visual scenes contain text that carries crucial information, and it is thus essential to understand text in images for downstream reasoning tasks. For example, a deep water label on a warning sign warns people about the danger in the scene. Recent work has explored the TextVQA task that requires reading and understanding text in images to answer a question. However, existing approaches for TextVQA are mostly based on custom pairwise fusion mechanisms between a pair of two modalities and are restricted to a single prediction step by casting TextVQA as a classification task. In this work, we propose a novel model for the TextVQA task based on a multimodal transformer architecture accompanied by a rich representation for text in images. Our model naturally fuses different modalities homogeneously by embedding them into a common semantic space where self-attention is applied to model inter- and intra- modality context. Furthermore, it enables iterative answer decoding with a dynamic pointer network, allowing the model to form an answer through multi-step prediction instead of one-step classification. Our model outperforms existing approaches on three benchmark datasets for the TextVQA task by a large margin.
As a prominent task for visual reasoning, the Visual Question Answering (VQA) task  has received wide attention in terms of both datasets (e.g. [4, 17, 22, 21, 20]) and methods (e.g. [14, 3, 6, 25, 33]). However, these datasets and methods mostly focus on the visual components in the scene. On the other hand, they tend to ignore a crucial modality – text in the images – that carries essential information for scene understanding and reasoning. For example, in Figure 1, deep water on the sign warns people about the danger in the scene. To address this drawback, new VQA datasets [44, 8, 37] have been recently proposed with questions that explicitly require understanding and reasoning about text in the image, which is referred to as the TextVQA task.
The TextVQA task distinctively requires models to see, read and reason over three modalities: the input question, the visual contents in the image such as visual objects, and the text in the image. Several approaches [44, 8, 37, 7] have been proposed for the TextVQA task, based on OCR results of the image. In particular, LoRRA  extends previous VQA models  with an OCR attention branch and adds OCR tokens as a dynamic vocabulary to the answer classifier, allowing copying a single OCR token from the image as the answer. Similarly in , OCR tokens are grouped into blocks and added to the output space of a VQA model.
While these approaches enable reading text in images to some extent, they typically rely on custom pairwise multimodal fusion mechanisms between two modalities (such as single-hop attention over image regions and text tokens, conditioned on the input question), which limit the types of possible interactions between modalities. Furthermore, they treat answer prediction as a single-step classification problem – either selecting an answer from the training set answers or copying a text token from the image – making it difficult to generate complex answers such as book titles or signboard names with multiple words, or answers with both common words and specific image text tokens, such as McDonald’s burger where McDonald’s is from text in the image and burger is from the model’s own vocabulary. In addition, the word embedding based image text features in previous work have limited representation power and miss important cues such as the appearance (e.g. font and color) and the location of text tokens in images. For example, tokens that have different fonts and are spatially apart from each other usually do not belong to the same street sign.
In this paper, we address the above limitations with our novel Multimodal Multi-Copy Mesh (M4C) model for the TextVQA task, based on the transformer  architecture accompanied by iterative answer decoding through dynamic pointers, as shown in Figure 1. Our model naturally fuses the three input modalities and captures intra- and inter- modality interactions homogeneously within a multimodal transformer, which projects all entities from each modality into a common semantic embedding space, and applies the self-attention mechanism [38, 48] to collect relational representations for each entity. Instead of casting answer prediction as a classification task, we perform iterative answer decoding in multiple steps and augment our answer decoder with a dynamic pointer network that allows selecting text in the image in a permutation-invariant way without relying on any ad-hoc position indices in previous work such as LoRRA . Furthermore, our model is capable of combining its own vocabulary with text in the image in a generated answer, as shown in examples in Figure 4 and 5. Finally, we introduce a rich representation for text tokens in the images based on multiple cues, including its word embedding, appearance, location, and character-level information.
Our contributions in this paper are as follows: 1) We show that multiple (more than two) input modalities can be naturally fused and jointly modeled through our multimodal transformer architecture. 2) Unlike previous work on TextVQA, our model reasons about the answer beyond a single classification step and predicts it through our pointer-augmented multi-step decoder. 3) We adopt a rich feature representation for text tokens in images and show that it is better than features based only on word embedding in previous work. 4) Our model significantly outperforms previous work on three challenging datasets for the TextVQA task: TextVQA  (+25% relative), ST-VQA  (+65% relative), and OCR-VQA  (+32% relative).
2 Related work
VQA based on reading and understanding image text.
Recently, a few datasets and methods [44, 8, 37, 7] have been proposed for visual question answering based on text in images (referred to as the TextVQA task). LoRRA , a prominent prior work on this task, extends the Pythia  framework for VQA and allows it to copy a single OCR token from the image as the answer, by applying a single attention hop (conditioned on the question) over the OCR tokens and including the OCR token indices in the answer classifier’s output space. A conceptually similar model is proposed in , where OCR tokens are grouped into blocks and added to both the input features and the output answer space of a VQA model. In addition, a few other approaches [8, 7] enable text reading by augmenting existing VQA models with OCR inputs. However, these existing methods are limited by their simple feature representation of image text, multimodal learning approaches, and one-step classification for answer outputs. In this work, we address these limitations with our M4C model.
Multimodal learning in vision-and-language tasks.
Early approaches on vision-and-language tasks often combined the image and text through attention over one modality conditioned on the other modality, such as image attention based on text (e.g. [51, 34]). Some approaches have explored multimodal fusion mechanisms such as bilinear models (e.g. [14, 25]), self-attention (e.g. ), and graph networks (e.g. ). Inspired by the success of Transformer  and BERT  architectures in natural language tasks, several recent works [33, 1, 47, 31, 29, 45, 53, 11] have also applied transformer-based fusion between image and text with self-supervision on large-scale datasets. However, most existing works treat each modality with a specific set of parameters, which makes them hard to scale to more input modalities. On the other hand, in our work we project all entities from each modality into a joint embedding space and treat them homogeneously with a transformer architecture over the list of all things. Our results suggest that joint embedding and self-attention are efficient when modeling multiple (more than two) input modalities.
Dynamic copying with pointers.
Many answers in the TextVQA task come from text tokens in the image such as book titles or street signs. As it is intractable to have every possible text token in the answer vocabulary, copying text from the image would often be an easier option for answer prediction. Prior work has explored dynamically copying the inputs in different tasks such as text summarization , knowledge retrieval , and image captioning  based on Pointer Networks  and its variants. For the TextVQA task, recent works [44, 37] have proposed to copy OCR tokens by adding their indices to classifier outputs. However, apart from their limitation of copying only a single token (or block), one drawback of these approaches is that they require a pre-defined number of OCR tokens (since the classifier has a fixed output dimension) and their output is dependent on the ordering of the tokens. In this work, we overcome this drawback using a permutation-invariant pointer network together with our multimodal transformer.
3 Multimodal Multi-Copy Mesh (M4C)
In this work, we present Multimodal Multi-Copy Mesh (M4C), a novel approach for the TextVQA task based on a pointer-augmented multimodal transformer architecture with iterative answer prediction. Given a question and an image as inputs, we extract feature representations from three modalities – the question, the visual objects in the image, and the text present in the image. These three modalities are represented respectively as a list of question words features, a list of visual object features from an off-the-shelf object detector, and a list of OCR token features based on an external OCR system.
Our model projects the feature representations of entities (in our case, question words, detected objects, and detected OCR tokens) from the three modalities as vectors in a learned common embedding space. Then, a multi-layer transformer  is applied on the list of all projected features, enriching their representations with intra- and inter- modality context. Our model learns to predict the answer through iterative decoding accompanied by a dynamic pointer network. During decoding, it feeds in the previous output to predict the next answer component in an auto-regressive manner. At each step, it either copies an OCR token from the image, or selects a word from its fixed answer vocabulary. Figure 2 shows an overview of our model.
3.1 A common embedding space for all modalities
Our model receives inputs from three modalities – question words, visual objects, and OCR tokens. We extract feature representations for each modality and project them into a common -dimensional semantic space through domain-specific embedding approaches as follows.
Embedding of question words.
Given a question as a sequence of words, we embed these words into the corresponding sequence of -dimensional feature vectors (where using a pretrained BERT model .111In our implementation, we extract question word features from the first 3 layers of BERT-BASE. We find it sufficient to use its first few layers instead of using all its 12 layers, which saves computation. During training, the BERT parameters are fine-tuned using the question answering loss.
Embedding of detected objects.
Given an image, we obtain a set of visual objects through a pretrained detector (Faster R-CNN  in our case). Following prior work [3, 43, 44], we extract appearance feature using the detector’s output from the -th object (where . To capture its location in the image, we introduce a 4-dimensional location feature from -th object’s relative bounding box coordinates , where and are image width and height respectively. Then, the appearance feature and the location feature are projected into the -dimensional space with two learned linear transforms (where is the same as in the question word embedding above), and are summed up as the final object embedding as
where and are learned projection matrices. is layer normalization , added on the output of the linear transforms to ensure that the object embedding has the same scale as the question word embedding. We fine-tune the last layer of the Faster R-CNN detector during training.
Embedding of OCR tokens with rich representations.
Intuitively, to represent text in images, one needs to encode not only its characters, but also its appearance (e.g. color, font, and background) and spatial location in the image (e.g. words appearing on the top of a book cover are more likely to be book titles). We follow this intuition in our model and use a rich OCR representation consisting of four types of features, which is shown in our experiments to be significantly better than word embedding (such as FastText) alone in prior work . After obtaining a set of OCR tokens in an image through external OCR systems, from the -th token (where we extract 1) a 300-dimensional FastText  vector , which is a word embedding with sub-word information, 2) an appearance feature from the same Faster R-CNN detector in the object detection above, extracted via RoI-Pooling on the OCR token’s bounding box, 3) a 604-dimensional Pyramidal Histogram of Characters (PHOC)  vector , capturing what characters are present in the token – this is more robust to OCR errors and can be seen as a coarse character model, and 4) a 4-dimensional location feature based on the OCR token’s relative bounding box coordinates . We linearly project each feature into -dimensional space, and sum them up (after layer normalization) as the final OCR token embedding as below
where , , and are learned projection matrices and is layer normalization.
3.2 Multimodal fusion and iterative answer prediction with pointer-augmented transformers
After embedding all entities (question words, visual objects, and OCR tokens) from each modality as vectors in the -dimensional joint embedding space as described in Sec. 3.1, we apply a stack of transformer layers  with a hidden dimension of over the list of all entities from , , and . Through the multi-head self-attention mechanism in transformers, each entity is allowed to freely attend to all other entities, regardless of whether they are from the same modality or not. For example, an OCR token is allowed to attend to another OCR token, a detected object, or a question word. This enables modeling both inter- and intra- modality relations in a homogeneous way through the same set of transformer parameters. The output from our multimodal transformer is a list of -dimensional feature vectors for entities in each modality, which can be seen as their enriched embedding in multimodal context.
We predict an answer to the question through iterative decoding, using exactly the same transformer layers as a decoder. We decode the answer word by word in an auto-regressive manner for a total of steps, where each decoded word may be either an OCR token in the image or a word from our fixed vocabulary of frequent answer words. As illustrated in Figure 2, at each step during decoding, we feed in an embedding of the previously predicted word, and predict the next answer word based on the transformer output with a dynamic pointer network.
Let be the -dimensional transformer outputs of the OCR tokens in the image. Assume we have a vocabulary of words that frequently appear in the training set answers. At the -th decoding step, the transformer model outputs a -dimensional vector corresponding to the input at step (explained later in this section). From , we predict both the -dimensional scores of choosing a word from fixed answer vocabulary and the -dimensional scores of selecting an OCR token from the image at decoding step . In our implementation, the fixed answer vocabulary score for the -th word (where ) is predicted as a simple linear layer as
where is a -dimensional parameter for the -th word in the answer vocabulary, and is a scalar parameter.
To select a token from the OCR tokens in the image, we augment the transformer model with a dynamic pointer network, predicting a copying score (where ) for each token via bilinear interaction between the decoding output and each OCR token’s output representation as
where and are matrices, and and are -dimensional vectors.
During prediction, we take the argmax on the concatenation of fixed answer vocabulary scores and dynamic OCR-copying scores, selecting the top scoring element (either a vocabulary word or an OCR token) from all candidates.
In our iterative auto-regressive decoding procedure, if the prediction at decoding time-step is an OCR token, we feed in its OCR representation as the transformer input to the next prediction step . Otherwise (the previous prediction is a word from the fixed answer vocabulary), we feed in its corresponding weight vector in Eqn. 3 as the next step’s input . In addition, we add two extra -dimensional vectors as inputs – a positional embedding vector corresponding to step , and a type embedding vector corresponding to whether the previous prediction is a fixed vocabulary word or an OCR token. Similar to machine translation, we augment our answer vocabulary with two special tokens, <begin> and <end>. Here <begin> is used as the input to the first decoding step, and we stop the decoding process after <end> is predicted.
To ensure causality in answer decoding, we mask the attention weights in the self-attention layers of the transformer architecture  such that question words, detected objects and OCR tokens cannot attend to any decoding steps, and all decoding steps can only attend to previous decoding steps in addition to question words, detected objects and OCR tokens. This is similar to prefix LM in .
During training, we supervise our multimodal transformer at each decoding step. Similar to sequence prediction tasks such as machine translation, we use teacher-forcing  (i.e. using ground-truth inputs to the decoder) to train our multi-step answer decoder, where each ground-truth answer is tokenized into a sequence of words. Given that an answer word can appear in both fixed answer vocabulary and OCR tokens, we apply multi-label sigmoid loss (instead of softmax loss) over the concatenated scores .
We evaluate our model on three challenging datasets for the TextVQA task, including the TextVQA dataset , the ST-VQA dataset , and the OCR-VQA dataset . Our model outperforms previous work by a significant margin on all the three datasets.
4.1 Evaluation on the TextVQA dataset
|#||Method||Question enc.||OCR||OCR token||Output||Accu.||Accu.|
|pretraining||system||representation||module||on val||on test|
|2||M4C w/o dec.||(none)||Rosetta-ml||FastText||classifier||29.55||–|
|3||M4C w/o dec.||BERT||Rosetta-ml||FastText||classifier||30.15||–|
|4||M4C w/o dec.||BERT||Rosetta-en||FastText||classifier||31.28||–|
|5||M4C w/o dec.||BERT||Rosetta-en||FastText + bbox||classifier||33.32||–|
|6||M4C w/o dec.||BERT||Rosetta-en||FastText + bbox + FRCN||classifier||34.38||–|
|7||M4C w/o dec.||BERT||Rosetta-en||FastText + bbox + FRCN + PHOC||classifier||35.70||–|
|8||M4C (ours - ablation)||(none)||Rosetta-ml||FastText + bbox + FRCN + PHOC||decoder||36.06||–|
|9||M4C (ours - ablation)||BERT||Rosetta-ml||FastText + bbox + FRCN + PHOC||decoder||37.06||–|
|10||M4C (ours)||BERT||Rosetta-en||FastText + bbox + FRCN + PHOC||decoder||39.40||39.01|
|11||DCD_ZJU (ensemble) ||–||–||–||–||31.48||31.44|
|13||M4C (ours; w/ ST-VQA)||BERT||Rosetta-en||FastText + bbox + FRCN + PHOC||decoder||40.55||40.46|
The TextVQA dataset  contains 28,408 images from the Open Images dataset , with human-written questions asking to reason about text in the image. Similar to VQAv2 , each question in the TextVQA dataset has 10 human annotated answers, and the final accuracy is measured via soft voting of the 10 answers.222See https://visualqa.org/evaluation for details.
We use as the dimensionality of the joint embedding space and extract question word features with BERT-BASE using the 768-dimensional outputs from its first three layers, which are fine-tuned during training.
For visual objects, following Pythia  and LoRRA , we detect objects with a Faster R-CNN detector  pretrained on the Visual Genome dataset , and keeps 100 top-scoring objects per image. Then, the fc6 feature vector is extracted from each detected object. We apply the Faster R-CNN fc7 weights on the extracted fc6 features to output 2048-dimensional fc7 appearance features and fine-tune fc7 weights during training. However, we do not use the ResNet-152 convolutional features  as in LoRRA.
Finally, we extract text tokens on each image using the Rosetta OCR system . Different from the prior work LoRRA  that uses a multilingual Rosetta version, in our model we use an English-only version of Rosetta that we find has higher recall. We refer to these two versions as Rosetta-ml and Rosetta-en, respectively. As mentioned in Sec. 3.1, from each OCR token we extract FastText  feature, appearance feature from Faster R-CNN (FRCN), PHOC  feature, and bounding box (bbox) feature.
In our multimodal transformer, we use layers of multimodal transformer with 12 attention heads. Other hyper-parameters (such as dropout ratio) follow BERT-BASE . However, we note that the multimodal transformer parameters are initialized from scratch rather than from a pretrained BERT model. We use maximum decoding step in answer prediction unless otherwise specified, which is sufficient to cover almost all answers.
We collect the top 5000 frequent words from the answers in the training set as our answer vocabulary. During training, we use a batch size of 128, and train for a maximum of 24,000 iterations. Our model is trained using the Adam optimizer, with a learning rate of 1e-4 and a staircase learning rate schedule, where we multiply the learning rate by 0.1 at 14000 and at 19000 iterations. The best snapshot is selected using the validation set accuracy. The entire training takes approximately 10 hours on 4 Nvidia Tesla V100 GPUs.
As a notable prior work on this dataset, we show a step-by-step comparison with the LoRRA model . LoRRA uses two single-hop attention layers over image visual features and OCR features. The attended visual and OCR features are then fused with a vector encoding of the question and fed into a single-step classifier to select either a frequent answer from the training set or a single OCR token from the image. Different from our rich OCR representation in Sec. 3.1, in the LoRRA model each OCR token is only represented as a 300-dimensional FastText vector.
Ablations on pretrained question encoding and OCR systems.
We first experiment with a restricted version of our model using the multimodal transformer architecture but without iterative decoding in answer prediction, i.e. M4C (w/o dec.) in Table 1. In this setting, we only decode for one step, and either select a frequent answer333In this case, we predict the entire (multi-word) answer, instead of a single word from our answer word vocabulary as in our full model. from the training set or copy a single OCR token in the image as the answer. As a step-by-step comparison with LoRRA, we start with extracting OCR tokens from Rosetta-ml, representing OCR tokens only with FastText vectors, and initializing question encoding parameters in Sec. 3.1 from scratch (rather than from a pretrained BERT-BASE model). The result is shown in line 2 of Table 1. Compared with LoRRA in line 1, this restricted version of our model already outperforms LoRRA by around 3% (absolute) on TextVQA validation set. Given that LoRRA uses pretrained GloVe  for question encoding while we learn question encoding from scratch in line 2, this result shows that our multimodal transformer architecture is more efficient for jointly modeling the three input modalities. We then switch to a pretrained BERT for question encoding in line 3, and Rosetta-en for OCR extraction in line 4. Comparing line 2 to 4, we see that a pretrained BERT leads to around 0.6% higher accuracy, and Rosetta-en gives another 1% improvement.
Ablations on OCR feature representation.
We analyze the impact of our rich OCR representation in Sec. 3.1 through ablations in Table 1 line 4 to 7. We see that OCR location (bbox) features and the RoI-pooled appearance features (FRCN) both improve the performance by a noticeable margin. In addition, we find that PHOC is also helpful as a character-level representation of the OCR token. Our rich OCR representation gives around 4% (absolute) accuracy improvement compare with using only FastText features as in LoRRA (line 7 vs 4). We note that our extra OCR features do not require more pretrained models, as we apply exactly the same Faster R-CNN model use in object detection for OCR appearance features, and PHOC is a manually-designed feature that does not need pretraining.
|What does the light sign read on the farthest right window?||Who is usa today’s bestselling author?||What is the name of the band?||what is the time?|
|LoRRA: exit||LoRRA: roger zelazny||LoRRA: 7||LoRRA: 1:45|
|M4C (ours): bud light||M4C (ours): cathy williams||M4C (ours): soul doubt||M4C (ours): 3:44|
|human: bud light; all 2 liters||human: cathy williams||human: soul doubt; h. michael karshis; unanswerable||human: 5:40; 5:41; 5:42; 8:00|
Iterative answer decoding.
We then apply our full M4C model with iterative answer decoding to the TextVQA dataset. The results are shown in Table 1 line 10, which is around 4% (absolute) higher than its counterpart in line 7 using a single-step classifier and 13% (absolute) higher than LoRRA in line 1. In addition, we ablate our model using Rosetta-ml and randomly initialized question encoding parameters in line 8 and 9. Here, we see that our model in line 8 still outperforms LoRRA (line 1) by as much as 9.5% (absolute) when using the same OCR system as LoRRA and even fewer pretrained components. We also analyze the performance of our model with respect to the maximum decoding steps, shown in Figure 3, where decoding for multiple steps greatly improves the performance compared with a single step. Figure 4 shows qualitative examples of our M4C model on the TextVQA dataset in comparison to LoRRA , where our model is capable of selecting multiple OCR tokens and combining them with its fixed vocabulary in predicted answers.
TextVQA Challenge 2019.
We also compare to the winning entries in the TextVQA Challenge 2019.444https://textvqa.org/challenge We compare our method to DCD  (the challenge winner, based on ensemble) and MSFT_VTI  (the top entry after the challenge), both relying on one-step prediction. We show that our single model (line 10) significantly outperforms these challenge winning entries on the TextVQA test set by a large margin. We also experiment with using the ST-VQA dataset  as additional training data (a practice used by some of the previous challenge participants), which gives another 1% improvement and 40.46% final test accuracy – a new state-of-the-art on the TextVQA dataset.
4.2 Evaluation on the ST-VQA dataset
|What is the name of the street on which the Stop sign appears?||What does the white sign say?||What is the license plate number on the red car?||What kind of stop sign is in the image?|
|prediction: 45th parallel dr||prediction: tokyo station||prediction: gsv 820||prediction: stop all way|
|GT: 45th parallel dr||GT: tokyo station||GT: gsv 820||GT: all way|
The ST-VQA dataset  contains natural images from multiple sources including ICDAR 2013 , ICDAR 2015 , ImageNet , VizWiz , IIIT STR , Visual Genome , and COCO-Text .555We notice that many images from COCO-Text  in the downloaded ST-VQA data (around 1/3 of all images) are resized to for unknown reasons, which degrades the image quality and distorts their aspect ratios. In our experiments, we replace these images with their original sized versions from COCO-Text as inputs to object detection and OCR systems. The format of the ST-VQA dataset is similar to the TextVQA dataset in Sec. 4.1. However, each question is accompanied by only one or two ground-truth answers provided by the question writer. The dataset involves three tasks, and its Task 3 - Open Dictionary (containing 18,921 training-validation images and test 2,971 images) corresponds to our general TextVQA setting where no answer candidates are provided at test time.
The ST-VQA dataset adopts Average Normalized Levenshtein Similarity (ANLS)6 as its official evaluation metric, defined as scores (where and are prediction and ground-truth answers and is edit distance) averaged over all questions. Also, all scores below the threshold 0.5 are truncated to 0 before averaging. To facilitate comparison, we report both accuracy and ANLS in our experiments.
As the ST-VQA dataset does not have an official split for training and validation, we randomly select 17,028 images as our training set and use the remaining 1,893 images as our validation set. We train our model on the ST-VQA dataset following exactly the same setting (line 10 in Table 1) as in our TextVQA experiments in Sec. 4.1, where we extract image text tokens using Rosetta-en, use FastText + bbox + FRCN + PHOC as our OCR representation, and initialize question encoding parameters from a pretrained BERT-BASE model. The results are shown in Table 2.
|module||on val||on val||on test|
|3||M4C w/o dec.||classifier||33.52||0.397||–|
Ablations of our model.
We train two versions of our model, one restricted version (M4C w/o dec. in Table 2) with a fixed one-step classifier as output module (similar to line 7 in Table 1) and one full version (M4C) with iterative answer decoding. Comparing the results of these two models, it can be seen that there is a large improvement from our iterative answer prediction mechanism.
Comparison to previous work.
We compare with two previous methods on this dataset: 1) SAN+STR , which combines SAN for VQA  and Scene Text Retrieval  for answer vocabulary retrieval, and 2) VTA , the ICDAR 2019 ST-VQA Challenge666https://rrc.cvc.uab.es/?ch=11&com=tasks winner, based on BERT  for question encoding and BUTD  for VQA. From Table 2, it can be seen that our restricted model (M4C w/o dec.) already achieves higher ANLS than these two models, and our full model achieves as much as +0.18 (absolute) ANLS boost over the best previous work.
4.3 Evaluation on the OCR-VQA dataset
|module||on val||on test|
|5||M4C w/o dec.||classifier||46.3||–|
|Who is the author of this book?||Is this a pharmaceutical book?|
|prediction: the new york times||prediction: no|
|GT: the new york times||GT: no|
The OCR-VQA dataset  contains 207,572 images of book covers, with template-based questions asking about the title, author, edition, genre, year or other information about the book. Each question is has a single ground-truth answer, and the dataset assumes that the answers to these questions can be inferred from the book cover images.
We train our model using the same hyper-parameters as in Sec. 4.1 and 4.2, but use the total iterations and adapted learning rate schedule since the OCR-VQA dataset contains more images. The results are shown in Table 3. Compared to using a one-step classifier (M4C w/o dec.), our full model with iterative decoding achieves significantly better accuracy, which coincides with Figure 3 that having multiple decoding steps is greatly beneficial on this dataset. This is likely because the OCR-VQA dataset often contains multi-word answers such as book titles and author names.
We compare to four baseline approaches from , which are VQA systems based on 1) visual features from a convolutional network (CNN), 2) grouping OCR tokens into text blocks (BLOCK) with manually defined rules, 3) an averaged word2vec (W2V) feature over all the OCR tokens in the image, and 4) their combinations. Note that while the BLOCK baseline can also select multiple OCR tokens, it relies on manually defined rules to merge tokens into groups and can only select one group as answer, while our method learns from data how to copy OCR tokens to compose answers. Compare to these baselines, our M4C has over 15% (absolute) higher test accuracy. Figure 6 shows qualitative examples of our model on this dataset.
In this paper, we present Multimodal Multi-Copy Mesh (M4C) for visual question answering based on understanding and reasoning about text in images. M4C adopts rich representations for text in the images, jointly models all modalities through a pointer-augmented multimodal transformer architecture over a joint embedding space, and predicts the answer through iterative decoding, outperforming previous work by a large margin on three challenging datasets for the TextVQA task. Our results suggest that it is efficient to handle multiple modalities through domain-specific embedding followed by homogeneous self-attention and to generate complex answers as multi-step decoding instead of one-step classification.
-  Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. Fusion of detected objects in text for visual question answering. arXiv preprint arXiv:1908.05054, 2019.
-  Jon Almazán, Albert Gordo, Alicia Fornés, and Ernest Valveny. Word spotting and recognition with embedded attributes. IEEE transactions on pattern analysis and machine intelligence, 36(12):2552–2566, 2014.
-  Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6077–6086, 2018.
-  Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2425–2433, 2015.
-  Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
-  Hedi Ben-Younes, Rémi Cadene, Matthieu Cord, and Nicolas Thome. Mutan: Multimodal tucker fusion for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2612–2620, 2017.
-  Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusiñol, Minesh Mathew, CV Jawahar, Ernest Valveny, and Dimosthenis Karatzas. Icdar 2019 competition on scene text visual question answering. arXiv preprint arXiv:1907.00490, 2016.
-  Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusiñol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
-  Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135–146, 2017.
-  Fedor Borisyuk, Albert Gordo, and Viswanath Sivakumar. Rosetta: Large scale system for text detection and recognition in images. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 71–79. ACM, 2018.
-  Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740, 2019.
-  Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
-  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, 2019.
-  Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 457–468, 2016.
-  Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and inter-modality attention flow for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6639–6648, 2019.
-  Lluís Gómez, Andrés Mafla, Marçal Rusinol, and Dimosthenis Karatzas. Single shot scene text retrieval. In Proceedings of the European Conference on Computer Vision (ECCV), pages 700–715, 2018.
-  Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904–6913, 2017.
-  Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3608–3617, 2018.
-  Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
-  Drew A Hudson and Christopher D Manning. Gqa: a new dataset for compositional question answering over real-world images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
-  Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901–2910, 2017.
-  Kushal Kafle and Christopher Kanan. An analysis of visual question answering algorithms. In Proceedings of the IEEE International Conference on Computer Vision, pages 1965–1973, 2017.
-  Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pages 1156–1160. IEEE, 2015.
-  Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras. Icdar 2013 robust reading competition. In 2013 12th International Conference on Document Analysis and Recognition, pages 1484–1493. IEEE, 2013.
-  Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. Bilinear attention networks. In Advances in Neural Information Processing Systems, pages 1564–1574, 2018.
-  Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32–73, 2017.
-  Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. arXiv preprint arXiv:1811.00982, 2018.
-  Alex M Lamb, Anirudh Goyal Alias Parth Goyal, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. Professor forcing: A new algorithm for training recurrent networks. In Advances In Neural Information Processing Systems, pages 4601–4609, 2016.
-  Gen Li, Nan Duan, Yuejian Fang, Daxin Jiang, and Ming Zhou. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. arXiv preprint arXiv:1908.06066, 2019.
-  Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. Relation-aware graph attention network for visual question answering. arXiv preprint arXiv:1903.12314, 2019.
-  Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557, 2019.
-  Yuetan Lin, Hongrui Zhao, Yanan Li, and Donghui Wang. DCD_ZJU, TextVQA Challenge 2019 winner. https://visualqa.org/workshop.html.
-  Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems, 2019.
-  Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Hierarchical question-image co-attention for visual question answering. In Advances In Neural Information Processing Systems, pages 289–297, 2016.
-  Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural baby talk. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7219–7228, 2018.
-  Anand Mishra, Karteek Alahari, and CV Jawahar. Image retrieval using textual cues. In Proceedings of the IEEE International Conference on Computer Vision, pages 3040–3047, 2013.
-  Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In Proceedings of the International Conference on Document Analysis and Recognition, 2019.
-  Ankur P Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. A decomposable attention model for natural language inference. arXiv preprint arXiv:1606.01933, 2016.
-  Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
-  Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019.
-  Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pages 91–99, 2015.
-  Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368, 2017.
-  Amanpreet Singh, Vivek Natarajan, Yu Jiang, Xinlei Chen, Meet Shah, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Pythia-a platform for vision & language research. In SysML Workshop, NeurIPS, volume 2018, 2018.
-  Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019.
-  Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530, 2019.
-  Anonymous submission. MSFT_VTI, TextVQA Challenge 2019 top entry (post-challenge). https://evalai.cloudcv.org/web/challenges/challenge-page/244/.
-  Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490, 2019.
-  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008, 2017.
-  Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie. Coco-text: Dataset and benchmark for text detection and recognition in natural images. arXiv preprint arXiv:1601.07140, 2016.
-  Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692–2700, 2015.
-  Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 21–29, 2016.
-  Semih Yavuz, Abhinav Rastogi, Guan-Lin Chao, and Dilek Hakkani-Tur. Deepcopy: Grounded response generation with hierarchical pointer networks. arXiv preprint arXiv:1908.10731, 2019.
-  Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason J Corso, and Jianfeng Gao. Unified vision-language pre-training for image captioning and vqa. arXiv preprint arXiv:1909.11059, 2019.