Describing Natural Images Containing Novel Objects with
Knowledge Guided Assitance
Images in the wild encapsulate rich knowledge about varied abstract concepts and cannot be sufficiently described with models built only using image-caption pairs containing selected objects. We propose to handle such a task with the guidance of a knowledge base that incorporate many abstract concepts. Our method is a two-step process where we first build a multi-entity-label image recognition model to predict abstract concepts as image labels and then leverage them in the second step as an external semantic attention and constrained inference in the caption generation model for describing images that depict unseen/novel objects. Evaluations show that our models outperform most of the prior work for out-of-domain captioning on MSCOCO and are useful for integration of knowledge and vision in general.
Content on the Web is highly heterogeneous comprising mostly visual and textual modalities. In most cases, these modalities complement each other to illustrate the semantics of concepts and objects. Many approaches have leveraged such multi-view content for grounding textual with visual information or vice versa. Natural language processing (NLP) tasks like monolingual word similarity (Lazaridou et al., 2015) and language learning (Chrupała et al., 2015) were improved by grounding textual with visual information. The grounding of visual with textual information has shown improvement for existing computer vision (CV) tasks such as image annotation (Srivastava and Salakhutdinov, 2012). The aforementioned tasks can be further subdivided into two broad categories where the first category of tasks combines the semantics of multiple modalities to achieve common representation for representational tasks such as generation of descriptions for images or videos (Venugopalan et al., 2014; Vinyals et al., 2017) and visual question answering (Antol et al., 2015). While the second category of tasks leverages cross-modal semantics to identify the relationship between visual and textual content for attaining referential grounding (Boleda et al., 2017).
Several approaches are proposed to solve varied tasks separately with methods from both categories. However, procedures from both categories can benefit by complementing each other. For example, many approaches for the representational task of generating captions for images are inspired by encoder-decoder architecture or its variations that encompass an attention mechanism (Bahdanau et al., 2014) learn model using only (visual)image-(textual)caption parallel corpora. This causes the model to fail in describing those images which contain unseen/novel objects and concepts not part of parallel captions. Also, the vocabulary is limited to frequent words in the captions and often fails to incorporate rare or infrequent words. Given such challenges, we strive towards a solution that addresses the bottleneck in the aforementioned representational task.
Usage of structured information provided by a knowledge base (KB) (Lehmann et al., 2015) has shown to assist several textual tasks such as question answering over structured data (Bordes et al., 2014), language modeling (Ahn et al., 2016), and generation of factoid questions (Serban et al., 2016). Our hypothesis is that caption generation for images containing unseen/novel objects can significantly benefit from employing structured information (henceforth called knowledge) provided by a KB.
Thus in this paper, we aim to address the task of generating captions that include unseen visual objects in images with our proposed solution termed as knowledge guided assistance (KGA). The aim of KGA is to operate as referential grounding by providing external semantic attention during training and also work as a dynamic constraint while inference of a caption generation model. In particular, KGA assists to generate captions for unseen/novel objects in images which lack parallel captions. This makes KGA differ from earlier approaches that perform similar task such as deep compositional captioning (DCC) (Hendricks et al., 2016) and novel object captioner (NOC) (Venugopalan et al., 2017) by not depending solely on the corpus specific word semantics for next word prediction in a caption generation model. Also, when compared with constrained beam search (CBS) (Anderson et al., 2017) KGA incorporates more information from textual caption data by not constraining solely on image tags. In this respect, KGA is closer to LSTM-C (Yao et al., 2017) which uses object classifiers and copying mechanism, but KGA diverges by using attention mechanism and dynamic weight transfer as opposed to word copying. Figure 1 presents an overview and the main contributions of this paper are summarized as follows:
We designed a novel approach to improve the representational task of caption generation including unseen/novel visual objects with the assistance from a knowledge base.
We created a multi-label image classifier for grounding depicted visual objects to knowledge base entities.
We conducted an extensive experimental evaluation showing effectiveness of KGA.
2. Related Work
Our related work can be drawn from many closely aligned areas.
2.1. Grounding Natural Language in Images
The grounding of natural language in images is employed to comprehend objects and their relationships. Flickr30k Entities (Plummer et al., 2015) is one such approach that augments the Flickr30k dataset images with bounding boxes using all noun phrases present in their parallel textual descriptions. We also leveraged textual descriptions for grounding in images, but to explicitly relate objects with their knowledge base entities. Other approaches (Fereshteh et al., 2015; Johnson et al., 2015) also tried to relate knowledge to images, but not by explicitly linking it to KB. However to extract visual knowledge for supporting tasks such as question answering and image retrieval.
2.2. Attention Mechanism in Caption Generation
Initially, attention mechanism was applied to tasks such as image caption generation (Xu et al., 2015) with two different possibilities i.e. soft and hard attention. Another recent improvement is seen in performing adaptive attention with visual sentinel (Lu et al., 2016) by identifying when to look inside an image for cues and not every time as aforementioned would do it. As both of the aforementioned approaches would look entire image to add global context, region-based attention (Jin et al., 2015) was proposed to experience visual perception where the attention shifting among the visual regions imposes a thread of visual ordering. Slightly, deviating from earlier visual feature centric approaches, the attribute based attention (You et al., 2016) extracted attributes from an image and used them as input vectors resembles our approach.
2.3. Dealing with Rare/OoV Words
Usage of external vocabulary or the structured data is becoming prominent in many neural network models. Goal of these approaches is to copy information from external sources whenever the neural network models fail to predict with certain confidence. Recently, a neural knowledge language model (Ahn et al., 2016) was proposed to improve language modeling with external structured data to improve tasks which are dependent on entities and also extended for text generation (Lebret et al., 2016). Some approaches (Gu et al., 2016) have used copying mechanism to deal with out-of-vocabulary (OoV) words, while few adopted pointing mechanism (Merity et al., 2016). Our approach for constrained inference fall in-line with such approaches, but rather prefer to enhance neural model weights than direct copying.
3. Describing Images with Novel Objects Using Knowledge Guided Assistance (KGA)
In this section, we present our caption generation model for generating captions for unseen/novel image objects with support from KGA. The core goal of KGA is to introduce external semantic attention (ESA) into the learning caption generation models and KGA also work as a constraint for transferring learned weights between seen and unseen semantic and word image labels during inference.
3.1. Caption Generation Model
Our image caption generation model (henceforth, KGA-CGM) combines three important components: a language model pre-trained on unpaired textual corpora, external semantic attention (ESA) and image features with a textual (T), semantic (S) and visual (V) layer (i.e. TSV layer) for predicting the next word in the sequence when learned using image-caption pairs. In the following, we present each of these components separately while Figure 2 presents the overall architecture of KGA-CGM.
3.1.1. Language Model
This component is crucial to transfer the sentence structure for unseen visual objects. Language model is implemented with two long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) layers to predict the next word given previous words in a sentence. If represent the input to the forward LSTM of layer-1 for capturing forward input sequences into hidden sequence vectors (), where is the final time step. Then encoding of input word sequences into hidden layer-1 sequences at each time step is achieved as follows:
Similarly, layer-1 hidden sequences supplied as input to layer-2 are encoded as follows:
where represent hidden layer parameters. Finally, the encoded hidden sequence () at time step is then used for predicting the probability distribution of the next word given by Equation 3.
The softmax layer is only used while training with unpaired textual corpora and not used when learned with image captions.
3.1.2. External Semantic Attention (ESA)
Attention mechanism was first introduced by Bahdanau et al. (2014) in an encoder-decoder architecture. It was particularly useful for dynamically changing context while decoding. Later, the attention mechanism was extended to other tasks such as speech recognition (Chorowski et al., 2015) and multimodal representational tasks. A feasible case of multimodal representational task is image caption generation, where attention is perceived from two different aspects i.e. visual attention (Xu et al., 2015) and attribute attention (You et al., 2016). Visual attention leveraged spatial image features, while attribute attention leveraged attributes obtained from an image.
Our objective from ESA is similar to attribute attention, but leverages entity annotations as semantic labels obtained using a multi-entity-label image classifier (discussed in the later section). Entity annotations obtained are analogous to image patches and attributes.
In formal terms, if is an entity annotation label and the entity annotation vector among set of entity annotation vectors () and the attention weight of then is calculated at each time step using Equation 4.
where given by Equation 5 and represent scoring function which conditions on hidden state () of a caption language model. It can be observed that the scoring function is crucial for deciding attention weights. Also, relevance of the hidden state with each entity annotation is calculated using Equation 6.
where , is a bilinear parameter matrix. Once the attention weights are calculated, the soft attention weighted annotation vector of the context , which is a dynamic representation of the caption at time step is given by Equation 7
Here, and represent the cardinality of entity class annotations per image-caption pair instance.
3.1.3. Image Features
The image aligned with captions are used to extract visual features using multi-word-label image classifier (discussed more in later sections). To be consistent with other approaches (Hendricks et al., 2016; Venugopalan et al., 2017) and for a fair comparison, our visual features () also have each index corresponding to the probability of word-label annotation in the image.
3.1.4. TSV layer
Once the output from all components is acquired, the TSV layer is employed to integrate their features i.e. textual (), semantic () and visual () yielded by language model, ESA and images respectively. Thus, TSV acts as a transformation layer for molding three different feature spaces into a single common space for prediction of next word in the sequence.
If , and represent vectors acquired at each time step from language model, ESA and images respectively. Then the integration at TSV layer of KGA-CGM is provided by Equation 8.
where and are linear conversion matrices and is the image-caption pair training dataset vocabulary size.
3.1.5. Word Prediction
The output from the TSV layer at each time step is further used for predicting the next word in the sequence using a softmax layer given by Equation 9.
3.2. KGA-CGM Training
To learn parameters of KGA-CGM, first we freeze the parameters of the language model trained using unpaired textual corpora. Thus, enabling only those parameters to be learned with image-caption pairs emerging from ESA and TSV layer such as and . KGA-CGM is now trained to optimize the cost function that minimizes the sum of the negative log likelihood of the appropriate word at each time step given by Equation 10.
Where represent the length of sentence (i.e. caption) with beginning of sentence (BOS), end of sentence (EOS) tokens at -th training sample and as a number of samples used for training.
3.3. KGA-CGM Constrained Inference
Inference is not straightforward as in earlier image caption generation approaches (Vinyals et al., 2017). Unseen/novel image objects have no parallel captions throughout training, hence they will never be generated in a caption during inference. Thus, unseen/novel objects require guidance from similar objects or external sources during inference. Earlier approaches such as DCC (Hendricks et al., 2016) have leveraged similar concepts (i.e image word-labels) to transfer weights between seen and unseen word-labels. However, similar labels are found only using word embeddings of textual corpora and are not constrained on images. This obstructs the view from an image leading to spurious results. We resolve such issues by constraining the weight transfer between seen and unseen image labels (i.e. both semantic and word) with help from KGA. As the first step, we identify closest similar word-label of unseen objects with their Glove embeddings (Pennington et al., 2014) learned using unpaired textual corpora. Now, for transferring weights between seen and unseen image labels, we perform dynamic weight transfer during test image caption generation with help of entity annotation semantic labels () provided by multi-entity-label image classifier. Whenever the word predicted by our KGA-CGM model is the closest similar word of an unseen object then the unseen object is checked for its presence in . If the unseen object is part of , then direct transfer of weights is performed between seen and unseen with [unseen,:]=[closest,:] and [unseen,:]=[closest,:] and then setting [unseen,closest], [closest,unseen]=0 to remove mutual dependencies of their presence in an image. For the next image test sample, weights are again set back to their initial states. Beam search is used to consider the best sentences at time to identify the sentence at next time step. In this research, we perform experiments with =1 and =3.
4. Learning Multi-Label Image Classifiers
It can be perceived from earlier sections that the important constituents that influence KGA are the image semantic labels and features. Image features embody objects/actions/scenes identified in an image, while semantic labels capture the external semantic attention and entity class labels. In this section, we present the approach to extract both image features and semantic labels.
4.1. Multi-Word-label Image Classifier
To extract image features, emulating Hendricks et al. (Hendricks et al., 2016) a multi-word-label classifier is built using the caption aligned to an image by extracting part-of-speech (POS) tags such as nouns, verbs and adjectives attained for each word. For example, the caption “A young child brushes his teeth at the sink” contains word-labels such as “young (JJ)”, “child (NN)”, “teeth (NN)” etc., that represent abstract concepts in an image. An image classifier is trained now with multiple word-labels using a sigmoid cross-entropy loss by fine-tuning VGG-16 (Simonyan and Zisserman, 2014) pre-trained on the training part of the ILSVRC-2012.
4.2. Multi-Entity-label Image Classifier
To extract semantic labels which are analogous to the word-labels, the multi-entity-label classifier is build with entity labels attained from a knowledge base annotation tool such as DBpedia spotlight111https://github.com/dbpedia-spotlight/. Considering the caption presented in the aforementioned section, entities extracted from the caption are “Brush222http://dbpedia.org/resource/Brush” and “Tooth333http://dbpedia.org/resource/Tooth”, which will be treated as semantic labels. An image classifier is now trained with multiple entity-labels using sigmoid cross-entropy loss by fine-tuning VGG-16 (Simonyan and Zisserman, 2014) pre-trained on the training part of the ILSVRC-2012.
5. Experimental Setup
In this section, we describe the experimental setup employed for our experiments.
5.1. Resources and Datasets
Our approaches are dependent on several resources such as toolkits etc., while datasets are utilized to conduct evaluations. In the following, resources, datasets and evaluation measures are presented.
5.1.1. Knowledge Bases (KBs)
There are several openly available KBs such as DBpedia444http://wiki.dbpedia.org/, Wikidata555https://www.wikidata.org/wiki/Wikidata:Main_Page, and YAGO666http://www.mpi-inf.mpg.de/departments/databases-and-information-systems/research/yago-naga/yago/downloads/ which cover general knowledge about entities and their relationships. We choose DBpedia as our KB for entity annotation, as it is one of the extensively used resource for semantic annotation and disambiguation (Lehmann et al., 2015).
5.1.2. Unseen Objects in Image-Caption Dataset
To evaluate KGA-CGM, we use the subset of the MSCOCO (Lin et al., 2014) dataset proposed by Hendricks et al. (2016). The dataset is obtained by clustering 80 image object category labels into 8 clusters and then selecting one object from each cluster to be held out from the training set. Now the training set does not contain the images and sentences of those 8 objects represented by bottle, bus, couch, microwave, pizza, racket, suitcase and zebra. Thus making the MSCOCO training dataset to constitute 70,194 image-caption pairs. While validation set of 40504 image-caption pairs are again divided into 20252 each for testing and validation. Now, the goal of KGA-CGM is to generate caption for those test images which contain these 8 unseen object categories.
5.2. Evaluation Measures
To evaluate the representational task of caption generation, we use evaluation metrics same as earlier approaches (Hendricks et al., 2016; Venugopalan et al., 2017; Yao et al., 2017) such as METEOR and also SPICE (Anderson et al., 2017) to check the effectiveness of generated caption. CIDEr (Vedantam et al., 2015) metric is not used as it is required to calculate inverse document frequency used by this metric across the entire test set and not just unseen object subsets. F1-score is also calculated to measure the presence of unseen objects in the generated captions when compared against reference captions.
The experiments are conducted to evaluate the efficacy of the proposed approaches and their dependencies.
6.1. Implementing Multi-Label Classifiers
The key component of unseen image object caption generation is comprehending visual information. To realize it, we re-use existing and also build our own image multi-label classifier.
6.1.1. Multi-Word-Label Classifier
The principal role of the multi-word-label classifier is to provide image features for caption generation. We use the pre-trained model (Hendricks et al., 2016) trained on the subset of MSCOCO using the approach presented earlier. The image features extracted represent the probabilities of 471 image labels occurring in a given image.
6.1.2. Multi-Entity-label Classifier
The goal of multi-entity-label classifier is to recognize multiple semantic labels per image. To build the classifier, training set of MSCOCO dataset constituting 82,783 training image-caption pairs are used to extract around 812 unique labels with an average of 3.2 labels annotated per image. Additionally, features for the images are extracted using different layers such as pool5, fc6 and fc7 of VGG-16 (Simonyan and Zisserman, 2014) pre-trained on ILSVRC-2012 for fine-tuning with our semantic labels. Also, to comprehend the contribution of aforementioned layers to the accuracy of the classifier, we throughly analyzed separately the classifiers built using pool5, fc6 and fc7 as the initialization layers before fine-tuning. Our analysis revealed that pool5 features overfit even with regularization.
To address this challenge, we trained a classifier with Caffe777http://caffe.berkeleyvision.org/ by fine-tuning the layers above fc6 and fc7 which gave us an improvement in the accuracy as observed in Table 1.
The classifier fine-tuned on fc6 features constitute two fully connected layers of 4096 and 812 neurons above fc6, with the first layer having 50% dropout and ReLU activation while the output layer comprises a sigmoid activation. Similarly, the classifier fine-tuned with fc7 features have an output layer of 812 neurons comprising a sigmoid activation. The loss function used during training is sigmoid cross-entropy, while only sigmoid is used during prediction for exhibiting the presence of label probabilities. Figure 3 shows the predictions on the test dataset. It can be observed that fc6 gave the best result with an accuracy around 70% for top-12 and 74.4% top-16.
6.2. Entity-Label Embeddings
We presented that the acquisition of labels for multi-entity-label classifiers was obtained by the DBpedia spotlight entity annotation and disambiguation tool. These labels (i.e. entities) are expected to encapsulate general knowledge (e.g. encyclopedic knowledge) which is inter linked. Approaches (Socher et al., 2013) earlier have transformed such entities in a knowledge base into embeddings to capture their relational information for tasks such as knowledge base completion. In our research as well, we see the efficacy of these embeddings for caption generation. Entity embeddings leverages external semantic information to be used for attention. To obtain entity-label embeddings, we adopted the RDF2Vec (Ristoski and Paulheim, 2016) approach and generated 500 dimensional vector representations for all 812 labels used to represent images in the entire MSCOCO.
We further qualitatively evaluate these entity-label embeddings to check there affect on caption generation. Most images are respresented with more than one entity-label, thus providing multi-label information for each image. However, directly using their embeddings for ESA can affect caption generation if the label embeddings are not closely related. To check for their closely relatedness, we perform entity similarity. Table 2 shows the results of uneen/novel mscoco objects.
|Unseen Object||Top-5 Closely Related Entities|
|Bottle||Wine_bottle, Wine_glass, Table_setting|
|Bus||Truck, Double-decker_bus, Transit_bus|
|Couch||Pillow, Cupboard, Bathtub|
|Microwave||Blender, Oven, Paper_bag|
|Pizza||Pasta, Pepperoni, Salad|
|Racket||Ball, Flying_disc, Snowboard|
|Suitcase||Baggage, Backpack, Hair_dryer|
|Zebra||Giraffe, Elephant, Horn_(anatomy)|
It can be perceived from the Table 2 that most of the closely related entities always co-occur in an image as shown with few examples in the paper. Thus enhancing the caption generation model with ESA proven to be effective.
We also performed t-SNE visualization of all entity-labels to check how cluster together. It can be seen from the Figure 4 visualization some of the closely related objects which occur in the same context cluster close to each other.
6.3. Novel Objects Description
In this section, we evaluate KGA-CGM for unseen/novel image objects caption generation.
One of the important component of KGA-CGM model is language model. Even though its weights are fixed during learning, the words in a caption are initially set with pre-trained Glove (Pennington et al., 2014) embeddings of 256 dimensions while the hidden layer dimensions are set to 512. Information about other components i.e. image features and semantic labels are already discussed in earlier sections. KGM-CGM is then trained with Adam optimizer with gradient clipping having maximum norm of 1.0 for about 1550 epochs. Validation data is used for fine tuning parameters and model selection.
6.3.2. Ablation Study
To understand how different aspects of training KGA-CGM influence the unseen image objects caption generation, we perform ablation study by removing different components of KGA-CGM. Table 3 present the results obtained.
It can be noticed that None, which refers to the no usage of either ESA or constrained inference (CI) in the KGA-CGM model have F1 measure 0. Enabling ESA into this basic caption generation model has shown an increase in the METEOR and SPICE as observed in Only ESA. However, F1 measure has remained 0 due to no transfer of weights between seen and unseen image objects. Alternatively, enabling CI showed a jump in F1 measure as seen in Only CI. However, both METEOR and SPICE are lower than Only ESA due to missing weights for ESA. Enabling both ESA and CI make our complete KGA-CGM model equipped with both semantic attention from the image as well as constrained transfer of weights providing highest METEOR and SPICE scores of 22.2 and 14.6 respectively as observed in ESA+CI. Also, it has increased F1 measure when compared to Only CI. This shows that the coherent and accurately generated caption is important for presence of an object in caption. We also analyzed the effect of beam size on KGA-CGM and it can be observed that increasing beam size during inference has shown a drop in all measures. This can be attributed to the usage of terms which are outside unseen objects captions vocabulary and are more general to entire caption dataset.
6.3.3. Quantitative Analysis
We compared our complete KGA-CGM model with the other existing models that generated captions for the unseen MSCOCO image objects. To have a fair comparison, only those results are compared that used VGG-16 to generate image features. Table 4 shows the comparison of average scores based on METEOR, SPICE and F1 on all 8 unseen image objects with beam size 1 and greater than that (). It can be noticed that KGA-CGM with beam size 1 was comparable to other approaches even though it used fixed vocabulary and image tags. For example, CBS (Anderson et al., 2017) used expanded vocabulary of 21,689 when compared to 8802 by us. Also, our word-labels per image are fixed, while CBS uses a varying size of predicted image tags (T1-4). This makes it nondeterministic and can increase uncertainty, as varying tags will either increase or decrease the performance. In Table 5, we also present individual scores from all 8 unseen objects separately. Though our average scores were comparable to other approaches, for some of the unseen objects we attain state of the art results. We also add more analysis about the important components used in the KGA-CGM model in the Appendix.
|DCC (Hendricks et al., 2016)||1||21.0||13.4||39.7|
|NOC (Venugopalan et al., 2017)||1||20.7||-||50.5|
|NOC (Venugopalan et al., 2017)||1||21.3||-||48.8|
|LSTM-C (Yao et al., 2017)||1||23.0||-||55.6|
|CBS(T4) (Anderson et al., 2017)||1||23.3||15.9||54.0|
|DCC (Hendricks et al., 2016)||1||4.6||29.8||45.9||28.1||64.6||52.2||13.2||79.9|
|F1||NOC (Venugopalan et al., 2017)||1||17.7||68.7||25.5||24.7||69.3||55.3||39.8||89.0|
|CBS(T4) (Anderson et al., 2017)||1||16.3||67.8||48.2||29.7||77.2||57.1||49.9||85.7|
|LSTM-C (Yao et al., 2017)||1||29.6||74.4||38.7||27.8||68.1||70.2||44.7||91.4|
|DCC (Hendricks et al., 2016)||1||18.1||21.6||23.1||22.1||22.2||20.3||18.3||22.3|
|METEOR||NOC (Venugopalan et al., 2017)||1||21.2||20.4||21.4||21.5||21.8||24.6||18.0||21.8|
6.3.4. Qualitative Analysis
In the Table 6, sample predictions of our best KGA-CGM model is presented. It can be observed that entity-labels has shown an influence for caption generation. Since, entities as image labels are already disambiguated, it attained high similarity in the prediction of a word thus adding useful semantics.
|Image||Unseen Object||NOC||Our Predicted Caption||Predicted Entity-Labels (Top-3)|
|bottle||A wine bottle sitting on a table next to a wine bottle||A bottle of wine sitting on top of a table||Wine_glass, Wine_bottle, Bottle|
|bus||Bus driving down a street next to a bus stop.||A white bus is parked on the street||Bus,Public_Transport,Transit_Bus|
|couch||A woman sitting on a chair with a large piece of cake on her arm||A woman sitting on a couch with a remote||Cake,Couch,Glass|
|microwave||A kitchen with a refrigerator, refrigerator, and refrigerator.||A kitchen with a microwave, oven and a refrigerator||Refrigerator,Oven,Microwave_Oven|
|pizza||A man standing next to a table with a pizza in front of it.||A man is holding a pizza in his hands||Pizza,Restaurant,Hat|
|racket||A woman court holding a tennis racket on a court||A woman playing tennis on a tennis court with a racket||Tennis, Racket_(sports_equipment), Court|
|suitcase||A cat laying on a suitcase on a bed.||A cat laying inside of a suitcase on a bed||Cat,Baggage,Black_Cat|
|zebra||Zebras standing together in a field with zebras||A group of zebras standing in a line||Zebra,Enclosure,Zoo|
7. Conclusion and Future Work
In this paper, we presented an approach to generate captions for images that lack parallel captions during training. Experimental results on unseen/novel image objects captioning exhibit that usage of structured information encapsulated in the form of relational knowledge (i.e KB) has unveiled a way to build connection between real world objects to their visual information. In future, we plan to expand our models to build multimedia knowledge bases that automatically can be queried based on relational information between images.
Appendix A Appendix
This appendix provides information on some of the quantitative and qualitative results of individual components in KGA-CGM. Also, more qualitative results of the generated captions on held-out MSCOCO objects is presented.
a.1. Language Model Hidden Layers
As presented in the paper, language model in our caption generation model (i.e KGA-CGM) is a 2-layer forward LSTM. For learning KGA-CGM with image-caption pairs, input caption word embeddings are chosen to be 256 dimensions, while the LSTM hidden layer dimensions for both layer-1 and layer-2 is selected as 512. However, varying hidden layer dimensions can show an influence on the caption generation results. In this section, we vary the hidden layer dimensions and analyze the consequences. Table 7 shows the METEOR, SPICE and F1 average measures on 8 unseen MSCOCO objects.
a.2. KGA-CGM More Qualitative Results
- Ahn et al. (2016) Sungjin Ahn, Heeyoul Choi, Tanel Pärnamaa, and Yoshua Bengio. 2016. A neural knowledge language model. arXiv preprint arXiv:1608.00318 (2016).
- Anderson et al. (2017) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017. Guided Open Vocabulary Image Captioning with Constrained Beam Search. In EMNLP.
- Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In ICCV. 2425–2433.
- Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 (2014).
- Boleda et al. (2017) Gemma Boleda, Sebastian Padó, Nghia The Pham, and Marco Baroni. 2017. Living a discrete life in a continuous world: Reference with distributed representations. arXiv preprint arXiv:1702.01815 (2017).
- Bordes et al. (2014) Antoine Bordes, Sumit Chopra, and Jason Weston. 2014. Question answering with subgraph embeddings. arXiv preprint arXiv:1406.3676 (2014).
- Chorowski et al. (2015) Jan K Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio. 2015. Attention-based models for speech recognition.. In NIPS. 577–585.
- Chrupała et al. (2015) G. Chrupała, A. Kádár, and A. Alishahi. 2015. Learning language through pictures. arXiv preprint arXiv:1506.03694 (2015).
- Fereshteh et al. (2015) Sadeghi Fereshteh, Santosh K. Kumar Divvala, and Ali Farhadi. 2015. Viske: Visual knowledge extraction and question answering by visual verification of relation phrases.. In CVPR. 1456–1464.
- Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. arXiv preprint arXiv:1603.06393 (2016).
- Hendricks et al. (2016) Lisa Anne Hendricks, Subhashini Venugopalan, Marcus Rohrbach, Raymond Mooney, Kate Saenko, and Trevor Darrell. 2016. Deep compositional captioning: Describing novel object categories without paired training data. In CVPR. 1–10.
- Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9, 8 (1997), 1735–1780.
- Jin et al. (2015) Junqi Jin, Kun Fu, Runpeng Cui, Fei Sha, and Changshui Zhang. 2015. Aligning where to see and what to tell: image caption with region-based attention and scene factorization. arXiv preprint arXiv:1506.06272 (2015).
- Johnson et al. (2015) Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma Shamma, Michael Bernstein, and Li. Fei-Fei. 2015. Image retrieval using scene graphs.. In CVPR. 3668–3678.
- Lazaridou et al. (2015) A. Lazaridou, N.T. Pham, and M. Baroni. 2015. Combining language and vision with a multimodal skip-gram model. arXiv preprint arXiv:1501.02598 (2015).
- Lebret et al. (2016) Rémi Lebret, David Grangier, and Michael Auli. 2016. Generating text from structured data with application to the biography domain. ArXiv e-prints, March (2016).
- Lehmann et al. (2015) Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, et al. 2015. DBpedia–a large-scale, multilingual knowledge base extracted from Wikipedia. Semantic Web (2015).
- Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In ECCV. Springer, 740–755.
- Lu et al. (2016) Jiasen Lu, Caiming Xiong, Devi Parikh, and Richard Socher. 2016. Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning. arXiv preprint arXiv:1612.01887 (2016).
- Merity et al. (2016) Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer Sentinel Mixture Models. arXiv preprint arXiv:1609.07843 (2016).
- Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation.. In EMNLP. 1532–1543.
- Plummer et al. (2015) Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.. In ICCV. 2641–2649.
- Ristoski and Paulheim (2016) Petar Ristoski and Heiko Paulheim. 2016. Rdf2vec: Rdf graph embeddings for data mining. In International Semantic Web Conference. Springer, 498–514.
- Serban et al. (2016) Iulian Vlad Serban, Alberto García-Durán, Caglar Gulcehre, Sungjin Ahn, Sarath Chandar, Aaron Courville, and Yoshua Bengio. 2016. Generating factoid questions with recurrent neural networks: The 30m factoid question-answer corpus. arXiv preprint arXiv:1603.06807 (2016).
- Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
- Socher et al. (2013) Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013. Reasoning with neural tensor networks for knowledge base completion. In Advances in neural information processing systems. 926–934.
- Srivastava and Salakhutdinov (2012) N. Srivastava and R.R. Salakhutdinov. 2012. Multimodal learning with deep boltzmann machines.. In NIPS. 2222–2230.
- Vedantam et al. (2015) Ramakrishna Vedantam, Lawrence C. Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In CVPR. 4566–4575.
- Venugopalan et al. (2017) Subhashini Venugopalan, Lisa Anne Hendricks, Marcus Rohrbach, Raymond Mooney, Trevor Darrell, and Kate Saenko. 2017. Captioning images with diverse objects. In CVPR.
- Venugopalan et al. (2014) Subhashini Venugopalan, Huijuan Xu, Jeff Donahue, Marcus Rohrbach, Raymond Mooney, and Kate Saenko. 2014. Translating videos to natural language using deep recurrent neural networks. arXiv preprint arXiv:1412.4729 (2014).
- Vinyals et al. (2017) Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2017. Show and tell: Lessons learned from the 2015 mscoco image captioning challenge. IEEE transactions on pattern analysis and machine intelligence 39, 4 (2017), 652–663.
- Xu et al. (2015) Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C Courville, Ruslan Salakhutdinov, Richard S Zemel, and Yoshua Bengio. 2015. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. In ICML, Vol. 14. 77–81.
- Yao et al. (2017) Ting Yao, Pan Yingwei, Li Yehao, and Tao Mei. 2017. Incorporating Copying Mechanism in Image Captioning for Learning Novel Objects. In CVPR.
- You et al. (2016) Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image captioning with semantic attention. In CVPR. 4651–4659.