A Verb Prediction

Situation Recognition with Graph Neural Networks


We address the problem of recognizing situations in images. Given an image, the task is to predict the most salient verb (action), and fill its semantic roles such as who is performing the action, what is the source and target of the action, \etc. Different verbs have different roles (\eg attacking has weapon), and each role can take on many possible values (nouns). We propose a model based on Graph Neural Networks that allows us to efficiently capture joint dependencies between roles using neural networks defined on a graph. Experiments with different graph connectivities show that our approach that propagates information between roles significantly outperforms existing work, as well as multiple baselines. We obtain roughly 3-5% improvement over previous work in predicting the full situation. We also provide a thorough qualitative analysis of our model and influence of different roles in the verbs.


1 Introduction

Object [14, 33, 36], action [35, 41], and scene classification [51, 52] have come a long way, with performance in some of these tasks almost reaching human agreement. However, in many real world applications such as robotics we need a much more detailed understanding of the scene. For example, knowing that an image depicts a repairing action is not sufficient to understand what is really happening in the scene. We thus need additional information such as the person repairing the house, and the tool that is used.

Figure 1: Understanding an image involves more than just predicting the most salient action. We need to know who is performing this action, what tools (s)he may be using, \etc. Situation recognition is a structured prediction task that aims to predict the verb and its frame that consists of multiple role-noun pairs. The figure shows a glimpse of our model that uses a graph to model dependencies between the verb and its roles.

Several datasets have recently been collected for such detailed understanding [22, 27, 48]. In [22], the Visual Genome dataset was built containing detailed relationships between objects. A subset of the scenes were further annotated with scene graphs [17] to capture both unary (\eg attributes) and pairwise (\eg relative spatial info) object relationships. Recently, Yatskar et al. [48] extended this idea to actions by labeling action frames where a frame consists of a fixed set of roles that define the action. Fig. 1 shows a frame for action repairing. The challenge then consists of assigning values (nouns) to these roles based on the image content. The number of different role types, their possible values, as well as the number of actions are very large, making it a very challenging prediction task. As shown in Fig. 2, the same verb can appear in very different image contexts, and nouns that fill the roles are vastly different.

In [48], the authors proposed a Conditional Random Field (CRF) to model dependencies between verb-role-noun pairs. In particular, a neural network was trained in an end-to-end fashion to both, predict the unary potentials for verbs and nouns, and to perform inference in the CRF. While their model captured the dependency between the verb and role-noun pairs, dependencies between the roles were not modeled explicitly.

In this paper, we aim to jointly reason about verbs and their roles using a Graph Neural Network (GNN), a generalization of graphical models to neural networks. A GNN defines observation and output at each node in the graph, and propagates messages along the edges in a recurrent manner. In particular, we exploit the GNNs to also model dependencies between roles and predict a consistent structured output. We explore different connectivity structures among the role nodes, and show that our approach significantly improves performance over existing work. In addition, we compare with strong baseline methods using Recurrent Neural Networks (RNNs) that have been shown to work well on joint prediction tasks, such as semantic [50] and object instance [3] segmentation, as well as on group activity recognition [8]. We also visualize the learned models to further investigate dependencies between roles.

Figure 2: Images corresponding to the same verb can be quite different in their content involving verb roles. This makes situation recognition difficult.

2 Related Work

Situation recognition generalizes action recognition to include actors, objects, and location in the activity. There has been work to combine activity recognition with scene or object labels [7, 12, 45, 46]. In [13, 31], visual semantic role labeling tasks were proposed where datasets are built to study action along with localization of people and objects. In another line of work, Yatskar \etal [48] created the imSitu dataset that uses linguistic resources from FrameNet [10] and WordNet [29] to associate images not only with verbs, but also with specific role-noun pairs that describe the verb with more details. As a baseline approach, in [48], a Conditional Random Field (CRF) jointly models prediction of the verb and verb-role-noun triplets. Further, considering that the large output space and sparse training data could be problematic, a tensor composition function was used [47] to share nouns across different roles. The authors also proposed to augment the training data by searching images using query phrases built from the structured situation.

Different from these methods, our work focuses on explicitly modeling dependencies between roles for each verb through the use of different neural architectures.

Understanding Images.

There is a surge of interest in joint vision and language tasks in recent years. Visual Question Answering in images and videos [1, 38] aims to answer questions related to image or video content. In image captioning [19, 40, 43, 26], a natural language sentence is generated to describe the image. Approaches for these tasks often use the CNN-RNN pipelines to provide a caption, or a correct answer to a specific question. Dependencies between verbs and nouns are typically being implicitly learned with the RNN. An alternative is to list all important objects with their attributes and relationships. Johnson \etal [17] created scene graphs, which are being used for visual relationship detection [27, 30, 49] tasks. In [25], the authors exploit scene graphs to generate image captions.

In Natural Language Processing (NLP), semantic role labeling [11, 18, 20, 32, 44, 53] involves annotating a sentence with thematic or semantic roles. Building upon resources from NLP, and leveraging collections such as FrameNet [10] and WordNet [29], visual semantic role labeling, or situation recognition, aims to interpret details for one particular action with verb-role-noun pairs.

Graph Neural Networks.

There are a few different ways for applying neural networks to graph-structured data. We divide them into two categories. The first group defines convolutions on graphs. Approaches like [2, 6, 21] utilized the graph Laplacian and applied CNNs to spectral domain. Differently, Duvenaud et al. [9] designed a special hash function such that a CNN can be used on the original graphs.

The second group applies feed-forward neural networks to every node of the graph recurrently. Information is propagated through the network by dynamically updating the hidden state of each node based on their history and incoming messages from their neighborhood. The Graph Neural Network (GNN) proposed by [34] utilized multi-layer perceptrons (MLP) to update the hidden state. However, their learning algorithm is restrictive due to the contraction map assumption. In the following work, the Gated Graph Neural Network (GGNN) [23] used a recurrent gating function [4] to perform the update, and effectively learned model parameters using back-propagation through time (BPTT).

Other work [24, 37] designed special update functions based on the LSTM [16] cell and applied the model to tree-structured or general graph data. In [28], knowledge graphs and GGNNs are used for image classification. Here we use GGNNs for situation recognition.

3 Graph-based Neural Models for Situation Recognition

Task Definition.

Situation recognition as per the imSitu dataset [48] assumes a discrete set of verbs , nouns , roles , and frames . The verb and its corresponding frame that contains roles are obtained from FrameNet [10], while nouns come from WordNet [29]. Each verb is associated with a frame that contains a set of semantic roles . Each role is paired with a noun value . Here, indicates that the noun is unknown or not applicable. A set of semantic roles and their nouns is called a realized frame, denoted as , where each role is with a noun.

Given an image, the task is to predict the structured situation , specified by a verb and its corresponding realized frame . For example, as shown on the right of Fig. 2, the verb riding is associated with three role-noun pairs, i.e., {agent:dog, vehicle:surfboard, place:sidewalk}.

Figure 3: The architecture of fully-connected roles GGNN. The undirected edges between all roles of a verb-frame allows to fully capture the dependencies between them.

3.1 Graph Neural Network

The verb and semantic roles of a situation depend on each other. For example, in the verb carrying, the roles agent and agent-part are tightly linked with the item being carried. Small items can be carried by hand, while heavy items may be carried on the back. We propose modeling these dependencies through a graph . The nodes in our graph are of two types of verb or role, and take unique values of or , respectively. Since each image in the dataset is associated with one unique verb, every graph has a single verb node. Edges in the graph encode dependencies between role-role or verb-role pairs, and can be directed or undirected. Fig. 1 shows an example of such a graph where verb and role nodes are connected to each other.


Modeling structure and learning representation on graphs have prior work. Gated Graph Neural Networks (GGNNs) [23] is one approach that learns the representation of a graph, which is then used to predict node- or graph-level output. Each node of a GGNN is associated with a hidden state vector that is updated in a recurrent fashion. At each time step, the hidden state of a node is updated based on its history and incoming messages from its neighbors. These updates are applied simultaneously to all nodes in the graph at each propagation step. The hidden states after propagation steps are used to predict the output. In contrast, a standard unrolled RNN only moves information in one direction and updates one “node” per time step.

GGNN for Situation Recognition.

We adopt the GGNN framework to recognize situations in images. Each image is associated with one verb that corresponds to a frame with a set of roles . We instantiate a graph for each image that consists of one verb node, and (number of roles associated with the frame) role nodes. To capture the dependency between roles to the full extent, we propose creating undirected edges between all pairs of roles. Fig. 3 shows two example graph structures of this type. We explore other edge configurations in the evaluation.

To initialize the hidden states for each node, we use features derived from the image. In particular, for every image , we compute representations and using the penultimate fully-connected layer of two convolutional neural network (CNN) pre-trained to predict verbs and nouns, respectively. We initialize the hidden states of the verb node and role node as


where corresponds to a one-hot encoding of the predicted verb and is a one-hot encoding of the role that the node corresponds to. is the verb embedding matrix, and is the role embedding matrix. and are parameters that transform image features to the space of hidden representations. corresponds to element-wise multiplication, and is a non-linear function such as or ReLU (). We normalize the initialized hidden states to unit-norm prior to propagation.

For any node , at each time step, the aggregation of incoming messages at time is determined by the hidden states of its neighbors :


Note that we use a shared linear layer of weights and biases to compute incoming messages across all nodes.

After aggregating the messages, the hidden state of the node is updated through a gating mechanism similar to the Gated Recurrent Unit [4, 23] as follows:


This allows each node to softly combine the influence of the aggregated incoming message and its own memory. , , , , , , , , and are the weights and biases of the update function.

Output and Learning.

We run propagation steps. After propagation, we extract node-level outputs from GGNN to predict the verb and nouns. Specifically, for each image, we predict the verb and a set of nouns for each role associated with the verb frame using a softmax layer:


Note that the softmax function is applied across the class space for verbs and nouns . can be treated as the probability of assigning noun to role .

Figure 4: The architecture of chain RNN for verb riding. The time-steps at which different roles are predicted needs to be decided manually, and has an influence on the performance.

Each image in the imSitu dataset comes with three sets of annotations (from three annotators) for the nouns. During training, we accumulate the cross-entropy loss at verb and noun nodes for every annotation as


where and correspond to the ground-truth verb for image and the ground-truth noun for role of the image, respectively. Different to the Soft-OR loss in [48], we encourage the model to predict all three annotations for each image. We use back-propagation through time (BPTT) [42] to train the model.


At test time, our approach first predicts the verb to choose a corresponding frame and obtain the set of associated roles . We then propagate information among role nodes and choose the highest scoring noun for each role. Thus our predicted situation is


To reduce reliance on the quality of verb prediction, we explore beam search over verbs as discussed in Experiments.

3.2 Simpler Graph Architectures

An alternative to model dependencies between nodes is to use recurrent neural networks (RNN). Here, situation recognition can be considered as a sequential prediction problem of choosing the verb and corresponding noun-role pairs. The hidden state of the RNN carries information across the verb and noun-role pairs, and the input at each time-step dictates what the RNN should predict.

Chain RNN.

An unrolled RNN can be seen as a special case of a GGNN, where nodes form a chain with directed edges between them. However, there are a few notable differences, wherein the nodes receive information only once from their (left) neighbor. In addition, the nodes do not perform steps of propagation among each other and predict output immediately after the information arrives.

In the standard chain structure of a RNN, we need to manually specify the order of the verb and roles. As the choice of the verb dictates the set of roles in the frame, we predict the verb at the first time step. We observe that the imSitu dataset and any verb-frame in general, commonly consist of place and agent-like roles (\eg semantic role teacher can be considered as the agent for the verb teaching). We thus predict place and agent roles as the second and third roles in the chain 1. We make all other roles for the frame to follow subsequently in descending order of the number of times they occur across all verb-frames. Fig. 4 shows an example of such a model.

For a fair comparison to the fully connected roles GGNN, we employ the GRU update in our RNN. The input to the hidden states matches node initialization (Eqs. 1 and 2). We follow the same scheme for predicting the output (linear layer with softmax), and train the model with the same cross-entropy loss.

Method top-1 predicted verb top-5 predicted verbs ground truth verbs
verb value value-all verb value value-all value value-all mean
1 Unaries 36.32 23.74 13.86 61.51 38.57 20.76 58.32 27.57 35.08
2 Unaries, BS=10 36.39 23.74 14.01 61.65 38.64 20.96 58.32 27.57 35.16
3 FC Graph, =1 36.25 25.99 17.02 61.60 42.91 26.44 64.87 35.52 38.83
4 FC Graph, =2 36.43 26.08 17.22 61.52 42.86 26.38 65.31 35.86 38.96
5 FC Graph, =4 36.46 26.26 17.48 61.42 43.06 26.74 65.73 36.43 39.19
6 FC Graph, =4, BS=10 36.70 26.52 17.70 61.63 43.34 27.09 65.73 36.43 39.39
7 FC Graph, =4, BS=10, vOH 36.93 27.52 19.15 61.80 45.23 29.98 68.89 41.07 41.32
8 FC Graph, =4, BS=10, vOH, =ReLU 36.26 27.22 19.10 62.14 45.59 30.32 69.35 41.71 41.46
9 FC Graph, =4, BS=10, vOH, Soft-OR 36.75 27.33 18.94 61.69 44.91 29.41 68.29 40.25 40.95
Table 1: Situation prediction results on the development set. We compare several variants of our fully-connected roles model to show the improvements achieved at every step. refers to the number of time-steps of propagation in the fully connected roles GGNN (FC Graph). BS=10 indicates the use of beam-search with beam-width of 10. vOH (verb, one-hot) is included when the embedding of the predicted verb is used to initialize the hidden state of the role nodes. =ReLU refers to the non-linear function used after initialization. All other rows use =. Finally, Soft-OR refers to the loss function used in [48]. Best performance is in bold and second-best is italicized.
Figure 5: The architecture of tree-structured RNN. Like the Chain RNN, verb prediction is at the root of the tree, and semantic roles agent-like and place are parents of all other roles.

Tree-structured RNN.

As mentioned above, the place and agent semantic roles occur more frequently. We propose a structure where they have a larger chance to influence prediction of other roles. In particular, we create a tree-structured RNN [37] where the hidden states first predict the verb, followed by agent and place, and all other roles. Fig. 5 shows examples of resulting structures.

The tree-structured RNN can be deemed as a special case of GGNN, where nodes have the following directed edges:


where , and represents all roles in that frame other than agent and place. Similar to the chain RNN, we use GRU update and follow the same learning and inference procedures.

4 Evaluation

We evaluate our methods on the imSitu dataset [48] and use the standard splits with 75k, 25k, and 25k images for the train, development, and test subsets, respectively. Each image in imSitu is associated with one verb and three annotations for the role-noun pairs.

We follow [47] and report three metrics: (i) verb: the verb prediction performance; (ii) value: the semantic verb-role-value tuple prediction performance that is considered to be correct if it matches any of the three ground truth annotators; and (iii) value-all: the performance when the entire situation is correct and all the semantic verb-role-value pairs match at least one ground truth annotation.

4.1 Implementation Details

Image Representations.

We adopt two pre-trained VGG-16 CNNs [36] for extracting image features by removing the last fully-connected and softmax layers, and fine-tuning all weights. The first CNN () is trained to predict verbs, and second CNN () predicts the top most frequent nouns ( cover about 95% of nouns) in the dataset.


Creating a graph with no edges, or equivalently with steps of propagation corresponds to using the initialized features to perform prediction. We refer to this approach as Unaries, which will be used as the simplest baseline to showcase the benefit of modeling dependencies between the roles.


We implement the proposed models in Torch [5]. The network is trained using RMSProp [15] with mini-batches of 256 samples. We choose the hidden state dimension , and train image (), verb () and role () embeddings. The image features are extracted before training the GGNN or RNN models.

The initial learning rate is and starts to decay after 10 epochs by a factor of . We use dropout with a probability of on the output prediction layer (\cf Eqs. 5 and 6) and clip the gradients to range .

Mapping agent Roles.

The imSitu dataset [48] has situations for 504 verbs. Among them, we notice that 19 verbs do not have the semantic role agent but instead with roles of similar meaning (\eg verb educating has role teacher). We map these alternative roles to agent when determining their position in the RNN architecture. Such a mapping is not used for the fully connected GGNN model.

Variable Number of Roles.

A verb has a maximum of 6 roles associated with it. We implement our proposed model with fixed-size graphs involving 7 nodes. To deal with verbs with less than 6 roles, we zero the hidden states at each time-step of propagation, making them not receive or send any information.

top-1 predicted verb top-5 predicted verbs ground truth verbs
verb value value-all verb value value-all value value-all mean
1 Unaries 36.39 23.74 14.01 61.65 38.64 20.96 58.32 27.57 35.16
2 Chain RNN 34.62 24.67 17.94 61.09 41.67 27.80 62.58 36.57 38.36
3 Tree-structured RNN 34.62 24.24 16.04 58.86 39.15 23.65 60.44 30.91 35.98
4 Chain GGNN, =8 36.63 27.27 19.03 61.88 44.97 29.44 68.20 40.21 40.95
5 Tree-structured GGNN, =6 36.78 27.48 19.54 61.75 45.12 30.11 68.54 41.01 41.29
6 Fully-connected GGNN, =4 36.93 27.52 19.15 61.80 45.23 29.98 68.89 41.07 41.32
Table 2: Situtation prediction results on the development set for models with different graph structures. All models use beam search, predicted verb embedding, and . Best performance is highlighted in bold, and second-best in each table section is italicized.
top-1 predicted verb top-5 predicted verbs ground truth verbs
verb value value-all verb value value-all value value-all mean
dev CNN+CRF [48] 32.25 24.56 14.28 58.64 42.68 22.75 65.90 29.50 36.32
Tensor Composition [47] 32.91 25.39 14.87 59.92 44.50 24.04 69.39 33.17 38.02
Tensor Composition + DataAug [47] 34.20 26.56 15.61 62.21 46.72 25.66 70.80 34.82 39.57
Chain RNN 34.62 24.67 17.94 61.09 41.67 27.80 62.58 36.57 38.36
Fully-connected Graph 36.93 27.52 19.15 61.80 45.23 29.98 68.89 41.07 41.32
test CNN+CRF [48] 32.34 24.64 14.19 58.88 42.76 22.55 65.66 28.96 36.25
Tensor Composition [47] 32.96 25.32 14.57 60.12 44.64 24.00 69.20 32.97 37.97
Tensor Composition + DataAug [47] 34.12 26.45 15.51 62.59 46.88 25.46 70.44 34.38 39.48
Chain RNN 34.63 24.65 17.89 61.06 41.73 28.15 62.94 37.32 38.54
Fully-connected Graph 36.72 27.52 19.25 61.90 45.39 29.96 69.16 41.36 41.40
Table 3: We compare situation prediction results on the development and test sets against state-of-the-art models. Each model was run on the test set only once. Our model shows significant improvement in the top-1 prediction on all metrics, and performs better than a baseline that uses data augmentation. The performance improvement on the value-all metric is important for applications, such as captioning and QA. Best performance is highlighted in bold, and second-best is italicized.

4.2 Results

We first present a quantitative analysis comparing different variants of our proposed model. We then evaluate the performance of different architectures, and compare results with state-of-the-art approaches.

Ablative Analysis

A detailed study of the GGNN model with fully connected roles (referred to as FC Graph) is shown in Table 1. An important hyper-parameter for the GGNN model is the number of propagation steps . We found that the performance increases by a small amount when increasing , and saturates soon (in rows 3, 4, and 5). We believe that this is due to the use of a fully-connected graph, and all nodes sharing most of the information at the first-step propagation. Nevertheless, the propagation is important, as revealed in the comparison between Unaries () from row 1 and in row 3. We obtain a mean improvement of 3.8% in all metrics.

During test we have the option of using beam search, where we hold best verb predictions and compute the role-noun predictions for each of the corresponding graphs (frames). Finally, we select the top prediction using the highest log-probability across all options. We use a beam width of in our experiments, which yields small improvement. Rows 1 and 2 of Table 1 show the improvement using beam search on a graph without propagation. Rows 5 and 6 show the benefit after multiple steps of propagation.

Rows 6 and 7 of Table 1 demonstrate the impact of using embeddings of the predicted verb (vOH) to initialize the role nodes’ hidden states in Eq. (2). Notable improvement is obtained when using the ground-truth verb (3-4%). The value-all for the top-1 predicted verb increases from 17.70% to 19.15%. We also tested different non-linear functions for initialization, \ie, (row 7) or ReLU (row 8), however, the impact is almost negligible. We thus use for all experiments.

Finally, comparing rows 7 and 9 of Table 1 reveals that our loss function to predict all annotations in Eq. (7) performs slightly better than the Soft-OR loss that aims to fit at least one of the annotations [48].

Figure 6: Example images with their predictions listed from all methods. Roles are marked with a blue background, and predicted nouns are in green boxes when correct, and red when wrong. Using the FC Graph corrects mistakes made by the Unaries or Chain RNN prediction models.

Baseline RNNs.

Table 2 summarizes the results with different structures on the dev set. As expected, Unaries perform consistently worse than models with information propagation between nodes on the value and value-all metrics. The Tree-structured RNN provides a boost in value-all for top-1 predicted verb, while the Chain RNN provides a improvement. Owing to the better connectivity between the roles in a Chain RNN (especially place and agent), we observe better performance compared to the Tree-structured RNN. Note that as the RNNs are trained jointly to predict both verbs and nouns, and as the noun gradients dominate, the verb prediction takes a hit.

Different Graph Structures.

We can also use chain or tree-structured graphs in GGNN. Along with the FC graph in row 6 of Table 2, rows 4 and 5 present the results for different GGNN structures. They show that connecting roles with each other is critical and sharing information helps. Interestingly, the Chain GGNN needs more propagation steps (=8), as it takes time for the left-most and right-most nodes to share information. Smaller values of are possible when nodes are well-connected as in Tree-structured (=6) or FC Graph (=4). Fig. 6 presents prediction from all models for two images. The FC Graph is able to reason about associating cheese and pizza rather than sprinkling meat or food on it.

Comparison with State-of-the-art.

We compare the performance of our models against state-of-the-art on both the dev and test sets in Table 3. Our CNN predicts the verb well. Beam search leads to even better performance (2-4% higher) in verb prediction. We note that Tensor Composition + DataAug actually uses more data to train models. Nevertheless, we achieve the best performance on all metrics when using the top-1 predicted verb.

Another advantage of our model is in improvement for the value-all metric. It yields +8% when using the ground-truth verb, +6% with top-5 predicted verbs, and +4.5% with top-1 predicted verb, compared with the baseline without data augmentation. Interestingly, even with data augmentation, we outperform [47] by 3-4% in value-all for top-1 predicted verb. This property attributes to information sharing between role nodes, which helps in correcting errors and better predicts frames. Note that value-all is an important metric to measure a full understanding of the image. Models with higher value-all will likely lead to better captioning or question-answering results.

4.3 Further Discussion

We delve deeper into our model and discuss why the FC Graph outperforms baselines.

Figure 7: We present the “amount” of information that is propagated between roles for two verbs along with sample images. Blue corresponds to high, and green to zero. Each element of the matrix corresponds to the norm of the incoming message from different roles (normalized column sum to 1). Left: verb fastening needs to pay attention to the tool used. Right: important components to describe jumping are the agent and obstacles along the path.
Figure 8: Images with ground-truth and top-1 predictions from the development set. Roles are marked with blue background. Ground-truth (GT) nouns are in yellow and predicted (PRED) nouns with green when correct, or red when wrong. Although the predicted verb is different from the ground-truth, it is very plausible. Some of the verbs refer to the same frame (\eg sitting and slouching), and contain the same set of roles, which our model is able to correctly infer.
Figure 9: Images with top-1 predictions from the development set. For all samples, the predicted verb is correct, shown below the image in bold. Roles are marked with a blue background, and predicted nouns are in green when correct, and red when wrong. Top row: We are able to correctly predict the situation (verb and all role-noun pairs) for all samples. Bottom row: The first three samples contain errors in prediction (\eg the agent for the verb pawing is clearly a dog). However, the latter three samples are in fact correct predictions that are not found in the ground-truth annotations (\eg people are in fact camping in the forest).

Learned Structure.

A key emphasis of this model is on information propagation between roles. In Fig. 7, we present the norms of the propagation matrices. Each element in the matrix is the norm of the incoming message from role to averaged across all images (in dev set) at the first time-step, i.e., regarding Eq. (3). In this example, tool is important for the verb fastening and influences all other roles, while agent and obstacle influence roles in jumping.

Wrong Verb Predictions.

We present a few examples of top scoring results where the verb prediction is wrong in Fig. 8. Note that in fact these predicted verbs are plausible options for the given images. The metric value treats them as wrong, and yet we can correctly predict the role-noun pairs. One example is the middle one of slouching vs. sitting. Fig. 8 (bottom) shows that choosing a different verb might lead to the selection of different roles (goalitem vs. item, destination). Nevertheless, predicting book for browsing is a good choice.

Predictions with Correct Verb.

Fig. 9 shows several examples of prediction obtained by FC Graph, where the predicted verb matches the ground-truth one. The top row corresponds to samples where the metric value-all scores correctly as all role-noun pairs are correct. Note that the roles are closely related (\eg(agent, clungto) and (material, dye)) and help each other choose the correct nouns. In the bottom row, we show some failure cases in predicting role-noun pairs. First, the model favors predicting place as outdoor (a majority of place is outdoor in the training set). Second, for the sample with verb picking, we predict the crop as apple, which appears 79 times in the dataset compared with cotton that appears 14 times. Providing more training samples (\eg [47]) could help remedy such issues.

In the latter three samples of the bottom row, although the model makes reasonable predictions, they do not match the ground-truth. For example, the ground-truth annotation for the verb taxiing is agent:jet and for the verb camping is agent:persons. Therefore, even though each image comes with three annotations, synonymous nouns and verbs make the task still challenging.

5 Conclusion

We presented an approach for recognizing situations in images that involves predicting the correct verb along with its corresponding frame consisting of role-noun pairs. Our Graph Neural Network (GNN) approach explicitly models dependencies between verb and roles, allowing nouns to inform each other. On a benchmark dataset imSitu, we achieved 4.5% accuracy improvement on a metric that evaluates correctness of the entire frame (value-all). We presented analysis of our model, demonstrating the need to capture the dependencies between roles, and compared it with RNN models and other related solutions.


This work is in part supported by a grant from the Research Grants Council of the Hong Kong SAR (project No. 413113). We also acknowledge support from NSERC, and GPU donations from NVIDIA.

Supplementary Material

We present additional analysis and results of our approach in the supplementary material. First, we analyze the verb prediction performance in Sec. A. In Sec. B, we present t-SNE [39] plots to visualize the verb and role embeddings. We present several examples of the influence of different roles on predicting the verb-frame correctly. This is visualized in Sec. C through propagation matrices similar to Fig. 7 of the main paper. Finally, in Sec. D we include several example predictions that our model makes.

Appendix A Verb Prediction

We present the verb prediction accuracies for our fully-connected model on the development set in Fig. 10. The random performance is close to 0.2% (504 verbs). About 22% of all verbs are classified correctly over 50% of the time. These include taxiing, erupting, flossing, microwaving, \etc. On the other hand, verbs such as attaching, making, placing can have very different image representations, and show prediction accuracies of less than 10%.

Our model helps improve the role-noun predictions by sharing information across all roles. Nevertheless, if the verb is predicted incorrectly, the whole situation is treated as incorrect. Thus, verb prediction performance plays a crucial role.

Figure 10: Verb prediction accuracy on the development set. Some verbs such as taxiing typically have a similar image (a plane on the tarmac), while verbs such as rubbing or twisting can have very different corresponding images.

Confusion between similar verbs.

We analyze the confusion between similar verbs, that according to the metrics, leads to incorrect situation recognition. In the main paper, Fig. 8 presents a few examples where we are able to correctly predict the roles, but the situation is classified as wrong since the verb is incorrect.

The imSitu dataset consists of 504 verbs, and while we do have a complete confusion matrix, visualizing the results is hard. As explained in the dataset [48], the verb frames were obtained using FrameNet. We notice that the 504 verbs from the imSitu dataset are grouped into 161 FrameNet verbs [10]. For example, several verbs such as walking, climbing, skipping, prowling and 26 others are clustered together to the FrameNet verb: self_motion. The clusters need not be large, and 73 of 161 clusters consist of just one verb.

We use this as a clustering, and present several confusion matrices for verb clusters in Fig. 11. All verb predictions that do not belong to the cluster are grouped as others. While, the others column does collect most of the predictions, there is significant confusion between similar verbs.

Figure 11: Confusion matrices for verb prediction. Each row indicates the expected ground-truth, and the columns are predictions (each row sums to 100%). As it is not possible to show all 504 verbs, we pick verb clusters based on their FrameNet labels (shown in the title). Confusion between remaining verbs not in the cluster is grouped in the last column as others. The examples show significant confusion between verbs which are hard to differentiate visually: colliding-crashing-ramming, or crying-giggling-laughing-weeping.

Appendix B Verb and Role Embeddings

We initialize the hidden states of our role nodes (\cf Eq. 2 of the main paper) with


where, and are verb and role embeddings respectively, and and are one-hot vectors representing the noun for a specific role, and the predicted verb. is the image representation using the noun-prediction CNN. Note that both verbs and roles are embedded to a space.

Figure 12: 2D t-SNE representation of the all the learned verb embeddings. While the number of labels is quite large, it is still possible to see small clusters of verbs forming at the periphery of the figure. top: farming-harvesting, pouring-emptying-milking, slicing-chopping-peeling. top-right: carting-wheeling-heaving, pinching-poking. right: providing-giving, offering-begging-serving, reading-squinting-staring. bottom-right: betting-gambling, grieving-mourning, baptizing-praying. bottom: glowing-flaming, bubbling-overflowing, sniffing-smelling. bottom-left: landing-taxiing, dialing-calling-phoning-typing, boating-rowing. left: drinking-lapping, microwaving-baking, mining-climbing-descending. top-left: dusting-scrubbing-cleaning-wiping, drying-hanging, repairing-fixing-installing.
Figure 13: 2D t-SNE representation of the learned verb embeddings of the verbs belonging to 11 largest clusters (using FrameNet verb clustering). The clusters are: attaching, body_movement, cause_harm, cause_motion, closure, filling, manipulation, operate_vehicle, placing, removing, self_motion. Each cluster is assigned a unique color from the jet colormap. Our model is even able to learn to embed similar verbs across these FrameNet groupings. For example, it brings together whirling (FrameNet: cause_motion) and dancing (FN: self_motion); raking (FN: cause_motion) and shoveling (FN: removing); packing (FN: placing) and unpacking (FN: filling); throwing (FN: cause_motion) and kicking (FN: cause_harm); and many others.


The dataset consists of 504 verbs. We first show a plot depicting all verbs in Fig. 12. Owing to the number of verbs, this is quite hard to see, nevertheless, we can still observe clusters of similar verbs (\eg dusting-cleaning- scrubbing-wiping, recording-singing-performing\etc.).

Additionally, we use the verb clustering afforded by the FrameNet verb associations, and select a set of 196 verbs from the 11 largest clusters (cluster size ). We present their embeddings in Fig. 13. The learned embeddings not only discover the clustering, but are also able to associate across clusters. For example, (in the top-left corner), applying and smearing belong to the Placing FrameNet verb, while spreading and buttering correspond to Filling in FrameNet. Nevertheless, our model is able to learn that these verbs may have similar context (\eg buttering bread), and brings their representations close.

Figure 14: 2D t-SNE representation of the learned role embeddings. Note how semantic roles capturing similar themes are brought together. For example, blocked-blocker, or recipients-distributed, or payment-goods. Additionally, related semantic roles that apply across verbs are also brought together. For example, components-instrument-object-part, or liquid- drencheditem, foodcontainer-glue-connector. As most roles do not present a natural clustering, we are unable to color all roles, and they are shown in black. Colored roles are associated with one unique verb.


The dataset comes with 190 roles, however, 139 of them are unique to one verb. For example, the roles top and bottom appear only once, in the frame for the verb stacking. Similarly, roles shape and cloth appear only when the verb is folding. We present two-dimensional t-SNE [39] representations of the learned role embeddings in Fig. 14. We associate same colors with role pairs that are associated with only one verb (there are only 12 such pairs, accounting for 24 of 190 roles). All other roles are shown in black. In the Fig. 14, we see that the strongly related pairs that are unique to one verb (and colored) are very close to each other. Additionally, other semantic roles that are related, \eg food, heatsource, container (right side of figure) are also close together.

Appendix C Visualizing the propagation matrices.

We visualize the propagation matrix for 15 more verbs (extending Fig. 7 of the main paper). Note that, even though we choose the verbs randomly, we see that many verbs do have dominant roles that influence others. Each row consists of the matrix, and 4 randomly chosen images corresponding to the verb.

Our model propagates information between all roles, and we present the norm of the message sent by each role to the other in the propagation matrix. The verb and list of roles is displayed at the beginning of each row for simplicity. The rows and columns of the propagation matrix follow this ordering of roles.

Roles: agent, place, item, feature, tool


Roles: agent, place, item, receiver

Roles: agent, place, target, tool, substance


Roles: agent, place, target


Roles: agent, place, patient, aspect, tool


Roles: agent, place, created, instrument

Roles: agent, place, item, substance


Roles: agent, place, tool, distributed, recipients


Roles: agent, place, item, tool


Roles: agent, place, item, recipient

Roles: agent, place, surface


Roles: agent, place, coagent, coagentpart, agentpart


Roles: agent, place, source, tool, destination


Roles: agent, place, item, container

Roles: agent, place, event, stage, tool


Appendix D Prediction Results

We round up the supplementary material with several more example predictions from our model. Fig. 15 shows predictions that are completely correct. Fig. 16 shows examples where we are able to predict the correct verb, but not all the role-noun pairs. Such examples are counted towards the value metric, but not value-all. Finally, Fig. 17 shows top-scoring (log-probability) examples where the verb is wrongly predicted, but is mostly plausible (the correct noun predictions are not captured by any metric). The role-noun pairs here are often correct.

Figure 15: Images with top-1 predictions from the development set. For all samples, the predicted verb is correct, and is shown below the image in bold. Roles are marked with a blue background, and predicted nouns with green when correct, and red when wrong. We are able to correctly predict the situation (verb and all role-noun pairs) for all example images shown here.
Figure 16: Images with top-1 predictions from the development set. For all samples, the predicted verb is correct, and is shown below the image in bold. Roles are marked with a blue background, and predicted nouns with green when correct, and red when wrong. We show examples with genuine errors in prediction (\eg the telephone for the verb pressing is clearly a remote control). However, some examples are marked wrong due to the lack of matching ground-truth annotations (\eg the woman smelling the flower is outdoors (GT: field)).
Figure 17: Images with ground-truth and top-1 predictions from the development set. Roles are marked with blue background. Ground-truth (GT) nouns with yellow, and predicted (PRED) nouns with green when correct, or red when wrong. Although the predicted verb is different from the ground-truth, it is very plausible. Some of the verbs refer to the same frame (\eg sprinting, racing), and contain the same set of roles, which our model is able to correctly infer.


  1. Predicting place requires a more global view of the image compared to agent. Changing the order to verb agent place results in drop of performance.


  1. S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh. VQA: Visual Question Answering. In ICCV, 2015.
  2. J. Bruna, W. Zaremba, A. Szlam, and Y. LeCun. Spectral networks and locally connected networks on graphs. ICLR, 2014.
  3. L. Castrejon, K. Kundu, R. Urtasun, and S. Fidler. Annotating object instances with a polygon-rnn. In CVPR, 2017.
  4. K. Cho, B. van Merriënboer, C. Gulcehre, B. Dzmitry, F. Bougares, H. Schwenk, and Y. Bengio. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In EMNLP, 2014.
  5. R. Collobert, K. Kavukcuoglu, and C. Farabet. Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop, 2011.
  6. M. Defferrard, X. Bresson, and P. Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In NIPS, 2016.
  7. V. Delaitre, I. Laptev, and J. Sivic. Recognizing human actions in still images: a study of bag-of-features and part-based representations. In BMVC, 2010.
  8. Z. Deng, A. Vahdat, H. Hu, and G. Mori. Structured Inference Machines: Recurrent Neural Networks for Analyzing Relations in Group Activity Recognition. In CVPR, 2016.
  9. D. K. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, and R. P. Adams. Convolutional networks on graphs for learning molecular fingerprints. In NIPS, 2015.
  10. C. J. Fillmore, C. R. Johnson, and M. R. L. Petruck. Background to FrameNet. International Journal of Lexicography, 16(3):235–250, 2003.
  11. H. Fürstenau and M. Lapata. Graph Alignment for Semi-Supervised Semantic Role Labeling. In EMNLP, 2009.
  12. A. Gupta and L. S. Davis. Beyond Nouns: Exploiting Prepositions and Comparative Adjectives for Learning Visual Classifiers. In ECCV, 2008.
  13. S. Gupta and J. Malik. Visual Semantic Role Labeling. arXiv:1505.04474, 2015.
  14. K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
  15. G. Hinton, N. Srivastava, and K. Swersky. Lecture 6a overview of mini-batch gradient descent.
  16. S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation, 9(8):1735–1780, 1997.
  17. J. Johnson, R. Krishna, M. Stark, L.-J. Li, D. A. Shamma, M. S. Bernstein, and L. Fei-Fei. Image Retrieval using Scene Graphs. In CVPR, 2015.
  18. D. Jurafsky and J. H. Martin. Speech and Language Processing, chapter 22. Semantic Role Labeling. 3, draft edition, 2017.
  19. A. Karpathy and L. Fei-Fei. Deep Visual-Semantic Alignments for Generating Image Descriptions. In CVPR, 2015.
  20. P. Kingsbury and M. Palmer. From TreeBank to PropBank. 2002.
  21. T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. ICLR, 2017.
  22. R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li, D. A. Shamma, M. Bernstein, and L. Fei-Fei. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations. arXiv:1602.07332, 2016.
  23. Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel. Gated Graph Sequence Neural Networks. In ICLR, 2016.
  24. X. Liang, X. Shen, J. Feng, L. Lin, and S. Yan. Semantic object parsing with graph lstm. In ECCV, 2016.
  25. D. Lin, S. Fidler, C. Kong, and R. Urtasun. Generating multi-sentence lingual descriptions of indoor scenes. In BMVC, 2015.
  26. H. Ling and S. Fidler. Teaching machines to describe images via natural language feedback. In arXiv:1706.00130, 2017.
  27. C. Lu, R. Krishna, M. Bernstein, and L. Fei-Fei. Visual Relationship Detection with Language Priors. In ECCV, 2016.
  28. K. Marino, R. Salakhutdinov, and A. Gupta. The more you know: Using knowledge graphs for image classification. CVPR, 2017.
  29. G. A. Miller. WordNet: A Lexical Database for English. Communications of the ACM, 38(11):39–41, 1995.
  30. B. A. Plummer, A. Mallya, C. M. Cervantes, and J. Hockenmaier. Phrase Localization and Visual Relationship Detection with Comprehension Linguistic Cues. arXiv:1611.06641, 2017.
  31. M. R. Ronchi and P. Perona. Describing Common Human Visual Actions in Images. In BMVC, 2015.
  32. M. Roth and M. Lapata. Neural Semantic Role Labeling with Dependency Path Embeddings. In ACL, 2016.
  33. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV, 115(3):211–252, 2015.
  34. F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, and G. Monfardini. The Graph Neural Network Model. IEEE Transactions on Neural Networks, 20(1):61–80, 2009.
  35. K. Simonyan and A. Zisserman. Two-stream convolutional networks for action recognition in videos. In NIPS, 2014.
  36. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
  37. K. S. Tai, R. Socher, and C. D. Manning. Improved semantic representations from tree-structured long short-term memory networks. ACL, 2015.
  38. M. Tapaswi, Y. Zhu, R. Stiefelhagen, A. Torralba, R. Urtasun, and S. Fidler. MovieQA: Understanding Stories in Movies through Question-Answering. In CVPR, 2016.
  39. L. J. P. van der Maaten and G. E. Hinton. Visualizing High-Dimensional Data using t-SNE. Journal of Machine Learning Research, 2008.
  40. O. Vinyals, A. Toshev, S. Bengio, and Dumitru Erhan. Show and Tell: A Neural Image Caption Generator. In CVPR, 2015.
  41. H. Wang, A. Kläser, C. Schmid, and C.-L. Liu. Action recognition by dense trajectories. In CVPR, 2011.
  42. P. J. Werbos. Generalization of backpropagation with application to a recurrent gas market model. Neural networks, 1(4):339–356, 1988.
  43. K. Xu, J. L. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio. Show, Attend and Tell: Neural Image Caption Generation with Visual Attention. JMLR, 2015.
  44. S. Yang, Q. Gao, C. Liu, C. Xiong, S.-C. Zhu, and J. Y. Chai. Grounded Semantic Role Labeling. In NAACL, 2016.
  45. B. Yao and L. Fei-Fei. Grouplet: A Structured Image Representation for Recognizing Human and Object Interactions. In CVPR, 2010.
  46. B. Yao, X. Jiang, A. Khosla, A. L. Lin, L. Guibas, and L. Fei-Fei. Human Action Recognition by Learning Bases of Action Attributes and Parts. In ICCV, 2011.
  47. M. Yatskar, V. Ordonez, L. Zettlemoyer, and A. Farhadi. Commonly Uncommon: Semantic Sparsity in Situation Recognition. In CVPR, 2017.
  48. M. Yatskar, L. Zettlemoyer, and A. Farhadi. Situation Recognition: Visual Semantic Role Labeling for Image Understanding. In CVPR, 2016.
  49. H. Zhang, Z. Kyaw, S.-F. Chang, and T.-S. Chua. Visual Translation Embedding Network for Visual Relation Detection. arXiv:1702.08319, 2017.
  50. S. Zheng, S. Jayasumana, B. Romera-Paredes, V. Vineet, Z. Su, D. Du, C. Huang, and P. H. S. Torr. Conditional Random Fields as Recurrent Neural Networks. In ICCV, 2015.
  51. B. Zhou, A. Khosla, A. Lapedriza, A. Torralba, and A. Oliva. Places: An Image Database for Deep Scene Understanding. arXiv:1610.02055, 2016.
  52. B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning Deep Features for Scene Recognition using Places Database. In NIPS, 2014.
  53. J. Zhou and W. Xu. End-to-end Learning of Semantic Role Labeling using Recurrent Neural Networks. In ACL, 2015.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description