Analyzing representations

Analyzing machine-learned representations: A natural language case study


As modern deep networks become more complex, and get closer to human-like capabilities in certain domains, the question arises of how the representations and decision rules they learn compare to the ones in humans. In this work, we study representations of sentences in one such artificial system for natural language processing. We first present a diagnostic test dataset to examine the degree of abstract composable structure represented. Analyzing performance on these diagnostic tests indicates a lack of systematicity in the representations and decision rules, and reveals a set of heuristic strategies. We then investigate the effect of the training distribution on learning these heuristic strategies, and study changes in these representations with various augmentations to the training set. Our results reveal parallels to the analogous representations in people. We find that these systems can learn abstract rules and generalize them to new contexts under certain circumstances – similar to human zero-shot reasoning. However, we also note some shortcomings in this generalization behavior – similar to human judgment errors like belief bias. Studying these parallels suggests new ways to understand psychological phenomena in humans as well as informs best strategies for building artificial intelligence with human-like language understanding.


[figure]style=plain,subcapbesideposition=top \floatsetup[table]capposition=top \fourauthorsIshita DasguptaDemi GuoSamuel J. GershmanNoah D. Goodman \fouraffiliationsHarvard UniversityHarvard UniversityHarvard UniversityStanford University \authornoteCorrespondence concerning this article should be addressed to Ishita Dasgupta, Department of Physics and Center for Brain Science, Harvard University, 52 Oxford Street, Cambridge, MA 02138, United States. E-mail: A preliminary version of this work was previously reported in [Dasgupta \BOthers. (\APACyear2018)]

1 Introduction

Recent years have seen a vast improvement in the capabilities of artificial intelligence systems, driven primarily by developments in deep neural networks (see LeCun \BOthers., \APACyear2015, for a review). These have allowed artifical system to reach human-level performance at video games (Mnih \BOthers., \APACyear2015), object recognition (Russakovsky \BOthers., \APACyear2015), and voice generation (Oord \BOthers., \APACyear2016), as well as produced impressive performance in several other domains. However, some serious concerns haunt deep learning approaches and their promise as a general solution to artificial intelligence. Many of these concerns surround the lack of structure in the representations and decision criteria these systems learn (Marcus, \APACyear2018; Lake \BOthers., \APACyear2018). This problem has been implicated in deep learning’s data inefficiency and inability to learn abstract structure from few examples, its difficulty in utilizing hierarchical structure and fostering transfer between tasks and domains, as well as the challenge of integrating established prior information into deep learning systems. It also presents serious concerns about the interpretability of its representations and decision criteria, making them less dependable and risky for deployment in sensitive or highly variable domains.

All of this points to a crucial problem: how can we better understand the representations learned by these systems? Existing studies (e.g., Karpathy \BOthers., \APACyear2015; Li \BOthers., \APACyear2015; Yosinski \BOthers., \APACyear2015; Zeiler \BBA Fergus, \APACyear2014) primarily use approaches inspired by neuroscience methods developed to understand the brain, for example the statistical analysis of unit activations, and ablation studies where specific units are disconnected or deactivated. These methods promise interesting bottom-up insights into the inner workings of these systems. Cognitive science provides another set of tools to approach this problem from the top down (Ritter \BOthers., \APACyear2017; Kádár \BOthers., \APACyear2017; McCoy \BOthers., \APACyear2019), by decomposing cognitive processes into their computational components, building models that incorporate these components, and testing these by making predictions about behavior on carefully selected test problems that distinguish different hypotheses.

The cognitive science approach has yielded huge benefits in understanding higher-level cognition in humans, a prime example of which is the human ability to learn, understand, and produce language (Chomsky \BBA Lightfoot, \APACyear2002; Linzen, \APACyear2019). This domain exemplifies a hallmark of human intelligence: the ability, in the words of von Humboldt, to “make infinite use of finite means.” Specifically, human cognitive abilities have been characterized as systematic (Fodor \BBA Pylyshyn, \APACyear1988; Lake \BOthers., \APACyear2019) – this indicates an algebraic capacity to produce new combinations from known components. For example, when a person learns a word in a specific context as part of a particular sentence, they can immediately use this new word in an infinity of other sentences in which this word has never previously been encountered. Systematicity therefore allows humans an impressive capacity to generalize, transferring knowledge from one context to others. This ability requires the representations underlying this newly learned word for example, to be abstract (not tied to specific contexts) and compositional (possible to combine with other words and sentences). The failure of neural network models to achieve such systematicity has been a recurring (and controversial) theme in cognitive science (Fodor \BBA Pylyshyn, \APACyear1988; Lake \BOthers., \APACyear2018). This concern has previously been studied specifically in the domain of natural language (Lake \BBA Baroni, \APACyear2017; S. Gershman \BBA Tenenbaum, \APACyear2015; Belinkov \BBA Glass, \APACyear2019), demonstrating the lack of abstract compositional reasoning in certain networks. These analyses are often carried out on toy systems, and while they demonstrate conclusively the lack of systematicity, they largely neglect a deeper analysis of what the systems do learn.

In this paper, we carry out an analysis of the representations learned by a state-of-the-art model for a difficult natural language processing task. We discover that its representations are not systematic; instead, the model uses various heuristic strategies. We then investigate how these heuristics might arise. Analyses of the training distribution reveal that it is very biased, containing many unintended structural regularities that can be exploited by these much simpler heuristics. These simple rules are therefore easily acquired by the neural network, since they explain a substantial amount of variance without having to invoke a more complex systematic representations. We then carry out various augmentations to the training set to better understand if the system can learn abstract composable representations, given the right training distribution. We find parallels between our findings and studies of human representations in terms of how systematic they are under certain circumstances, as well as in terms of when and where this systematicity breaks down. We discuss how such analyses can be fruitful to both cognitive science and machine learning.

2 Background

In this section we review some background on the kinds of representations we will be studying (vector space embeddings of sentences). We also review the three key factors in how such embeddings are generated: the task that they are optimized for, the architecture of the model used to perform that task, and the training distribution on which performance is optimized.111The details and implementation of the optimization algorithm also contribute (see Ruder, \APACyear2016, for an overview), but as long as the optimization reaches convergence this has relatively little effect, and we leave this out of our current discussion.

2.1 Vector space embeddings

Vector space models represent items as vectors in some metric space. These have a long history in cognitive science as models of semantic representations (Steyvers, \APACyear2006; Beals \BOthers., \APACyear1968; Pereira \BOthers., \APACyear2016). In particular, in the domain of language, vector space models of words (also known as word embeddings) that are learned using distributional information (statistics of text corpora) have been shown to encode syntactic as well as semantic structure, and have been used in psychological models for syntactic category acquisition (Redington \BOthers., \APACyear1998), inductive vocabulary learning (Landauer \BBA Dumais, \APACyear1997), analogical reasoning (Rumelhart \BBA Abrahamson, \APACyear1973), categorization (Jones \BBA Mewhort, \APACyear2007), and high-level associative judgments (Bhatia, \APACyear2017). Modern machine learning has allowed the mining of very large datasets to produce vector space embeddings that are now commonly used as the word representations in artificial intelligence systems for natural language processing (Pennington \BOthers., \APACyear2014; Mikolov \BOthers., \APACyear2013).

Understanding language requires understanding not only words, but also their relations within a sentence. These relations are abstract and composable, allowing language to be combinatorially productive – with a finite set of words, one can systematically produce an infinite set of sentences simply by creating new and longer combinations of these known words. The number of sentences in a language therefore far exceed the number of words. For this reason, generating similar vector embeddings for sentences has proven challenging. Recent papers have developed several supervised as well as unsupervised approaches to learning vector space representations of sentences using recurrent neural networks (RNNs) that are able to represent the order of words in a sentence (Kiros \BOthers., \APACyear2015; Hill \BOthers., \APACyear2016; Conneau \BOthers., \APACyear2017). These are intended to capture sentence-level semantic content, and have been shown to perform reasonably well on transfer tasks (sentence-level semantic tasks on which the embeddings were not specifically trained). In particular, the performance of these sentence models exceeds the performance of representations that treat sentences as bags of words (BOW models) – these patently lack any order information about the words, therefore ignoring the abstract and composable relational structure at the sentence level. However, it is unclear exactly what relational information between words is actually represented in such RNN sentence models. In this work, we start to shed light on this question.

2.2 Natural Language Inference

The sentence embeddings we analyze are trained on the natural language inference (NLI) task. The goal is to classify pairs of sentences (a premise and a hypothesis) into ‘entailment’, ‘contradiction’, or ‘neutral’, depending on the semantic relation between the two sentences. This is a popular domain for studying artificial representations since it has a lot of relatively interpretable underlying structure (Glockner \BOthers., \APACyear2018; McCoy \BOthers., \APACyear2019; Nie \BOthers., \APACyear2019). For example, it is a simple domain in which abstract and composable relational structure is required – word-level information is not generally sufficient to perform well on this task. The premise sentence “Anne is more cheerful than Bob” contradicts the hypothesis sentence “Anne is less cheerful than Bob”, but entails the hypothesis sentence “Bob is less cheerful than Anne”. Here, both the hypothesis sentences have the exact same words, and would be indistinguishable if we were just comparing the words in them. More generally, X is more Y than Z entails that Z is less Y than X, for any X, Y and Z. In this case, the specific words used almost don’t even matter, and the bulk of the information is in the relations between the words in the sentence. Encoding abstract rules like this allows us to systematically carry out natural language inference on combinatorially many different sentences, with different Xs, Ys, and Zs.

The human ability to carry out abstract reasoning of this sort is a richly studied topic. Some of these abilities however are so obvious, that they are often simply taken for granted without formal study. For example, it is reasonable to assume that any adult human (in the absence of time pressure or cognitive load) can fairly easily process that if X is more Y than Z, then in general Z is less Y than X irrespective of the specific meanings of X, Y and Z. In this paper, we investigate to what extent certain machine-learned sentence embeddings can represent and use such abstract rules in natural language inference.

Despite the generally acknowledged power of human abstract reasoning, a number of studies indicate that humans are not perfect: semantic content (for example the specific meanings of the X, Y and Zs above) has been shown to interfere with systematic inferences in an effect often termed ‘belief bias’ (Braine, \APACyear1978; Johnson-Laird \BBA Steedman, \APACyear1978). This effect is especially noticeable in children (Evans \BBA Perry, \APACyear1995), as well as adults under time pressure or cognitive load (Evans, \APACyear2013). In the last part of this paper, we discuss similarities between humans and machines in how they fail certain tests of systematicity.

Figure 1: InferSent architecture (Conneau \BOthers., \APACyear2017).

2.3 Models for sentence embeddings

The sentence embeddings we study in this paper are from a highly successful NLI system, InferSent (Conneau \BOthers., \APACyear2017). Each premise and hypothesis sentence are input to a sentence encoder as a sequence of pre-trained 300-dimensional GloVe word embeddings (Pennington \BOthers., \APACyear2014). These word embeddings already contain a lot of information about the semantic and syntactic roles of the words (see section on Vector space embeddings for details), and therefore a large part of the lexical information is already represented. Therefore the bulk of the work InferSent has to do is to learn and represent how these words relate to one another in a sentence to provide meanings. The sentence encoder takes in this variable length input and, after passing it through various recurrent and convolutional layers (see Conneau \BOthers., \APACyear2017, for details), provides a 4096-dimensional vector as output. This output vector serves as a sentence embedding. To make the final inference, these sentence embeddings for the premise and hypothesis are fed to a simple classifier described in Figure 1 that labels each pair as entailment, neutral or contradiction. The network is trained end-to-end with supervised learning, using a large labelled dataset for NLI (see next section for details on this dataset). The learned embeddings were shown to perform well on other sentence-level tasks (such as sentiment analysis, semantic textual similarity and other natural language inference datasets) by re-using the sentence encoder and training only the classifier for the specific task at hand. This indicates that the sentence encoder does capture some semantic content in the embeddings.

For our tasks, we replicate the procedure in Conneau \BOthers. (\APACyear2017) to obtain sentence embeddings. These are henceforth referred to as the InferSent sentence embeddings. Our trained InferSent model gives us 84.73% accuracy on validation and 84.84% accuracy on the test dataset, which is comparable to the performance of the classifier reported in Conneau \BOthers. (\APACyear2017). For comparison, we also train a bag-of-words (BOW) baseline model that averages the pre-trained GloVe word embeddings for all the words in the sentence to form a sentence embedding. These embeddings cannot represent abstract relational structure, since the architecture of the model used to generate them (a simple average of the word embeddings) cannot express word order. We then train a simply classifier on these embeddings to perform natural language inference. This model achieves accuracy on the SNLI test set (comparable to the BOW performance reported in Conneau \BOthers., \APACyear2017).

Neural networks can act as universal function approximators (Siegelmann \BBA Sontag, \APACyear1995; Hornik, \APACyear1991), and given sufficient capacity, they can represent any arbitrarily complex set of relations between the words in the sentence. The InferSent model has a very large capacity due to a large number of layers and hidden units (see Conneau \BOthers., \APACyear2017), so a lot of abstract compositional structure is in theory within the representational capacity of these sentence embeddings. In this paper, we analyze how much systematic structure is actually learned and utilized for the NLI task at hand.

2.4 Training datasets

To understand sentence embeddings like the ones learned by InferSent, it is imperative to not only consider the model specifications for the system that produces them (in this case the specific end-to-end architecture of the network in InferSent), but also the learning signals it receives from the training set. For many deep learning based methods, very little information about the structure of the task is baked into the architecture of the models – the only structure about language that it is endowed with before training are the biases that come with using a recurrent neural network as the architecture. This specifies that sentences have variable-length, sequential structure. These embedding models are therefore fairly ‘tabula rasa’, and most of what they represent about the structure of the task (in this case natural language inference) is learned from training data. As elaborated in the previous section, some abstract compositional structure is within the representational capacity of the InferSent sentence embeddings – but whether or not the right structure is actually learned and represented depends largely on the training data. The significance of the training set on the representations learned by flexible deep learning methods is often not adequately considered. One contribution of this work is to highlight and analyze this issue.

InferSent was trained on the Stanford Natural Language Inference (SNLI) dataset (Bowman \BOthers., \APACyear2015), a popular labelled dataset for natual language inference. SNLI consists of  550k premise-hypothesis sentence pairs, and is balanced (consists of equal number of pairs with entailment, contradiction and neutral relationships). The dataset was generated with a crowd-sourcing framework. Workers were presented with a scene description from a corpus of image captions that act as the premise, and asked to supply hypothesis sentences that have each of the three possible NLI relations (entailment, neutral, and contradiction) to the given premise. The freedom to produce entirely novel hypotheses leads to a rich set of sentences; however, it also leads to some artifacts that can strongly bias the representations learned by a ‘tabula rasa’ system. We discuss these in later sections.

3 A test dataset of minimal cases: The Comparisons dataset

Our goal is to understand the representations and decision criteria learned by InferSent, in particular how much systematic relational information they encode and utilize – do they represent abstract rules for the ways words combine to give meaning to sentences? In the machine learning literature on natural language processing, any performance above the bag-of-words (BOW) baseline (that only receives the words in the sentence with no order information) is often seen as proof of the encoding and utilization of relational information. However, this is an unwarranted conclusion—the BOW baseline usually receives only averaged word vectors for the sentence, and therefore also loses some of the lexical information. It often does not actually reach the best possible performance with only the words. Performance above this baseline therefore does not license the conclusion that relational information is being encoded and used at all.

Here, we pursue an alternative approach, inspired by traditions in cognitive psychology and psycholinguistics of building diagnotic test sets to investigate the underlying representations and decision rules. The goal is to generate a set of sentence pairs such that encoding the relations between words (in addition to the words themselves) is required to correctly classify them into the three NLI classes. Diagnostic test datasets such as these, that posit a hard baseline for performance without relational information, provide a more foolproof way to test whether such information is being used.

We considered pairs of sentences such that the NLI relation between the sentences can be changed without changing any of the words in the sentence, only their order. We generated our test dataset using comparisons as these are easy to fit into the NLI framework, and yield many simple examples of sentence pairs that require more than word-level data to understand. For example, the premise sentence “the woman is more cheerful than the man” contradicts one hypothesis sentence, “the woman is less cheerful than the man”, but entails another hypothesis sentence, “the man is less cheerful than the woman”. Since both hypothesis sentences have the exact same words, they would be indistinguishable if we were just comparing their bag-of-words representations. Therefore, a model based only on the words, and not considering the relations between them, would at most get one of the two classifications right. This caps the bag-of-words performance at , and some relational rules must be learned to perform above this baseline.

Generation of several such sentence pairs can be easily automated. We considered three sub-types, described below and summarized in Tables 1 and 2.

3.1 Same type

Premise-Hypothesis pairs differ only in the order of the words.
Premise: The woman is more cheerful than the man
Hypothesis: The man is more cheerful than the woman
Premise: The woman is more cheerful than the man
Hypothesis: The woman is more cheerful than the man

3.2 More-Less type

Premise-Hypothesis pairs differ by whether they contain the words ‘more’ or ‘less’.
Premise: The woman is more cheerful than the man
Hypothesis: The woman is less cheerful than the man
Premise: The woman is more cheerful than the man
Hypothesis: The man is less cheerful than the woman

3.3 Not type

Premise-Hypothesis pairs differ by whether they contain the word ‘not’.
Premise: The woman is more cheerful than the man
Hypothesis: The woman is not more cheerful than the man
Premise: The woman is more cheerful than the man
Hypothesis: The man is not more cheerful than the woman

Type Entailment hypothesis Contradiction hypothesis

X is more Y than Z Z is more Y than X

Z is less Y than X X is less Y than Z

Z is not more Y than X X is not more Y than Z
Table 1: Rules in Comparisons dataset for Premise: X is more Y than Z
Type Number of sentence pairs
Comparisons (same) 14670
Comparisons (more-less) 14670
Comparisons (not) 14670
Table 2: Comparisons dataset summary.

To facilitate comparison with the SNLI dataset, we ensured that the vocabulary distribution of our Comparisons dataset is similar to the original SNLI training dataset.222Only a few words differed by more than from their occurrence rate in SNLI, such as not, a, than, the, is, less, more. This was inevitable given the general structure of the comparison sentence pairs we use. All of these words however did still occur in the SNLI training corpus, and were not new to the model at test time. This ensured that we are only manipulating the relational structure of the test set, and poor performance cannot be attributed to not having experienced the specific words before.

4 Testing the sentence embeddings

We tested the two classifiers based on two different sentence embeddings (the InferSent sentence embeddings, and the BOW sentence embeddings) on the constructed test set (the Comparisons dataset, Table 2). Both of these classifiers were trained for the same task (Natural Language Inference), on the same training dataset (SNLI), and differed only in the model used to generate them. The InferSent embeddings had access to word order, while the BOW embeddings did not (see Section ‘Models for sentence embeddings’ for details). The overall performance of each of the two classifiers on the Comparisons dataset are given in Table 3, and analyzed in greater detail in the following sections.

Type BOW InferSent
same 50.0 50.37
more/less 30.24 50.35
not 48.98 45.24
Table 3: Performance on the Comparisons dataset.

4.1 Performance of Bag of Words

We found that the BOW embeddings make classifications that are exactly symmetric across the two true labels (entailment and contradiction) in each task (rows in Figure 2). This is expected since the sentence pairs with one label are just permuted versions of the sentence pairs with the other label. Therefore BOW cannot distinguish them, and necessarily classifies both of them the same way. This also ensures that the performance is capped at . Asymmetry between the classifications of the two categories can occur only when relational information is encoded in the sentence embedding.

Considering the aggregate performance of BOW in Table 3, we found that performance, particularly on the ‘more/less’ type subset of the test dataset (), was significantly below . This highlights the trouble with using BOW embeddings as a baseline for the encoding and use of relational information. Up to performance is achievable on this dataset without using any relational information; therefore performance above the BOW baseline of does not necessarily imply the use of relational information.

Figure 2: BOW embedding confusion matrices, with normalized rows.
Figure 3: InferSent embedding confusion matrices, with normalized rows.

4.2 Performance of InferSent

The performance of the InferSent embeddings was slightly asymmetric (Figure 3), indicating that it was able to distinguish sentences slightly, based on relational information. Yet overall the InferSent embeddings were extremely poor at this task (Table 3), achieving performances slightly above for two of the three sub-types of sentence pairs in the Comparisons dataset, and even less than in a third sub-type. This indicates that InferSent embeddings do not correctly encode and utilize the kinds of abstract relational rules we tested with the Comparisons dataset.

However, InferSent’s performance on another test dataset (the SNLI test dataset) is as high as – so it is clearly encoding some relevant information about natural language inference. Further, a quick glance at Figure 3 indicates that InferSent does not respond randomly to the queries in our Comparisons dataset, but rather in some structured (though incorrect) way. Rather than simply conclude that InferSent embeddings are not systematic and leaving things at that, we can study patterns in the incorrect classifications made to better understand the underlying representations and decisions rules. Since our test dataset is highly structured, it allows a controlled way to generate and test hypotheses about the heuristic representations and decision rules InferSent implements.

Apart from isolating and characterizing these heuristics, it is also instructive to consider how InferSent might come to encode them in the first place. To answer this, we look to the study of heuristic strategies in humans. The theory of ecological rationality (Todd \BBA Gigerenzer, \APACyear2007; Simon, \APACyear1991) posits that a system can exploit structural regularities in its learning environment using heuristics that achieve close to optimal performance in that specific environment. These might be much simpler than the most general strategy that performs well in all environments. Heuristics that leverage these structural regularities are therefore termed ‘ecologically valid’ in that environment. This suggests that we can better understand how heuristic strategies might arise in InferSent by examining if they are ecologically valid in its ‘learning environment’ (i.e., the training set). In the following sections, we delve into the heuristic strategies that explain performance on our Comparisons datset, as well as how InferSent might have come to encode them by testing their ecological validity in the SNLI training dataset.

4.2.1 Overlap Heuristic

We note in Figure 3 that almost all the sentence pairs in the same-type comparisons were classified as entailments, despite half of them being true contradictions. A distinguishing feature of the same-type comparisons is that the premise and hypothesis sentences have full word overlap (they both contain exactly the same words). This observation allows us to hypothesize an overlap heuristic: high overlap in words between premise and hypothesis biases InferSent against classifying the pair as a contradiction.

While we have seen some evidence that this heuristic is indeed at play (based on the performance on the same-type comparisons), the question remains as to why it encodes this rule. With our knowledge of language, we know this simple rule to reflect on incorrect understanding of natural language inference. However, all the knowledge about the NLI task that InferSent encodes is from its training dataset. If the dataset has underlying structural regularities that can be exploited by simple heuristic strategies, then a tabula rasa model for NLI such as InferSent that is trained on this dataset will learn to encode it.

We carried out an analysis of the SNLI dataset to determine if the overlap heuristic is ecologically valid in it. First, we observed anecdotally that indeed several contradictory sentence pairs have no overlap in words. For example, a contradictory sentence pair in SNLI is:
Premise: Several people are trying to climb a ladder in a tree.
Hypothesis: People are watching a ball game.
To quantitatively verify this observation, we ranked all the sentence pairs in SNLI by overlap rate: (in non-increasing order). We then considered the top X sentences with highest overlap for different Xs. As shown in Table 4, when considering the full dataset, the distribution is balanced (the percentage of entailments, contradictions and neutral sentences are equal). However, we found that as the word overlap in the sentences increases, the percentage of contradictions drops. When considering only the top 1000 sentence pairs for overlap, we found that of them have entailment or neutral labels, with only the remaining having a contradiction label.

Top Entailment Neutral Contradiction
All 33.4 33.3 33.3
10000 39.5 35.7 24.8
1000 50.8 40.7 8.5
Table 4: Percentage of entailments split by overlap rate of words in SNLI.

It is therefore natural that InferSent encodes the simple overlap heuristic as a predictor of contradiction. This explains not only the failure of InferSent to generalize its good performance on SNLI to the same-type comparisons in our test dataset, but also matches the specific failure mode we observe in its responses.

4.2.2 Antonyms Heuristic

We note in Figure 3 the opposite trend for the more/less-type comparisons, where almost all the sentence pairs were classified as contradictions, despite half of them being true entailments. A distinguishing feature of the more/less-type comparisons is that the the premise and hypothesis always differ by one word – if the premise contains the word ‘more’ (‘less’) then the hypothesis always contain the word ‘less’ (‘more’). This observation allows us to hypothesize an antonyms heuristic: sentences differing in the presence of words that have opposing meanings (antonyms) tend to be classified by InferSent as contradictions, irrespective of the other words or their order in the sentence.

Similarly to the previous section, we investigated the training dataset to elucidate if this heuristic is ecologically valid in InferSent’s training set. Anecdotally, we saw that the contradicting hypotheses provided by crowd workers to generate SNLI do follow this pattern. For example, a contradictory sentence pair in SNLI is:
Premise: A man in a white t-shirt takes a picture in the middle of the street with two public buses in the background.
Hypothesis: A man is wearing a black t-shirt.

To verify this observation quantitatively, we analyzed the statistics of antonym usage in SNLI. To test whether a sentence pair (A,B) contains antonyms, we went through each word in sentence A, and considered all synonyms of that word, and considered all antonyms of those synonyms. Finally, we checked if sentence B contained any of those antonyms. These synonyms and antonyms were found using the NLTK WordNet software (Bird \BBA Loper, \APACyear2004). We then considered two different statistics. First, we calculated P(Contradiction | Antonym), which is the probability that a sentence pair is a contradiction given that its premise and hypothesis contain an antonym pair. This measures how well the presence of antonyms predicts a contradiction label in the training set. Second, we calculated P(Antonym | Contradiction), which is the probability that a contradictory sentence pair contains antonyms. This measures how well a contradiction label predicts antonyms. Both statistics were compared with the equivalent statistic for entailment, to provide a baseline for comparison. Table 5 shows that the presence of antonyms strongly predicts a contradiction label in the SNLI dataset (61.2% compared to chance at 33.3%). We also found that a contradiction label predicts the presence of an antonym pair (12.2%) more strongly than entailment did (3.5%). This indicates that the antonyms heuristic can explain significant variance for the contradiction label in the training set.

P(Antonym X) P( X Antonym)
X = Contradiction 12.2% 61.2%
X = Entailment 3.5% 18.0%
Table 5: Percentage of entailments split by antonym word pair in the SNLI dataset.

Since most of our Comparisons dataset contained a large amount of overlap between premise and hypothesis, the rules InferSent applies when responding to these test questions might be biased towards those learned in similar high-overlap settings during training. We checked the statistics of antonymy in the the high overlap subset of SNLI (top 10,000 highest overlap) to provide a closer comparison (Table 6). Here, contradiction predicts the presence of an antonym pair (43.7%) more strongly than in the whole dataset (12.2%). The difference between P(Antonym | Contradiction) and P(Antonym | Entailment) is also more pronounced in this high overlap subset. The presence of an antonym pair no longer predicts contradictions at a high rate (28.9 %), but this is possibly due to the very low base rate of contradictions in the high overlap subset of SNLI, as compared to entailments.

P(Antonym X) P( X Antonym)
X = Contradiction 43.5% 28.9%
X = Entailment 8.7% 34.3%
Table 6: Percentage of entailments split by antonyms in high overlap SNLI subset.

These results suggest again, that the underlying statistics of the SNLI dataset allow models, including InferSent, to perform well with simple lexical heuristics that ignore the order of words and their relations.

4.2.3 Negation Heuristic

We see in Figure 3 that the not-type comparisons are preferentially classified as contradictions. A distinguishing feature of the not-type comparisons is that the premise and the hypothesis differ by the presence of the negation ‘not’. This observation allows us to hypothesize a negation heuristic where sentence pairs that differ in the presence of negations are preferentially classified as contradictions.

Following procedures analogous to previous sections, we first noted anecdotally, that this heuristic seems to have validity in the contradicting hypotheses in SNLI. For example, a contradictory sentence pair in SNLI is:
Premise: Men turn to the camera to smile on the middle of three long tables in a refectory.
Hypothesis: The man is not smiling.

We verified this observation quantitatively by looking at the statistics for negation in SNLI. We collected all sentence pairs that contain “negating N-grams”: no, not, n’t (by considering “n’t”, we included words such as “don’t” or “doesn’t”). We then carried out analyses similar to the previous section, where we checked (1) the predictive power of negations on contradictions (P(Contradiction | Negation)), and (2) the predictive power of contradiction on negations, P(Negation | Contradiction), and compare both of these to statistics for entailment as a baseline. We found (Table 7) that the presence of a negation strongly predicts contradiction in the SNLI dataset (58.4% compared to chance at 33.3%). We also found that while both numbers are very low, a contradiction predicts the presence of a negation (3.3%) slightly more strongly than entailment does (1.1%). We also carried out the same analysis for a high-overlap subset (top 10,000 highest overlap) of SNLI to maximize similarity with our comparisons dataset and saw similar results (Table 8). In fact, the presence of negation predicts a contradiction, P(Negation | Contradiction) = , at rates comparable to that in the full dataset, P(Negation | Contradiction) = , despite the much lower base rates of contradiction in this subset of the data. This indicates strong ecological validity for this heuristic in the high overlap subset of the SNLI dataset.

P(Negation X) P( X Negation)
X = Contradiction 3.3 58.4
X = Entailment 1.1 20.0
Table 7: Percentage of entailments split by negation in SNLI dataset.
P(Negation X) P( X Negation)
X = Contradiction 1.3 60.0
X = Entailment 0.1 7.5
Table 8: Percentage of entailments split by negation in high overlap SNLI subset.

4.2.4 Summary of heuristics

We found evidence for three heuristics that explain the bulk of the patterns seen in the performance of InferSent on our Comparisons dataset, all of which are ecologically valid in the SNLI dataset. First, we identified the overlap heuristic where a large overlap in words between two sentences leads InferSent to not classify them as contradictions. Second, we identified the antonyms heuristic and the negation heuristic, where the premise and hypothesis differ in the presence of an antonym or a negation, which leads InferSent to classify them as contradictions.

These illustrate a disproportionate dependence on lexical (rather than relational) meaning in the representations and decision rules used by InferSent. While these heuristics serve well in certain domains, for example in SNLI, they don’t amount to a more general encoding of entailment and contradiction between sentence pairs, as evidenced by InferSent’s poor performance on our Comparisons dataset.

The analysis so far has highlighted word-level heuristics that InferSent might be using. Yet the confusion matrix results (Figure 3) show a slight asymmetry, indicating at least minor multi-word effects. This suggests that InferSent might be using some (potentially also heuristic) encodings for word order. However, a systematic analysis of the effect of word order, and how much variance such heuristics might explain, is challenging due to the combinatorial explosion in the number of possibilities. We leave a thorough investigation of this to future work.

5 Augmenting the learning environment

The foregoing results suggest that such ecological validity of simple heuristics in the SNLI training data (InferSent’s learning environment) could explain why InferSent acquires them over a more abstract, systematic representation of the relations between words in a sentence. This leaves open the question of whether architectures such as InferSent are capable of learning the abstract relational rules needed to succeed at our task given a different training set where simple heuristics no longer explain so much of the variance. RNN architectures like the one in InferSent can in theory represent the relational structure required to encode the abstract rules of the sort in Table 1 (see Section ‘Models for sentence embeddings’ for details). But how might we get them to learn and use them? In this section, we explore this question by training the InferSent model on part of the Comparisons dataset, and testing on a held-out subset of it. This serves to test whether simple training on examples of the rules in Table 1, will enable InferSent to encode some abstract relational rules.

The total training subset of our Comparisons dataset consists of 40k sentence pairs ( the size of the 550k pair SNLI training set). Validation and test sets consist of 2000 sentence pairs each. There are no overlapping sentence pairs between any of these sets, therefore simply memorizing the training set will not allow good test performance. Good test performance requires the encoding and utilization of an abstract relational rule.

Epoch Performance (%)
Train(Comp) Test(Comp) Test(SNLI)
0 47.81 45.36 84.84
13 99.91 99.8 56.37
Table 9: Results of fine-tuning InferSent on the Comparisons dataset.

We started with the original InferSent embeddings already trained on the SNLI dataset, and then fine-tuned it on our new Comparisons dataset (using the same protocols used in Conneau \BOthers., \APACyear2017, to train InferSent). Results are shown in Table 9. We found that using this method, performance on the SNLI data task degrades over the course of fine-tuning on the new Comparisons dataset from 84.84% to 56.37%. This points to over-fitting to the Comparisons data, at the cost of representing information necessary for SNLI. We found however, that performance on the Comparisons test set is much higher (99.8 %) than when trained only on SNLI (47.81%). Note that this test set consists of sentence pairs InferSent has never seen before. We thus find that the model architecture for InferSent, given the right training data, can encode some form of abstract relational structure that allows it to learn rules of the form in Table 1 and apply them to new sentence pairs – in particular sentence pairs with Xs, Ys and Zs that it has never seen in that combination before.

Epoch Performance (%)
Train(Combined) Test(Comp) Test(SNLI)
0 33.33 33.33 33.33
12 90.99 100.00 84.96
Table 10: Results of retraining Infersent on both SNLI and the Comparisons dataset.

We then checked whether InferSent can represent this relational structure without losing the information necessary for SNLI. We started with an untrained network, and then trained on an augmented version of the original training data. Here, examples from the SNLI training set were randomly interleaved with examples from our Comparisons training dataset, otherwise using the same training protocols reported in (Conneau \BOthers., \APACyear2017). The test results are reported in Table 10. We found that the accuracy obtained this way on the SNLI test set (84.96 %) is comparable to the model trained only on SNLI (84.84 %). Moreover, test accuracy on the Comparisons dataset is close to perfect (99.55 %) and is much higher than the model trained only on SNLI (47.81 %). This establishes that in this case the model has enough capacity to achieve high performance on specially designed edge-cases like the Comparisons dataset, without loss of performance on the more general SNLI dataset.

This result also verifies that the heuristics we find in the original InferSent are an ecologically rational response to a training environment that licenses these ‘shortcut’ strategies, and not because of shortcomings in representational or learning abilities of the model itself. This points to the benefits of understanding the learning environment in greater detail, and potentially including specially designed data to guard against incorrect heuristics that don’t generalize. Research on the generation of adversarial examples targets this intuition. The idea is to have a separate ‘adversarial’ model that generates edge-case training examples optimized to try and fool the main model into giving the wrong answer (Goodfellow \BOthers., \APACyear2014; Zhao \BOthers., \APACyear2017). It does so by generating examples that violate the heuristics the main model has learned from training thus far. Subsequently, the training environment for the model is augmented to include these edge cases making the current heuristics no longer ecologically valid. The main model therefore updates its representations and decision rules accordingly and the process is continued. Our work provides some insight into how we can leverage a top-down understanding of the structure of language and systematic stimulus design, to generate such edge-case training data and potentially improve the representations learned by machine learning systems.

A key hurdle for the scalability for such augmentation as a solution to improving artificial representations of language however is that there are an infinite number of possible stimuli, with brand new combinations of words that may never have been encountered before. No finite amount of augmentation will allow a system to represent and process this infinite space of natural language sentences unless it can also generalize its knowledge gained from the examples observed thus far to new examples. In this section we saw that InferSent can generalize rules like those in Table 1 to never previously observed combinations of X, Y and Z to perform well on the test set of the Comparisons dataset. In the following sections we further discuss the generalization capacities of the representations learned by InferSent, and focus in particular on their differences and similarities to human generalization.

6 Generalization

An important and well-studied aspect of human-like representations is that rules learned with one set of tokens can be systematically generalized to other tokens (Fodor \BBA Pylyshyn, \APACyear1988; Lake \BOthers., \APACyear2019). In the section on ‘zero-shot reasoning’ we study if our machine-learned representations can perform such generalization to tokens that have never previously been observed. More often however, the tokens to which we want to generalize learned rules have previously been observed, but simply in a different context. The historical contexts of tokens can determine some of their properties – like syntactic category, and semantic content – which in turn inform how humans generalize rules to them, sometimes deviating from entirely systematic generalization. In our section on ‘context-tying’, we examine how the historical context of tokens influences systematic generalization in our machine-learned representations, and how these effects compare to those in humans.

Throughout this section, we will only consider sentence pairs that are similar in structure to ones in our Comparisons dataset, and will no longer consider performance on SNLI. We will predominantly be studying the model that has been trained jointly with our Comparisons dataset in addition to SNLI (referred henceforth to as the augmented-InferSent model).

6.1 Zero-shot reasoning

Zero-shot reasoning is the ability to solve tasks involving a term that has never been seen before. This (often also called zero-shot learning) has commonly been used as a test for systematicity (Lake \BOthers., \APACyear2018) – a human can carry out inferences like “Anne is more boffy than Bob” entails that “Bob is less boffy than Anne” without ever having encountered the word “boffy” before.

But this ability requires the representation learned to be abstract, and not be tied to the Xs, Ys, and Z’s seen in training. Instead it has to encode encode an abstract relational rule where “X is more Y than Z” entails “Z is more Y than X” for all possible X, Y and Z, irrespective of their specific values. If the representation are tied to the observed values of Xs, Ys and Zs and cannot generalize to new values for these, each possible X, Y and Z has to have occurred in the training dataset. However, these can be arbitrarily complex (e.g., “The old woman with a flower in her hair is more deliriously happy than the tall young man wearing the blue bowler hat” implies that “The tall young man wearing the blue bowler hat is less deliriously happy than the old woman with a flower in her hair”). Ensuring that every possible such X, Y and Z have been seen in the training data is impossible, and this kind of generalization is key to human-like language understanding.

In this section we consider the performance of the augmented-InferSent model. We already know that this model performs well on both SNLI, and generalizes to new combinations of X, Y, and Z in our Comparisons dataset (see Table 10), where each X, Y and Z have previously been seen. In this section, we analyze its ability to generalize to 3 different kinds of Xs and Zs that have never been encountered during training.

  • Held out nouns: Nouns (from the GloVe dataset) that never occur in the training data (neither SNLI nor our Comparisons dataset).

  • Made up “words”: Directly using a 300 dimensional vector randomly sampled from an uncorrelated Gaussian distribution, as a stand-in for a real GloVe vector.

  • Long noun phrases: The Xs and Zs used in training as part of the Comparisons dataset were of the type “the man”. Here we generate longer noun phrases of the form “the grumpy man in front of us” consisting of randomly sampled adjectives, nouns and prepositional phrases.

For each sub-type in the Comparisons dataset (same, more-less and not types), we generated a test set of 1,000 sequences by substituting Xs and Zs of the above kinds. The Ys were sampled in the same way as in the Comparison dataset (random adjectives that appear in SNLI). We then tested on these sentences, and reported the average accuracy. Note that not only had these specific sentences (combinations of X, Y and Z) never been seen during training, even the individual Xs and Zs had not been seen. We found that InferSent generalizes to all three new kinds of Xs and Zs quite well (Table 11). The held-out nouns are the most similar to the Xs and Zs seen during training since they are also exactly one word, and are nouns sampled from GloVe. It is notable that generalization performance with these is comparable to that with the very different kinds of Xs and Zs such as the made-up words, or longer noun phrases, indicating a fairly abstract representation of relational rules that are not tied to the specific value of X and Z.

Test set InferSent () augmented-InferSent ()
Held-out nouns 47.9 82.0
Made up words 48.0 83.2
Long noun phrases 49.1 84.9
Table 11: Zero-shot reasoning: Performance on previously unobserved Xs and Zs.

This indicates that the representation learned by augmented-InferSent is partially abstract and composable, allowing some systematic generalization to a variety of Xs and Zs that have never been seen before. In the next section we further probe contextuality of generalization and how that interacts with the training set / learning environment, making comparisons to human generalization.

6.2 Context Tying

We saw in the previous section that augmented-InferSent has some of the the central human-like capacity of zero-shot reasoning. This indicates some systematicity in its representations. However, even humans do not always succeed at fully systematic generalization. In this section we investigate these exceptions and qualifications to the widest interpretation of systematic generalization, focusing on the role of context in generalization. We do this in two ways: using type violations and biased exposure.

6.2.1 Type violations

One extreme of learning a purely abstract rule like in Table 1 is to be completely insensitive to any properties of the Xs, Ys and Zs, and generalize this rule to all possible tokens. However, this very strong generalization may not always match human intuitions. For example the sentence pair
Premise: The punctual is more cheerful than the man
Hypothesis: The punctual is not more cheerful than the man
does not seem to have a right answer. The rule applies easily only to Xs, Ys and Zs that are of the right type – in this case the right syntactic category.

While syntactic structure is not directly provided to the embedding model, some notion of syntactic category will be implicit. Information about the syntactic category of a word can be gleaned from its contexts, i.e. the other words around it (Chomsky, \APACyear1993; Redington \BOthers., \APACyear1998; Socher \BOthers., \APACyear2010), and in some cases can be decoded from word embeddings directly (Pennington \BOthers., \APACyear2014).

We investigated generalization of rules in augmented-Infersent to test items which, unlike in the previous section, had been previously seen, but had only occurred in a different syntactic role (i.e., a different context). We generated a test set of ungrammatical sentences using Xs and Zs that are random non-nouns, in our case random adjectives from SNLI. Crucially, these words had been seen before, but never in the position/context that X and Z occupy in the Comparisons dataset, since appearing in those positions violates syntax. We then evaluated the performance of the augmented-InferSent model in the same way as in the previous section on zero-shot generalization. We found that accuracy on such sentence pairs is low, giving poor performance (Table 12). This indicates that the rules learned, though at least partially abstract as indicated by generalization to held-out nouns, come with restrictions on the type of (known) items they will apply to. This follows closely how humans generalize – that learned rules don’t generalize indiscriminately to all tokens, but rather only within some fixed categories. These categories in turn, like syntactic categories, can be gleaned from the contexts in which these tokens usually appear. In the next section, we examine the role of semantic content in the context of tokens, and how that influences generalization.

Test set InferSent () augmented-InferSent ()
Held-out nouns 47.9 82.0
Non-noun words 47.9 49.3
Table 12: Type violations: Performance with tokens from the wrong syntactic category, versus with held-out tokens from the right syntactic category

6.2.2 Biased exposure

In this section, we manipulate the context of various tokens, without violating the syntactic rules, to study its effect on generalization. In all the augmentations we have used so far, some token X is equally likely to occur in the context of a same-type sentence pair as it is in the context of a more/less-type sentence pair. Similarly, X is as likely to occur in the context where it is ‘more cheerful than the man’ as it is to occur in the context where it is ‘less cheerful than the man’. Therefore, apart from the restrictions of syntactically correct placement, there is no additional structure around which contexts which tokens occur in – they are all randomly distributed. However, in the real world, tokens are not uniformly sampled into contexts even within constraints of syntax; a word is much more likely to be sampled repeatedly in certain contexts than others. This is because the appearance of tokens in naturally occurring sentences is not determined solely by their syntactic role, but also by their semantic role. For example, one is more much likely to encounter the sentence “broccoli is more nutritious than candy” than the sentence “candy is more nutritious than broccoli”, since one is true of the real world, and the other is not. Nonetheless, the premise “candy is more nutritious than broccoli” still logically entails the hypothesis “broccoli is less nutritious than candy”. Statistics of how often certain implications and inferences are made in the learning environment (that will be reflected in semantic beliefs about the real-world) can interfere with such logical inferences in humans in both deductive (Evans \BOthers., \APACyear1983) and probabilistic (Evans \BOthers., \APACyear2002) reasoning. This is often termed ‘belief bias’.

In this section, we test if the representations we are studying exhibit belief bias. We manipulate the uniformity in the co-occurrence of tokens with contexts (subject to syntactic constraints), and examine if a newly augmented InferSent model can generalize a token it has seen in one context, to cases where it appears in a different context. We compare this to a zero-shot control condition, where the test token has never been seen before.

To this end, we first generated variants of our Comparisons dataset where tokens are no longer uniformly sampled into contexts. We considered only two sub-types of the comparison types summarized in Table 1: the same-type () and the more/less-type (). These consist the two contexts and in which tokens can appear. Noun phrases were generated using the same procedure used for the long noun phrases in the section on zero-shot reasoning—phrases (tokens) of the form “the grumpy man in front of us”. These tokens were then randomly divided into -type and -type (460 each). Therefore there is no structural difference between the the and tokens, only the context in which they are seen will differ across conditions.

We built four sets of sentence pairs that vary in their context-token combination: consisted of combinations of tokens in a context, so on and so forth for , , and . Each such context-token combination set was independently divided into train and test sets (each of size 5000). The sentence pairs in each of the four test sets had never been seen before in any of the four training sets.

We augmented the original InferSent embeddings with different combinations of samples from the four different train sets.333In this experiment we only make comparisons between the performances of differently augmented models, rather than considering the overall performance like in previous experiments. The influence on performance from the SNLI training data is irrelevant since it will affect all four augmented models equally. Therefore we can neglect SNLI performance and carry out our experiments using fine-tuned augmentation rather than full retraining (see the Section ‘Augmenting the learning environment’ for details on these). This is computationally a lot cheaper. We then compared their performance on all four of the test sets to examine how different context-token combinations seen during training influenced test generalization. The three different embeddings that result are as follows:

  • Zero-shot control condition: Only the tokens were seen in training; no token were seen at all. Therefore testing with tokens from is analogous to zero-shot reasoning. The training set consisted of the full training sets from and .

  • Experimental conditions: Both and tokens were seen in training, therefore testing with tokens from is not analogous to zero-shot reasoning. However, the contexts in which and tokens appear during training differed. There are two different embeddings we trained of this kind.

    • Exposed– : This embedding saw tokens in both and contexts (as with the control condition), and additionally also saw tokens – but only in the context. In order to balance the number of training examples from each context between conditions, the training set consisted of the full training sets from and half (randomly selected) of the training set from each of the and context-token combination sets.

    • Exposed– : This embedding saw tokens in both and contexts, but saw tokens only in the context. The training set was balanced across contexts here as well.

Test set Performance (%)
Zero-shot Exposed– Exposed–
97.44 97.02 98.0

95.72 99.7 61.16

95.78 67.71 99.96

Table 13: Biased exposure: Results from InferSent embeddings augmented with different training sets that manipulate the co-occurrence of context and token.

All three models received the same number of training examples, with equal numbers of sentence pairs from both contexts and . They all also saw noun phrases appear in both contexts. The three models only differed in which contexts noun phrases appeared during training. The control model never saw noun phrases, Exposed– only saw them in the context and Exposed– only saw them in the context. All of these were then tested on the same held-out test set. We see from Table 13 that all three models generalize well to held-out test examples involving previously unobserved combinations of noun phrases in both contexts (first row). This is consistent with our initial results on augmentation (see section ‘Augmenting the learning environment’). Further, the control (zero-shot reasoning) condition that never saw noun phrases in training generalizes well to all the test examples with noun phrases (first column). This is consistent with our results on zero-shot generalization (see section ‘Zero-shot reasoning’).

We now turn to generalization performance when tokens were seen before but only in a specific context (second and third columns in Table 13). We discuss the results for the model Exposed– (that saw noun phrases in type comparisons), a symmetric discussion applies also to Exposed–. W found that Exposed– performs well on held-out test examples from the category (99.7 %) – as consistent with our original experiments with augmentation. However, we found that it fails to generalize very well to type noun phrases in the context, with a significant drop in performance (67.71 %). The crucial comparison is that this low performance is also significantly worse that that of the zero-shot control on the same test set (95.78 %). Neither of these have seen phrases in the context – yet the control generalizes very well, while the Exposed– fails to. This indicates that while the representations learned can generalize well to previously unseen tokens, this generalization is poorer to tokens that have in fact been seen before, but only in a different context.

This indicates that our representations do learn something akin to belief bias, where the context in which tokens have been seen (even within the right syntactic category) can influence how abstract logical rules (like in Table 1) generalize to them. This suggests potential directions for research on modeling how belief bias in humans arises. However, it is crucial to point out that although humans do exhibit such context tying, the effects are mostly observed in children (Evans \BBA Perry, \APACyear1995) and under time pressure / cognitive load (Evans \BBA Curtis-Holmes, \APACyear2005). The co-existence of such a fast heuristic strategy (that potentially suffers from belief bias), and a slower deliberative strategy (that can perform abstract reasoning) is a well-studied and popular model for representations and decision rules in humans (Evans \BBA Curtis-Holmes, \APACyear2005; Kahneman, \APACyear2011; Groves \BBA Thompson, \APACyear1970). Thus, although people have a tendency towards belief bias, they are able to overcome it and engage in abstract reasoning, which our machine-learned representations cannot do.

This raises a new concern about the scalability of augmentation as a general approach to learning systematic representations in such tabula rasa machine-learning systems. There are infinitely many possible sentences that all follow the rules of syntax, so observing tokens in contexts that one has not often seen them in, but where they are syntactically valid, is likely to occur often. Our new findings show that while zero-shot reasoning to previously unobserved tokens works in certain cases, these tabula rasa systems may tie an observed token to the small fraction of contexts in which it has been seen. This hinders generalization to cases where this token occurs in a new context. In order for every token to have been observed in every context, a combinatorially large amount of augmented training data would be required, potentially making this approach unfeasible for achieving the kinds of systematic representations humans have.

7 Discussion and Future Work

In this paper, we carried out a case study in the use of methods from cognitive science and psycholinguistics to better understand machine-learned representations. We developed minimal cases in a natural language inference task that test for some aspects of abstract relational structure in sentences. We used this diagnostic tool on large-scale state-of-the-art sentence embeddings (Conneau \BOthers., \APACyear2017) to not only demonstrate its lack of abstract composable structure, but also provide insight into the representations and decision criteria actually learned. This approach led us to isolate the use of some simple heuristics, which we then traced to structural regularities in the training distribution. This allowed us to demonstrate the strong effect the training data has on the representations learned. We then augmented this training environment with so-called adversarial examples such that simple heuristics like the ones we found are no longer ecologically valid. We found that such augmentation leads the system to learn some forms of abstract relational structure. Notably, we found that one of the traditional holy grails of systematicity —zero-shot generalization of learned rules to new, previously unseen words—can be partially achieved using appropriate augmentation. Further tests, however, revealed limitations to the breadth of this generalization. We found that while zero-shot generalization to previously unseen words works, generalizations to words that have previously been seen in a different context, suffers. This gives us another measure for the extent of systematicity in representations—a phenomena we call ‘context-tying’. We discussed the relationship between this effect and findings in human cognitive psychology where semantic beliefs about the real-world can interfere with flexible inferences supported by abstract logical representations (Evans, \APACyear2013). This parallel suggests new ways to model this psychological phenomena (Dasgupta \BOthers., \APACyear2019). The presence of context-tying in the machine-learned representations indicates that combinatorially large amounts of augmentation will likely be required for a tabula rasa unstructured neural network model to learn an entirely systematic representation from data.

These results suggest many directions for future work. We showed how the issue of context-tying bodes poorly for the scalability of using only training set augmentations to achieve human-like systematic representations. Recent work, however, suggests such adversarial mechanisms in the human brain (S\BPBIJ. Gershman, \APACyear2019). This motivates further research on how this approach might be made more scalable. We studied the representations learned from a fixed amount of augmentation and training. An important step forward is to better understand how systematicity in these representations evolves over the course of augmented training, and exactly how much augmentation is really needed. Another important problem is to understand what augmentations work best. To that end, a promising direction is to integrate our approach, where augmentations are generated using existing knowledge about analogous representations in humans, with approaches that learn to generate such adversarial augmentations (Kang \BOthers., \APACyear2018; Goodfellow \BOthers., \APACyear2014; Zhao \BOthers., \APACyear2017).

Human infants are not as tabula rasa as models like InferSent but rather encode useful inductive biases (Mitchell, \APACyear1980; Pearl \BBA Goldwater, \APACyear2016; Chomsky \BBA Lightfoot, \APACyear2002; Lightfoot \BBA Julia, \APACyear1984; Seidenberg, \APACyear1997). Building such biases into our models (Lake \BOthers., \APACyear2018; Gandhi \BBA Lake, \APACyear2019; Dubey \BOthers., \APACyear2018; Battaglia \BOthers., \APACyear2018) is a promising direction towards scalably learning systematic representations. We also showed how analysis and controlled testing for heuristic strategies in the learning environment can provide rich insights into the representations learned. Such analyses could also be used to improve learning and subsequent performance by leveraging this underlying structure (Şimşek, \APACyear2013; Şimşek \BOthers., \APACyear2016; Gigerenzer \BBA Todd, \APACyear1999; Martignon \BBA Hoffrage, \APACyear2002). Finally, we leverage methods from cognitive psychology to introduce a new structured test dataset (the Comparisons dataset) as well as a new metric (context-tying) for sentence representations. Rather than the traditional single-dimensional metrics of the accuracy achieved on ad-hoc test datasets, our approach provides insights into the kinds of mistakes made and therefore a more principled and nuanced ways to benchmark artificial systems against humans (White \BOthers., \APACyear2017; Marelli \BOthers., \APACyear2014; Lake \BBA Baroni, \APACyear2017; Linzen \BOthers., \APACyear2016; McCoy \BOthers., \APACyear2019; Glockner \BOthers., \APACyear2018). A metric like context-tying is not bound to the domain of language, and can also be used to benchmark systematicity in other domains that benefit from abstract compositional representations – like scene understanding (Ommer \BBA Buhmann, \APACyear2009; Johnson \BOthers., \APACyear2017) or structured planning (Burridge \BOthers., \APACyear1999; Singh, \APACyear1992). Future work should pursue other such diagnostic metrics, to build towards a comprehensive suite of testable criteria for exactly what constitutes human-like representations, and also to further inform which aspects of these we wish to emulate in artificial systems.

7.1 Acknowledgements

We are grateful to Anatole Gershman, Tim O’Donnell, Joshua Greene and Fiery Cushman for helpful discussions. ID is supported by Microsoft Research. SJG is supported by the Office of Naval Research (N00014-17-1-2984). NDG is supported by DARPA agreement number FA8750-14-2-0009, and a Sloan Foundation Research Fellowship.


  • Battaglia \BOthers. (\APACyear2018) \APACinsertmetastarbattaglia2018relational{APACrefauthors}Battaglia, P\BPBIW., Hamrick, J\BPBIB., Bapst, V., Sanchez-Gonzalez, A., Zambaldi, V., Malinowski, M.\BDBLothers  \APACrefYearMonthDay2018. \BBOQ\APACrefatitleRelational inductive biases, deep learning, and graph networks Relational inductive biases, deep learning, and graph networks.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1806.01261. \PrintBackRefs\CurrentBib
  • Beals \BOthers. (\APACyear1968) \APACinsertmetastarbeals1968foundations{APACrefauthors}Beals, R., Krantz, D\BPBIH.\BCBL \BBA Tversky, A.  \APACrefYearMonthDay1968. \BBOQ\APACrefatitleFoundations of multidimensional scaling. Foundations of multidimensional scaling.\BBCQ \APACjournalVolNumPagesPsychological review752127. \PrintBackRefs\CurrentBib
  • Belinkov \BBA Glass (\APACyear2019) \APACinsertmetastarbelinkov2019analysis{APACrefauthors}Belinkov, Y.\BCBT \BBA Glass, J.  \APACrefYearMonthDay2019. \BBOQ\APACrefatitleAnalysis methods in neural language processing: A survey Analysis methods in neural language processing: A survey.\BBCQ \APACjournalVolNumPagesTransactions of the Association for Computational Linguistics749–72. \PrintBackRefs\CurrentBib
  • Bhatia (\APACyear2017) \APACinsertmetastarbhatia2017associative{APACrefauthors}Bhatia, S.  \APACrefYearMonthDay2017. \BBOQ\APACrefatitleAssociative judgment and vector space semantics. Associative judgment and vector space semantics.\BBCQ \APACjournalVolNumPagesPsychological review12411. \PrintBackRefs\CurrentBib
  • Bird \BBA Loper (\APACyear2004) \APACinsertmetastarbird2004nltk{APACrefauthors}Bird, S.\BCBT \BBA Loper, E.  \APACrefYearMonthDay2004. \BBOQ\APACrefatitleNLTK: the natural language toolkit Nltk: the natural language toolkit.\BBCQ \BIn \APACrefbtitleProceedings of the ACL 2004 on Interactive poster and demonstration sessions Proceedings of the acl 2004 on interactive poster and demonstration sessions (\BPG 31). \PrintBackRefs\CurrentBib
  • Bowman \BOthers. (\APACyear2015) \APACinsertmetastarsnli:emnlp2015{APACrefauthors}Bowman, S\BPBIR., Angeli, G., Potts, C.\BCBL \BBA Manning, C\BPBID.  \APACrefYearMonthDay2015. \BBOQ\APACrefatitleA large annotated corpus for learning natural language inference A large annotated corpus for learning natural language inference.\BBCQ \BIn \APACrefbtitleProceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). Proceedings of the 2015 conference on empirical methods in natural language processing (emnlp). \APACaddressPublisherAssociation for Computational Linguistics. \PrintBackRefs\CurrentBib
  • Braine (\APACyear1978) \APACinsertmetastarbraine1978relation{APACrefauthors}Braine, M\BPBID.  \APACrefYearMonthDay1978. \BBOQ\APACrefatitleOn the relation between the natural logic of reasoning and standard logic. On the relation between the natural logic of reasoning and standard logic.\BBCQ \APACjournalVolNumPagesPsychological review8511. \PrintBackRefs\CurrentBib
  • Burridge \BOthers. (\APACyear1999) \APACinsertmetastarburridge1999sequential{APACrefauthors}Burridge, R\BPBIR., Rizzi, A\BPBIA.\BCBL \BBA Koditschek, D\BPBIE.  \APACrefYearMonthDay1999. \BBOQ\APACrefatitleSequential composition of dynamically dexterous robot behaviors Sequential composition of dynamically dexterous robot behaviors.\BBCQ \APACjournalVolNumPagesThe International Journal of Robotics Research186534–555. \PrintBackRefs\CurrentBib
  • Chomsky (\APACyear1993) \APACinsertmetastarchomsky1993lectures{APACrefauthors}Chomsky, N.  \APACrefYear1993. \APACrefbtitleLectures on government and binding: The Pisa lectures Lectures on government and binding: The pisa lectures (\BNUM 9). \APACaddressPublisherWalter de Gruyter. \PrintBackRefs\CurrentBib
  • Chomsky \BBA Lightfoot (\APACyear2002) \APACinsertmetastarchomsky2002syntactic{APACrefauthors}Chomsky, N.\BCBT \BBA Lightfoot, D\BPBIW.  \APACrefYear2002. \APACrefbtitleSyntactic structures Syntactic structures. \APACaddressPublisherWalter de Gruyter. \PrintBackRefs\CurrentBib
  • Conneau \BOthers. (\APACyear2017) \APACinsertmetastarConneau:2017uf{APACrefauthors}Conneau, A., Kiela, D., Schwenk, H., Barrault, L.\BCBL \BBA Bordes, A.  \APACrefYearMonthDay2017\APACmonth05. \BBOQ\APACrefatitleSupervised Learning of Universal Sentence Representations from Natural Language Inference Data Supervised Learning of Universal Sentence Representations from Natural Language Inference Data.\BBCQ \PrintBackRefs\CurrentBib
  • Dasgupta \BOthers. (\APACyear2018) \APACinsertmetastardasgupta2018evaluating{APACrefauthors}Dasgupta, I., Guo, D., Stuhlmüller, A., Gershman, S\BPBIJ.\BCBL \BBA Goodman, N\BPBID.  \APACrefYearMonthDay2018. \BBOQ\APACrefatitleEvaluating compositionality in sentence embeddings Evaluating compositionality in sentence embeddings.\BBCQ \APACjournalVolNumPagesProceedings of the 39th Annual Conference of the Cognitive Science Society. \PrintBackRefs\CurrentBib
  • Dasgupta \BOthers. (\APACyear2019) \APACinsertmetastardasgupta2019theory{APACrefauthors}Dasgupta, I., Schulz, E., Tenenbaum, J\BPBIB.\BCBL \BBA Gershman, S\BPBIJ.  \APACrefYearMonthDay2019. \BBOQ\APACrefatitleA theory of learning to infer A theory of learning to infer.\BBCQ \APACjournalVolNumPagesBioRxiv644534. \PrintBackRefs\CurrentBib
  • Dubey \BOthers. (\APACyear2018) \APACinsertmetastardubey2018investigating{APACrefauthors}Dubey, R., Agrawal, P., Pathak, D., Griffiths, T\BPBIL.\BCBL \BBA Efros, A\BPBIA.  \APACrefYearMonthDay2018. \BBOQ\APACrefatitleInvestigating human priors for playing video games Investigating human priors for playing video games.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1802.10217. \PrintBackRefs\CurrentBib
  • Evans (\APACyear2013) \APACinsertmetastarevans2013psychology{APACrefauthors}Evans, J\BPBIS\BPBIB.  \APACrefYear2013. \APACrefbtitleThe psychology of deductive reasoning (Psychology revivals) The psychology of deductive reasoning (psychology revivals). \APACaddressPublisherPsychology Press. \PrintBackRefs\CurrentBib
  • Evans \BOthers. (\APACyear1983) \APACinsertmetastarevans1983conflict{APACrefauthors}Evans, J\BPBIS\BPBIB., Barston, J\BPBIL.\BCBL \BBA Pollard, P.  \APACrefYearMonthDay1983. \BBOQ\APACrefatitleOn the conflict between logic and belief in syllogistic reasoning On the conflict between logic and belief in syllogistic reasoning.\BBCQ \APACjournalVolNumPagesMemory & cognition113295–306. \PrintBackRefs\CurrentBib
  • Evans \BBA Curtis-Holmes (\APACyear2005) \APACinsertmetastarevans2005rapid{APACrefauthors}Evans, J\BPBIS\BPBIB.\BCBT \BBA Curtis-Holmes, J.  \APACrefYearMonthDay2005. \BBOQ\APACrefatitleRapid responding increases belief bias: Evidence for the dual-process theory of reasoning Rapid responding increases belief bias: Evidence for the dual-process theory of reasoning.\BBCQ \APACjournalVolNumPagesThinking & Reasoning114382–389. \PrintBackRefs\CurrentBib
  • Evans \BOthers. (\APACyear2002) \APACinsertmetastarevans2002background{APACrefauthors}Evans, J\BPBIS\BPBIB., Handley, S\BPBIJ., Over, D\BPBIE.\BCBL \BBA Perham, N.  \APACrefYearMonthDay2002. \BBOQ\APACrefatitleBackground beliefs in Bayesian inference Background beliefs in bayesian inference.\BBCQ \APACjournalVolNumPagesMemory & Cognition302179–190. \PrintBackRefs\CurrentBib
  • Evans \BBA Perry (\APACyear1995) \APACinsertmetastarevans1995belief{APACrefauthors}Evans, J\BPBIS\BPBIB.\BCBT \BBA Perry, T\BPBIS.  \APACrefYearMonthDay1995. \BBOQ\APACrefatitleBelief bias in children’s reasoning. Belief bias in children’s reasoning.\BBCQ \APACjournalVolNumPagesCahiers de Psychologie Cognitive/Current Psychology of Cognition. \PrintBackRefs\CurrentBib
  • Fodor \BBA Pylyshyn (\APACyear1988) \APACinsertmetastarfodor88{APACrefauthors}Fodor, J\BPBIA.\BCBT \BBA Pylyshyn, Z\BPBIW.  \APACrefYearMonthDay1988. \BBOQ\APACrefatitleConnectionism and cognitive architecture: A critical analysis Connectionism and cognitive architecture: A critical analysis.\BBCQ \APACjournalVolNumPagesCognition283–71. \PrintBackRefs\CurrentBib
  • Gandhi \BBA Lake (\APACyear2019) \APACinsertmetastargandhi2019mutual{APACrefauthors}Gandhi, K.\BCBT \BBA Lake, B\BPBIM.  \APACrefYearMonthDay2019. \BBOQ\APACrefatitleMutual exclusivity as a challenge for neural networks Mutual exclusivity as a challenge for neural networks.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1906.10197. \PrintBackRefs\CurrentBib
  • S. Gershman \BBA Tenenbaum (\APACyear2015) \APACinsertmetastargershman15{APACrefauthors}Gershman, S.\BCBT \BBA Tenenbaum, J\BPBIB.  \APACrefYearMonthDay2015. \BBOQ\APACrefatitlePhrase similarity in humans and machines Phrase similarity in humans and machines.\BBCQ \BIn \APACrefbtitleProceedings of the 37th Annual Conference of the Cognitive Science Society. Proceedings of the 37th annual conference of the cognitive science society. \PrintBackRefs\CurrentBib
  • S\BPBIJ. Gershman (\APACyear2019) \APACinsertmetastargershman2019generative{APACrefauthors}Gershman, S\BPBIJ.  \APACrefYearMonthDay2019. \BBOQ\APACrefatitleThe generative adversarial brain The generative adversarial brain.\BBCQ \PrintBackRefs\CurrentBib
  • Gigerenzer \BBA Todd (\APACyear1999) \APACinsertmetastargigerenzer1999simple{APACrefauthors}Gigerenzer, G.\BCBT \BBA Todd, P\BPBIM.  \APACrefYear1999. \APACrefbtitleSimple heuristics that make us smart Simple heuristics that make us smart. \APACaddressPublisherOxford University Press, USA. \PrintBackRefs\CurrentBib
  • Glockner \BOthers. (\APACyear2018) \APACinsertmetastarglockner2018breaking{APACrefauthors}Glockner, M., Shwartz, V.\BCBL \BBA Goldberg, Y.  \APACrefYearMonthDay2018. \BBOQ\APACrefatitleBreaking nli systems with sentences that require simple lexical inferences Breaking nli systems with sentences that require simple lexical inferences.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1805.02266. \PrintBackRefs\CurrentBib
  • Goodfellow \BOthers. (\APACyear2014) \APACinsertmetastargoodfellow2014explaining{APACrefauthors}Goodfellow, I\BPBIJ., Shlens, J.\BCBL \BBA Szegedy, C.  \APACrefYearMonthDay2014. \BBOQ\APACrefatitleExplaining and harnessing adversarial examples Explaining and harnessing adversarial examples.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1412.6572. \PrintBackRefs\CurrentBib
  • Groves \BBA Thompson (\APACyear1970) \APACinsertmetastargroves1970habituation{APACrefauthors}Groves, P\BPBIM.\BCBT \BBA Thompson, R\BPBIF.  \APACrefYearMonthDay1970. \BBOQ\APACrefatitleHabituation: a dual-process theory. Habituation: a dual-process theory.\BBCQ \APACjournalVolNumPagesPsychological review775419. \PrintBackRefs\CurrentBib
  • Hill \BOthers. (\APACyear2016) \APACinsertmetastarHill:2016uu{APACrefauthors}Hill, F., Cho, K.\BCBL \BBA Korhonen, A.  \APACrefYearMonthDay2016\APACmonth02. \BBOQ\APACrefatitleLearning Distributed Representations of Sentences from Unlabelled Data Learning Distributed Representations of Sentences from Unlabelled Data.\BBCQ \PrintBackRefs\CurrentBib
  • Hornik (\APACyear1991) \APACinsertmetastarhornik1991approximation{APACrefauthors}Hornik, K.  \APACrefYearMonthDay1991. \BBOQ\APACrefatitleApproximation capabilities of multilayer feedforward networks Approximation capabilities of multilayer feedforward networks.\BBCQ \APACjournalVolNumPagesNeural networks42251–257. \PrintBackRefs\CurrentBib
  • Johnson \BOthers. (\APACyear2017) \APACinsertmetastarjohnson2017clevr{APACrefauthors}Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C.\BCBL \BBA Girshick, R.  \APACrefYearMonthDay2017. \BBOQ\APACrefatitleClevr: A diagnostic dataset for compositional language and elementary visual reasoning Clevr: A diagnostic dataset for compositional language and elementary visual reasoning.\BBCQ \BIn \APACrefbtitleProceedings of the IEEE Conference on Computer Vision and Pattern Recognition Proceedings of the ieee conference on computer vision and pattern recognition (\BPGS 2901–2910). \PrintBackRefs\CurrentBib
  • Johnson-Laird \BBA Steedman (\APACyear1978) \APACinsertmetastarjohnson1978psychology{APACrefauthors}Johnson-Laird, P\BPBIN.\BCBT \BBA Steedman, M.  \APACrefYearMonthDay1978. \BBOQ\APACrefatitleThe psychology of syllogisms The psychology of syllogisms.\BBCQ \APACjournalVolNumPagesCognitive psychology10164–99. \PrintBackRefs\CurrentBib
  • Jones \BBA Mewhort (\APACyear2007) \APACinsertmetastarjones2007representing{APACrefauthors}Jones, M\BPBIN.\BCBT \BBA Mewhort, D\BPBIJ.  \APACrefYearMonthDay2007. \BBOQ\APACrefatitleRepresenting word meaning and order information in a composite holographic lexicon. Representing word meaning and order information in a composite holographic lexicon.\BBCQ \APACjournalVolNumPagesPsychological review11411. \PrintBackRefs\CurrentBib
  • Kádár \BOthers. (\APACyear2017) \APACinsertmetastarkadar2017representation{APACrefauthors}Kádár, A., Chrupała, G.\BCBL \BBA Alishahi, A.  \APACrefYearMonthDay2017. \BBOQ\APACrefatitleRepresentation of linguistic form and function in recurrent neural networks Representation of linguistic form and function in recurrent neural networks.\BBCQ \APACjournalVolNumPagesComputational Linguistics434761–780. \PrintBackRefs\CurrentBib
  • Kahneman (\APACyear2011) \APACinsertmetastarkahneman2011thinking{APACrefauthors}Kahneman, D.  \APACrefYear2011. \APACrefbtitleThinking, fast and slow Thinking, fast and slow. \APACaddressPublisherMacmillan. \PrintBackRefs\CurrentBib
  • Kang \BOthers. (\APACyear2018) \APACinsertmetastarkang2018adventure{APACrefauthors}Kang, D., Khot, T., Sabharwal, A.\BCBL \BBA Hovy, E.  \APACrefYearMonthDay2018. \BBOQ\APACrefatitleAdventure: Adversarial training for textual entailment with knowledge-guided examples Adventure: Adversarial training for textual entailment with knowledge-guided examples.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1805.04680. \PrintBackRefs\CurrentBib
  • Karpathy \BOthers. (\APACyear2015) \APACinsertmetastarkarpathy2015visualizing{APACrefauthors}Karpathy, A., Johnson, J.\BCBL \BBA Fei-Fei, L.  \APACrefYearMonthDay2015. \BBOQ\APACrefatitleVisualizing and understanding recurrent networks Visualizing and understanding recurrent networks.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1506.02078. \PrintBackRefs\CurrentBib
  • Kiros \BOthers. (\APACyear2015) \APACinsertmetastarkiros2015skip{APACrefauthors}Kiros, R., Zhu, Y., Salakhutdinov, R\BPBIR., Zemel, R., Urtasun, R., Torralba, A.\BCBL \BBA Fidler, S.  \APACrefYearMonthDay2015. \BBOQ\APACrefatitleSkip-thought vectors Skip-thought vectors.\BBCQ \BIn \APACrefbtitleAdvances in neural information processing systems Advances in neural information processing systems (\BPGS 3294–3302). \PrintBackRefs\CurrentBib
  • Lake \BBA Baroni (\APACyear2017) \APACinsertmetastarlake17{APACrefauthors}Lake, B\BPBIM.\BCBT \BBA Baroni, M.  \APACrefYearMonthDay2017. \BBOQ\APACrefatitleStill not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1711.00350. \PrintBackRefs\CurrentBib
  • Lake \BOthers. (\APACyear2019) \APACinsertmetastarlake2019human{APACrefauthors}Lake, B\BPBIM., Linzen, T.\BCBL \BBA Baroni, M.  \APACrefYearMonthDay2019. \BBOQ\APACrefatitleHuman few-shot learning of compositional instructions Human few-shot learning of compositional instructions.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1901.04587. \PrintBackRefs\CurrentBib
  • Lake \BOthers. (\APACyear2018) \APACinsertmetastarlake18{APACrefauthors}Lake, B\BPBIM., Ullman, T\BPBID., Tenenbaum, J\BPBIB.\BCBL \BBA Gershman, S\BPBIJ.  \APACrefYearMonthDay2018. \BBOQ\APACrefatitleBuilding machines that learn and think like people Building machines that learn and think like people.\BBCQ \APACjournalVolNumPagesBehavioral and Brain Sciences40. \PrintBackRefs\CurrentBib
  • Landauer \BBA Dumais (\APACyear1997) \APACinsertmetastarlandauer1997solution{APACrefauthors}Landauer, T\BPBIK.\BCBT \BBA Dumais, S\BPBIT.  \APACrefYearMonthDay1997. \BBOQ\APACrefatitleA solution to Plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge. A solution to plato’s problem: The latent semantic analysis theory of acquisition, induction, and representation of knowledge.\BBCQ \APACjournalVolNumPagesPsychological review1042211. \PrintBackRefs\CurrentBib
  • LeCun \BOthers. (\APACyear2015) \APACinsertmetastarlecun2015deep{APACrefauthors}LeCun, Y., Bengio, Y.\BCBL \BBA Hinton, G.  \APACrefYearMonthDay2015. \BBOQ\APACrefatitleDeep learning Deep learning.\BBCQ \APACjournalVolNumPagesnature5217553436. \PrintBackRefs\CurrentBib
  • Li \BOthers. (\APACyear2015) \APACinsertmetastarli2015visualizing{APACrefauthors}Li, J., Chen, X., Hovy, E.\BCBL \BBA Jurafsky, D.  \APACrefYearMonthDay2015. \BBOQ\APACrefatitleVisualizing and understanding neural models in nlp Visualizing and understanding neural models in nlp.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1506.01066. \PrintBackRefs\CurrentBib
  • Lightfoot \BBA Julia (\APACyear1984) \APACinsertmetastarlightfoot1984language{APACrefauthors}Lightfoot, D.\BCBT \BBA Julia, P.  \APACrefYearMonthDay1984. \BBOQ\APACrefatitleThe language lottery: Toward a biology of grammars The language lottery: Toward a biology of grammars.\BBCQ \PrintBackRefs\CurrentBib
  • Linzen (\APACyear2019) \APACinsertmetastarlinzen2019can{APACrefauthors}Linzen, T.  \APACrefYearMonthDay2019. \BBOQ\APACrefatitleWhat can linguistics and deep learning contribute to each other? Response to Pater What can linguistics and deep learning contribute to each other? response to pater.\BBCQ \APACjournalVolNumPagesLanguage. \PrintBackRefs\CurrentBib
  • Linzen \BOthers. (\APACyear2016) \APACinsertmetastarlinzen2016assessing{APACrefauthors}Linzen, T., Dupoux, E.\BCBL \BBA Goldberg, Y.  \APACrefYearMonthDay2016. \BBOQ\APACrefatitleAssessing the ability of LSTMs to learn syntax-sensitive dependencies Assessing the ability of lstms to learn syntax-sensitive dependencies.\BBCQ \APACjournalVolNumPagesTransactions of the Association for Computational Linguistics4521–535. \PrintBackRefs\CurrentBib
  • Marcus (\APACyear2018) \APACinsertmetastarmarcus2018deep{APACrefauthors}Marcus, G.  \APACrefYearMonthDay2018. \BBOQ\APACrefatitleDeep learning: A critical appraisal Deep learning: A critical appraisal.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1801.00631. \PrintBackRefs\CurrentBib
  • Marelli \BOthers. (\APACyear2014) \APACinsertmetastarmarelli2014semeval{APACrefauthors}Marelli, M., Bentivogli, L., Baroni, M., Bernardi, R., Menini, S.\BCBL \BBA Zamparelli, R.  \APACrefYearMonthDay2014. \BBOQ\APACrefatitleSemeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment.\BBCQ \BIn \APACrefbtitleProceedings of the 8th international workshop on semantic evaluation (SemEval 2014) Proceedings of the 8th international workshop on semantic evaluation (semeval 2014) (\BPGS 1–8). \PrintBackRefs\CurrentBib
  • Martignon \BBA Hoffrage (\APACyear2002) \APACinsertmetastarmartignon2002fast{APACrefauthors}Martignon, L.\BCBT \BBA Hoffrage, U.  \APACrefYearMonthDay2002. \BBOQ\APACrefatitleFast, frugal, and fit: Simple heuristics for paired comparison Fast, frugal, and fit: Simple heuristics for paired comparison.\BBCQ \APACjournalVolNumPagesTheory and Decision52129–71. \PrintBackRefs\CurrentBib
  • McCoy \BOthers. (\APACyear2019) \APACinsertmetastarmccoy2019right{APACrefauthors}McCoy, R\BPBIT., Pavlick, E.\BCBL \BBA Linzen, T.  \APACrefYearMonthDay2019. \BBOQ\APACrefatitleRight for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1902.01007. \PrintBackRefs\CurrentBib
  • Mikolov \BOthers. (\APACyear2013) \APACinsertmetastarmikolov2013distributed{APACrefauthors}Mikolov, T., Sutskever, I., Chen, K., Corrado, G\BPBIS.\BCBL \BBA Dean, J.  \APACrefYearMonthDay2013. \BBOQ\APACrefatitleDistributed representations of words and phrases and their compositionality Distributed representations of words and phrases and their compositionality.\BBCQ \BIn \APACrefbtitleAdvances in neural information processing systems Advances in neural information processing systems (\BPGS 3111–3119). \PrintBackRefs\CurrentBib
  • Mitchell (\APACyear1980) \APACinsertmetastarmitchell1980need{APACrefauthors}Mitchell, T\BPBIM.  \APACrefYear1980. \APACrefbtitleThe need for biases in learning generalizations The need for biases in learning generalizations. \APACaddressPublisherDepartment of Computer Science, Laboratory for Computer Science Research …. \PrintBackRefs\CurrentBib
  • Mnih \BOthers. (\APACyear2015) \APACinsertmetastarmnih2015human{APACrefauthors}Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A\BPBIA., Veness, J., Bellemare, M\BPBIG.\BDBLothers  \APACrefYearMonthDay2015. \BBOQ\APACrefatitleHuman-level control through deep reinforcement learning Human-level control through deep reinforcement learning.\BBCQ \APACjournalVolNumPagesNature5187540529. \PrintBackRefs\CurrentBib
  • Nie \BOthers. (\APACyear2019) \APACinsertmetastarnie2019analyzing{APACrefauthors}Nie, Y., Wang, Y.\BCBL \BBA Bansal, M.  \APACrefYearMonthDay2019. \BBOQ\APACrefatitleAnalyzing compositionality-sensitivity of nli models Analyzing compositionality-sensitivity of nli models.\BBCQ \BIn \APACrefbtitleProceedings of the AAAI Conference on Artificial Intelligence Proceedings of the aaai conference on artificial intelligence (\BVOL 33, \BPGS 6867–6874). \PrintBackRefs\CurrentBib
  • Ommer \BBA Buhmann (\APACyear2009) \APACinsertmetastarommer2009learning{APACrefauthors}Ommer, B.\BCBT \BBA Buhmann, J.  \APACrefYearMonthDay2009. \BBOQ\APACrefatitleLearning the compositional nature of visual object categories for recognition Learning the compositional nature of visual object categories for recognition.\BBCQ \APACjournalVolNumPagesIEEE Transactions on Pattern Analysis and Machine Intelligence323501–516. \PrintBackRefs\CurrentBib
  • Oord \BOthers. (\APACyear2016) \APACinsertmetastaroord2016wavenet{APACrefauthors}Oord, A\BPBIv\BPBId., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A.\BDBLKavukcuoglu, K.  \APACrefYearMonthDay2016. \BBOQ\APACrefatitleWavenet: A generative model for raw audio Wavenet: A generative model for raw audio.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1609.03499. \PrintBackRefs\CurrentBib
  • Pearl \BBA Goldwater (\APACyear2016) \APACinsertmetastarpearl2016statistical{APACrefauthors}Pearl, L.\BCBT \BBA Goldwater, S.  \APACrefYearMonthDay2016. \BBOQ\APACrefatitleStatistical learning, inductive bias, and Bayesian inference in language acquisition Statistical learning, inductive bias, and bayesian inference in language acquisition.\BBCQ \PrintBackRefs\CurrentBib
  • Pennington \BOthers. (\APACyear2014) \APACinsertmetastarpennington14{APACrefauthors}Pennington, J., Socher, R.\BCBL \BBA Manning, C.  \APACrefYearMonthDay2014. \BBOQ\APACrefatitleGlove: Global vectors for word representation Glove: Global vectors for word representation.\BBCQ \BIn \APACrefbtitleProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) Proceedings of the 2014 conference on empirical methods in natural language processing (emnlp) (\BPGS 1532–1543). \PrintBackRefs\CurrentBib
  • Pereira \BOthers. (\APACyear2016) \APACinsertmetastarpereira2016comparative{APACrefauthors}Pereira, F., Gershman, S., Ritter, S.\BCBL \BBA Botvinick, M.  \APACrefYearMonthDay2016. \BBOQ\APACrefatitleA comparative evaluation of off-the-shelf distributed semantic representations for modelling behavioural data A comparative evaluation of off-the-shelf distributed semantic representations for modelling behavioural data.\BBCQ \APACjournalVolNumPagesCognitive neuropsychology333-4175–190. \PrintBackRefs\CurrentBib
  • Redington \BOthers. (\APACyear1998) \APACinsertmetastarredington1998distributional{APACrefauthors}Redington, M., Crater, N.\BCBL \BBA Finch, S.  \APACrefYearMonthDay1998. \BBOQ\APACrefatitleDistributional information: A powerful cue for acquiring syntactic categories Distributional information: A powerful cue for acquiring syntactic categories.\BBCQ \APACjournalVolNumPagesCognitive science224425–469. \PrintBackRefs\CurrentBib
  • Ritter \BOthers. (\APACyear2017) \APACinsertmetastarritter17{APACrefauthors}Ritter, S., Barrett, D\BPBIG., Santoro, A.\BCBL \BBA Botvinick, M\BPBIM.  \APACrefYearMonthDay2017. \BBOQ\APACrefatitleCognitive Psychology for Deep Neural Networks: A Shape Bias Case Study Cognitive psychology for deep neural networks: A shape bias case study.\BBCQ \BIn \APACrefbtitleInternational Conference on Machine Learning International conference on machine learning (\BPGS 2940–2949). \PrintBackRefs\CurrentBib
  • Ruder (\APACyear2016) \APACinsertmetastarruder2016overview{APACrefauthors}Ruder, S.  \APACrefYearMonthDay2016. \BBOQ\APACrefatitleAn overview of gradient descent optimization algorithms An overview of gradient descent optimization algorithms.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1609.04747. \PrintBackRefs\CurrentBib
  • Rumelhart \BBA Abrahamson (\APACyear1973) \APACinsertmetastarrumelhart1973model{APACrefauthors}Rumelhart, D\BPBIE.\BCBT \BBA Abrahamson, A\BPBIA.  \APACrefYearMonthDay1973. \BBOQ\APACrefatitleA model for analogical reasoning A model for analogical reasoning.\BBCQ \APACjournalVolNumPagesCognitive Psychology511–28. \PrintBackRefs\CurrentBib
  • Russakovsky \BOthers. (\APACyear2015) \APACinsertmetastarrussakovsky2015imagenet{APACrefauthors}Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S.\BDBLothers  \APACrefYearMonthDay2015. \BBOQ\APACrefatitleImagenet large scale visual recognition challenge Imagenet large scale visual recognition challenge.\BBCQ \APACjournalVolNumPagesInternational journal of computer vision1153211–252. \PrintBackRefs\CurrentBib
  • Seidenberg (\APACyear1997) \APACinsertmetastarseidenberg1997language{APACrefauthors}Seidenberg, M\BPBIS.  \APACrefYearMonthDay1997. \BBOQ\APACrefatitleLanguage acquisition and use: Learning and applying probabilistic constraints Language acquisition and use: Learning and applying probabilistic constraints.\BBCQ \APACjournalVolNumPagesScience27553061599–1603. \PrintBackRefs\CurrentBib
  • Siegelmann \BBA Sontag (\APACyear1995) \APACinsertmetastarsiegelmann1995computational{APACrefauthors}Siegelmann, H\BPBIT.\BCBT \BBA Sontag, E\BPBID.  \APACrefYearMonthDay1995. \BBOQ\APACrefatitleOn the computational power of neural nets On the computational power of neural nets.\BBCQ \APACjournalVolNumPagesJournal of computer and system sciences501132–150. \PrintBackRefs\CurrentBib
  • Simon (\APACyear1991) \APACinsertmetastarsimon1991bounded{APACrefauthors}Simon, H\BPBIA.  \APACrefYearMonthDay1991. \BBOQ\APACrefatitleBounded rationality and organizational learning Bounded rationality and organizational learning.\BBCQ \APACjournalVolNumPagesOrganization science21125–134. \PrintBackRefs\CurrentBib
  • Şimşek (\APACyear2013) \APACinsertmetastarcsimcsek2013linear{APACrefauthors}Şimşek, Ö.  \APACrefYearMonthDay2013. \BBOQ\APACrefatitleLinear decision rule as aspiration for simple decision heuristics Linear decision rule as aspiration for simple decision heuristics.\BBCQ \BIn \APACrefbtitleAdvances in neural information processing systems Advances in neural information processing systems (\BPGS 2904–2912). \PrintBackRefs\CurrentBib
  • Şimşek \BOthers. (\APACyear2016) \APACinsertmetastarcsimcsek2016most{APACrefauthors}Şimşek, Ö., Algorta, S.\BCBL \BBA Kothiyal, A.  \APACrefYearMonthDay2016. \BBOQ\APACrefatitleWhy most decisions are easy in tetris-and perhaps in other sequential decision problems, as well Why most decisions are easy in tetris-and perhaps in other sequential decision problems, as well.\BBCQ \APACjournalVolNumPagesProceedings of Machine Learning Research481757–1765. \PrintBackRefs\CurrentBib
  • Singh (\APACyear1992) \APACinsertmetastarsingh1992transfer{APACrefauthors}Singh, S\BPBIP.  \APACrefYearMonthDay1992. \BBOQ\APACrefatitleTransfer of learning by composing solutions of elemental sequential tasks Transfer of learning by composing solutions of elemental sequential tasks.\BBCQ \APACjournalVolNumPagesMachine Learning83-4323–339. \PrintBackRefs\CurrentBib
  • Socher \BOthers. (\APACyear2010) \APACinsertmetastarsocher2010learning{APACrefauthors}Socher, R., Manning, C\BPBID.\BCBL \BBA Ng, A\BPBIY.  \APACrefYearMonthDay2010. \BBOQ\APACrefatitleLearning continuous phrase representations and syntactic parsing with recursive neural networks Learning continuous phrase representations and syntactic parsing with recursive neural networks.\BBCQ \BIn \APACrefbtitleProceedings of the NIPS-2010 Deep Learning and Unsupervised Feature Learning Workshop Proceedings of the nips-2010 deep learning and unsupervised feature learning workshop (\BVOL 2010, \BPGS 1–9). \PrintBackRefs\CurrentBib
  • Steyvers (\APACyear2006) \APACinsertmetastarsteyvers2006multidimensional{APACrefauthors}Steyvers, M.  \APACrefYearMonthDay2006. \BBOQ\APACrefatitleMultidimensional scaling Multidimensional scaling.\BBCQ \APACjournalVolNumPagesEncyclopedia of cognitive science. \PrintBackRefs\CurrentBib
  • Todd \BBA Gigerenzer (\APACyear2007) \APACinsertmetastartodd2007environments{APACrefauthors}Todd, P\BPBIM.\BCBT \BBA Gigerenzer, G.  \APACrefYearMonthDay2007. \BBOQ\APACrefatitleEnvironments that make us smart: Ecological rationality Environments that make us smart: Ecological rationality.\BBCQ \APACjournalVolNumPagesCurrent directions in psychological science163167–171. \PrintBackRefs\CurrentBib
  • White \BOthers. (\APACyear2017) \APACinsertmetastarwhite2017inference{APACrefauthors}White, A\BPBIS., Rastogi, P., Duh, K.\BCBL \BBA Van Durme, B.  \APACrefYearMonthDay2017. \BBOQ\APACrefatitleInference is everything: Recasting semantic resources into a unified evaluation framework Inference is everything: Recasting semantic resources into a unified evaluation framework.\BBCQ \BIn \APACrefbtitleProceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers) Proceedings of the eighth international joint conference on natural language processing (volume 1: Long papers) (\BPGS 996–1005). \PrintBackRefs\CurrentBib
  • Yosinski \BOthers. (\APACyear2015) \APACinsertmetastaryosinski2015understanding{APACrefauthors}Yosinski, J., Clune, J., Nguyen, A., Fuchs, T.\BCBL \BBA Lipson, H.  \APACrefYearMonthDay2015. \BBOQ\APACrefatitleUnderstanding neural networks through deep visualization Understanding neural networks through deep visualization.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1506.06579. \PrintBackRefs\CurrentBib
  • Zeiler \BBA Fergus (\APACyear2014) \APACinsertmetastarzeiler2014visualizing{APACrefauthors}Zeiler, M\BPBID.\BCBT \BBA Fergus, R.  \APACrefYearMonthDay2014. \BBOQ\APACrefatitleVisualizing and understanding convolutional networks Visualizing and understanding convolutional networks.\BBCQ \BIn \APACrefbtitleEuropean conference on computer vision European conference on computer vision (\BPGS 818–833). \PrintBackRefs\CurrentBib
  • Zhao \BOthers. (\APACyear2017) \APACinsertmetastarzhao2017generating{APACrefauthors}Zhao, Z., Dua, D.\BCBL \BBA Singh, S.  \APACrefYearMonthDay2017. \BBOQ\APACrefatitleGenerating natural adversarial examples Generating natural adversarial examples.\BBCQ \APACjournalVolNumPagesarXiv preprint arXiv:1710.11342. \PrintBackRefs\CurrentBib
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description