Brain-Like Object Recognition with High-Performing Shallow Recurrent ANNs

Brain-Like Object Recognition with
High-Performing Shallow Recurrent ANNs

Jonas Kubilius Martin Schrimpf Ha Hong Bay Labs Inc., San Francisco, CA 94102 Najib J. Majaj Center for Neural Science, New York University, New York, NY 10003 Rishi Rajalingham McGovern Institute for Brain Research, MIT, Cambridge, MA 02139 Elias B. Issa Department of Neuroscience, Zuckerman Mind Brain Behavior Institute, Columbia University, New York, NY 10027 Kohitij Kar Pouya Bashivan Jonathan Prescott-Roy McGovern Institute for Brain Research, MIT, Cambridge, MA 02139 Kailyn Schmidt McGovern Institute for Brain Research, MIT, Cambridge, MA 02139 Aran Nayebi Neurosciences PhD Program, Stanford University, Stanford, CA 94305 Daniel Bear Department of Psychology, Stanford University, Stanford, CA 94305 Daniel L. K. Yamins James J. DiCarlo

Deep convolutional artificial neural networks (ANNs) are the leading class of candidate models of the mechanisms of visual processing in the primate ventral stream. While initially inspired by brain anatomy, over the past years, these ANNs have evolved from a simple eight-layer architecture in AlexNet to extremely deep and branching architectures, demonstrating increasingly better object categorization performance, yet bringing into question how brain-like they still are. In particular, typical deep models from the machine learning community are often hard to map onto the brain’s anatomy due to their vast number of layers and missing biologically-important connections, such as recurrence. Here we demonstrate that better anatomical alignment to the brain and high performance on machine learning as well as neuroscience measures do not have to be in contradiction. We developed CORnet-S, a shallow ANN with four anatomically mapped areas and recurrent connectivity, guided by Brain-Score, a new large-scale composite of neural and behavioral benchmarks for quantifying the functional fidelity of models of the primate ventral visual stream. Despite being significantly shallower than most models, CORnet-S is the top model on Brain-Score and outperforms similarly compact models on ImageNet. Moreover, our extensive analyses of CORnet-S circuitry variants reveal that recurrence is the main predictive factor of both Brain-Score and ImageNet top-1 performance. Finally, we report that the temporal evolution of the CORnet-S "IT" neural population resembles the actual monkey IT population dynamics. Taken together, these results establish CORnet-S, a compact, recurrent ANN, as the current best model of the primate ventral visual stream.

Figure 1: Synergizing machine learning and neuroscience through Brain-Score (top). By quantifying brain-likeness of models, we can compare models of the brain and use insights gained to inform the next generation of models. DenseNet, CORnet-S and ResNet architectures are the current winning models on Brain-Score. Green dots represent popular deep neural networks while gray dots correspond to various exemplary small-scale models that demonstrate the relationship between ImageNet performance and Brain-Score on a wider range of performances. CORnet-S area architecture (bottom left). The model consists of four areas which are pre-mapped to cortical areas V1, V2, V4, and IT in the ventral stream. V1COR is feed-forward and acts as a pre-processor to reduce the input complexity. V2COR, V4COR and ITCOR are recurrent (within area) to reduce the need for many layers and incorporate skip-connections, following the observation that ResNets and DenseNets are strong models on Brain-Score.

1 Introduction

Although notorious for their superior performance in object recognition tasks, artificial neural networks (ANNs) have also witnessed a tremendous success in the neuroscience community as currently the best class of models of the neural mechanisms of visual processing. Surprisingly, after training deep, feedforward ANNs to perform the standard ImageNet categorization task (Deng2009ImageNet), intermediate layers in ANNs can partly account for how neurons in intermediate layers of the primate visual system will respond to any given image, even ones that the model has never seen before (yamins2013hierarchical; yamins2014performance; khaligh2014deep; gucclu2015deep; cichy2016deep; yamins2016using). Moreover, these networks also partly predict human and non-human primate object recognition performance and object similarity judgments (rajalingham2018large; kubilius2016deep). Having strong models of the brain opened up unexpected possibilities of brain-response control by showing optimal stimuli to primates (bashivan2019neural), pointing the way to a potentially exciting future of noninvasive brain-machine interfaces.

How can we push these models to capture brain processing even more stringently? Continued architectural optimization on ImageNet alone no longer seems like a viable option. Indeed, more recent and deeper ANNs have not been shown to further improve on measures of brain-likeness (rajalingham2018large), even though their ImageNet performance has vastly increased (ILSVRC15). Models have evolved from an eight-layer AlexNet (krizhevsky2012imagenet) to extremely deep ResNet (he2016deep) and complex, branched Inception and NASNet architectures (szegedy2017inception; Liu2017PNASNet). While the initial limited number of layers in the breakthrough models (krizhevsky2012imagenet) was inspired by the primate ventral visual stream and thus could easily be assigned to the different areas of the ventral stream, the link between several hundred layers and a the handful of ventral stream areas is not obvious. More importantly, while high-performing models for object recognition remain feedforward, we already know this is not the case in primate ventral visual stream where recurrent processes play an important role in object recognition (tang2018recurrent; kar2019evidence).

We propose that aligning ANNs to neuroanatomy might lead to more compact, interpretable and, most importantly, functionally brain-like ANNs. To test this, we here demonstrate that a neuroanatomically more aligned ANN, CORnet-S, improves its functional match to measurements from the ventral stream, while maintaining high performance on ImageNet. CORnet-S commits to a shallow recurrent anatomical structure of the ventral visual stream, and thus achieves a much more compact architecture, yet retains a strong ImageNet top-1 performance of 73.1% and is the top model on measures of brain-likeness, such that it is the new state-of-the-art model in predicting neural firing rates and image-by-image human behavior on Brain-Score, a novel large-scale benchmark composed of neural recordings and behavioral measurements. We identify that these results are primarily driven by recurrent connections, in line with our understanding of how the primate visual system processes visual information (tang2018recurrent; kar2019evidence). In fact, comparing the high level ("IT") neural representations between recurrent steps in the model and time-varying primate IT recordings, we find that CORnet-S partly captures these neural response trajectories - the first model to achieve this neural benchmark.

2 CORnet-S: Brain-driven model architecture

Using benchmarks of brain-likeness in Brain-Score and Feedforward Simplicity as our guiding measures, we built CORnet-S. Specifically, our model aims to be (based on kubilius2018predict):

(1) Predictive, so that it is a mechanistic model of the brain. We are not only interested in having correct model outputs (behaviors) but also internals that match the brain’s anatomical and functional constraints. We prefer ANNs because neurons are the units of online information transmission and models without neurons cannot be obviously mapped to neural spiking data (yamins2016using).

(2) Compact, i.e. we prefer simpler models among models with similar scores as they are potentially easier to understand and more efficient to experiment with. The human and non-human primate ventral visual pathway consists of only a handful of areas that process visual inputs: retina, LGN, V1, V2, V4, and a set of areas in the inferior temporal cortex (IT). While the exact number of areas is hard to establish, we ask that models have few areas (though each area may perform multiple operations). The model should thus obtain strong scores on Feedforward Simplicity. Also, we have no strong reason to believe that circuitry should differ across areas in the ventral visual pathway.

(3) Recurrent: while core object recognition was originally believed to be largely feedforward because of its fast time scale (dicarlo2012does), it has long been suspected that recurrent connections must be relevant for some aspects of object perception (lamme2000distinct; bar2006top; wagemans2012century), and recent studies have shown their role even at short time scales (kar2019evidence; tang2018recurrent; schrimpf2017thesis; rajaei2018beyond). Moreover, responses in the visual system have a temporal profile, so models at least should be able to produce responses over time too.

2.1 CORnet-S model specifics

CORnet-S (Fig. 1) aims to rival the best models on Brain-Score by transforming very deep feedforward architectures into a shallow recurrent model. Specifically, CORnet-S draws inspiration from ResNets that are some of the best models on our behavioral benchmark (Fig. 1; rajalingham2018large) and can be thought of as unrolled recurrent networks (liao2016bridging). Recent studies further demonstrated that weight sharing in ResNets was indeed possible without a significant loss in CIFAR and ImageNet performance (jastrzebski2017residual; leroux2018iamnn).

Moreover, CORnet-S specifically commits to an anatomical mapping to brain areas. While for comparison models we establish this mapping by searching for the layer in the model that best explains responses in a given brain area, ideally such mapping would already be provided by the model, leaving no free parameters. Thus, CORnet-S has four computational areas, conceptualized as analogous to the ventral visual areas V1, V2, V4, and IT, and a linear category decoder that maps from the population of neurons in the model’s last visual area to its behavioral choices.

Each visual area implements a particular neural circuitry with neurons performing simple canonical computations: convolution, addition, nonlinearity, response normalization or pooling over a receptive field. The circuitry is identical in each of its visual areas (except for V1), but we vary the total number of neurons in each area. The details of the layers depicted in Figure 1 are as follows. Due to high computational demands, first area V1COR performs a convolution with stride 2, max pooling with stride 2, and a convolution. Areas V2COR, V4COR and ITCOR perform two convolutions, a convolution with stride 2 and a convolution. To implement recurrence, outputs of an area are passed through several times to yield the final output of that area. For instance, after V2COR processed the input once, that result is passed into V2COR again and treated as a new input. Input changes over time are thus not implemented ("gate" in Fig. 1). V2COR and ITCOR are repeated twice, V4COR is repeated four times. Batch normalization (ioffe2015batch) was not shared over time as suggested by jastrzebski2017residual. There are no across-area bypass or across-area feedback connections in the current definition of CORnet-S and retinal and LGN processing are omitted.

The decoder part of a model implements a simple linear classifier – a set of weighted linear sums with one sum for each object category. To reduce the amount of neural responses projecting to this classifier, we first average responses over the entire receptive field per feature map.

The model is available at

2.2 Comparison to other models

Liang_2015_CVPR introduced perhaps the first deep recurrent neural network intended for object recognition by adding a variant of a simple recurrent cell to a shallow five-layer convolutional neural network backbone. zamir2017feedback built a more powerful version by employing LSTM cells, and a similar approach was used by spoerer2017recurrent who showed that a simple version of a recurrent net can improve network performance on an MNIST-based task. liao2016bridging argued that ResNets can be thought of as recurrent neural networks unrolled over time with non-shared weights, and demonstrated the first working version of a folded ResNet, also explored by jastrzebski2017residual.

However, all of these networks were only tested on CIFAR-100 at best. As noted by nayebi2018task, while many networks may do well on a simpler task, they may differentiate once the task becomes sufficiently difficult. Moreover, our preliminary testing indicated that non-ImageNet-trained models do not appear to score high on Brain-Score, so even for practical purposes we needed models that could be trained on ImageNet. leroux2018iamnn proposed probably the first recurrent architecture that performed well on ImageNet. In an attempt to explore the recurrent net space in a more principled way, nayebi2018task performed a large-scale search in the LSTM-based recurrent cell space by allowing the search to find the optimal combination of local and long-range recurrent connections. The best model demonstrated a strong ImageNet performance while being shallower than feedforward controls. In this project, we wanted to go one step further and build a maximally compact model that would nonetheless yield top Brain-Score and outperform other recurrent networks on ImageNet.

3 Brain-Score: Comparing models to brain

To obtain quantified scores for brain-likeness, we built Brain-Score, a composite benchmark that measures how well models can predict (a) the mean neural response of each neural recording site to each and every tested naturalistic image in non-human primate visual areas V4 and IT (data from majaj2015simple); (b) mean pooled human choices when reporting a target object to each tested naturalistic image (data from rajalingham2018large), and (c) when object category is resolved in non-human primate area IT (data from kar2019evidence).

3.1 Neural predictivity

A total of 2760 images containing a single object pasted randomly on a natural background were presented centrally to passively fixated monkeys for 100 ms and neural responses were obtained from 88 V4 sites and 168 IT sites. For our analyses, we used normalized time-averaged neural responses in the 70-170 ms window. A regression model was constructed for each neuron using 90% of image responses and tested on the remaining 10% in a 10-fold cross-validation strategy. The median over neurons of the Pearson’s  between the predicted and actual response constituted the final neural fit score for each visual area. In our model CORnet-S, we used designated model areas and the best time point to predict corresponding neural data. In other models, we used the most predictive layer.

3.2 Behavioral predictivity

A total of 2400 images containing a single object pasted randomly on a natural background were presented to 1472 humans for 100 ms and they were asked to choose from two options which object they saw. 240 of those images with around 60 responses per object-distractor pair were used further (3̃000 thousand unique responses). We used the outputs of the layer just prior to 1000-value category vectors to construct a linear (logistic regression) decoder from model features. We used the regression’s probabilities for each class to compare model choices against actual human responses. This is a correlational measure, meaning that models that do better on behavioral predictivity are better aligned with human responses, making similar correct choices and committing similar errors.

3.3 Object solution times

A total of 1320 images, containing images from Section 3.1 and MS COCO (lin2014microsoft) were presented centrally to behaving monkeys for 100 ms and neural responses were obtained from 424 IT sites. We fit a logistic regression on 80% of data at each 10 ms and tested on 10% at which time point the normalized scores per image (rajalingham2018large) surpassed a threshold values defined by the monkey’s behavioral output for that image, which we refer to as "object solution times", or OSTs (see kar2019evidence, for details), linearly interpolating the exact millisecond between the surpassing and previous timestep. The remaining 10% of the data was used for validation to establish how many recurrent steps in CORnet-S are needed. We report a Spearman correlation between the model OSTs and the actual monkey OSTs.

3.4 Feedforward Simplicity

Given equally predictive models, we prefer a simpler one. We considered several alternative metrics to measure model simplicity, ultimately choosing to compute the number of convolutions and fully-connected layers along the longest path of information flow. For instance, the circuits in each of CORnet-S areas have the length of four since information is passed sequentially through four convolutional layers. Note that we counted recurrent (shared parameter) paths only once. If recurrent paths were counted the number of times information was passed through them, models with shared parameters would be no simpler than those using unique parameters at every time (i.e., feedforward models), which is counterintuitive to us. We also wanted to emphasize that the difference between a path length of 5 and 10 is much greater than between 105 and 110, so the path length was transformed by a natural logarithm. Finally, we wanted a measure of simplicity, not complexity, so the resulting value was inverted, resulting in the following formulation of Feedforward Simplicity:

Given the lack of consensus how many areas the primate ventral visual pathway contains, Feedforward Simplicity is currently not included in the composite Brain-Score and is reported separately here.

4 Results

4.1 Brain-Score is correlated with classification performance

We performed a large-scale model comparison using most commonly used neural network families: AlexNet (krizhevsky2012imagenet), VGG (simonyan2014very), ResNet (he2016deep), Inception (Szegedy2015c; Szegedy2015b; szegedy2017inception), SqueezeNet (Iandola2016aSqueezeNet), DenseNet (huang2017densely), MobileNet (Howard2017MobileNet), and (P)NASNet (Zoph2017NASNetMobile; Liu2017PNASNet). These networks were taken from publicly available checkpoints: AlexNet, SqueezeNet, ResNet-{18,34} from PyTorch (paszke2017automatic); Inception, ResNet-{50,101,152}, (P)NASNet, MobileNet from TensorFlow-Slim (tf-slim); and Xception, DenseNet, VGG from Keras (chollet2015keras). As such, the training procedure is different between models and our results should be related to those model instantiations and not to architecture families. To further map out the space of possible architectures, we included a family of models termed basenets: lightweight AlexNet-like architectures with six convolutional layers and a single fully-connected layer, captured at various stages of training. Various hyperparameters were varied between basenets, such as kernel sizes, nonlinearities, learning rate etc. To rank models in Figure 1 on an overall score, we take the mean of the behavioral score, the V4 neural score, the IT neural score, and the neural dynamics score introduced in Section 4.6.

Figure 1 shows how models perform on Brain-Score and ImageNet. CORnet-S outperforms other alternatives by a large margin with the Brain-Score of .456. Top ImageNet models also perform well, with leading models stemming from the DenseNet and ResNet families, whereas e.g. the Inception family of models has seemingly decreased in its Brain-Score over subsequent versions.

Figure 2: Brain-Score generalization across datasets: (a) to neural recordings in new subjects with the same stimulus set, (b) to neural recordings in new subjects with a very different stimulus set (MS COCO), (c) to behavioral responses in new subjects with new object categories, (d) to CIFAR-100.

Interestingly, models that score high on brain data are also not the ones ranking the highest on ImageNet performance, suggesting a potential disconnect between ImageNet performance and fidelity to brain mechanisms. For instance, despite its superior performance of 82.90% top-1 accuracy on ImageNet, PNASNet only ranks \nth13 on the overall Brain-Score. Models with an ImageNet top-1 performance below 70% show a strong correlation with Brain-Score of .92 () but above 70% ImageNet performance there was no significant correlation (, cf. Figure 1).

4.2 Brain-Score is a robust measure of generalization

We further asked if Brain-Score reflects idiosyncracies of the particular datasets we included in this benchmark or instead, more desirably, provides an overall evaluation of how brain-like models are. To address this question, we performed four different tests with various generalization demands (Fig. 2). First, we compared the scores of models predicting IT neural responses to a set of new IT neural recordings (kar2019evidence) where new monkeys were shown the same images as before. We observed a strong correlation between the two sets (Pearson ). When compared on predicting IT responses to a very different image set (1600 MS COCO images; lin2014microsoft), model rankings were still strong (Pearson ). We also found a strong correlation between model scores on our original behavioral set and a newly obtained set of behavioral responses to 20 new categories (200 images total; Pearson ). Finally, we evaluated model feature generalization to CIFAR-100 without fine-tuning (following Kornblith2018a). Again, we observed a compelling correlation to Brain-Score values (Pearson ). Overall, we expect that adding more benchmarks to Brain-Score will further lead scores to converge.

4.3 CORnet-S is the best yet much simpler brain-predicting models

CORnet-S is strong at neural as well as behavioral predictions (Fig. 3),

Figure 3: Comparison of brain scores on several popular models and CORnet-S. CORnet-S is comparable to the state-of-the-art models on neural predictivity and a top model on behavioral predictivity.

making it one of the best models tested on Brain-Score so far. Critically, CORnet-S is substantially simpler than other top-performing models on Brain-Score (Fig. 4, left) and commits to a particular mapping between model and brain areas.

Figure 4: Feedforward Simplicity versus Brain-Score (left) and ImageNet performance (right). Most simple models perform poorly on Brain-Score and ImageNet, while best models for explaining brain data are complicated. CORnet-S offers the best of both worlds with the best Brain-Score and ImageNet performance and the highest degree of simplicity we could achieve to date. Note that dots were slightly jittered along the x-axis to improve visibility.

4.4 CORnet-S is best on ImageNet among compact models

Due to anatomical constraints imposed by the brain, CORnet-S’s architecture is much more compact than the majority of deep models in computer vision (Fig. 4, right). Compared to similar models with a path length of less than 50, CORnet-S is better in terms of Feedforward Simplicity and outperforms other models on ImageNet top-1 classification accuracy. AlexNet and IamNN are simpler models (simplicity .48 (path length 8) and .38 (14) respectively) but suffer on classification accuracy (57.7 and 69.6 top-1 respectively) – CORnet-S provides a trade-off between the two with a Feedforward Simplicity of .37 (path length 15) and top-1 accuracy of 73.1. Several epochs later in training top-1 accuracy actually climbed to 74.4 but since we are optimizing for the brain, we chose the epoch with maximum Brain-Score. For reference, the state-of-the-art large model (mahajan2018exploring) achieves top-1 accuracy of 85.4, but at the cost of a Feedforward Simplicity of only .22 (path length 101).

4.5 CORnet-S mediates between compactness and high performance through recurrence

Figure 5: CORnet-S circuitry analysis. Each row indicates how ImageNet top-1 and Brain-Score change with respect to the baseline model (in bold) when a particular hyperparameter is changed. The OSTs of IT are not included in the Brain-Score here.

To determine which elements in the circuitry are critical to CORnet-S, we attempted to alter its block structure and record changes in Brain-Score Fig. 5. We only used V4, IT and behavioral predictivity in this analysis in order to understand the non-temporal value of CORnet-S structure. We found that the most important factor was the presence of at least a few steps of recurrence in each block. Having a fairly wide bottleneck (at least 4x expansion) and a skip connection were other important factors. On the other hand, adding more recurrence or having five areas in the model instead of four did not improve the model or hurt its Brain-Score. Other factors affected mostly ImageNet performance, including using two convolutions instead of three within a block, having more areas in the model and using batch normalization per time step instead of a global group normalization (wu2018group). The type of gating did not seem to matter. However, note that we kept training hyperparameters identical for all these model variants. We therefore cannot rule out that the reported differences could be minimized if more optimal hyperparameters were found.

4.6 CORnet-S captures neural dynamics in primate IT

Figure 6: CORnet-S captures neural dynamics. A logistic decoder is fit to predict object category at each 10 ms window of IT responses in model and monkey. We then tested object solution times (OST) per image, i.e. when scores of model and monkey surpass the threshold of monkey behavioral response. is computed on the raw data, whereas the plot visualizes binned OSTs. Error bars denote s.e.m. across images.

Feed-forward networks cannot make any dynamic predictions over time, and thus cannot capture a critical property of the primate visual system (kar2019evidence; tang2018recurrent). By introducing recurrence, CORnet-S is capable of producing temporally-varying response trajectories. Recent experimental results (kar2019evidence) reveal that the linearly decodable solutions to object recognition are not all produced in the IT neural population at the same time – images that are particularly challenging for deep ANNs take longer to evolve in IT. This timing provides a strong test for the model: does it predict image-by-image temporal trajectories in IT neural responses over time? We thus estimated for each image when explicit object category information becomes available in CORnet-S – termed "object solution time" (OST) – and compared it with the same measurements obtained from monkey IT cortex (kar2019evidence). Importantly, the model was not trained to predict monkey OSTs. Rather, a logistic classifier was trained to decode object category from neural responses and from model’s responses at each 10 ms window. OST is defined as the time when this decoding accuracy reaches monkey’s accuracy when it is performing object recognition task on a given image. We evaluated how well CORnet-S could capture these fine-grain temporal dynamics (binning images by their model OST) and report a correlation score of .19 (; Figure 6). Feed-forward models cannot capture neural dynamics and thus scored 0.

5 Discussion

We developed a relatively shallow recurrent model CORnet-S that follows neuroanatomy more closely than standard machine learning ANNs, and is among the top models on Brain-Score yet remains competitive on ImageNet, combining the best of both neuroscience desiderata and machine learning engineering requirements, and demonstrating that models that satisfy both communities can be developed. While we believe that CORnet-S is a closer approximation to the anatomy of the ventral visual stream than current state-of-the-art deep ANNs because we specifically limit the number of areas and include recurrence, it is still far from complete in many ways. From a neuroscientist’s point of view, on top of the lack of biologically-plausible learning mechanisms (self-supervised or unsupervised), a better model of ventral visual pathway would include more anatomical and circuitry-level details, such as retina or lateral geniculate nucleus. Similarly, adding a skip connection was not informed by cortical circuit properties but rather proposed by he2016deep as a means to alleviate the degradation problem in very deep architectures. But we note that not just any architectural choices work. We have tested hundreds of architectures before finding CORnet-S type of circuitries (Figure 5).

A critical component in establishing that models such as CORnet-S are strong candidate models for the brain is Brain-Score, a framework for quantitatively comparing any artificial neural network to the brain’s neural network for visual processing. With even the relatively small number of brain benchmarks that we have included so far, the framework already reveals interesting patterns. First, it extends prior work showing that performance correlates with brain similarity. However, adding recurrence allows us to break from this trend and achieve much better alignment to the brain. Even when the OST measure is not included in Brain-Score, CORnet-S remains one of the top models, indicating its general utility. On the other hand, we also find a potential disconnect between ImageNet performance and Brain-Score, with PNASNet, a state-of-the-art model on ImageNet not performing well on brain measures, whereas even small networks with poor ImageNet performance achieve reasonable scores. We further observed that models that score high on Brain-Score also tend to score high on other datasets, supporting the idea that Brain-Score reflects how good a model is overall, not just on the three particular neural and behavioral benchmarks that we used.

However, it is possible that the observed lack of correlation is only specific to the way models were trained, as reported recently by Kornblith2018a. For instance, they found that the presence of auxiliary classifiers or label smoothing does not affect ImageNet performance too much but significantly decreases transfer performance, in particular affecting Inception and NASNet family of models, i.e., the ones that performed worse on Brain-Score than their ImageNet performance would imply. Kornblith2018a reported that retraining these models with optimal settings markedly improved transfer accuracy. Since Brain-Score is also a transfer learning task, we cannot rule out that Brain-Score might change if we retrained the affected models classes. Thus, we reserve our claims only about the specific pre-trained models rather than the whole architecture classes.

More broadly, we suggest that models of brain processing are a promising opportunity for collaboration between neuroscience and machine learning. These models ought to be compared through quantified scores on how brain-like they are, which we here evaluate with a composite of many neural and behavioral benchmarks in Brain-Score. With CORnet-S, we showed that neuroanatomical alignment to the brain in terms of compactness and recurrence can better capture brain processing by predicting neural firing rates, image-by-image behavior, and even neural dynamics, while simultaneously maintaining high ImageNet performance and outperforming similarly compact models.


We thank Simon Kornblith for helping to conduct transfer tests to CIFAR, and Maryann Rui and Harry Bleyan for the initial prototyping of the CORnet family.

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 705498 (J.K.), US National Eye Institute (R01-EY014970, J.J.D.), Office of Naval Research (MURI-114407, J.J.D), the Simons Foundation (SCGB [325500, 542965], J.J.D; 543061, D.L.K.Y), the James S. McDonnell foundation (220020469, D.L.K.Y.) and the US National Science Foundation (iis-ri1703161, D.L.K.Y.). This work was also supported in part by the Semiconductor Research Corporation (SRC) and DARPA. The computational resources and services used in this work were provided in part by the VSC (Flemish Supercomputer Center), funded by the Research Foundation - Flanders (FWO) and the Flemish Government – department EWI.


Appendix A Numerical Brain-Scores

neural predictivity OST behavioral top-1 accuracy
Brain-Score model V4 IT IT predictivity ImageNet
.456 CORnet-S .653 .600 .189 .380 74.70
.412 densenet-169 .663 .606 0 .380 75.90
.407 resnet-101_v2 .661 .585 0 .383 77.00
.407 densenet-121 .657 .597 0 .372 74.50
.406 densenet-201 .655 .601 0 .369 77.00
.404 resnet-50_v2 .663 .586 0 .366 75.60
.403 resnet-152_v2 .663 .586 0 .364 77.80
.402 inception_v2 .658 .595 0 .354 73.90
.400 best mobilenet .645 .600 0 .356 71.80
.400 pnasnet_large .650 .587 0 .364 82.90
.399 inception_v1 .661 .576 0 .358 69.80
.399 resnet-18 .648 .584 0 .364 69.76
.399 inception_resnet_v2 .652 .592 0 .351 80.40
.397 xception .671 .565 0 .352 79.00
.397 inception_v4 .639 .576 0 .371 80.20
.396 vgg-19 .672 .566 0 .345 71.10
.396 nasnet_large .659 .589 0 .334 82.70
.395 inception_v3 .660 .589 0 .332 78.00
.393 resnet-34 .632 .560 0 .378 73.30
.392 vgg-16 .669 .572 0 .326 71.50
.382 nasnet_mobile .651 .594 0 .281 74.00
.378 best basenet .663 .594 0 .256 47.64
.368 alexnet .631 .589 0 .253 57.70
.353 squeezenet1_1 .654 .556 0 .201 57.50
.345 squeezenet1_0 .653 .546 0 .180 57.50
Table 1: Brain-Scores and individual performances for state-of-the-art models

Appendix B Brain-Score Benchmark details

In the following section we outline the benchmarks that models are measured against. A benchmark consists of a metric applied to a specific set of experimental data, which here can be either neural recordings or behavioral measurements.

b.1 Neural

The purpose of neural metrics is to establish how well internal representations of a source system (e.g., a neural network model) match the internal representations in a target system (e.g., a primate). Unlike typical machine learning benchmarks, these metrics provide a principled way to prefer some models over others even if their outputs are identical. We outline here one common metric, Neural Predictivity, which is a form of a linear regression.

Neural Predictivity: Image-Level Neural Consistency

Neural Predictivity is used to evaluate how well responses to given images in a source system (e.g., a deep ANN) predict the responses in a target system (e.g., a single neuron’s response in visual area IT). As inputs, this metric requires two assemblies of the form where neuroids can either be neural recordings or model activations. First, source neuroids are mapped to each target neuroid using a linear transformation:

where denotes linear regression weights and is the noise in the neural recordings. This mapping procedure is performed on multiple train-test splits across stimuli. In each run, the weights are fit to map from source neuroids to a target neuroid using training images, and then using these weights predicted responses are obtained for the held-out images. We used the neuroids from V4 and IT separately to compute these fits.

To obtain a neural predictivity score for each neuroid, we compare predicted responses with the measured neuroid responses by computing the Pearson correlation coefficient :


A median over all individual neuroid neural predictivity values (e.g., all measured target sites in a target brain region) is computed to obtain a predictivity score for that train-test split (median is used since responses are typically distributed non-normally). The final neural predictivity score for the target brain region is computed as the mean across all train-test splits.

We further estimate the internal consistency between neural responses by splitting neural responses in half across repeated presentations of the same image and computing Spearman-Brown-corrected Pearson correlation coefficient (Eq. 1) between the two splits across images for each neuroid.

In practice, we found that standard linear regression is comparably slow given a large dimensionality of the source system and not sufficiently robust. Thus, following yamins2014performance, we use a partial least squares (PLS) regression with 25 components. We further optimized this procedure by first projecting source features into a lower-dimensional space using principal components analysis. The projection matrix is obtained for the features of a selection of ImageNet images, so that the projection is constant across train-test splits. This projection matrix is then used to transform source features. Results reported here were obtained by retaining 1000 principal components from the feature responses per layer to 1000 ImageNet validation images that captured the most variance of a source model.

Neural Recordings

The neural dataset currently used in both neural benchmarks included in this version of Brain-Score is comprised of neural responses to 2,560 naturalistic stimuli in 88 V4 neurons and 168 IT neurons, collected by majaj2015simple. The image set consists of 2,560 grayscale images in eight object categories (animals, boats, cars, chairs, faces, fruits, planes, tables). Each category contains eight unique objects (for instance, the “face” category has eight unique faces). The image set was generated by pasting a 3D object model on a naturalist background. In each image, the position, pose, and size of an object was randomly selected in order to create a challenging object recognition task both for primates and machines. A circular mask was applied to each image (see majaj2015simple for details on image generation).

Two macaque monkeys were implanted three arrays each, with one array placed in area V4 and the other two placed on the posterior-anterior axis of IT cortex. The monkeys passively observed a series of images (100 ms image duration with 100 ms of gap between each image) that each subtended approximately 8 deg visual angle. To obtain a stable estimate of the neural responses to each image, each each image was re-tested about 50 times (re-tests were randomly interleaved with other images). In the benchmarks used here, we used an average neural firing rate (normalized to a blank gray image response) in the window between 70 ms and 170 ms after image onset where the majority of object category-relevant information is contained (majaj2015simple).

b.2 Behavioral

The purpose of behavioral benchmarks it to compute the similarity between source (e.g., an ANN model) and target (e.g., human or monkey) behavioral responses in any given task. For core object recognition tasks, primates (both human and monkey) exhibit behavioral patterns that differ from ground truth labels. Thus, our primary benchmark here is a behavioral response pattern metric, not an overall accuracy metric, and higher scores are obtained by ANNs that produce and predict the primate patterns of successes and failures. One consequence of this is that ANNs that achieve 100% accuracy will not achieve a perfect behavioral similarity score.

Even within the visual behavioral domain of core object recognition, there are many possible behavioral metrics. We here use the metric of the image-by-image patterns of difficulty, broken down by the object choice alternatives (termed I2n), because recent work (rajalingham2018large) suggests that it has the most power to distinguish among alternative ANNs (assuming that sufficient amounts of behavioral data are available).

I2n: Normalized Image-Level Behavioral Consistency

Source data (model features) for a total of images are transformed first into a matrix of object categories and images with behavioral data available using the following procedure. First, images where behavioral responses are not available (namely, images) are used to build a -way logistic regression from source data to a -value probability vector for each image, where each probability is the probability that a given object is in the image. This regression is then used to estimate probabilities for the held-out images. For each image, all normalized target-distractor pair probabilities are computed from the -way probability vector. For instance, if an image contains a dog and the distractor is a bear, the target-distractor score is .

In order to compare source and target data, we first transform these raw accuracies in the response matrix to a measure for each cell in the matrix:

where is the estimated z-score of responses, Hit Rate is the accuracy of a given target-distractor pair while the False Alarms Rate corresponds to how often the observers incorrectly reported seeing that target object in images where another object was presented. For instance, if a given image contains a dog and distractor is a bear, the Hit Rate for the dog-bear pair for that image comes straight from the matrix, while in order to obtain the False Alarms Rate, all cells from that matrix that did not have dogs in the image but had a dog as a distractor are averaged, and 1 minus that value is used as a False Alarm Rate. All above 5 were clipped. This transformation helps to remove bias in responses and also to diminish ceiling effects (since many primate accuracies were close to 1), but empirically observed benefits of in this dataset are small; see rajalingham2018large for a thorough explanation.

The resulting response matrix is further refined by subtracting the mean across trials of the same target-distractor pair (e.g., for dog-bear trials, their mean is subtracted from each trial). Such normalization exposes variance unique to each image and removes global trends that may be easier for models to capture. For instance, dog-bear trials on average could have been harder than dog-zebra trials. Without this normalization, a model might score very well by only capturing this tendency. After normalization, all responses are centered around zero, and thus capturing only global trends but not each image’s idiosyncrasies would be insufficient for a model to rank well.

After normalization, a Pearson correlation coefficient between source and target data is computed using Eq. 1. We further estimate noise ceiling, that is, how well an ideal model could perform given the noise in the measured behavioral responses, by dividing target data in half across trials, computing the normalized matrices for each half, and computing the Pearson correlation coefficient between the two halves. If source data is produced by a stochastic process, the same procedure can be carried out on the source data, resulting in the source’s reliability .

The final behavioral predictivity score of each ANN is then computed by:

All models that we tested so far produced deterministic responses, thus in our scoring.

Primate behavioral data

The behavioral data used in the current round of benchmarks was obtained by rajalingham2015comparison and rajalingham2018large. Here we focus on only the human behavioral data, but the human and non-human primate behavioral patterns are very similar to each other (rajalingham2015comparison; rajalingham2018large).

The image set used in this data collection was generated in a similar way as the images for V4 and IT using 24 object categories. In total, the dataset contains 2,400 images (100 per object). For this benchmark, we used 240 (10 per object) of these images for which the most trials were obtained. 1,472 human observers responded to briefly presented images on Amazon Mechanical Turk. At each trial, an image was presented for 100 ms, followed by two response choices, one corresponding to the target object present in the image and the other being one of the remaining 23 objects (i.e., a distractor object). Participants responded by choosing which object was presented in the image. Thus, over three hundred thousand responses for each target-distractor pair were obtained from multiple participants, resulting in a response matrix when averaged across participants.

b.3 Predictors of neural scores

We compared model scores on neural (V4, IT) recordings with the scores on behavioral recordings to see if e.g. a behavioral benchmark alone would already be sufficient or if the entire set of benchmarks is necessary. We found that there was a correlation to behavior ( for V4 and or IT) which is strong enough to connect neurons to behavior but not sufficient for behavior alone to explain the entire neural population, warranting a composite set of benchmarks.

Moreover, we tested if the number of features in model layers might predict the neural scores. Even though we PCA all features to 1,000 components, higher dimensionality might result in better scores. Following Figure 7, we found this not be the case:

Figure 7: Neural Scores do not depend on number of features. We plot the number of features in models’ highest-scoring layers against their neural (V4 and IT) scores. The number of neurons does not appear to be a predictor of better brain-likeness.

models with the same number of neurons have scores across the board, for all number of features.

b.4 Feedforward Simplicity

Network Simplicity (path length) ImageNet top-1 / 5
AlexNet (krizhevsky2012imagenet) .48 (8) 57.7 / 79.1
VGG-16 (simonyan2014very) .36 (16) 71.5 / 90.4
VGG-19 (simonyan2014very) .34 (19) 72.4 / 90.9
ResNet-18 (he2016deep) .35 (18) 69.8 / 89.1
SqueezeNet (Iandola2016aSqueezeNet) .35 (18) 57.5 / 80.3
IamNN (leroux2018iamnn) .38 (14) 69.6 / 89.0
MobileNet-224 (Howard2017MobileNet) .27 (41) 70.6 / 89.5
CORnet-S .37 (15) 73.1 / 91.1
Table 2: Comparison of compact models on Feedforward Simplicity and classification accuracy. CORnet-S outperforms comparable models on simplicity and ImageNet top-1 performance.

From a neuroscience point of view, simpler models can be better mapped to cortex and be better analyzed and understood with regard to the brain. Simpler models can also be better made sense of sense in terms of what components constitute a strong model by reducing models to their most essential elements. One possibility was to use the total number of parameters (weights). However, it did not seem to map well to simplicity in neuroscience terms. For instance, a single convolutional layer with many filter maps could have many parameters yet it seems much simpler than a multilayer branching structure, like the Inception block (szegedy2017inception), that may have less parameters overall.

Moreover, our models are always tested on independent data sampled from different distributions than the train data. Thus, after training a model, all these parameters were fixed for the purposes of brain benchmarks, and the only free parameters are the ones introduced by the linear decoder that is trained on top of the frozen model’s parameters (see above for decoder details).

We also considered computing the total number of convolutional and fully-connected layers, but some models, like Inception, perform some convolutions in parallel, while others, like ResNeXt (xie2017aggregated), group multiple convolutions into a single computation. We thus decided to you the "longest path" definition as described in the main text.

b.5 Brain-Score

To evaluate how well a model is doing overall, we computed the global Brain-Score as a composite of neural V4 predictivity score, neural IT predictivity score, object solution times in IT, and behavioral I2n predictivity score (each of these scores was computed as described above and main text). The Brain-Score presented here is the mean of the four scores. This approach of taking the does not normalize by different scales of the scores so it may be penalizing scores with low variance. However, the alternative approach of ranking models on each benchmark separately and then taking the mean rank would impose the strong assumption that for any two models with (even insignificantly) different scores, their ranks are also different. We thus chose to take the mean score to preserve the distance in values.

Appendix C CORnet-S Details

c.1 Implementation Details

We used PyTorch 0.4.1 and trained the model using ImageNet 2012 (ILSVRC15). Images were preprocessed (1) for training, with random crops to 224 x 224 pixels, randomly flipped left and right and normalized by mean subtraction and division by standard deviation of the dataset; (2) for validation, with central crops to 224 x 224 pixels and normalized by mean subtraction and division by standard deviation of the dataset. We used a batch size of 256 images and trained on 2 GPUs (NVIDIA Titan X / GeForce 1080Ti) for 43 epochs. We use similar learning rate scheduling to ResNet with more variable learning rate updates (primarily in order to train faster): 0.1, divided by 10 every 20 epochs. For optimization, we use Stochastic Gradient Descent with momentum .9, a cross-entropy loss between image labels and model predictions (logits). We will open-source our code and weights through GitHub.

c.2 Model Search

Figure 8: ImageNet top-1 performance vs. behavioral benchmark on various CORnets. We manually tried many different configurations of CORnet circuitry. The figure is showing how behavioral benchmark of Brain-Score is related to ImageNet top-1 performance in 106 CORnet configurations. Each dot corresponds to a particular CORnet at a particular point during training. The correlation between ImageNet top-1 performance and CORnet is robust but there is also high variance in this relationship. In particular, notice how some models achieve close to 75% ImageNet performance but show only a mediocre behavioral score. Thus, optimizing solely for ImageNet is not guaranteed at all to lead to a good alignment to brain data.

Appendix D Generalization to Other Datasets

d.1 Neural: New Neurons, Old Images

We evaluated models on an independently collected neural dataset (288 neurons, 2 monkeys, 63 trials per image; kar2019evidence) where new monkeys were presented with a subset of 640 images from the 2760 images we used for neural predictivity.

d.2 Behavioral: New Images

We collected a new behavioral dataset, consisting of 200 images (20 objects 10 images) from Amazon Mechanical Turk users (185,106 trials in total). We used the same experimental paradigm as in our original behavioral test but none of the objects were from the same category as before.

d.3 Neural: New Neurons, COCO Images

We obtained a neural dataset from (kar2019evidence) for a selection of 1600 of MS COCO images (lin2014microsoft). These images are very dissimilar from the synthetic images we used in other tests, providing a strong means to test Brain-Score generalization. The dataset consisted of 288 neurons from 2 monkeys and 45 trials per image. Unlike our previous datasets, this one had a low internal consistency between neural responses, presumably due to the electrodes being near their end of life and producing unreasonably high amounts of noise. We therefore only used the 86 neurons with internal consistency of at least 0.9.

d.4 Cifar-100

Following the procedure described in Kornblith2018a, we tested how well these models generalize to CIFAR-100 dataset by only allowing a linear classifier to be retrained for the 100-way classification task (that is, without doing any fine-tuning). As in Kornblith2018a, we used a scikit-learn implementation of a multinomial logistic regression using L-BFGS (scikit-learn), with the best C parameter found by searching a range from .0005 to .05 in 10 logarithmic steps (40,000 images from CIFAR-100 train set were used for training and the remaining 10,000 for testing; the search range was reduced from Kornblith2018a because in our earlier tests we found that all models had their optimal parameters in this range). Accuracies reported on the 10,000 test images.

d.5 Neural: Early and Late Predictions

Focusing on the temporal aspect of our neural data, we divided spike rates into an early time bin ranging from 90-110 ms and a late time bin from 190-210 ms. We found that this early-late division highlighted functional model difference more prominently than the mean temporal prediction in nayebi2018task. For instance, Figure 9 shows how IT is predicted well by strong ImageNet models at a late stage, but not at early stages. CORnet-S does well on both of these predictions.

Figure 9: Prediction correlations on early and late spike rates. We compare ImageNet performance against pearson correlation of predicted spike rates with neural data binned into early (90-110 ms) and late (190-210 ms). Model mappings are performed separately per bin, layers are chosen based on 70-170 ms scores. Notice how better ImageNet models are better at predicting late IT responses, but not early ones.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description