# Learning Generative Models with Visual Attention

###### Abstract

Attention has long been proposed by psychologists to be important for efficiently dealing with the massive amounts of sensory stimulus in the neocortex. Inspired by the attention models in visual neuroscience and the need for object-centered data for generative models, we propose a deep-learning based generative framework using attention. The attentional mechanism propagates signals from the region of interest in a scene to an aligned canonical representation for generative modeling. By ignoring scene background clutter, the generative model can concentrate its resources on the object of interest. A convolutional neural net is employed to provide good initializations during posterior inference which uses Hamiltonian Monte Carlo. Upon learning images of faces, our model can robustly attend to the face region of novel test subjects. More importantly, our model can learn generative models of new faces from a novel dataset of large images where the face locations are not known.^{1}^{1}1In the proceedings of Neural Information Processing Systems, 2014, Montréal, Québec, Canada.

Learning Generative Models with Visual Attention

## 1 Introduction

Building rich generative models that are capable of extracting useful, high-level latent representations from high-dimensional sensory input lies at the core of solving many AI-related tasks, including object recognition, speech perception and language understanding. These models capture underlying structure in data by defining flexible probability distributions over high-dimensional data as part of a complex, partially observed system. Some of the successful generative models that are able to discover meaningful high-level latent representations include the Boltzmann Machine family of models: Restricted Boltzmann Machines, Deep Belief Nets [1], and Deep Boltzmann Machines [2]. Mixture models, such as Mixtures of Factor Analyzers [3] and Mixtures of Gaussians, have also been used for modeling natural image patches [4]. More recently, denoising auto-encoders have been proposed as a way to model the transition operator that has the same invariant distribution as the data generating distribution [5].

Generative models have an advantage over discriminative models when part of the images are occluded or missing. Occlusions are very common in realistic settings and have been largely ignored in recent literature on deep learning. In addition, prior knowledge can be easily incorporated in generative models in the forms of structured latent variables, such as lighting and deformable parts. However, the enormous amount of content in high-resolution images makes generative learning difficult [6, 7]. Therefore, generative models have found most success in learning to model small patches of natural images and objects: Zoran and Weiss [4] learned a mixture of Gaussians model over image patches; Salakhutdinov and Hinton [2] used centered and uncluttered stereo images of toy objects on a clear background; Tang et al. [8] used images of centered and cropped faces. The fact that these models require curated training data limits their applicability on using the (virtually) unlimited unlabeled data.

In this paper, we propose a framework to infer the region of interest in a big image for generative modeling. This will allow us to learn a generative model of faces on a very large dataset of (unlabeled) images containing faces. Our framework is able to dynamically route the relevant information to the generative model and can ignore the background clutter. The need to dynamically and selectively route information is also present in the biological brain. Plethora of evidence points to the presence of attention in the visual cortex [9, 10]. Recently, in visual neuroscience, attention has been shown to exist not only in extrastriate areas, but also all the way down to V1 [11].

Attention as a form of routing was originally proposed by Anderson and Van Essen [12] and then extended by Olshausen et al. [13]. Dynamic routing has been hypothesized as providing a way for achieving shift and size invariance in the visual cortex [14, 15]. Tsotsos et al. [16] proposed a model combining search and attention called the Selective Tuning model. Larochelle and Hinton [17] proposed a way of using third-order Boltzmann Machines to combine information gathered from many foveal glimpses. Their model chooses where to look next to find locations that are most informative of the object class. Reichert et al. [18] proposed a hierarchical model to show that certain aspects of covert object-based attention can be modeled by Deep Boltzmann Machines. Several other related models attempt to learn where to look for objects [19, 20] and for video based tracking [21]. Inspired by Olshausen et al. [13], we use 2D similarity transformations to implement the scaling, rotation, and shift operation required for routing. Our main motivation is to enable the learning of generative models in big images where the location of the object of interest is unknown a-priori.

## 2 Gaussian Restricted Boltzmann Machines

Before we describe our model, we briefly review the Gaussian Restricted Boltzmann Machine (GRBM) [22], as it will serve as the building block for our attention-based model. GRBMs are a type of Markov Random Field model that has a bipartite structure with real-valued visible variables connected to binary stochastic hidden variables . The energy of the joint configuration of the Gaussian RBM is defined as follows:

(1) |

where are the model parameters. The marginal distribution over the visible vector is and the corresponding conditional distributions take the following form:

(2) | |||||

(3) |

Observe that conditioned on the states of the hidden variables (Eq. 3), each visible unit is modeled by a Gaussian distribution, whose mean is shifted by the weighted combination of the hidden unit activations. Unlike directed models, an RBM’s conditional distribution over hidden nodes is factorial and can be easily computed.

We can also add a binary RBM on top of the learned GRBM by treating the inferred as the “visible” layer together with a second hidden layer . This results in a 2-layer Gaussian Deep Belief Network (GDBN) [1] that is a more powerful model of .

Specifically, in a GDBN model, is modeled by the energy function of the 2nd-layer RBM, while is given by Eq. 3. Efficient inference can be performed using the greedy approach of [1] by treating each DBN layer as a separate RBM model. GDBNs have been applied to various tasks, including image classification, video action and speech recognition [6, 23, 24, 25].

## 3 The Model

Let be a high resolution image of a scene, e.g. a
image. We want to use attention to propagate regions of interest from
up to a canonical representation. For example, in order to learn
a model of faces, the canonical representation could be a aligned and
cropped frontal face image. Let represent this
low resolution canonical image. In this
work, we focus on a Deep Belief Network^{2}^{2}2Other generative
models can also be used with our attention framework. to model .

This is illustrated in the diagrams of Fig. 1. The left panel displays the model of Olshausen et.al. [13], whereas the right panel shows a graphical diagram of our proposed generative model with an attentional mechanism. Here, and represent the latent hidden variables of the DBN model, and (position, rotation, and scale) are the parameters of the 2D similarity transformation.

The 2D similarity transformation is used to rotate, scale, and translate the canonical image onto the canvas that we denote by . Let be a pixel coordinate (e.g. or ) of the canonical image . Let be the set of all coordinates of . For example, if is , then ranges from to . Let the “gaze” variables be the parameter of the Similarity transformation. In order to simplify derivations and to make transformations be linear w.r.t. the transformation parameters, we can equivalently redefine , where and (see [26] for details). We further define a function as the transformation function to warp points to :

(4) |

We use the notation to denote the bilinear interpolation of at coordinates with anti-aliasing. Let be the extracted low-resolution image at warped locations :

(5) |

Intuitively, is a patch extracted from according to the shift,
rotation and scale parameters of , as shown in Fig. 1, right panel.
It is this patch of data that we seek to model generatively.
Note that the dimensionality of is equal to the cardinality of , where denotes
the set of pixel coordinates of the canonical image .
Unlike standard generative learning tasks, the data is not static
but changes with the latent variables .
Given and , we model
the top-down generative process over^{3}^{3}3We will often omit dependence of on for
clarity of presentation. with
a Gaussian distribution having a diagonal covariance matrix :

(6) |

The fact that we do not seek to model the rest of the regions/pixels of is by design. By using 2D similarity transformation to mimic attention, we can discard the complex background of the scene and let the generative model focus on the object of interest. The proposed generative model takes the following form:

(7) |

where for we use a flat prior that is constant for all , and is defined by a 2-layer Gaussian Deep Belief Network. The conditional is given by a Gaussian distribution as in Eq. 6. To simplify the inference procedure, and the GDBN model of , , will share the same noise parameters .

## 4 Inference

While the generative equations in the last section are straightforward and intuitive, inference in these models is typically intractable due to the complicated energy landscape of the posterior. During inference, we wish to compute the distribution over the gaze variables and canonical object given the big image . Unlike in standard RBMs and DBNs, there are no simplifying factorial assumptions about the conditional distribution of the latent variable . Having a 2D similarity transformation is reminiscent of third-order Boltzmann machines with performing top-down multiplicative gating of the connections between and . It is well known that inference in these higher-order models is rather complicated.

One way to perform inference in our model is to resort to Gibbs sampling by computing the set of alternating conditional posteriors: The conditional distribution over the canonical image takes the following form:

(8) |

where is the top-down influence of the DBN. Note that if we know the gaze variable and the first layer of hidden variables , then is simply defined by a Gaussian distribution, where the mean is given by the average of the top-down influence and bottom-up information from . The conditional distributions over and given are given by the standard DBN inference equations [1]. The conditional posterior over the gaze variables is given by:

(9) |

Using Bayes’ rule, the unnormalized log probability of is defined in Eq. 9. We stress that this equation is atypical in that the random variable of interest actually affects the conditioning variable (see Eq. 5) We can explore the gaze variables using Hamiltonian Monte Carlo (HMC) algorithm [27, 28]. Intuitively, conditioned on the canonical object that our model has in “mind”, HMC searches over the entire image to find a region with a good match to .

If the goal is only to find the MAP estimate of , then we may want to use second-order methods for optimizing . This would be equivalent to the Lucas-Kanade framework in computer vision, developed for image alignment [29]. However, HMC has the advantage of being a proper MCMC sampler that satisfies detailed balance and fits nicely with our probabilistic framework.

The HMC algorithm first specifies the Hamiltonian over the position variables and auxiliary momentum variables : , where the potential function is defined by and the kinetic energy function is given by . The dynamics of the system is defined by:

(10) |

(11) | ||||

(12) |

Observe that Eq. 12 decomposes into sums over single coordinate positions . Let us denote to be the coordinate warped by . For the first term on the RHS of Eq. 12,

(13) |

where denotes the sampling of the gradient images of at the warped location . For the second term on the RHS of Eq. 12, we note that we can re-write Eq. 4 as:

(14) |

giving us

(15) |

HMC simulates the discretized system by performing leap-frog updates of and using Eq. 10. Additional hyperparameters that need to be specified include the step size , number of leap-frog steps, and the mass of the variables (see [28] for details).

### 4.1 Approximate Inference

HMC essentially performs gradient descent with momentum, therefore it is prone to getting stuck at local optimums. This is especially a problem for our task of finding the best transformation parameters. While the posterior over should be unimodal near the optimum, many local minima exist away from the global optimum. For example, in Fig. 2(a), the big image is enclosed by the blue box, and the canonical image is enclosed by the green box. The current setting of aligns together the wrong eyes. However, it is hard to move the green box to the left due to the local optima created by the dark intensities of the eye. Resampling the momentum variable every iteration in HMC does not help significantly because we are modeling real-valued images using a Gaussian distribution as the residual, leading to quadratic costs in the difference between and (see Eq. 9). This makes the energy barriers between modes extremely high.

To alleviate this problem we need to find good initializations of . We use a Convolutional Network (ConvNet) to perform efficient approximate inference, resulting in good initial guesses. Specifically, given , and , we predict the change in that will lead to the maximum . In other words, instead of using the gradient field for updating , we learn a ConvNet to output a better vector field in the space of . We used a fairly standard ConvNet architecture and the standard stochastic gradient descent learning procedure.

We note that standard feedforward face detectors seek to model , while completely ignoring the canonical face . In contrast, here we take into account as well. The ConvNet is used to initialize for the HMC algorithm. This is important in a proper generative model because conditioning on is appealing when multiple faces are present in the scene. Fig. 2(b) is a hypothesized Euclidean space of , where the black manifold represents canonical faces and the blue manifold represents cropped faces . The blue manifold has a low intrinsic dimensionality of 4, spanned by . At A and B, the blue comes close to black manifold. This means that there are at least two modes in the posterior over . By conditioning on , we can narrow the posterior to a single mode, depending on whom we want to focus our attention. We demonstrate this exact capability in Sec. 6.3.

Fig. 3 demonstrates the iterative process of how approximate inference works in our model. Specifically, based on , the ConvNet takes a window patch around () and () as input, and predicts the output . In step 2, is updated accordingly, followed by step 3 of alternating Gibbs updates of and , as discussed in Sec. 4. The process is repeated. For the details of the ConvNet see the supplementary materials.

## 5 Learning

While inference in our framework localizes objects of interest and is akin to object detection, it is not the main objective. Our motivation is not to compete with state-of-the-art object detectors but rather propose a probabilistic generative framework capable of generative modeling of objects which are at unknown locations in big images. This is because labels are expensive to obtain and are often not available for images in an unconstrained environment.

To learn generatively without labels we propose a simple Monte Carlo based Expectation-Maximization algorithm. This algorithm is an unbiased estimator of the maximum likelihood objective. During the E-step, we use the Gibbs sampling algorithm developed in Sec. 4 to draw samples from the posterior over the latent gaze variables , the canonical variables , and the hidden variables , of a Gaussian DBN model. During the M-step, we can update the weights of the Gaussian DBN by using the posterior samples as its training data. In addition, we can update the parameters of the ConvNet that performs approximate inference. Due to the fact that the first E-step requires a good inference algorithm, we need to pretrain the ConvNet using labeled gaze data as part of a bootstrap process. Obtaining training data for this initial phase is not a problem as we can jitter/rotate/scale to create data. In Sec. 6.2, we demonstrate the ability to learn a good generative model of face images from the CMU Multi-PIE dataset.

## 6 Experiments

We used two face datasets in our experiments. The first dataset is a frontal face dataset, called the Caltech Faces from 1999, collected by Markus Weber. In this dataset, there are 450 faces of 27 unique individuals under different lighting conditions, expressions, and backgrounds. We downsampled the images from their native 896 by 692 by a factor of 2. The dataset also contains manually labeled eyes and mouth coordinates, which will serve as the gaze labels. We also used the CMU Multi-PIE dataset [30], which contains 337 subjects, captured under 15 viewpoints and 19 illumination conditions in four recording sessions for a total of more than 750,000 images. We demonstrate our model’s ability to perform approximate inference, to learn without labels, and to perform identity-based attention given an image with two people.

### 6.1 Approximate inference

We first investigate the critical inference algorithm of on the Caltech Faces dataset. We run 4 steps of approximate inference detailed in Sec. 4.1 and diagrammed in Fig. 3, followed by three iterations of 20 leap-frog steps of HMC. Since we do not initially know the correct , we initialize to be the average face across all subjects.

Fig. 4 shows the image of and during inference for a test subject. The initial gaze box is colored yellow on the left. Subsequent gaze updates progress from yellow to blue. Once ConvNet-based approximate inference gives a good initialization, starting from step 5, five iterations of 20 leap-frog steps of HMC are used to sample from the the posterior.

Fig. 5 shows the quantitative results of Intersection over Union (IOU) of the ground truth face box and the inferred face box. The results show that inference is very robust to initialization and requires only a few steps of approximate inference to converge. HMC clearly improves model performance, resulting in an IOU increase of about 5% for localization. This is impressive given that none of the test subjects were part of the training and the background is different from backgrounds in the training set.

Our method | OpenCV | NCC | template | |

IOU 0.5 | 97% | 97% | 93% | 78% |

# evaluations |

We also compared our inference algorithm to the template matching in the task of face detection. We took the first 5 subjects as test subjects and the rest as training. We can localize with 97% accuracy (IOU 0.5) using our inference algorithm^{4}^{4}4 is randomly initialized at 30 pixels, scale range from 0.5 to 1.5.. In comparison, a near state-of-the-art face detection system from OpenCV 2.4.9 obtains the same 97% accuracy. It uses Haar Cascades, which is a form of AdaBoost^{5}^{5}5OpenCV detection uses pretrained model from haarcascade_frontalface_default.xml, scaleFactor=1.1, minNeighbors=3 and minSize=30.. Normalized Cross Correlation [31] obtained 93% accuracy, while Euclidean distance template matching achieved an accuracy of only 78%.
However, note that our algorithm looks at a constant number of windows while the other baselines are all based on scanning windows.

### 6.2 Generative learning without labels

nats | No CMU training | CMU w/o labels | CMU w/ labels |

Caltech Train | 617 0.4 | 627 0.5 | 569 0.6 |

Caltech Valid | 512 1.1 | 503 1.8 | 494 1.7 |

CMU Train | 96 0.8 | 499 0.1 | 594 0.5 |

CMU Valid | 85 0.5 | 387 0.3 | 503 0.7 |

454.6 | 687.8 | 694.2 |

The main advantage of our model is that it can learn on large images of faces without localization label information (no manual cropping required). To demonstrate, we use both the Caltech and the CMU faces dataset. For the CMU faces, a subset of 2526 frontal faces with ground truth labels are used. We split the Caltech dataset into a training and a validation set. For the CMU faces, we first took 10% of the images as training cases for the ConvNet for approximate inference. This is needed due to the completely different backgrounds of the Caltech and CMU datasets. The remaining 90% of the CMU faces are split into a training and validation set. We first trained a GDBN with 1024 and 256 hidden units on the Caltech training set. We also trained a ConvNet for approximate inference using the Caltech training set and 10% of the CMU training images.

Table 2 shows the estimates of the variational lower-bounds on the
average log-density (higher is better) that the GDBN models assign to the ground-truth cropped face images from the training/test sets
under different scenarios.
In the left column,
the model is only trained on Caltech faces.
Thus it gives very low probabilities to the CMU faces.
Indeed, GDBNs achieve a variational lower-bound of only 85 nats per test image.
In the middle column, we use our
approximate inference to estimate the location of the CMU training faces and further trained the GDBN on the
newly localized faces.
This gives a dramatic increase of the model performance on the CMU Validation set^{6}^{6}6We note that we still made use of labels coming from
the 10% of CMU Multi-PIE training set in order to pretrain our ConvNet. "w/o labels" here means that no labels for the CMU Train/Valid images are given.,
achieving a lower-bound of 387 nats per test image.
The right column gives the best possible results if we can train with the CMU manual localization labels.
In this case, GDBNs achieve a lower-bound of 503 nats. We used Annealed Importance Sampling (AIS) to estimate the partition function for the top-layer RBM. Details on estimating the variational lower bound are in the supplementary materials.

Fig. 6(a) further shows samples drawn from the Caltech trained DBN, whereas Fig. 6(b) shows samples after training with the CMU dataset using estimated . Observe that samples in Fig. 6(b) show a more diverse set of faces. We trained GDBNs using a greedy, layer-wise algorithm of [1]. For the top layer we use Fast Persistent Contrastive Divergence [32], which substantially improved generative performance of GDBNs (see supplementary material for more details).

### 6.3 Inference with ambiguity

Our attentional mechanism can also be useful when multiple objects/faces are present in the scene. Indeed, the posterior is conditioned on , which means that where to attend is a function of the canonical object the model has in “mind” (see Fig. 2(b)). To explore this, we first synthetically generate a dataset by concatenating together two faces from the Caltech dataset. We then train approximate inference ConvNet as in Sec. 4.1 and test on the held-out subjects. Indeed, as predicted, Fig. 7 shows that depending on which canonical image is conditioned, the same exact gaze initialization leads to two very different gaze shifts. Note that this phenomenon is observed across different scales and location of the initial gaze. For example, in Fig. 7, right-bottom panel, the initialized yellow box is mostly on the female’s face to the left, but because the conditioned canonical face is that of the right male, attention is shifted to the right.

## 7 Conclusion

In this paper we have proposed a probabilistic graphical model framework for learning generative models using attention. Experiments on face modeling have shown that ConvNet based approximate inference combined with HMC sampling is sufficient to explore the complicated posterior distribution. More importantly, we can generatively learn objects of interest from novel big images. Future work will include experimenting with faces as well as other objects in a large scene. Currently the ConvNet approximate inference is trained in a supervised manner, but reinforcement learning could also be used instead.

## Acknowledgements

The authors gratefully acknowledge the support and generosity from Samsung, Google, and ONR grant N00014-14-1-0232.

## References

- [1] G. E. Hinton, S. Osindero, and Y. W. Teh. A fast learning algorithm for deep belief nets. Neural Computation, 18(7):1527–1554, 2006.
- [2] R. Salakhutdinov and G. Hinton. Deep Boltzmann machines. In AISTATS, 2009.
- [3] Geoffrey E. Hinton, Peter Dayan, and Michael Revow. Modeling the manifolds of images of handwritten digits. IEEE Transactions on Neural Networks, 8(1):65–74, 1997.
- [4] Daniel Zoran and Yair Weiss. From learning models of natural image patches to whole image restoration. In ICCV. IEEE, 2011.
- [5] Yoshua Bengio, Li Yao, Guillaume Alain, and Pascal Vincent. Generalized denoising auto-encoders as generative models. In Advances in Neural Information Processing Systems 26, 2013.
- [6] H. Lee, R. Grosse, R. Ranganath, and A. Y. Ng. Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations. In ICML, pages 609–616, 2009.
- [7] Marc’Aurelio Ranzato, Joshua Susskind, Volodymyr Mnih, and Geoffrey Hinton. On Deep Generative Models with Applications to Recognition. In CVPR, 2011.
- [8] Yichuan Tang, Ruslan Salakhutdinov, and Geoffrey E. Hinton. Deep mixtures of factor analysers. In ICML. icml.cc / Omnipress, 2012.
- [9] M. I. Posner and C. D. Gilbert. Attention and primary visual cortex. Proc. of the National Academy of Sciences, 96(6), March 1999.
- [10] E. A. Buffalo, P. Fries, R. Landman, H. Liang, and R. Desimone. A backward progression of attentional effects in the ventral stream. PNAS, 107(1):361–365, Jan. 2010.
- [11] N Kanwisher and E Wojciulik. Visual attention: Insights from brain imaging. Nature Reviews Neuroscience, 1:91–100, 2000.
- [12] C. H. Anderson and D. C. Van Essen. Shifter circuits: A computational strategy for dynamic aspects of visual processing. National Academy of Sciences, 84:6297–6301, 1987.
- [13] B. A. Olshausen, C. H. Anderson, and D. C. Van Essen. A neurobiological model of visual attention and invariant pattern recognition based on dynamic routing of information. The Journal of neuroscience : the official journal of the Society for Neuroscience, 13(11):4700–4719, 1993.
- [14] Laurenz Wiskott. How does our visual system achieve shift and size invariance?, 2004.
- [15] S. Chikkerur, T. Serre, C. Tan, and T. Poggio. What and where: a Bayesian inference theory of attention. Vision Research, 50(22):2233–2247, October 2010.
- [16] J. K. Tsotsos, S. M. Culhane, W. Y. K. Wai, Y. H. Lai, N. Davis, and F. Nuflo. Modeling visual-attention via selective tuning. Artificial Intelligence, 78(1-2):507–545, October 1995.
- [17] Hugo Larochelle and Geoffrey E. Hinton. Learning to combine foveal glimpses with a third-order boltzmann machine. In NIPS, pages 1243–1251. Curran Associates, Inc., 2010.
- [18] D. P. Reichert, P. Seriès, and A. J. Storkey. A hierarchical generative model of recurrent object-based attention in the visual cortex. In ICANN (1), volume 6791, pages 18–25. Springer, 2011.
- [19] B. Alexe, N. Heess, Y. W. Teh, and V. Ferrari. Searching for objects driven by context. In NIPS 2012, December 2012.
- [20] Marc’Aurelio Ranzato. On learning where to look. arXiv, arXiv:1405.5488, 2014.
- [21] M. Denil, L. Bazzani, H. Larochelle, and N. de Freitas. Learning where to attend with deep architectures for image tracking. Neural Computation, 28:2151–2184, 2012.
- [22] G. E. Hinton and R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313:504–507, 2006.
- [23] A. Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis, University of Toronto, Toronto, Ontario, Canada, 2009.
- [24] Graham W. Taylor, Rob Fergus, Yann LeCun, and Christoph Bregler. Convolutional learning of spatio-temporal features. In ECCV 2010. Springer, 2010.
- [25] A. Mohamed, G. Dahl, and G. Hinton. Acoustic modeling using deep belief networks. IEEE Transactions on Audio, Speech, and Language Processing, 2011.
- [26] Richard Szeliski. Computer Vision - Algorithms and Applications. Texts in Computer Science. Springer, 2011.
- [27] S. Duane, A. D. Kennedy, B. J Pendleton, and D. Roweth. Hybrid Monte Carlo. Physics Letters B, 195(2):216–222, 1987.
- [28] R. M. Neal. MCMC using Hamiltonian dynamics. in Handbook of Markov Chain Monte Carlo (eds S. Brooks, A. Gelman, G. Jones, XL Meng). Chapman and Hall/CRC Press, 2010.
- [29] Simon Baker and Iain Matthews. Lucas-kanade 20 years on: A unifying framework. International Journal of Computer Vision, 56:221–255, 2002.
- [30] Ralph Gross, Iain Matthews, Jeffrey F. Cohn, Takeo Kanade, and Simon Baker. Multi-pie. Image Vision Comput., 28(5):807–813, 2010.
- [31] J. P. Lewis. Fast normalized cross-correlation, 1995.
- [32] T. Tieleman and G. E. Hinton. Using fast weights to improve persistent contrastive divergence. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML 2009, volume 382, page 130. ACM, 2009.

## Appendix

### Convolutional Neural Network

The training of ConvNet for approximate inference is standard and did not involve any special ’tricks’. We used SGD with minibatch size of 128 samples. We used a standard ConvNet architecture with convolution C layers followed by max-pooling S layers. The ConvNet takes as input and to predict change in such that to maximize . In order to better predict change of , as well as a bigger border around are used as the input to the ConvNet. Therefore, has resolution and has resolution of .

Two handle two different inputs with different resolutions, two different “streams" are used in this ConvNet architecture. One stream will process and another one for . These two streams will be combined multiplicatively after subsampling the stream by a factor of 3. The rest of the ConvNet is same as the standard classification ConvNets, except that we use mean squared error as our cost function. See Figure 8 for a visual diagram of what the convolutional neural network architecture used.

layer | type | latent variables | filter size | # weights |
---|---|---|---|---|

0 | input | maps:3 72x72 | - | - |

1 | input | maps:3 24x24 | - | - |

2 | Conv of layer 0 | maps:16 66x66 | 7x7 | 2352 |

3 | Pooling | maps:16 22x22 | 3x3 | - |

4 | Conv of layer 1 | maps:16 22x22 | 5x5 | 1200 |

5 | Combine layers 3,4 | maps:16 22x22 | - | - |

6 | Fully connected | 1024 | - | 7.9M |

7 | Fully connected | 4 | - | 4K |

Table 3 details the exact model architecture used. In layer 5, the two streams have the same number of hidden maps and hidden topography. We combine these two multiplicatively by multiplying their activation element-wise. This creates a third-order flavor and is more powerful for the task of determining where to shift attention to next.

### Gaussian Deep Belief Network

The training of the Gaussian Deep Belief Network is performed in a standard greedy layerwise fashion. The first layer Gaussian Restricted Boltzmann Machine is trained with the Contrastive Divergence algorithm where the standard deviation of each visible unit is learned as well. After training the first layer, we use Eq. 3 to obtain first hidden layer binary probabilities. We then train a 2nd binary-binary Restricted Boltzmann Machine using the fast persistent contrastive divergence learning algorithm. This greedy training leads us to the Gaussian Deep Belief Network. No finetuning of the entire network is performed.

#### Quantitative evaluation for Gaussian Deep Belief Network

For quantitative evaluation, we approximate the standard variational lower bound on the log likelihood of the Gaussian Deep Belief Network. The model is a directed model:

(16) |

For any approximating posterior distribution , the GDBN’s log-likelihood has this lower variational bound:

(17) |

The entropy can be calculated since we made the factorial assumption on .

(18) |

(19) |

In order to calculate the expectation of the approximating posteriors, we use Monte Carlo sampling.

(20) | ||||

(21) | ||||

(22) |

where is the -th sample from the posterior .

In order to calculate the partition function of the top-most layer of the GDBN, we use Annealed Importance Sampling (AIS). We used 100 chains with 50,000 intermediate distributions to estimate the partition function of the binary-binary RBM which forms the top layer of the GDBN. Even though AIS is an unbiased estimator of the partition function, it is prone to under-estimating it due to bad mixing of the chains. This causes the log probability to be over-estimated. Therefore the variational lower bounds reported in our paper are not strictly guaranteed to be lower bounds and are subject to errors. However, we believe that the margin of error is unlikely to be high enough to affect our conclusions.

### Additional Results

We present some more examples of inference process of our framework.

Below, we show some success cases and a failure case for inference on the CMU Multi-PIE dataset. The initial gaze variables of are highlighted in yellow and later iterations are highlighted with color gradually changing to blue.