# Emergence of Compositional Representations in Restricted Boltzmann Machines

## Abstract

Extracting automatically the complex set of features composing real high-dimensional data is crucial for achieving high performance in machine–learning tasks. Restricted Boltzmann Machines (RBM) are empirically known to be efficient for this purpose, and to be able to generate distributed and graded representations of the data. We characterize the structural conditions (sparsity of the weights, low effective temperature, nonlinearities in the activation functions of hidden units, and adaptation of fields maintaining the activity in the visible layer) allowing RBM to operate in such a compositional phase. Evidence is provided by the replica analysis of an adequate statistical ensemble of random RBMs and by RBM trained on the handwritten digits dataset MNIST.

Recent years have witnessed major progresses in supervised machine learning, e.g. in video, audio, image processing,… (1). Despite those impressive successes, unsupervised learning, in which the structure of data is learned without a priori knowledge still presents formidable challenges. A fundamental question is how to learn probability distributions that fit well complex data manifolds in high-dimensional spaces (2). Once learnt, such generative models can be used for denoising, completion, artificial data generation,… Hereafter we focus on one important generative model, Restricted Boltzmann Machines (RBM) (3); (4). In its simplest formulation a RBM is a Boltzmann machine on a bipartite graph, see Fig. 1(a), with a visible (v) layer that represents the data, connected to a hidden (h) layer meant to extract and explain their statistical features. The marginal distribution over the visible layer is fitted to the data through approximate likelihood maximization (5); (6); (7); (8). Once the parameters are trained each hidden unit becomes selectively activated by a specific data feature; owe to the bidirectionality of connections the probability to generate configurations of the visible layer where this feature is present is, in turn, increased. Multiple combinations of numbers of features, with varying degrees of activation of the corresponding hidden units allow for efficient generation of a large variety of new data samples. However, the existence of such ‘compositional’ encoding seems to depend on the values of the RBM parameters, such as the size of the hidden layer (9). Characterizing the conditions under which RBM can operate in this compositional regime is the purpose of the present work.

In the RBM shown in Fig. 1(a) the visible layer includes units , with , chosen here to be binary (). Visible units are connected to hidden units , through the weights . The energy of a configuration is defined through

(1) |

where is a potential acting on hidden unit ; due to the binary nature of the visible units their potential is fully characterized by a local field, in (1). Configurations are then sampled from the Gibbs equilibrium distribution associated to , , where is the partition function (3).

Given a visible configuration the most likely value of hidden unity is a function of its input : , where the activation function as can be seen from the minimization of . Examples of are shown in Fig. 1(b). When is linear, i.e. for quadratic potential the probability is Gaussian in the hidden units, and the marginal distribution of the visible configurations can be exactly computed (10). It coincides with the equilibrium distribution of a Boltzmann machine with a pairwise interaction matrix , or, equivalently, of a Hopfield model (11), whose patterns are the columns of the weight matrix .

Activation functions empirically known in machine–learning literature to provide good results are, however, nonlinear. Nonlinear produce effective Boltzmann machines with high order () multibody interactions between the visible units . Two examples are shown in Fig. 1(b): Bernoulli units, which take discrete 0,1 values, and Rectified Linear Units (ReLU) (1). Unlike Bernoulli units ReLU preserve information about the magnitudes of their inputs above threshold (12); this property is expected for real neurons and ReLU were first introduced in the context of theoretical neuroscience (13).

We first report results from a training experiment of RBM with ReLU on the handwritten digits dataset MNIST (14). Our goal is not to classify digits from 0 to 9, but to learn a generative model of digits from examples. Details about learning can be found in Supplemental Material, Section I. Figure 2(a) shows typical ‘features’ after learning. Each feature includes negative and positive weights, and is localized around small portions of the visible layer. These features look like elementary strokes, which are combined by the RBM to generate random digits (Fig. 2(b)). In each generated handwritten digit image hidden units are silent (), see histogram in Fig. 2(c). The remaining hidden units have largely varying activations, some weak and few very strong; we estimate the number of strongly activated ones through the participation ratio , with exponent as explained below. On average elementary strokes compose a generated digit, see Fig. 2(c). Different combinations of strokes correspond to different variants of the same digits. Many of those variants are not contained in the training set, and closely match digits in the test set (Supplemental Material, Fig. 1(b)), hence showing the generative power of RBM.

Learning is accompanied by structural changes in RBM, which we track with two parameters: and . Those parameters are proxies for, respectively, the fraction of nonzero weights and the effective inverse temperature, see Supplemental Material, Section III. Figure 2(d) shows that diminishes to small values , whereas increases. While most weights become very small and negligible the remaining ones get large, in agreement with Fig. 2(a). Notice that sparsity is not imposed to obtain a specific class of features, e.g. as in (15), but naturally emerges through likelihood maximization across training. The presence of large weights implies that flipping visible units is associated to large energy costs. Visible units are effectively at very low temperature, as can be seen from the quasi binary nature of conditional averages in Fig. 2(b) and Supplemental Material, Fig. 5.

We argue below that these structural changes are not specific to MNIST-trained RBM but are generically needed to bring RBM towards a compositional phase, in which visible configurations are composed from combinations of a large number (typically, , as in Fig. 2(c)) of features encoded by simultaneously, strongly activated hidden units. Our claim is supported by a detailed analysis of a Random RBM (R-RBM) ensemble, in which the weights are quenched random variables, with controlled sparsity and strength, and the magnitude of the visible fields and the values of the ReLU thresholds can be chosen. For adequate choices of these control parameters the compositional phase is thermodynamically favoured with respect to the ferromagnetic phase of the Hopfield model, where one pattern is activated (16), and to the spin-glass phase, in which all hidden units are weakly and incoherently activated (Fig. 1(c)).

In the R-RBM ensemble weights are independent random variables, equal to with probabilities equal to, respectively, ; sets the degree of sparsity of the weights attached to the visible unit , high sparsities corresponding to small . The estimator defined above (Fig. 2(d)) measures the fraction of nonzero weights, . This distribution was previously introduced to study parallel storage of multiple sparse items in the Hopfield model (17); (18). For simplicity the fields on visible units and the potentials acting on hidden units are chosen to be uniform, and (Fig. 1(a)). We define the ratio of the numbers of hidden and visible units, .

Given a visible layer configuration , hidden units coding for features present in will be strongly activated: their inputs will be strong and positive, comparable to the product of the norms of ( for large ) and (of the order of ), and, hence, will scale as , where , called magnetization, is finite. Most hidden units have, however, features essentially orthogonal to , and receive inputs fluctuating around 0, with finite variances. These scalings ensure that defined above (Fig. 2(c)) will coincide with the number of strongly activated units when ; choosing exponent in rather than would have introduced biases coming from weakly activated units (Supplemental Material, Section III.B).

The typical ground state (GS) energy (1) of R-RBM can be computed with the replica method within the replica-symmetric Ansatz (16), as the optimum of

(2) | |||||

over the order parameters (averaged over the quenched weights): and are, respectively, the magnetization and the number of feature-encoding hidden units, is the mean squared activity of the other hidden units, is the weighted activity of the visible layer in the GS, and are response functions, i.e. derivatives of the mean activity of, respectively, hidden and visible units with respect to their inputs (20). In (2) denotes the Gaussian measure, , and is the average over the sum of i.i.d. weights drawn as above.

We first fix , and optimize over all the other order parameters. At large the only solution has , and corresponds to the Spin-Glass phase. For intermediate values of , other solutions, with , exist. For the sake of simplicity we consider first the homogenous sparsity case, with . We show in Fig. 3(a), for fixed and various values of , the maximal value of below which a phase with magnetized hidden units exists. Importantly this critical value can be made arbitrarily large by increasing the ReLU threshold . This phenomenon is a consequence of the nonlinearity of ReLU, and can be understood as follows. The squared activity of non-magnetized hidden units obeys the saddle-point equation . The first factor is reminiscent of the expression arising for the Hopfield model (for which at zero temperature) (16), while the second factor comes from the nonlinearity of ReLU. being a rapidly decaying function of its argument large thresholds lead to small values. As the level of crosstalk due to nonmagnetized hidden units diminishes larger ratios can be supported by R-RBM without entering the glassy phase. Numerical simulations of R-RBMs at large confirm the existence and (meta)stability of phases with nonzero magnetizations (Fig. 3(b)). Moreover, the values of the average normalized magnetizations are in excellent agreement with those found by optimizing .

The nature of the large– phases and the selection of the value of are best understood in the limit case of highly sparse connections, . The R-RBM model exhibits an interesting limit behaviour, which we call hereafter compositional phase. In this regime the number of strongly magnetized hidden units is unbounded, and diverges as , with and finite. The normalized GS energy is a nonmonotonous function of the index , see Fig. 4(a). Minimization of leads to the selection of a well defined index . The magnetizations of the strongly activated units, , vanish linearly with ^{1}

The compositional phase competes with the ferromagnetic phase, in which but is a monotonously growing function of (hence, ), and the spin glass phase, in which and does not depend on , see Fig. 4(a). The phase diagram in the parameter space ( will be detailed in (20). Briefly speaking, given , should be large enough (as in Fig. 3) and should be neither too large to penalize the ferromagnetic phase, nor too small to avoid the spin glass regime.

Characteristic properties of our compositional phase are confronted to ReLU RBMs trained on MNIST in Fig. 4 (b,c). Compared to Fig. 2 we add a regularization penalty to control the final degree of sparsity; the case gives standard regularization, while, for , the effective penalty strength increases with the weights, hence promoting homogeneity among hidden units. After training we generate Monte Carlo samples of each RBM at equilibrium, and monitor the average number of active hidden units, , and the normalized magnetization, . Figure 4(b) shows vs. , in good agreement with the R-RBM theoretical scaling . Figure 4(c) shows that is a decreasing function of , as qualitatively predicted by theory, but quantitatively differs from the prediction of R-RBM with homogeneous . This disagreement can be partly explained by the heterogeneities in the sparsities in RBMs trained on MNIST, e.g. units on the borders are connected to only few hidden units, whereas units at the center of the grid are connected to many. We introduce a heterogeneous R-RBM model, where the distribution of the ’s is fitted from MNIST-trained RBMs (Supplemental Material, Section III.E). Its GS energy can be calculated from (2), see Fig. 4(a) (20). Results are shown in Fig. 4(b,c) to be in good agreement with RBM trained on MNIST.

RBMs, unlike the Hopfield or mixture model, may produce gradually different visible configurations through progressive changes in the hidden-layer activation pattern. R-RBMs enjoy the same property. We compute, through a real-replica approach (20), the average Hamming distance (per pixel) between the visible configurations minimizing the energy (1) for two hidden configurations sharing hidden units among the strongly activated ones. Figure 4(d) shows that monotonously increases from for up to (complete decorrelation of visible units) for , in very good quantitative agreement with results for RBM trained on MNIST.

The gradual change property has deep dynamical consequences. MCMC of MNIST-trained RBM (videos available in Supplemental Material) show that gradual changes may occasionally lead to another digit type, by passing through well-drawn, yet ambiguous digits. The progressive replacement of feature-encoding hidden units (small steps) along the transition path does not increase much the energy, and the transition process is fast compared to activated hopping between deep minima taking place in the Hopfield model.

Our study is related to several previous works. RBMs with linear activation function coincide with the Hopfield model. In this framework magnetized hidden units identify retrieved patterns, and corresponds to the capacity of the autoassociative memory. Tsodyks and Feigel’man showed how the critical capacity (for single pattern retrieval) could be dramatically increased with sparse weights () and appropriate tuning of the fields (21); however this effect could be achieved only with vanishingly low activities . Agliari and collaborators showed in a series of papers (17); (18) that multiple sparse patterns could be simultaneously retrieved in the case of linear and vanishing capacity (finite ). Finite capacity could be achieved at zero temperature in the limit of extreme sparsity, , only (19); for MNIST and would give . Our work shows that large values of can be reached even with moderate sparsity (as in realistic situations, see Fig. 2) provided that nonlinear (ReLU) and appropriate threshold values are considered. The presence of the fields acting on the visible units (absent in the model of (17); (18); (19)), is also crucial for the existence of our compositional phase as explained above.

It would be interesting to extend our work to more than one layers of hidden units, or to other types of nonlinear . While numerical studies of RBMs with Bernoulli hidden units show no qualitative change compared to ReLU, choosing growing asymptotically faster than could affect the nature of the extracted features (23). An important challenge would be to understand the training dynamics, i.e. how hidden units gradually extract features from data prototypes.

Acknowledgements. We are grateful to C. Fisher and G. Semerjian for useful discussions. This work was partly funded by the CNRS-Inphyniti Inferneuro and the HFSP RGP0057/2016 projects, and benefited from the support of NVIDIA Corporation with the donation of a Tesla K40 GPU card.

### Footnotes

- Solutions with nonhomogeneous magnetizations , varying from one strongly activated hidden unit to another, give additional contributions to of the order of with respect to the homogeneous solution , and do not affect the value of (20).

### References

- Y. LeCun, Y. Bengio, J. Hinton, Nature 521, 436-444 (2015).
- Y. Bengio, A. Courville, P. Vincent, IEEE transactions on pattern analysis and machine intelligence 35, 1798-1828 (2013).
- P. Smolensky, Chapter 6: Information Processing in Dynamical Systems: Foundations of Harmony Theory, in Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Volume 1: Foundations, MIT Press, 194-281 (1986).
- R. Salakhutdinov, A. Mnih, G. Hinton, Proceedings of the 24th international conference on Machine learning, p. 791-798 (2007).
- G. Hinton, Momentum 9, 926 (2010).
- T. Tieleman, Proceedings of the 25th international conference on Machine learning, pp. 1064-1071 (2008).
- G. Desjardins, A. Courville, Y. Bengio, P. Vincent, O. Delalleau, Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, pp. 145-152 (2010).
- M. Gabrie, E.W. Tramel, F. Krzakala, Advances in Neural Information Processing Systems, pp. 640-648 (2015).
- A. Fischer, C. Igel, Iberoamerican Congress on Pattern Recognition, pp. 14-36 (2012).
- A. Barra, A. Bernacchia, E. Santucci, P. Contucci, Neural Networks 34, 1-9 (2012).
- J.J. Hopfield, Proc. Nat. Acad. Sci. USA 79, 2554 (1982).
- V. Nair, G.E. Hinton, Proceedings of the 27th International Conference on Machine Learning, p. 807-814 (2010).
- A. Treves, J. Comp. Neurosci. 2, 259-272 (1995).
- Y. LeCun, C. Cortes, C.J. Burges, The MNIST database of handwritten digits (1998).
- B.A. Olshausen, D.J. Field, Nature 381, 607-609 (1996).
- D.J. Amit, H. Gutfreund, H. Sompolinsky, Phys. Rev. Lett. 55, 1530 (1985).
- E. Agliari, A. Barra, A. Galluzzi, F. Guerra, F. Moauro, Phys. Rev. Lett. 109, 268101 (2012).
- E. Agliari, A. Annibale, A. Barra, A.C.C. Coolen, D. Tantari, J. Phys. A 46, 415003 (2013).
- P. Sollich, D. Tantari, A. Annibale, A. Barra, Phys. Rev. Lett. 113, 238106 (2014).
- J. Tubiana, R. Monasson, in preparation (2017).
- M. Tsodyks, M.V. Feigel’man, Europhys. Lett. 6, 101-105 (1989).
- R. Salakhutdinov, I. Murray, Proceedings of the 25th international conference on Machine learning, pp. 872-879 (2008).
- D. Krotov, J.J. Hopfield, arXiv:1606.01164 (2016).
- See Supplemental Material [url], which includes Ref [25].
- Theano Development team Theano Development Team. arXiv:1605.02688. (2016).