# Unsupervised Feature Learning from Temporal Data

Unsupervised Feature Learning from Temporal Data

Ross Goroshin, Joan Bruna, Jonathan Tompson, David Eigen, Yann LeCun |
---|

Courant Institute of Mathematical Science 719 Broadway 12 Floor, New York, NY 10003 |

Facebook AI Research, 770 Broadway, New York, NY 10003 |

{goroshin,bruna,tompson,deigen,yann}@cims.nyu.edu |

Current state-of-the-art classification and detection algorithms rely on supervised training. In this work we study unsupervised feature learning in the context of temporally coherent video data. We focus on feature learning from unlabeled video data, using the assumption that adjacent video frames contain semantically similar information. This assumption is exploited to train a convolutional pooling auto-encoder regularized by slowness and sparsity. We establish a connection between slow feature learning to metric learning and show that the trained encoder can be used to define a more temporally and semantically coherent metric.

Our main assumption is that data samples that are temporal neighbors are also likely to be neighbors in the latent space. For example, adjacent frames in a video sequence are more likely to be semantically similar than non-adjacent frames. This assumption naturally leads to the slowness prior on features which was introduced in SFA (Wiskott & Sejnowski (2002)).

Temporal coherence can be exploited by assuming a prior on the features extracted from the temporal data sequence. One such prior is that the features should vary slowly with respect to time. In the discrete time setting this prior corresponds to minimizing an norm of the difference of feature vectors for temporally adjacent inputs. Consider a video sequence with frames, if represents the feature vector extracted from the frame at time then the slowness prior corresponds to minimizing . To avoid the degenerate solution , a second term is introduced which encourages data samples that are not temporal neighbors to be separated by at least a distance of -units in feature space, where is known as the margin. In the temporal setting this corresponds to minimizing , where . Together the two terms form the loss function introduced in Hadsell et al. (2006) as a dimension reduction and data visualization algorithm known as DrLIM. Assume that there is a differentiable mapping from input space to feature space which operates on individual temporal samples. Denote this mapping by and assume it is parametrized by a set of trainable coefficients denoted by . That is, . The per-sample loss function can be written as:

(1) |

The second contrastive term in Equation 1 only acts to avoid the degenerate solution in which is a constant mapping, it does not guarantee that the resulting feature space is informative with respect to the input. This discriminative criteria only depends on pairwise distances in the representation space which is a geometrically weak notion in high dimensions. We propose to replace this contrastive term with a term that penalizes the reconstruction error of both data samples. Introducing a reconstruction terms not only prevents the constant solution but also acts to explicitly preserve information about the input. This is a useful property of features which are obtained using unsupervised learning; since the task to which these features will be applied is not known a priori, we would like to preserve as much information about the input as possible.

What is the optimal architecture of for extracting slow features? Slow features are invariant to temporal changes by definition. In natural video and on small spatial scales these changes mainly correspond to local translations and deformations. Invariances to such changes can be achieved using appropriate pooling operators Bruna & Mallat (2013); LeCun et al. (1998). Such operators are at the heart of deep convolutional networks (ConvNets), currently the most successful supervised feature learning architectures Krizhevsky et al. (2012). Inspired by these observations, let be a two stage encoder comprised of a learned, generally over-complete, linear map () and rectifying nonlinearity , followed by a local pooling. Let the hidden activations, , be subdivided into potentially overlapping neighborhoods denoted by . Note that biases are absorbed by expressing the input in homogeneous coordinates. Feature produced by the encoder for the input at time can be expressed as . Training through a local pooling operator enforces a local topology on the hidden activations, inducing units that are pooled together to learn complimentary features. In the following experiments we will use . Although it has recently been shown that it is possible to recover the input when is sufficiently redundant, reconstructing from these coefficients corresponds to solving a phase recovery problem Bruna et al. (2014) which is not possible with a simple inverse mapping, such as a linear map . Instead of reconstructing from we reconstruct from the hidden representation . This is the same approach taken when training group-sparse auto-encoders Kavukcuoglu et al. (2009). In order to promote sparse activations in the case of over-complete bases we additionally add a sparsifying penalty on the hidden activations. Including the rectifying nonlinearity becomes critical for learning sparse inference in a hugely redundant dictionary, e.g. convolutional dictionaries Gregor & LeCun (2010). The complete loss functional is:

(2) |

Figure 2 shows a convolutional version of the proposed architecture and loss. By replacing all linear operators in our model with convolutional filter banks and including spatial pooling, translation invariance need not be learned LeCun et al. (1998). In all other respects the convolutional model is conceptually identical to the fully connected model described in the previous section.

## References

- Bengio et al. (2012) Bengio, Yoshua, Courville, Aaron C., and Vincent, Pascal. Representation learning: A review and new perspectives. Technical report, University of Montreal, 2012.
- Bromley et al. (1993) Bromley, Jane, Bentz, James W, Bottou, Léon, Guyon, Isabelle, LeCun, Yann, Moore, Cliff, Säckinger, Eduard, and Shah, Roopak. Signature verification using a âsiameseâ time delay neural network. International Journal of Pattern Recognition and Artificial Intelligence, 7(04):669–688, 1993.
- Bruna & Mallat (2013) Bruna, Joan and Mallat, Stéphane. Invariant scattering convolution networks. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 35(8):1872–1886, 2013.
- Bruna et al. (2014) Bruna, Joan, Szlam, Arthur, and LeCun, Yann. Signal recovery from pooling representations. In ICML, 2014.
- Cadieu & Olshausen (2012) Cadieu, Charles F. and Olshausen, Bruno A. Learning intermediate-level representations of form and motion from natural movies. Neural Computation, 2012.
- Goodfellow et al. (2013) Goodfellow, Ian J., Warde-Farley, David, Mirza, Mehdi, Courville, Aaron, and Bengio, Yoshua. Maxout networks. In ICML, 2013.
- Goroshin & LeCun (2013) Goroshin, Rostislav and LeCun, Yann. Saturating auto-encoders. In ICLR, 2013.
- Gregor & LeCun (2010) Gregor, Karol and LeCun, Yann. Learning fast approximations of sparse coding. In ICML’2010, 2010.
- Hadsell et al. (2006) Hadsell, Raia, Chopra, Soumit, and LeCun, Yann. Dimensionality reduction by learning an invariant mapping. In CVPR, 2006.
- Hyvärinen et al. (2004) Hyvärinen, Aapo, Karhunen, Juha, Oja, and Erkki. Independent component analysis, volume 46. John Wiley & Sons, 2004.
- Hyvärinen et al. (2003) Hyvärinen, Aapo, Hurri, Jarmo, and Väyrynen, Jaakko. Bubbles: a unifying framework for low-level statistical properties of natural image sequences. JOSA A, 20(7):1237–1252, 2003.
- Kavukcuoglu et al. (2009) Kavukcuoglu, Koray, Ranzato, MarcâAurelio, Fergus, Rob, and LeCun, Yann. Learning invariant features through topographic filter maps. In CVPR, 2009.
- Kayser et al. (2001) Kayser, Christoph, Einhauser, Wolfgang, Dummer, Olaf, Konig, Peter, and Kding, Konrad. Extracting slow subspaces from natural videos leads to complex cells. In ICANN’2001, 2001.
- Krizhevsky (2009) Krizhevsky, Alex. Learning multiple layers of features from tiny images. Master’s thesis, University of Toronto, April 2009.
- Krizhevsky et al. (2012) Krizhevsky, Alex, Sutskever, Ilya, and Hinton, Geoffrey E. Imagenet classification with deep convolutional neural networks. In NIPS, volume 1, pp. 4, 2012.
- LeCun et al. (1998) LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278–2324, 1998.
- Lies et al. (2014) Lies, Jorn-Philipp, Hafner, Ralf M, and Bethge, Matthias. Slowness and sparseness have diverging effects on complex cell learning. 10, 2014.
- Mobahi et al. (2009) Mobahi, Hossein, Collobert, Ronan, and Weston, Jason. Deep learning from temporal coherence in video. In ICML, 2009.
- Rifai et al. (2011) Rifai, Salah, Vincent, Pascal, Muller, Xavier, Galrot, Xavier, and Bengio, Yoshua. Contractive auto-encoders: Explicit invariance during feature extraction. In ICML, 2011.
- Vincent et al. (2008) Vincent, Pascal, Larochelle, Hugo, Bengio, Yoshua, and Manzagol, Pierre-Antoine. Extracting and composing robust features with denoising autoencoders. Technical report, University of Montreal, 2008.
- Wiskott & Sejnowski (2002) Wiskott, Laurenz and Sejnowski, Terrence J. Slow feature analysis: Unsupervised learning of invariances. Neural Computation, 2002.
- Zou et al. (2012) Zou, Will, Zhu, Shenghuo, Yu, Kai, and Ng, Andrew Y. Deep learning of invariant features via simulated fixations in video. In Advances in Neural Information Processing Systems, pp. 3212–3220, 2012.