Data-Efficient Learning of Feedback Policies from Image Pixels using Deep Dynamical Models
Data-efficient reinforcement learning (RL) in continuous state-action spaces using very high-dimensional observations remains a key challenge in developing fully autonomous systems. We consider a particularly important instance of this challenge, the pixels-to-torques problem, where an RL agent learns a closed-loop control policy (“torques”) from pixel information only. We introduce a data-efficient, model-based reinforcement learning algorithm that learns such a closed-loop policy directly from pixel information. The key ingredient is a deep dynamical model for learning a low-dimensional feature embedding of images jointly with a predictive model in this low-dimensional feature space. Joint learning is crucial for long-term predictions, which lie at the core of the adaptive nonlinear model predictive control strategy that we use for closed-loop control. Compared to state-of-the-art RL methods for continuous states and actions, our approach learns quickly, scales to high-dimensional state spaces, is lightweight and an important step toward fully autonomous end-to-end learning from pixels to torques.
Data-Efficient Learning of Feedback Policies from Image Pixels using Deep Dynamical Models
John-Alexander M. Assael††thanks: These authors contributed equally to this work. Department of Computing Imperial College London, UK email@example.com Niklas Wahlström11footnotemark: 1 Division of Automatic Control Linköping University, Sweden firstname.lastname@example.org Thomas B. Schön Department of Information Technology Uppsala University, Sweden email@example.com Marc Peter Deisenroth Department of Computing Imperial College London, UK firstname.lastname@example.org
The vision of fully autonomous and intelligent systems that learn by themselves has inspired artificial intelligence (AI) and robotics research for many decades. The pixels to torques problem identifies key aspects of such an autonomous system: autonomous thinking and decision making using (general-purpose) sensor measurements only, intelligent exploration and learning from mistakes. We consider the problem of efficiently learning closed-loop policies (“torques”) from pixel information end-to-end. Although, this problem falls into the general class of reinforcement learning (RL) , it is challenging for the following reasons: (1) The state-space (here defined by pixel values) is enormous (e.g., for a image, we are looking at continuous-valued dimensions); (2) In many practical applications, we need to find solutions data efficiently: When working with real systems, e.g., robots, we cannot perform millions of experiments because of time and hardware constraints.
One way of using data efficiently, and, therefore, reducing the number of experiments, is to learn predictive forward models of the underlying dynamical system, which are then used for internal simulations and policy learning. These ideas have been successfully applied to RL, control and robotics [2, 3, 4, 5, 6, 7], for instance. However, they often rely on heuristics, demonstrations or engineered low-dimensional features, and do not easily scale to data-efficient RL using pixel information only.
A common way of dealing with high-dimensional data is to learn low-dimensional feature representations. Deep learning architectures, such as deep neural networks , stacked auto-encoders [9, 10], or convolutional neural networks , are the current state-of-the-art in learning parsimonious representations of high-dimensional data. Since 2006, deep learning has produced outstanding empirical results in image, text and audio tasks .
Over the last months, there has been significant progress in the context of the pixels-to-torques problem. A first working solution was presented in 2015 , where an RL agent automatically learned to play Atari games purely based on pixel information. The key idea was to embed the high-dimensional pixel space into a lower-dimensional space using deep neural networks and apply Q-learning in this compact feature space. A potential issue with this approach is that it is not a data-efficient way of learning policies (weeks of training data are required), i.e., it will be impractical to apply it to a robotic scenario. This data inefficiency is not specific to Q-learning, but a general problem of model-free RL methods [3, 14].
To increase data efficiency, model-based RL methods aim to learn a model of the transition dynamics of the system/robot and subsequently use this model as a surrogate simulator. Recently, the idea of learning predictive models from raw images where only pixel information is available was exploited [15, 16]. The approach taken here follows the idea of Deep Dynamical Models (DDMs) : Instead of learning predictive models for images directly, a detour via a low-dimensional feature space is taken by embedding images into a lower-dimensional feature space, e.g., with a deep auto-encoder. This detour is promising since direct mappings between high-dimensional spaces require large data sets. Whereas Wahlström et al.  consider deterministic systems and nonlinear model predictive control (NMPC) techniques for online control, Watter et al.  use variational auto-encoders , local linearization, and locally linear control methods (iLQR  and AICO ).
To model the dynamical behavior of the system, the pixels of both the current and previous frame are used. Watter et al.  concatenate the input pixels to discover such features, whereas Wahlström et al.  concatenate the processed low-dimensional embeddings of the two states. The latter approach requires at least fewer parameters, which makes it a promising candidate for more data-efficient learning. Nevertheless, properties such as local linearization  can be attractive. However, the complex architecture proposed by Watter et al.  is based on very large neural network models with million parameters for learning to swing up a single pendulum. A vast number of training parameters results in higher model complexity, and, thus, decreased statistical efficiency. Hence, an excessive number of training samples, which might not be available, is required to learn the underlying system, taking several days to be trained. These properties make data-efficient learning complicated. Therefore, we propose a relatively lightweight architecture to address the pixels-to-torques problem in a data-efficient manner.
We propose a data-efficient model-based RL algorithm that addresses the pixels-to-torques problem. (1) We devise a data-efficient policy learning framework based on the DDM approach for learning predictive models for images. We use state-of-the-art optimization techniques for training the DDM. (2) Our model profits from a concatenation of low-dimensional features (instead of high-dimensional images) to model dynamical behavior, yielding – times fewer model parameters and faster training time than the complex E2C architecture . In practice, our model requires only a few hours of training, while E2C  requires days. (3) We introduce a novel training objective that encourages consistency in the latent space paving the way towards more accurate long-term predictions. Overall, we use an efficient model architecture, which can learn tasks of complex non-linear dynamics.
2 Problem Set-up and Objective
We consider an -step finite-horizon RL setting in which an agent attempts to solve a particular task by trial and error. In particular, our objective is to find a closed-loop policy , that minimizes the long-term cost , where denotes an immediate cost, is the continuous-valued system state, and are continuous control signals.
The learning agent faces the following additional two challenges: (a) The agent has no access to the true state , but perceives the environment only through high-dimensional pixel information (images); (b) A good control policy is required in only a few trials. This setting is practically relevant, e.g., when the agent is a robot that is monitored by a video camera based on which the robot has to learn to solve tasks fully autonomously. Therefore, this setting is an instance of the pixels-to-torques problem.
We solve this problem in three key steps, which will are detailed in the following sections: (a) Using a deep auto-encoder architecture we map the high-dimensional pixel information to a low-dimensional embedding/feature . (b) We combine the , and features with the control signal to learn a predictive model of the system dynamics for predicting future features . (a) and (b) form a Deep Dynamical Model (DDM) . (c) We apply an adaptive nonlinear model predictive control strategy for optimal closed-loop control and end-to-end learning from pixels to torques.
3 Learning a Deep Dynamical Model (DDM)
Our approach to solving the pixels-to-torques problem is based on a deep dynamical model (DDM), see Figure 1, which jointly (a) embeds high-dimensional images in a low-dimensional feature space via deep auto-encoders, and (b) learns a predictive forward model in this feature space, based on the work by Wahlström et al. . In particular, we consider a DDM with control signals and high-dimensional observations at time-step . We assume that the relevant properties of can be compactly represented by a feature variable . Furthermore, is the reconstructed high-dimensional measurement. The two components of the DDM, i.e., the low-dimensional feature and the predictive model, which predicts future features and observations based on past observations and control signals, are detailed in the following sections.
3.1 Predictive Forward Model
An encoder mapping high-dimensional observations onto low-dimensional features ,
A decoder mapping low-dimensional features back to high-dimensional observations , and
The predictive model , which takes as input and predicts the next latent feature .
The and functions of our DDM are neural network models performing the following transformations:
We now put these elements together to construct the DDM. The DDM architecture takes the raw images and as input and maps them to their low-dimensional features and respectively, using in (1a). These latent features are then concatenated and, together with the control signal , used to predict with in (1c). Finally, the predicted feature is passed through the decoder network , to compute the predicted image . The overall architecture is depicted in Figure 2.
The neural networks and that compose the DDM, are parameterized by and respectively. These parameters consist of the weights that perform linear transformations of the input data in each neuron.
For training the DDM in (1), we introduce a novel training objective, that encourages consistency in the latent space, paving the way toward accurate long-term predictions. More specifically, for our training objective we define the following cost functions
where is the squared deep auto-encoder reconstruction error and is the squared prediction error, both operating in image space. Note that depends on the parameters of the decoder, the predictive model and the encoder. Additionally, we introduce that enforces consistency between the latent spaces of the encoder and the prediction model . In the big-data regime, this additional penalty in latent space is not necessary, but if not much data is available, this additional term increases the data efficiency as the prediction model is forced to make predictions close to the next embedded feature . The overall training objective of the current dataset is given by
where is a parameter that controls the influence of . Finally, we train the DDM parameters by jointly minimizing the overall cost
Training jointly leads to good predictions as it facilitates the extraction and separation of the features describing the underlying dynamical system, and not only features for creating good reconstructions .
3.3 Network Architecture
The neural networks and are composed by linear layers, where each of the first are followed by Rectified Linear Unit (ReLU) activation functions . As it has been demonstrated , ReLU non-linearities allow the network to train faster than the conventional units, as evaluated on the CIFAR-10  dataset. Furthermore, similar to Watter et al. , we use Adam  to train the DDM, which is considered the state-of-the-art among the latest methods for stochastic gradient optimization. Finally, after evaluating different weight optimization methods, such as uniform and random Gaussian [27, 24], the weights of the DDM were initialized using orthogonal weight initialization , which demonstrated the most efficient training performance, leading to decoupled weights that evolve independently of each other.
4 Policy Learning
Our objective is to control the system to a state where a certain target frame without any prior knowledge of the system at hand. To accomplish this we use the DDM for learning a closed-loop policy by means of nonlinear model predictive control (NMPC).
4.1 NMPC using the DDM
NMPC finds an optimal sequence of control signals that minimizes a -step loss function, where is typically smaller than the full horizon. We choose to do the control in the low-dimensional embedded space to reduce the complexity of the control problem.
Our NMPC formulation relies on (a) a target feature and (b) the DDM that allows us to predict future features. The target feature is computed by encoding the target frame provided by the model. Further, with the DDM, future features can be predicted based on a sequence of future (and yet unknown) controls and two initial encoded features assuming that the current feature is denoted by .
Using the dynamical model, NMPC determines an optimal (open-loop) control sequence , such that the predicted features gets as close to the target feature as possible, which results in the objective
where is a cost associated with the deviation of the predicted features from the reference feature , and penalizes the amplitude of the control signals. Here, is a tuning parameter adjusting the importance of these two objectives. When the control sequence is determined, the first control is applied to the system. After observing the next feature, NMPC repeats the entire optimization and turns the overall policy into a closed-loop (feedback) control strategy.
Overall, we now have an online NMPC algorithm that, given a trained DDM, works indirectly on images by exploiting their feature representation.
4.2 Adaptive NMPC for Learning from Scratch
We will now turn over to describe how adaptive NMPC can be used together with our DDM to address the pixels-to-torques problem and to learn from scratch. At the core of our NMPC formulation lies the DDM, which is used to predict future features (and images) from a sequence of control signals. The quality of the NMPC controller is inherently bound to the prediction quality of the dynamical model, which is typical in model-based RL [14, 29, 5].
To learn models and controllers from scratch, we apply a control scheme that allows us to update the DDM as new data arrives. In particular, we use the NMPC controller in an adaptive fashion to gradually improve the model by collected data in the feedback loop without any specific prior knowledge of the system at hand. Data collection is performed in closed-loop (online NMPC), and it is divided into multiple sequential trials. After each trial, we add the data of the most recent trajectory to the data set, and the model is re-trained using all data that has been collected so far.
Simply applying the NMPC controller based on a randomly initialized model would make the closed-loop system very likely to converge to a point, which is far away from the desired reference value, due to the poor model that cannot extrapolate well to unseen states. This would in turn imply that no data is collected in unexplored regions, including the region that we are interested in. There are two solutions to this problem: either we use a probabilistic dynamical model [14, 5] to explicitly account for model uncertainty and the implied natural exploration, or we follow an explicit exploration strategy to ensure proper excitation of the system. In this paper, we follow the latter approach. In particular, we choose an -greedy exploration strategy where the optimal feedback at each time step is selected with a probability , and a random action is selected with probability .
Algorithm 1 summarizes our adaptive online NMPC scheme. We initialize the DDM with a random trial. We use the learned DDM to find an -greedy policy using predicted features within NMPC. This happens online while the collected data is added to the data set, and the DDM is updated after each trial.
5 Experimental Evaluation
In this section, we empirically assess the components of our proposed methodology for autonomous learning from high-dimensional synthetic image data, on learning the underlying dynamics of a single and a planar double pendulum. The main lines of the evaluation are: (a) the quality of the learned DDM and (b) the overall learning framework.
In both experiments, we consider the following setting: We take screenshots of a simulated pendulum system at a sampling frequency of . Each pixel is a component of the measurement and takes a continuous gray-value in the interval . The control signals are the torques applied to the system. No access to the underlying dynamics nor the state (angles and angular velocities) was available, i.e., we are dealing with a high-dimensional continuous time series. The challenge was to data-efficiently learn (a) a good dynamical model and (b) a good controller from pixel information only.
To speed up the training process, we applied PCA prior to model learning as a pre-processing step to reduce the dimensionality of the original problem. With these inputs, a -layer auto-encoder was employed, such that the dimensionality of the features is optimal to model the periodic angle of the pendulums. The features and were later passed to the -layer predictive feedforward neural network generating . Furthermore, during training, the parameter for encouraging consistent latent space predictions was set to for both experiments. While, in the adaptive NMPC, the tuning parameter that penalizes the amplitude of the control signals, was set to .
5.1 Planar Pendulum
The first experiment evaluates the performance of the DDM on a planar pendulum, assembled by 1-link robot arm with length , weight and friction coefficient .The screenshots consist of pixels, and the input dimension has been reduced to using PCA. These inputs are processed by an encoder with architecture: – ReLU – – ReLU – .
The low-dimensional features are of in order to model the periodic angle of the pendulum. To capture the dynamic properties, such as angular velocity, we concatenate two consecutive features with the control signal and pass them through the predictive model , with architecture: – ReLU – – ReLU – . Note that the dimensionality of the first layer is given by . Finally, the predicted feature , can be mapped back to using our decoder, with architecture: – ReLU – – ReLU – .
The performance of the DDM is illustrated in Figure 3 on a test data set. The top row shows the true images and the bottom row shows the DDM’s long-term predictions. The model yields a good predictive performance for both one-step ahead prediction and multiple-step ahead prediction, a consequence of (a) jointly learning predictor and auto-encoder, (b) concatenating features instead of images to model the dynamic behavior.
In Figure 4, we show the learned feature space for different pendulum angles between and . The DDM has learned to generate features that represent the angle of the pendulum, as they are mapped to a circle-like shape accounting for the wrap-around property of an angle.
Finally, in Figure 5, we report results on learning a policy that moves the pendulum from a start position to an upright target position . The reference signal was the screenshot of the pendulum in the target position. For the NMPC controller, we used a planning horizon of steps and a control penalty . For the -greedy exploration strategy we used .
Figure 5 shows the learning stages of the system, i.e., the different trials of the NMPC controller. Starting with a randomly initialized model, images were appended to the dataset in each trial. As it can be seen, starting already from the first controlled trial, the system managed to control the pendulum successfully and bring it to a position less than from the target position. This means, the solution is found very data efficiently, especially when we consider that the problem is learned from pixels information without access to the “true” state.
5.2 Planar Double Pendulum
In this experiment, a planar double pendulum is considered assembled by 2-link robot arm with length m and m respectively, weight kg and kg and friction coefficients . Torques can be applied at both joints. The screenshots consist of pixels, and the input dimension has been reduced to prior to model learning using PCA to speed up the training process. The encoder architecture is: – ReLU – – ReLU – , and the decoder vice versa. The low-dimensional embeddings and the architecture of the predictive model was: – ReLU – – ReLU – .
The predictive performance of the DDM is shown in Figure 3 on a test data set. The performance of the controller is depicted in Figure 5. We used trials with the downward initial position and upward target for the angle of both inner and outer pendulums. The figure shows the error after each trial (1000 frames) and clearly indicates that after three controlled trials a good solution is found, which brings both pendulums within a range to the target angles.
Despite the high complexity of the dynamical system, our learning framework manages to successfully control both pendulums after the third trial in nearly all cases.
5.3 Comparison with State-of-the-Art
The same experiments were executed employing PILCO , a state of the art policy search method, under the following settings: (a) PILCO has access to the true state, i.e., the angle and angular velocity ; (b) A deep auto-encoder is used to learn two-dimensional features from images, which are used by PILCO for policy learning. In the first setting (a) PILCO managed to successfully reach the target after the second and the third trial in the two experiments, respectively. However, in setting (b), PILCO did not manage to learn anything meaningful at all. The reason why PILCO could not learn on auto-encoder features is that these features were only trained to minimize the reconstruction error. However, the auto-encoder did not attempt to map similar images to similar features, which led to zig-zagging around in feature space (instead of following a smooth manifold as in Figure 4), making the model learning part in feature space incredibly hard .
We modeled and controlled equally complex models with E2C , but at the same time our DDM requires fewer neural network parameters if we use the same PCA pre-processing step within E2C. The reason lies in our efficient processing of the dynamics of the model in the feature space instead of the image space. This number increases up to fewer parameters than E2C without the PCA pre-processing step.
The number of parameters can be directly translated to reduced training time, and increased data efficiency. Employing the adaptive model predictive control, our proposed DDM model requires significantly less data samples, as it efficiently focuses on learning the latent space towards the reference target state. Furthermore, the control performance of our model is gradually improved in respect to the number of trials. As proved by our experimental evaluation we can successfully control a complex dynamical system, such as the planar double pendulum, with less than samples. This adaptive learning approach can be essential in problems with time and hardware constraints.
We proposed a data-efficient model-based RL algorithm that learns closed-loop policies in continuous state and action spaces directly from pixel information. The key components of our solution are (a) a deep dynamical model (DDM) that is used for long-term predictions via a compact feature space, (b) a novel training objective that encourages consistency in the latent space, paving the way toward more accurate long-term predictions, and (c) an NMPC controller that uses the predictions of the DDM to determine optimal actions on the fly without the need for value function estimation. For the success of this RL algorithm it is crucial that the DDM learns the feature mapping and the predictive model in feature space jointly to capture dynamical behavior for high-quality long-term predictions. Compared to state-of-the-art RL our algorithm learns fairly quickly, scales to high-dimensional state spaces and facilitates learning from pixels to torques.
We thank Roberto Calandra for valuable discussions in the early stages of the project. The Tesla K40 used for this research was donated by the NVIDIA Corporation.
- Sutton and Barto  R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction, volume 1. MIT Press, 1998.
- Schmidhuber  J. Schmidhuber. An on-line algorithm for dynamic reinforcement learning and planning in reactive environments. In International Joint Conference on Neural Networks (IJCNN), pages 253–258. IEEE, 1990.
- Atkeson and Schaal  C. G. Atkeson and S. Schaal. Learning tasks from a single demonstration. In Proceedings of the International Conference on Robotics and Automation (ICRA). IEEE, 1997.
- Bagnell and Schneider  J. A. Bagnell and J. G. Schneider. Autonomous helicopter control using reinforcement learning policy search methods. In Proceedings of the IEEE International Conference on Robotics and Automation, volume 2, pages 1615–1620, 2001.
- Deisenroth et al.  M. P. Deisenroth, D. Fox, and C. E. Rasmussen. Gaussian processes for data-efficient learning in robotics and control. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 37(2):408–423, 2015.
- Pan and Theodorou  Y. Pan and E. Theodorou. Probabilistic differential dynamic programming. In Advances in Neural Information Processing Systems, pages 1907–1915, 2014.
- Levine et al.  S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. arXiv preprint arXiv:1504.00702, 2015.
- Hinton and Salakhutdinov  G. Hinton and R. Salakhutdinov. Reducing the dimensionality of data with neural networks. Science, 313(5786):504–507, 2006.
- Bengio et al.  Y. Bengio, P. Lamblin, D. Popovici, and H. Larochelle. Greedy layer-wise training of deep networks. In Advances in Neural Information Processing Systems (NIPS), pages 153–160. MIT Press, 2007.
- Vincent et al.  P. Vincent, L. Hugo, Y. Bengio, and P.-A. Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning (ICML), pages 1096–1103. ACM, 2008. ISBN 978-1-60558-205-4.
- LeCun et al.  Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998.
- Schmidhuber  J. Schmidhuber. Deep learning in neural networks: An overview. Technical Report IDSIA-03-14 / arXiv:1404.7828v1 [cs.NE], The Swiss AI Lab IDSIA, 2014.
- Mnih et al.  V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529–533, 2015.
- Schneider  J. G. Schneider. Exploiting model uncertainty estimates for safe dynamic control learning. In Advances in Neural Information Processing Systems (NIPS). 1997.
- Wahlström et al. [2015a] N. Wahlström, T. B. Schön, and M. P. Deisenroth. From pixels to torques: Policy learning with deep dynamical models. arXiv preprint arXiv:1502.02251, 2015a.
- Watter et al.  M. Watter, J. T. Springenberg, J. Boedecker, and M. A. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. arXiv preprint arXiv:1506.07365, 2015.
- Wahlström et al. [2015b] N. Wahlström, T. B. Schön, and M. P. Deisenroth. Learning deep dynamical models from image pixels. In IFAC Symposium on System Identification (SYSID), 2015b.
- Jimenez Rezende et al.  D. Jimenez Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and variational inference in deep latent Gaussian models. In International Conference on Machine Learning (ICML), June 2014.
- Todorov and Li  E. Todorov and W. Li. A generalized iterative LQG method for locally-optimal feedback control of constrained nonlinear stochastic systems. In American Control Conference, pages 300–306. IEEE, 2005.
- Toussaint  M. Toussaint. Robot trajectory optimization using approximate inference. In International Conference on Machine Learning (ICML), Montreal, QC, Canada, June 2009.
- Rumelhart et al.  D. E. Rumelhart, G. E. Hinton, and R. J. Williams. Learning internal representations by error propagation. In D. E. Rumelhart and J. L. McClelland, editors, Parallel Distributed Processing, volume 1, pages 318–362. MIT Press, 1986.
- Bengio  Y. Bengio. Learning deep architectures for AI. Foundations and trends in Machine Learning, 2(1):1–127, 2009.
- Nair and Hinton  V. Nair and G. E. Hinton. Rectified linear units improve restricted Boltzmann machines. In International Conference on Machine Learning (ICML), pages 807–814, 2010.
- Krizhevsky et al.  A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS), pages 1097–1105, 2012.
- Krizhevsky  A. Krizhevsky. Learning multiple layers of features from tiny images. Master’s thesis, Computer Science Department, University of Toronto, 2009.
- Kingma and Ba  D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
- Glorot and Bengio  X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In Artificial Intelligence and Statistics (AISTATS), pages 249–256, 2010.
- Saxe et al.  A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120, 2013.
- Schaal  S. Schaal. Learning from demonstration. In Advances in Neural Information Processing Systems (NIPS), pages 1040–1046. 1997.
- Deisenroth and Rasmussen  M. P. Deisenroth and C. E. Rasmussen. PILCO: A model-based and data-efficient approach to policy search. In International Conference on Machine Learning (ICML), 2011.