Deep Equilibrium Models

Deep Equilibrium Models

Shaojie Bai
Carnegie Mellon University
&J. Zico Kolter
Carnegie Mellon University
Bosch Center for AI
&Vladlen Koltun
Intel Labs

We present a new approach to modeling sequential data: the deep equilibrium model (DEQ). Motivated by an observation that the hidden layers of many existing deep sequence models converge towards some fixed point, we propose the DEQ approach that directly finds these equilibrium points via root-finding. Such a method is equivalent to running an infinite depth (weight-tied) feedforward network, but has the notable advantage that we can analytically backpropagate through the equilibrium point using implicit differentiation. Using this approach, training and prediction in these networks require only constant memory, regardless of the effective “depth” of the network. We demonstrate how DEQs can be applied to two state-of-the-art deep sequence models: self-attention transformers and trellis networks. On large-scale language modeling tasks, such as the WikiText-103 benchmark, we show that DEQs 1) often improve performance over these state-of-the-art models (for similar parameter counts); 2) have similar computational requirements as existing models; and 3) vastly reduce memory consumption (often the bottleneck for training large sequence models), demonstrating an up-to 88% memory reduction in our experiments. The code is available at

1 Introduction

Most modern feedforward deep networks are built on the core concept of layers. In the forward pass, each network consists of a stack of some transformations, where is the depth of the network. To update these networks, the backward passes rely on backpropagating through the same layers via the chain rule, which typically necessitates that we store the intermediate values of these layers. The value for is usually a hyperparameter and is picked by model designers (e.g., ResNet-101 (He et al., 2016)). Among the many applications of deep networks, sequence modeling has witnesses continuous advances in deep architectures. Specifically, while recurrent networks have long been the dominant model for sequences (Elman, 1990; Hochreiter and Schmidhuber, 1997; Cho et al., 2014; Merity et al., 2018b), deep feedforward architectures based on temporal convolutions (Waibel et al., 1989; van den Oord et al., 2016; Bai et al., 2018) and self-attention (Vaswani et al., 2017; Dai et al., 2019; Child et al., 2019) have (re-)emerged to claim superior performance on a variety of sequence prediction tasks.

In very general terms, a deep feedforward sequence model can be written as the following iteration:


where is the layer index; is the hidden sequence of length at layer ; is the input sequence (i.e., we are choosing to explicitly model skip connections, for reasons we explain later); and is some nonlinear transformation which typically enforces causality (i.e., future time points cannot influence past ones). Our paper derives its motivation from surprising recent works that employ the same transformation in each layer (known as weight tying, with ) and still achieve results competitive with the state-of-the-art (Dehghani et al., 2019; Bai et al., 2019). This raises an interesting question: If the same transformation is applied at each layer of a deep network, what is the limit of this process, and how do we model it?

In this paper, we propose a new approach to “deep” modeling that addresses this question. Specifically, we introduce the deep equilibirum model (DEQ), a method that directly computes the fixed point of a nonlinear transformation, i.e., the solution to the nonlinear system


This solution corresponds to the eventual hidden layer values of an infinite depth network. But instead of finding this value by iterating the model, we propose to directly (and in practice, more quickly) find the equilibrium point via any black-box root-finding method. Importantly, we show that DEQ can directly differentiate through the fixed point equations via implicit differentation, which does not require storing any intermediate activation values. In other words, we can backpropagate through the infinite-depth network while using only constant memory, equivalent to a single layer’s activations.

After developing the generic DEQ approach, we study in detail the instantiation of DEQ via two feedforward sequence models: trellis networks (weight-tied temporal convolutions) (Bai et al., 2019) and memory-augmented universal transformers (weight-tied multi-head self-attention) (Dehghani et al., 2019; Dai et al., 2019), both of which have obtained state-of-the-art performance (SOTA) on various sequence tasks. We show how both the forward and the backward passes can be implemented efficiently via quasi-Newton methods. Finally, we demonstrate via experiments on large-scale high-dimensional sequence modeling benchmarks (e.g., WikiText-103 language modeling) that, despite only needing constant memory, DEQ can attain modeling accuracy on par with (or even slightly better than) corresponding layer-based networks. We believe that DEQ offers a novel perspective on the analysis of sequential data.

2 Background

Deep Sequence Models.

Given an input sequence , where (e.g., a word embedding) and is the sequence length, we define a sequence model as any function that produces output that satisfies the causality constraint: depends only on and not on any element of . Most recent progress on autoregressive sequence tasks has been based on deep learning, where three major families of sequence models stand out. Recurrent networks (RNNs) (Elman, 1990; Werbos, 1990) as well as their variants such as LSTM (Hochreiter and Schmidhuber, 1997) are universally applied and optimized in a variety of time-series tasks (Bradbury et al., 2017; Gal and Ghahramani, 2016; Merity et al., 2018b). Alternatively, prior work has shown that deeply stacked temporal convolutions (Waibel et al., 1989; van den Oord et al., 2016; Dauphin et al., 2017; Bai et al., 2018) can achieve competitive results, especially on long sequences. Finally, the self-attention transformer architecture (Vaswani et al., 2017; Dai et al., 2019) has also achieved SOTA on several NLP benchmarks (Devlin et al., 2019; Child et al., 2019). Efforts have also been devoted to drawing deeper connections among the three model families. (Bai et al., 2019) studies the underlying relationship between RNNs and ConvNets, proposing the trellis network that provably combines the benefits of both families. (Dehghani et al., 2019) introduces the recurrently-stacked universal transformer and shows its effectiveness on textual understanding and generation.

Memory-efficient Deep Networks.

An important factor that limits the development of high-capacity networks is limited memory on hardware devices used for training. To address this issue, (Chen et al., 2016) proposes gradient checkpointing that reduces an -layer network’s memory requirement to at the cost of extra forward passes (i.e., extra computations). Alternatively, (Gomez et al., 2017) proposes RevNet, where each layer’s activations can be reconstructed from the next layer to relax the memory cost. In contrast, DEQs reduce memory consumption to a constant (i.e., independent of network “depth”) by directly differentiating through the equilibrium point and thus circumventing the construction and maintenance of “layers”.

Continuous View of Deep Networks.

Some prior works have studied continuous views of deep networks. (Scellier and Bengio, 2017) proposes a biologically inspired equilibrium propagation framework for an energy-based model whose prediction is the fixed-point of the energy dynamics at its local minimum. (Haber and Ruthotto, 2017; Chen et al., 2018) model deep ResNets by black-box ODE solvers in forward and backward passes (as if the network has smaller “layer steps”) given the start- and end-points of a dynamical system. For deep sequence models, (Simard et al., 1989; Miller and Hardt, 2018) consider the RNN as a dynamical system and investigate its stability properties.

Our work takes a further step in the direction of the aforementioned areas. While some of the aforementioned work has primarily focused on the analysis of residual architectures or small symmetric-weight energy-based models, our work is not predicated on any specific type of interlayer transformation. We show that DEQs can be easily instantiated via two very different sequence learning architectures. More fundamentally, unlike ODE-based methods, which use the adjoint system to backpropagate through the entire latent trajectory, the DEQ model directly solves for sequence-level equilibria via a quasi-Newton method and backpropagates directly through this fixed point, without regard for the solution path that brought it there. Moreover, while ODE-based models (Haber and Ruthotto, 2017; Chen et al., 2018) were verified on numerical experiments and MNIST classification, computation and numerical stability issues challenge their application to large-scale problems. In comparison, we demonstrate the applicability of DEQ on realistic high-dimensional sequence tasks with competitive performance, while enjoying similar constant-memory benefits as (Chen et al., 2018).

Implicit layers in deep learning.

The DEQ model can be viewed as an infinitely deep weight-tied network, but interestingly can also be viewed as a single-layer network, with the caveat that the layer is defined implicitly, rather than explicitly. That is, the output of the layer, is defined as the value which solves some non-linear equation, in this case . There has been a growing interest in implicit layers in recent years (Niculae et al., 2018; Amos and Kolter, 2017; Niculae et al., 2018; Wang et al., 2019), but the precise formulation of the DEQ is quite different, and our current models represent the largest-scale practical application of implicit layers in deep learning of which we are aware. Concurrent work (El Ghaoui et al., 2019) also looks at such implicit layers in a broad sense, demonstrating similar representation capacity as we discuss in this paper, and focuses on training small models via Lagrangian methods; a combination of these approaches with our implicit-differentiation-based methods is a promising avenue for future work.

3 The Deep Equilibrium Sequence Model

We broadly consider the class of weight-tied deep sequence models (with passthrough connections from the input to each layer), which consist of the update


We note that this model encapsulates classes such as the trellis network (Bai et al., 2019) and the universal transformer (Dehghani et al., 2019) (which is typically not written with passthrough connections, but this is a trivial modification). Such weight-tying is generally considered to come with four major benefits: 1) it acts as a form of regularization that stabilizes training and supports generalization; 2) it significantly reduces the model size; 3) it is trivial to show that any deep network can be represented by a weight-tied deep network of equal depth and only a linear increase in width (see Appendix C); and 4) the network can be unrolled to any depth, typically with improved feature abstractions as depth increases (Bai et al., 2019; Dehghani et al., 2019). However, in practice almost all such models (and deep nets in general) are stacked, trained and evaluated by unrolling a pre-determined, fixed number of layers. One reason is the limited memory on training hardware: the models need to store intermediate hidden units for backpropagation and thus cannot be trained beyond a certain depth that depends on the available memory.

In principle, the network could have infinite depth. This is attained in the limit of unrolling a weight-tied model for an ever higher number of layers. What is the limit of this process? In practice, for certain classes of (discussed later), we hypothesize and observe that such weight-tied models tend to converge to a fixed point as depth increases towards infinity (see Appendix D for empirical evidence). In other words, as each layer refines the previous one by combining temporal features across the sequence, increasing depth towards infinity brings “diminishing returns”: each additional layer has a smaller and smaller contribution until the network reaches an equilibrium:


3.1 The Deep Equilibrium Approach

We introduce the deep equilibrium model (DEQ) which, instead of iteratively stacking , directly solves for and differentiates through the equilibrium state.

3.1.1 Forward Pass

Unlike a conventional network where the output is the activations from the layer, the output of a DEQ is the equilibrium point itself. Therefore, the forward evaluation could be any procedure that solves for this equilibrium point. Conventional deep sequence networks, if they converge to an equilibrium, can be considered a form of fixed-point iterations:


One can alternatively use other methods that provide faster convergence guarantees. For notational convenience, we define and re-write Eq. (4) as: . The equilibrium state is thus the root of , which we can solve more easily with Newton’s method or quasi-Newton methods (e.g., Broyden’s method (Broyden, 1965)):


where is the Jacobian inverse (or its low-rank approximation) at , and is the step size. But generally, one can exploit any black-box root-finding algorithm to solve for the equilibrium point in the forward pass, given an initial estimate (which we set to ):

3.1.2 Backward Pass

A major problem with using a black-box is that we are no longer able to rely on explicit backpropagation through the exact operations in the forward pass. While one can certainly fix an algorithm (say Newton’s method) to obtain the equilibrium, and then store and backpropagate through all the Newton iterations, we provide below an alternative procedure that is much simpler, requires constant memory, and assumes no knowledge of the black-box .

Theorem 1.

(Gradient of the Equilibrium Model)  Let be an equilibrium hidden sequence with length and dimensionality , and the ground-truth (target) sequence. Let be any differentiable function and let be a loss function (where are applied in vectorized manner) that computes


Then the loss gradient w.r.t. (for instance, or ) is


where is the inverse Jacobian of evaluated at .

The proof is provided in Appendix A. The insight provided by Theorem 1 is at the core of our method and its various benefits. Importantly, the backward gradient through the “infinite” stacking can be represented as one step of matrix multiplication that involves the Jacobian at equlibrium. For instance, an SGD update step on model parameters would be


Note that this result is independent of the root-finding algorithm we choose or the internal structure of the transformation , and thus does not require any storage of the intermediate hidden states, which is necessary for backpropagation in conventional deep networks.

3.1.3 Accelerating DEQ by Approximating the Inverse Jacobian

One challenge of enforcing the forward and backward passes described in Sections 3.1.1 and 3.1.2 is the cost of computing the exact inverse Jacobian at every intermediate Newton iteration. We propose to address this using Broyden’s method (Broyden, 1965), a quasi-Newton approach that makes low-rank updates to approximate via the Sherman-Morrison formula (Sherman and Morrison, 1950):


where and . Initially, we set and the Broyden iterations are stopped when either the norm of falls below a tolerance or when the maximum number of iterations is reached. This lets us avoid the cubic cost induced by the inverse operation.

A similar idea can be used for the backward pass as well. Specifically, to compute in Theorem 1, we can alternatively solve the linear system


where the first term (a vector-Jacobian product) can be efficiently computed via autograd packages (e.g., PyTorch) for any , without explicitly writing out the Jacobian matrix. Such linear systems can generally be solved by any indirect methods that leverage fast matrix-vector products; we thus propose to also rely on Broyden’s method (other indirect methods would also suffice) to solve for Eq. (11) and directly backpropagate through the equilibrium by Theorem 1 in the backward pass.

3.2 Properties of Deep Equilibrium Models

Section 3.1 develops a sequence model that, while still based on the deep learning philosophy, is quite different from other approaches in the field, as its output is agnostic to the choice of the algorithm in the forward pass. We now discuss some implications of the DEQ approach.

Memory Cost of DEQ.

An important benefit of DEQ is its extreme memory efficiency. As outlined in Section 3.1.3, since we are able to use any root-finding algorithm for both the forward and backward passes (e.g., Broyden’s method (Broyden, 1965)), a DEQ only needs to store (the equilibrium sequence), (input-related, layer-independent variables), and for the backward pass. Note that as we only need the vector-Jacobian product (with dimension , where is the minibatch size) in Eq. (11), we never need to explicitly construct the Jacobian , which could be prohibitively large on long and high-dimensional sequences (with dimension ). Compared to other deep networks, DEQs therefore offer a constant-memory alternative that enables models that previously required multiple GPUs and other implementation-based techniques (e.g., half-precision or gradient checkpointing (Chen et al., 2016; Child et al., 2019)) to fit easily into a single GPU.

The Choice of .

Our analysis in Sections 3.1.1, 3.1.2, and 3.1.3 is independent of the choice of , and the same kind of memory benefit is present regardless of the type of . However, to find the equilibrium in a reliable and efficient manner, generally needs to be stable and constrained. The two instantiations we provide in Section 4 are examples of stable transformations. (The gated activation in TrellisNet and layer normalization in the transformer constrain the output ranges.)

Stacking the DEQ?

A natural question arises: if one DEQ is good, can we get additional benefits by “stacking” DEQs (with potentially different classes of transformations)? The answer, somewhat surprisingly, is no, as evidenced by the following theorem, which is proved in Appendix B. The theorem essentially shows that stacking multiple DEQs does not create extra representational power over a single DEQ.

Theorem 2.

(Universality of “single-layer” DEQs.) Let be the input sequence, and let be sets of parameters for stable transformations and , respectively. Then there exists , where , s.t.


where denotes the last feature dimensions of .

4 Instantiations of DEQ

While the forward and backward analyses of DEQ do not depend on the internal structure of , in this section we briefly highlight two examples of as specific instantiations of DEQ. Both models (TrellisNet (Bai et al., 2019) and self-attention (Vaswani et al., 2017; Dehghani et al., 2019)) achieve state-of-the-art results on various sequence modeling benchmarks. Importantly, through these two very different models and their properties, we illustrate the compatibility of the DEQ approach with all three major families of existing deep sequence networks: transformers, RNNs, and temporal convolutional networks (TCNs).

Trellis Networks.

We briefly introduce the trellis network (TrellisNet) here and refer interested readers to (Bai et al., 2019) for a detailed description. Generally, TrellisNet is a TCN with two modifications. First, a linear transformation of the original input sequence is injected to the convolutional outputs at all layers. Second, the convolutional kernel weights are tied across the depth of the network (i.e., TrellisNet is a weight-tied TCN). Thus we can write TrellisNet with convolutional kernel size , dilation , and nonlinearity in DEQ form as

where is typically: 1) the last elements of the previous sequence’s output (if using history padding (Bai et al., 2019)); or 2) simply zero-padding. means concatenation along the temporal dimension. Following (Bai et al., 2019), we use the LSTM gated activation for .

Weight-tied Transformers.

At a high level, multi-head self-attention transformers (Vaswani et al., 2017) are very different from most deep networks. Instead of convolutions or recurrence, a self-attention layer maps the input into (query), (key), and (value) and computes the attention score between time-steps and as . This attention score is then normalized via softmax and multiplied with the sequence to produce the output. Since the transformer is order-invariant, prior work proposed to inject positional embeddings (PE) (Vaswani et al., 2017; Dai et al., 2019) to the self-attention operation. Following this design, (Dehghani et al., 2019) further proposed the universal transformer, which “recurrently stacks” the transformer’s self-attention and transition function block through a number of layers. Referring readers to (Vaswani et al., 2017; Dai et al., 2019; Dehghani et al., 2019) for more details, we write a weight-tied transformer in the DEQ form as

where produces the for the multi-head self-attention, and stands for layer normalization (Ba et al., 2016). Note that we add input injection to in addition to the positional embedding and initialize with . Following prior work (Vaswani et al., 2017; Devlin et al., 2019; Dai et al., 2019; Dehghani et al., 2019), we use a 2-layer positionwise feedforward residual block for . In our implementation, we use the memory-augmented transformer proposed by (Dai et al., 2019), where we feed (i.e., with history padding of length ) and relative positional embedding to the self-attention operation.

(a) A simple illustration of solving for an equilibrium point in 2D.
(b) A deep equilibrium model operates with significantly less memory than conventional deep nets due to an analytical backward pass.
Figure 1: Comparison of the DEQ with conventional weight-tied deep networks.

Figure 1 provides a comparison between conventional weight-tied deep networks (such as TrellisNet and weight-tied transformers) and the DEQ approach, highlighting the constant memory requirements of the latter (since only model parameters and input/output need to be stored).

5 Experiments

We evaluate DEQ on both synthetic stress tests and realistic large-scale language modeling tasks (where complex long-term temporal dependencies are involved). We use the two aforementioned instantiations of in DEQ. On both WikiText-103 (Merity et al., 2017) (which contains M words and a vocabulary size of K) and the smaller Penn Treebank corpus (where stronger regularizations are needed for conventional deep nets) for word-level language modeling, we show that DEQ achieves competitive performance even when compared to SOTA methods (of the same model size, both weight-tied and unweight-tied ones) while using significantly less memory. We provide a more detailed introduction of the tasks and datasets in Appendix F.


Both instantiations of DEQ use Broyden’s method (Broyden, 1965) to avoid direct computation of the inverse Jacobian, as described in Section 3.1.3. We note that the use of DEQ implicitly introduces a new “hyperparameter”— the stopping criterion for Broyden iterations. During training, we set this tolerance of forward and backward passes to and , respectively. At inference, we relax the tolerance to (or we can use a smaller maximum iteration limit for Broyden’s method; see discussions later). For the DEQ-TrellisNet instantiation, we roughly follow the settings of (Bai et al., 2019). For DEQ-Transformers, we employ the relative positional embedding proposed by (Dai et al., 2019), with sequences of length 150 at both training and inference on the WikiText-103 dataset. Implementations of DEQ can be found at

5.1 Copy Memory Task

Models (Size)
DEQ-Transformer (ours) (14K) TCN (Bai et al., 2018) (16K) LSTM (Hochreiter and Schmidhuber, 1997) (14K) GRU (Cho et al., 2014) (14K)
Copy Memory =400 Loss 3.5e-6 2.7e-5 0.0501 0.0491
Table 1: DEQ achieves strong performance on the long-range copy-memory task.

The goal of the copy memory task is simple: to explicitly test a sequence model’s ability to exactly memorize elements across a long period of time (see Appendix F for a more detailed specification). As shown in Table 1, DEQ demonstrates good memory retention over relatively long sequences (), with substantially better results than recurrent architectures such as LSTM/GRU (consistent with the findings in (Bai et al., 2018)).

5.2 Large-Scale Language Modeling

One issue encountered in prior works that take a continuous view of deep networks (Chen et al., 2018; Haber and Ruthotto, 2017) is the challenge of scaling these approaches to real, high-dimensional, large-scale datasets. In this subsection, we evaluate the DEQ approach on some large-scale language datasets and investigate its effectiveness as a practical “implicit-depth” sequence model.

Word-level Language Modeling w/ Penn Treebank (PTB)
Model # Params Non-embedding Test perplexity Memory
model size
Variational LSTM (Gal and Ghahramani, 2016) 66M - 73.4 -
NAS Cell (Zoph and Le, 2017) 54M - 62.4 -
NAS (w/ black-box hyperparameter tuner) (Melis et al., 2018) 24M 20M 59.7 -
AWD-LSTM (Merity et al., 2018b) 24M 20M 58.8 -
DARTS architecture search (second order) (Liu et al., 2019) 23M 20M 55.7 -
60-layer TrellisNet (w/ auxiliary loss, w/o MoS) (Bai et al., 2019) 24M 20M 57.0 8.5GB
DEQ-TrellisNet (ours) 24M 20M 57.1 1.2GB
Table 2: DEQ achieves competitive performance on word-level Penn Treebank language modeling (on par with SOTA results, without fine-tuning steps (Merity et al., 2018b)). The memory footprints are benchmarked (for fairness) on input sequence length 150 and batch size 15, which does not reflect the actual hyperparameters used; the values also do not include the memory for word embeddings.
Performance on Penn Treebank.

Following the set of hyperparameters used by (Bai et al., 2019) for TrellisNet, we evaluate the DEQ-TrellisNet instantiation on word-level language modeling with the PTB corpus. Note that without an explicit notion of “layer”, we do not add auxiliary losses, as was done in (Bai et al., 2019). As shown in Table 2, when trained from scratch, the DEQ-TrellisNet achieves a test perplexity on par with the original deeply supervised TrellisNet.

Word-level Language Modeling w/ WikiText-103 (WT103)
Model # Params Non-Embedding Test perplexity Memory
Model Size
Generic TCN (Bai et al., 2018) 150M 34M 45.2 -
Gated Linear ConvNet (Dauphin et al., 2017) 230M - 37.2 -
AWD-QRNN (Merity et al., 2018a) 159M 51M 33.0 7.1GB
Relational Memory Core (Santoro et al., 2018) 195M 60M 31.6 -
Transformer-XL (X-large, adaptive embed., on TPU) (Dai et al., 2019) 257M 224M 18.7 12.0GB
70-layer TrellisNet (+ auxiliary loss, etc.) (Bai et al., 2019) 180M 45M 29.2 24.7GB
70-layer TrellisNet with gradient checkpointing 180M 45M 29.2 5.2GB
DEQ-TrellisNet (ours) 180M 45M 29.0 3.3GB
Transformer-XL (medium, 16 layers) 165M 44M 24.3 8.5GB
DEQ-Transformer (medium, ours). 172M 43M 24.7 2.7GB
Transformer-XL (medium, 18 layers, adaptive embed.) 110M 72M 23.7 9.0GB
DEQ-Transformer (medium, adaptive embed., ours) 110M 70M 24.0 3.9GB
Transformer-XL (small, 4 layers) 139M 4.9M 35.8 4.8GB
Transformer-XL (small, weight-tied 16 layers) 138M 4.5M 34.9 6.8GB
DEQ-Transformer (small, ours). 138M 4.5M 32.4 1.1GB
Table 3: DEQ-based models are competitive with SOTA deep networks of the same model size on the WikiText-103 corpus, with significantly less memory. See Table 2 for more details on the memory benchmarking. Transformer-XL models are not weight-tied, unless specified otherwise.
Performance on WikiText-103.

On the much larger scale WT103 corpus (about 100x larger than PTB), the DEQ-TrellisNet achieves better test perplexity than the original deep TrellisNet. For the Transformer instantiation, we follow the design of the Transformer-XL model (Dai et al., 2019). We specifically compare to a “medium” Transformer-XL model (the largest released model that can fit on GPUs) and a “small” Transformer-XL model, while noting that the largest Transformer-XL network has massive memory requirements (due in part to very wide hidden features, batch sizes, and training-time sequence lengths, which would not be decreased by a DEQ) and can only be trained on TPUs (Dai et al., 2019). In Table 3, we show that the DEQs yield competitive performance (on par with the SOTA approaches on similar model sizes, while outperforming many prior results such as (Bradbury et al., 2017)), while consuming much less memory during training, as discussed below.

Memory Footprint of DEQ.

For conventional deep networks with layers, the training memory complexity is since all intermediate activations are stored for backpropagation. In comparison, DEQs have an (i.e., constant) memory footprint due to the root-finding formulation. We benchmark the reduced memory consumption in the last column of Tables 2 and 3, with controlled sequence lengths and batch sizes for fairness. On both instantiations, the DEQ approach leads to an over 80% (up to 88%) reduction in memory consumption by the model (excluding word embeddings, which are orthogonal to the comparison here). Moreover, we empirically verify (using a deep 70-layer TrellisNet) that DEQ consumes even less memory than gradient checkpointing (Chen et al., 2016), a popular technique that reduces the memory required to train a layer-based model to . Note that the DEQ’s memory footprint remains competitive even when compared with baselines that are not weight-tied (a reduction of over 60%), with similar or better accuracy.

Figure 2: Left: number of Broyden iterations in forward and backward passes gradually grows with epochs. Right: DEQ-Transformer finds the equilibrium in a stable and efficient manner (whereas the deep transformer could oscillate around the fixed point, even when one exists).
Initialization of DEQ.

To train DEQ models, it is critical to ensure that the model is stable, such that the equilibrium state can be reliably approximated via quasi-Newton methods. While we found that the most commonly used initialization schemes with small values (around 0) suffice, it is generally important to make sure that DEQ starts with a small operator norm in the weight matrices. For both DEQ-TrellisNet and DEQ-Transformer, we observe that they are not sensitive to any specific initialization scheme since non-linearities such as / and LayerNorm also help make contractive (and stable). We initialize the parameters of by sampling from .

Convergence to Equilibrium.

The deep equilibrium model does not have “layers”. One factor that affects computation time in DEQs is the number of Broyden iterations in forward/backward passes, where each forward Broyden step evaluates once, and a backward step computes a vector-Jacobian product. We find that in general the number of Broyden iterations gradually increases with training epochs (Figure 2, left, where the -axis is computed by ), an observation similar to the one reported for training Neural ODEs (Chen et al., 2018). One factor contributing to this phenomenon could be that the training pushes the operator norm of to larger values, making the fixed point harder to solve. Meanwhile, the backward pass requires much fewer iterations than the forward, primarily due to the simplicity of the linear system in Eq. (11). Regarding the convergence to equlibrium, we find that DEQs can almost always converge to the sequence-level fixed point, much more efficiently than original weight-tied transformers (Figure 2, right). Note that after 12 epochs, deeply stacked self-attention tends to oscillate around the fixed point, while DEQs exhibit stable convergence with the quasi-Newton method.

Broyden Iterations and the Runtime of DEQ.

Unlike conventional deep networks that come with a fixed number of layers, the runtime of DEQs depend strongly on the number of Broyden steps to reach the equilibrium. Therefore, it’s challenging to fairly compare the runtimes of implicit-depth models like DEQ with those of their corresponding weight-tied deep networks (e.g., using higher depth necessarily takes longer to run). Ideally, the values of should be as small as possible so as to ensure that the analytical gradients from Theorem 1 are accurate. However, we empirically observe that using a higher or a lower iteration limit allows the DEQ to be trained and evaluated much faster with only a small degradation in performance. For instance, generally we find or an iteration limit of 30 (on sequence length 75) to be sufficient for competitive performance. Figure 3 visualizes this tradeoff on DEQ-Transformer. Note that accuracy quickly diverges when tolerance is too large (Figure 3, left), suggesting that a poor estimate of the equilibrium can hurt DEQ training. Table 4 provides approximate runtimes for competitive-accuracy DEQs on WikiText-103. DEQs are typically slower than layer-based deep networks.

Additional empirical remarks as well as training tips are provided in Appendix E.

DEQ / 18-layer Transformer DEQ / 70-layer TrellisNet
Training Inference Training Inference
2.82 1.76 2.40 1.64
Table 4: Runtime ratios between DEQs and corresponding deep networks at training and inference ( implies DEQ is slower). The ratios are benchmarked on WikiText-103.
Figure 3: DEQ can be accelerated by leveraging higher tolerance (left) or a lower Broyden iteration limit (right). In general, poor estimates of the equilibrium can hurt DEQ performances.

6 Conclusion

Deep networks have predominantly taken the form of stacks of layers. We propose the deep equilibrium approach (DEQ), which models temporal data by directly solving for the sequence-level fixed point and optimizing this equilibrium for better representations. DEQ needs only memory at training time, is agnostic to the choice of the root solver in the forward pass, and is sufficiently versatile to subsume drastically different architectural choices. Our experiments have shown that DEQs have good temporal memory retention, are able to scale to realistic, large-scale sequence tasks, and perform competitively with SOTA methods. Overall, we believe that the DEQ approach provides an interesting and practical new perspective on designing and optimizing sequence models.


  • Al-Rfou et al. [2018] Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. arXiv:1808.04444, 2018.
  • Amos and Kolter [2017] Brandon Amos and J Zico Kolter. OptNet: Differentiable optimization as a layer in neural networks. In International Conference on Machine Learning (ICML), 2017.
  • Arjovsky et al. [2016] Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference on Machine Learning (ICML), 2016.
  • Ba et al. [2016] Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv:1607.06450, 2016.
  • Baevski and Auli [2019] Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. In International Conference on Learning Representations (ICLR), 2019.
  • Bai et al. [2018] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv:1803.01271, 2018.
  • Bai et al. [2019] Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. Trellis networks for sequence modeling. In International Conference on Learning Representations (ICLR), 2019.
  • Bradbury et al. [2017] James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. Quasi-recurrent neural networks. In International Conference on Learning Representations (ICLR), 2017.
  • Broyden [1965] Charles G Broyden. A class of methods for solving nonlinear simultaneous equations. Mathematics of Computation, 1965.
  • Chen et al. [2018] Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. In Neural Information Processing Systems, 2018.
  • Chen et al. [2016] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv:1604.06174, 2016.
  • Child et al. [2019] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv:1904.10509, 2019.
  • Cho et al. [2014] Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv:1409.1259, 2014.
  • Dai et al. [2019] Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. Transformer-XL: Attentive language models beyond a fixed-length context. In Annual Meeting of the Association for Computational Linguistics (ACL), 2019.
  • Dauphin et al. [2017] Yann N. Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International Conference on Machine Learning (ICML), 2017.
  • Dehghani et al. [2019] Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Łukasz Kaiser. Universal transformers. International Conference on Learning Representations (ICLR), 2019.
  • Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
  • El Ghaoui et al. [2019] Laurent El Ghaoui, Fangda Gu, Bertrand Travacca, and Armin Askari. Implicit Deep Learning. arXiv:1908.06315, 2019.
  • Elman [1990] Jeffrey L Elman. Finding structure in time. Cognitive Science, 14(2), 1990.
  • Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. In Neural Information Processing Systems, 2016.
  • Gomez et al. [2017] Aidan N Gomez, Mengye Ren, Raquel Urtasun, and Roger B Grosse. The reversible residual network: Backpropagation without storing activations. In Neural Information Processing Systems, 2017.
  • Haber and Ruthotto [2017] Eldad Haber and Lars Ruthotto. Stable architectures for deep neural networks. Inverse Problems, 2017.
  • He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Computer Vision and Pattern Recognition (CVPR), 2016.
  • Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 9(8), 1997.
  • Liu et al. [2019] Hanxiao Liu, Karen Simonyan, and Yiming Yang. DARTS: Differentiable architecture search. In International Conference on Learning Representations (ICLR), 2019.
  • Marcus et al. [1993] Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. Building a large annotated corpus of English: The Penn treebank. Computational Linguistics, 19(2), 1993.
  • Melis et al. [2018] Gábor Melis, Chris Dyer, and Phil Blunsom. On the state of the art of evaluation in neural language models. In International Conference on Learning Representations (ICLR), 2018.
  • Merity et al. [2017] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations (ICLR), 2017.
  • Merity et al. [2018a] Stephen Merity, Nitish Shirish Keskar, and Richard Socher. An analysis of neural language modeling at multiple scales. arXiv:1803.08240, 2018a.
  • Merity et al. [2018b] Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. In International Conference on Learning Representations (ICLR), 2018b.
  • Miller and Hardt [2018] John Miller and Moritz Hardt. When recurrent models don’t need to be recurrent. arXiv:1805.10369, 2018.
  • Niculae et al. [2018] Vlad Niculae, Andre Martins, Mathieu Blondel, and Claire Cardie. SparseMAP: Differentiable sparse structured inference. In International Conference on Machine Learning (ICML), 2018.
  • Salimans and Kingma [2016] Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Neural Information Processing Systems, 2016.
  • Santoro et al. [2018] Adam Santoro, Ryan Faulkner, David Raposo, Jack Rae, Mike Chrzanowski, Theophane Weber, Daan Wierstra, Oriol Vinyals, Razvan Pascanu, and Timothy Lillicrap. Relational recurrent neural networks. In Neural Information Processing Systems, 2018.
  • Scellier and Bengio [2017] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in Computational Neuroscience, 2017.
  • Sherman and Morrison [1950] Jack Sherman and Winifred J Morrison. Adjustment of an inverse matrix corresponding to a change in one element of a given matrix. The Annals of Mathematical Statistics, 1950.
  • Simard et al. [1989] Patrice Y Simard, Mary B Ottaway, and Dana H Ballard. Fixed point analysis for recurrent networks. In Neural Information Processing Systems, 1989.
  • Srivastava et al. [2014] Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR), 15(1), 2014.
  • Trinh et al. [2018] Trieu H Trinh, Andrew M Dai, Thang Luong, and Quoc V Le. Learning longer-term dependencies in RNNs with auxiliary losses. In International Conference on Machine Learning (ICML), 2018.
  • van den Oord et al. [2016] Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv:1609.03499, 2016.
  • Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017.
  • Waibel et al. [1989] Alex Waibel, Toshiyuki Hanazawa, Geoffrey Hinton, Kiyohiro Shikano, and Kevin J Lang. Phoneme recognition using time-delay neural networks. IEEE Transactions on Acoustics, Speech, and Signal Processing, 37(3), 1989.
  • Wang et al. [2019] Po-Wei Wang, Priya Donti, Bryan Wilder, and Zico Kolter. SATNet: Bridging deep learning and logical reasoning using a differentiable satisfiability solver. In International Conference on Machine Learning (ICML), 2019.
  • Werbos [1990] Paul J Werbos. Backpropagation through time: What it does and how to do it. Proceedings of the IEEE, 78(10), 1990.
  • Yang et al. [2018] Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. Breaking the softmax bottleneck: A high-rank RNN language model. International Conference on Learning Representations (ICLR), 2018.
  • Zhang et al. [2016] Saizheng Zhang, Yuhuai Wu, Tong Che, Zhouhan Lin, Roland Memisevic, Ruslan R Salakhutdinov, and Yoshua Bengio. Architectural complexity measures of recurrent neural networks. In Neural Information Processing Systems, 2016.
  • Zoph and Le [2017] Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. In International Conference on Learning Representations (ICLR), 2017.

Appendix A Backward Pass of the Deep Equilibrium Model

One of the core benefits of the DEQ approach comes from its analytical backward gradient at equilibrium. In this section, we provide a proof to Theorem 1 (which we restate here).

Theorem 1.

(Gradient of the Equilibrium Model)  Let be an equilibrium hidden sequence with length and dimensionality , and the ground-truth (target) sequence. Let be any differentiable function and be a loss function (where are applied in vectorized manner) that computes


Then the loss gradient w.r.t. (for instance, or ) is


where is the inverse Jacobian of evaluated at .

Proof of Theorem 1.

We first write out the equilibrium sequence condition: . By implicitly differentiating two sides of this condition with respect to :

Since , we have

which implies

Appendix B Sufficiency of a Single DEQ “Layer”

A hypothetical extension to the DEQ idea follows from the “deep” philosophy: if one DEQ works so well, why don’t we stack multiple DEQ modules with different parameters ()? We (re-)state and prove the following theorem, which demonstrates the universality of the DEQ model (i.e., sufficiency of exactly one DEQ “layer”).

Theorem 2.

(Universality of “Single-layer” DEQs.) Let be the input sequence, and let be sets of parameters for stable transformations and , respectively. Then there exists , where s.t.


where denotes the last feature dimensions of .

Proof of Theorem 2.

Assume is the equilibrium of the first DEQ module under transformation . Define , and by:


Then is a fixed point of , which completes the proof. ∎

Appendix C Universality of weight-tied, input-injected networks

Although the DEQ model corresponds to an infinite-depth network, as mentioned above it applies only to the specific case of weight-tied, input-injected infinite-depth models. This seems at first glance a substantial restriction over traditional deep networks, which have no requirement that the weights at each layer be identical. However, as we show below, this is not an actual restriction on the representational capacity from a mathematical point of view. Specifically, any deep network can be represented as a deep weight-tied network with no increase in depth and only a linear increase in the size of the hidden layer. This argument is equivalent to that presented in the TrellisNet work (Bai et al., 2019, Theorem 1), but we include it here in a slightly simpler and more general form. We emphasize that in practice we do not use the sparse structure below to construct the weight-tied layers for DEQ, but instead just use dense matrices and . However, the theorem below is important in establishing that there is no notable representational loss.

Theorem 3.

(Universality of weight-tied deep networks.) Consider a traditional -layer deep network defined by the relation


where denotes the hidden layer at depth , , are parameters of the network, and is the non-linearity at depth . Then the same network can be represented by a weight-tied, input-injected network of equivalent depth


where , , and are constant over all layers.

Proof of Theorem 3.

The proof is constructive: we build the weight-tied network equivalent to the original network by contructing the relevant matrices using a simple “shift” operation. In particular, we define the network parameters as


It is clear from inspection that after applications of the layer


using these parameters, the hidden vector will take on the value


Thus the weight-tied network computes all the same terms as the original network, using the same depth as the original network, and with a hidden unit size that is just the sum of the individual hidden unit sizes in the original network. This establises the claim of the theorem. ∎

Appendix D Empirical Convergence Observation on Weight-tied Deep Nets

Figure 4: The convergence of intermediate activations in TrellisNet (with kernel size 2) and weight-tied transformers on different sequence lengths.

As mentioned in Section 3, one motivation for optimizing the sequence-level equilibrium comes from our empirical observations that, starting at some point of the deep stacking, weight-tied deep networks begin to converge to a fixed point. We show in Figure 4 the convergence of trained layer-based TrellisNet (weight-tied temporal convolutions) and universal transformer (weight-tied self-attention) on randomly selected test samples of different lengths 100, 200, 400 and 800. In both cases, we see a tendency of the activations to converge. Notably, for transformers, we find stacked iterations usually lead to a oscillatory behavior on the plots (Figure 4), with values fluctuating around the actual fixed point (which we empirically verify can be found much more easily with Newton or quasi-Newton methods).

In practice, due to limited computation, we usually set the number of layers to a predetermined number (e.g, 60) and rarely reach the inference depths analyzed in Figure 4. Moreover, in the specific case of transformers, (Al-Rfou et al., 2018) stablizes the training of deep transformers (64-layer) on character-level language modeling with the help of various strong auxiliary losses at intermediate layers. In a certain sense, the addition of auxiliary losses have a similar effect as solving for equilibrium: we want intermediate-level hidden units to be both close to the target and as stable as possible (without drastic interlayer differences).

Appendix E More Remarks on DEQ

Applicability of Other Deep Techniques.

While the DEQ approach does not preclude specific architectural choices of (which means techniques such as layer normalization (Ba et al., 2016) or weight normalization (Salimans and Kingma, 2016) can work as is), it is not clear how certain regularizations such as auxiliary losses (Trinh et al., 2018; Bai et al., 2019) could be applied on DEQ, since there are no more “layers”. For dropout (Srivastava et al., 2014), we follow the practice of (Bai et al., 2019), which adapts the RNN variational dropout (Gal and Ghahramani, 2016) scheme to feedforward networks by applying the same mask at all levels. We empirically find that adding dropout makes the quasi-Newton methods slower to converge (i.e., inference-time equilibria are easier to find without the presence of noisy zeros). Since the weights of (and thus its operator norm) are directly related to the stability of root-finding, we observe that weight normalization (Salimans and Kingma, 2016) typically finds more stable parameters and slows down the growth of forward/backward Broyden’s iterations (as described in Figure 2).

Imbalances within Mini-batch.

Not all sequences in a minibatch converge to the equilibrium with the same number of iterations. However, with standard batched CUDA operations, the sequences that converge faster essentially need to “wait” for the slower ones. Though we empirically find such imbalance to be relatively small in scale, it could mean an inefficient GPU utilization at times.

DEQ Models Warmup with Very Shallow Nets.

Instead of training the DEQ from scratch, empirically we find that one can accelerate the DEQ training by pretraining a shallow weight-tied stack of (e.g., 2 layers), and using the resulting parameters to initialize the DEQ. In general, a shallow model plateaus at much lower accuracy than corresponding DEQs or deeper weight-tied networks. However, given the very small number of layers, a shallow model offers a memory- and computation-efficient starting point for DEQ training.

Training DEQs with Subsequences.

On extremely long sequences (e.g., ), the forward-pass fixed points can be challenging to solve accurately (especially at the start of the training) even with the help of the root-finding methods. Therefore, in practice, we suggest breaking these long sequences into a few subsequences when needed (recall that the forward pass can be any black-box root-finder). Moreover, with the help of Theorem 1, such subsequence technique can be used in the backward pass as well (where we solve for Eq. (11)). For instance, on a sequence :


where terms (A) and (B) require one evaluation of and term (C) requires one evaluation of . Hence the total memory needed is equivalent to that of applying once on the entire (but with the subsequences’ equilibrium likely easier to optimize).

Appendix F Task Descriptions

We briefly introduce the three sequence prediction tasks/datasets that we employ to evaluate the DEQ approach in Section 5.

Copy Memory Task.

The copy memory task is a small but challenging synthetic stress test that has been frequently used in prior work to test a sequence model’s memory retention ability (Zhang et al., 2016; Arjovsky et al., 2016; Bai et al., 2018). In this task, each sequence is 1-dimensional and has length , with randomly selected from integers (with repetition). The rest of the input elements are all filled with zeros, except for the . The goal of this task is to produce such that and . In other words, a sequence model trained on this task is expected to “recall” the first 10 elements of the sequence once it sees the delimiter , and copy them to the end of the sequence. We generate 20K training samples and 2K testing samples. In prior works, (Bai et al., 2018) have shown that RNNs generally struggle with the task, especially when , whereas feedforward models tend to have better memory.

Penn Treebank.

The Penn Treebank (PTB) corpus (Marcus et al., 1993) is a commonly used dataset for character- and word-level language modeling. When used for word-level language modeling, PTB contains about 888K words at training, with a vocabulary size of 10,000. As this is a comparatively small language corpus (with punctuations and capitalization removed), prior work has shown that well-designed regularizations are required for best results (Merity et al., 2018b; Yang et al., 2018).


The training corpus of WikiText-103 (WT103) (Merity et al., 2017) is about 110 times larger than PTB, with a vocabulary size over 260K. In general, this dataset is considered much more realistic than many others because it contains many rare words and retains punctuation, numbers, and capitalization from the original Wikipedia articles. WT103 is thus used to evaluate how well a sequence model scales to long sequences from a large vocabulary. This dataset has been frequently used in recent work with high-capacity sequence models (Bradbury et al., 2017; Bai et al., 2019; Dai et al., 2019; Baevski and Auli, 2019).

Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description