Lower bounds over Boolean inputs for deep neural networks with ReLU gates.
Motivated by the resurgence of neural networks in being able to solve complex learning tasks we undertake a study of high depth networks using ReLU gates which implement the function . We try to understand the role of depth in such neural networks by showing size lowerbounds against such network architectures in parameter regimes hitherto unexplored. In particular we show the following two main results about neural nets computing Boolean functions of input dimension ,
We use the method of random restrictions to show almost linear, , lower bound for completely weight unrestricted LTF-of-ReLU circuits to match the Andreev function on at least fraction of the inputs for for any
We use the method of sign-rank to show exponential in dimension lower bounds for ReLU circuits ending in a LTF gate and of depths upto with with some restrictions on the weights in the bottom most layer. All other weights in these circuits are kept unrestricted. This in turns also implies the same lowerbounds for LTF circuits with the same architecture and the same weight restrictions on their bottom most layer.
Along the way we also show that there exists a Sum-of-ReLU-of-ReLU function which Sum-of-ReLU neural nets can never represent no matter how large they are allowed to be.
There has been a recent surge of activity in using neural networks for complex artificial intelligence tasks (like this very recent spectacular demonstration  of the power of neural nets). This has rekindled interest in understanding neural networks from a complexity theory perspective. A myriad of hard mathematical questions have surfaced in the attempts to rigorously explain the power of neural networks and a comprehensive overview of these can be found in this recent three part series of articles from The Center for Brains, Minds and Machines (CBMM), [26, 25, 42].There is a rich literature investigating the complexity of the function classes represented by neural networks with various kinds of gates (or “activation functions" which is the more common parlance in machine learning). Many papers, a canonical example being the classic paper by Maass , establish complexity results for the entire class of functions represented by circuits where the gates can come from a very general family. This is complemented by papers that study a very specific family of gates such as the sigmoid gate or the LTF gate , ,  , , , , . Many associated results can also be found in these reviews [20, 27]. Recent circuit complexity results in , , ,  stand out as significant improvements over known lower (and upper) bounds on circuit complexity with threshold gates. The results of Maass  also show that very general families of neural networks can be converted into circuits with only LTF gates with at most a constant factor blow up in depth and polynomial blow up in size of the circuits.
In the last 5 years or so, a particular family of gates called the Rectified Linear Unit (ReLU) gates have been reported to have significant advantages over more traditional gates in practical applications of neural networks. Such a gate with real inputs computes the following output,
where and are fixed parameters associated with the gate ( is called the bias of the gate). In comparison, the valued LTF gate mentioned above computes (for the same weights as above) the function, where is the indicator function for the stated halfspace condition.
Some of the prior results which apply to general gates, such as the ones in , also apply to ReLU gates, because those results apply to gates that compute a piecewise polynomial function (ReLU is a piecewise linear function with only two pieces). However, as witnessed by results on LTF gates, one can usually make much stronger claims about specific classes of gates. To the best of our knowledge, no prior results have been obtained for ReLU gates from the perspective of Boolean complexity theory, i.e., the study of such circuits when restricted to Boolean inputs. The main focus of this work is to study circuits computing Boolean functions mapping which use ReLU gates in their intermediate layers, and have an LTF gate at the output node (to ensure that the output is in ). We remark that using an LTF gate at the output node while allowing more general analog gates in the intermediate nodes is a standard practice when studying the Boolean complexity of analog gates (see, for example, ).
Although we are not aware of an analysis of lower bounds for ReLU circuits when applied to only Boolean inputs, there has been recent work on the analysis of such circuits when viewed as a function from to (i.e., allowing real inputs and output). From  and  (with restrictions on the domain and the weights) we know of (super-)exponential lowerbounds on the size of Sum-of-ReLU circuits for certain easy Sum-of-ReLU-of-ReLU functions . Depth v/s size tradeoffs for such circuits have recently also been studied in [39, 12, 21, 41, 30] and in a recent paper  by the current authors. To the best of our knowledge no lowerbounds scaling exponentially with the dimension are known for analog deep neural networks of depths more than .
In what follows, the depth of a circuit will be the length of the longest path from the output node to an input variable, and the size of a circuit will be the total number of gates in the circuit. We will also use the notation Sum-of-ReLU to refer to circuits whose inputs feed into a single layer of ReLU gates, whose outputs are combined into a weighted sum to give the final output. Similarly, Sum-of-ReLU-of-ReLU denotes the circuit with depth 3, where the output node is a simple weighted sum, and the intermediate gates are all ReLU gates in the two “hidden" layers. We analogously define Sum-of-LTF, LTF-of-LTF, LTF-of-ReLU, LTF-of-LTF-of-LTF, LTF-of-ReLU-of-ReLU and so on. We will also use the notation LTF-of-(ReLU) for a circuit of the form LTF-of-ReLU-of-RELU--ReLU with levels of ReLU gates.
2 Statement and discussion of results
Boolean v/s real inputs.
We begin our study with the following observation which shows that ReLU circuits have markedly different behaviour when the inputs are restricted to be Boolean, as opposed to arbitrary real inputs. Since AND and OR gates can both be implemented by ReLU gates, it follows that any Boolean function can be implemented by a ReLU-of-ReLU circuit. In fact, it is not hard to show something slightly stronger:
Any function can be implemented by a Sum-of-ReLU circuit using at most number of ReLU gates, where denotes the Fourier coefficient of for the set .
The Lemma follows by observing that the indicator functions of each vertex of the Boolean hypercube can be implemented by a single ReLU gate, and the parity function on variables can be implemented by ReLU gates (see Appendix C). Thus, if one does not restrict the size of the circuit, then Sum-of-ReLU circuits can represent any pseudo-Boolean function. In contrast, we will now show that if one allows real inputs, then there exist functions with just 2 inputs (i.e., ) which cannot be represented by any Sum-of-ReLU circuit, no matter how large.
The function cannot be computed by any Sum-of-ReLU circuit, no matter how many ReLU gates are used. It can be computed by a Sum-of-ReLU-of-ReLU circuit.
The first part of the above proposition (the impossibility result) is proved in Appendix A. The second part follows from Corollary of a previous paper by the authors , which states that any function that can be implemented by a circuit of ReLU gates, can always be implemented with at most layers of ReLU gates (with a weighted Sum to give the final output).
Restricting to Boolean inputs.
From this point on, we will focus entirely on the situation where the inputs to the circuits are restricted to . One motivation behind our results is the desire to understand the strength of the ReLU gates vis-a-vis LTF gates. It is not hard to see that any circuit with LTF gates can be simulated by a circuit with ReLU gates with at most a constant blow-up in size (because a single LTF gate can be simulated by 2 ReLU gates when the inputs are a discrete set – see Appendix B). The question is whether ReLU gates can do significantly better than LTF gates in terms of depth and/or size.
A quick observation is that Sum-of-ReLU circuits can be linearly (in the dimension ) smaller than Sum-of-LTF circuits. More precisely,
The function given by can be implemented by a Sum-of-ReLU circuit with 2 ReLU gates, and any Sum-of-LTF that implements needs gates.
The above result follows from the following two facts: 1) any linear function is implementable by 2 ReLU gates, and 2) any Sum-of-LTF circuit with LTF gates gives a piecewise constant function that takes at most different values. Since takes different values (it evaluates every vertex of the Boolean hypercube to the corresponding natural number expressed in binary), we need gates.
In the context of these preliminary results, we now state our main contributions. For the next result we recall the definition of the Andreev function  which has previously many times been used to prove computational lower bounds [24, 15, 14].
Definition 1 (Andreev’s function).
The Andreev’s function is the following mapping,
where “bin" is the function that gives the decimal number that can be represented by its input bit string.
We are particularly inspired by the most recent use of the Andreev function by Kane and Williams  to get the first super linear lower bounds for approximating it using LTF-of-LTF circuits. We will give an almost linear lower bound on the size of LTF-of-ReLU circuits approximating this Andreev function with no restriction on the weights for each gate.
For any , there exists such that for all and , any LFT-of-ReLU circuit on bits that matches the Andreev function on bits for at least fraction of the inputs, has size .
It is well known that proving lower bounds without restrictions on the weights is much more challenging even in the context of LTF circuits. In fact, the recent results in  are the first superlinear lower bounds for LTF circuits with no restrictions on the weights. With restrictions on some or all the weights, e.g., assuming bounds on the weights (typically termed the “small weight asssumption") in certain layers, exponential lower bounds have been established for LTF circuits [11, 16, 32, 33]. Our next results are of this flavor: under certain kinds of weight restrictions, we prove exponential size lower bounds on the size of LTF-of-(ReLU) circuits. One thing to note is that our weight restrictions are assumed only on the bottom layer (closest to the input). The other layers can have gates with unbounded weights. Nevertheless, our weight restrictions are somewhat unconventional.
[Weight restriction condition] Let and be any permutation of . Let us also consider an arbitrary sequencing of the vertices of the hypercube . Define the polyhedral cone
In words, is the set of all linear objectives that order the vertices of the -dimensional hypercube in the order specified by . We will impose the condition that there exists a such that for each ReLU gate in the bottom layer, the vector ( as defined in (1)) and all weights are integers with magnitude bounded by some .
We will prove our lower bounds against the function proposed by Arkadev and Nikhil in ,
which we will refer to as the Arkadev-Nikhil function in the remainder of the paper. Here OMB is the ODD-MAX-BIT function which is a threshold gate which evaluates to on say a bit input if . We show the following exponential lowerbound against this function,
Let . Any depth LTF-of-(ReLU) circuits on bits such that the weights in the bottom layer are restricted as per Definition 2 that implements the Arkadev-Nikhil function on bits will require a circuit size of
Consequently, one obtains the same size lower bounds for circuits with only LTF gates of depth .
Note that this is an exponential in dimension size lowerbound for even super-polynomially growing bottom layer weights (and additional constraints as per Definition 2) and upto depths scaling as with .
We note that the Arkadev-Nikhil function can be represented by an size LTF-of-LTF circuit with no restrictions on weights (see Theorem 2.6 below). In light of this fact, Theorem 2.5 is somewhat surprising as it shows that for the purpose of representing Boolean functions a deep ReLU circuit (ending in a LTF) gate can get exponentially weakened when just its bottom layer weights are restricted as per Definition 2, even if the integers are allowed to be super-polynomially large. Moreover, the lower bounds also hold of LTF circuits of arbitrary depth , under the same weight restrictions on the bottom layer. We are unaware of any exponential lower bounds on LTF circuits of arbitrary depth under any kind of weight restrictions.
We will use the method of sign-rank to obtain the exponential lowerbounds in Theorems 2.5. The sign-rank of a real matrix with all non-zero entries is the least rank of a matrix of the same dimension with all non-zero entries such that for each entry , . For a Boolean function mapping, one defines the “sign-rank of f" as the sign-rank of the dimensional matrix . This notion of a sign-rank has been used to great effect in diverse fields from communication complexity to circuit complexity to learning theory. Explicit matrices with a high sign-rank were not known till the breakthrough work by Forster, . Forster et. al. showed elegant use of this complexity measure to show exponential lowerbounds against LTF-of-MAJ circuits in . Lot of the previous literature about sign-rank has been reviewed in the book by Satya Lokam . Most recently the following result was obtained by Arkadev and Nikhil in  leading to a proof of strict containment of LTF-of-MAJ in LTF-of-LTF.
We will prove our theorem by showing a small upper bound on the sign-rank of LTF-of-(ReLU) circuits which have their bottom most layer’s weight restricted in the said way.
3 Lower bounds for LTF-of-ReLU against the Andreev function (Proof of Theorem 2.4)
We will use the classic “method of random restrictions" [37, 36, 13, 40, 29] to show a lowerbound for weight unrestricted LTF-of-ReLU circuits for representing the Andreev function. The basic philosophy of this method is to take any arbitrary LTF-of-ReLU circuit which supposedly matches the Andreev function on a large fraction of the inputs and to randomly fix the values on some of its input coordinates and also do the same fixing on the same coordinates of the input to the Andreev function. Then we show that upon doing this restriction the Andreev function collapses to an arbitrary Boolean function on the remaining inputs (what it collapses to depends on what values were fixed on its inputs that got restricted). But on the other hand we show that the LTF-of-ReLU collapses to a circuit which is of such a small size that with high-probability it cannot possibly approximate a randomly chosen Boolean function on the remaining inputs. This contradiction leads to a lowerbound.
There are two important concepts towards implementing the above idea. First one has to precisely define as to when can a ReLU gate upon a partial restriction of its inputs be considered to be removable from the circuit. Once this notion is clarified it will automatically turn out that doing random restrictions on ReLU is the same as doing random restriction on a LTF gate as was recently done in . The secondly it needs to be true that at any fixed size LTF-of-ReLU circuits cannot represent too many of all the Boolean functions possible at the same input dimension. For this very specific case of LTF-of-ReLU circuits where ReLU gates necessarily have a fan-out of , Theorem 2.1 in  applies and we have from there that LTF-of-ReLU circuits over bits with ReLU gates can represent at most number of Boolean functions. We note that slightly departing from the usual convention with neural networks here in this work by Wolfgaang Mass he allows for direct wires from the input nodes to the output LTF gate. This flexibility ties in nicely with how we want to define a ReLU gate to be becoming removable under the random restrictions that we use.
Random Boolean functions vs any circuit class
In everything that follows all samplings being done (denoted as ) are to be understood as sampling from an uniform distribution unless otherwise specified. Firstly we note this well-known lemma,
Let be any given Boolean function. Then the following is true,
From the above it follows that if is the total number of functions in any circuit class (whose members be called ) then we have by union bound,
Equipped with these basics we are now ready to begin the proof of the lowerbound against weight unrestricted LTF-of-ReLU circuits,
Let denote arbitrary LTF-of-ReLU circuits over bits.
For some and a size function denoted as we use equation 4 , the definition of above and the upperbound given earlier for the number of LTF-of-ReLU functions at a fixed circuit size (now used for circuits on bits) to get,
whereby in the last inequality above we have assumed that . This assumption is legitimate because we want to estimate certain large asymptotics. For any arbitrarily chosen constant we try to satisfy the following condition, . For any constant for large enough we would have and hence the above constraint on gets satisfied if we work in the regime, . So for this range of we have, . Now we want, . But on the otherhand for the upperbound on to make sense we need, . Its clear that both the conditions get satisfied if for asymptotically large we choose . And corresponding to this we have for
Definition 4 ().
Let be the subset of all these above for which the above event is true.
Now we recall the definition of the Andreev function in equation 1 for the following definition and the claim,
Definition 5 ().
Let denote the set of all possible “random restrictions" where one is fixing all the input bits of except bit in each row of the matrix . So the restricted function (call it by overloading the notation for simplicity) computes a function of the form,
From the definitions of and above the following is immediate,
The truth table of is the string in the input to that gets fixed by . Thus we observe that if is chosen uniformly at random then is a bit Boolean function chosen uniformly at random.
Let be any arbitrary member of . Let be the truth-table of . Let be restrictions on the input of which fix the part of its input to . So when we are sampling restrictions uniformly at random from the restrictions of the type these different instances differ in which bit of each row of the matrix (of the input to ) they left unfixed and to what values did they fix the other entries of . Let be a bit LTF-of-ReLU Boolean circuit of size say . Thus under the restriction both and are bit Boolean functions.
Now we note that a ReLU gate over bits upon a random restriction becomes redundant (and hence removable) iff its linear argument either reduces to a non-positive definite function or a positive definite function. In the former case the gate is computing the constant function zero and in the later case it is computing a linear function which can be simply implemented by introducing wires connecting the inputs directly to the output LTF gate. Thus in both the cases the resultant function no more needs the ReLU gate for it to be computed. (We note that such direct wires from the input to the output gate were allowed in how the counting was done of the total number of LTF-of-ReLU Boolean functions at a fixed circuit size.) Combining both the cases we note that the conditions for collapse (in this sense) of a ReLU gate is identical to that of the conditions of collapse for a LTF gate with the same linear argument. Hence corresponding to the random restrictions we can just directly utilize the random restriction lemma from  to say that,
The above definition of implies,
Now we compare with the definitions of and to observe that (a) with probability at least , is of the circuit type as in the event in equation and (b) by definition of the Andreev function it follows that has its truth table given by and hence it specifies the same function as . Hence this can as well write this as,
Circuits have low correlation with the Andreev function
We think of sampling a as a two step process of first sampling a , a bit Boolean function and fixing the first bits of to be the truth-table of and then we randomly assign values to the remaining bits of . Call these later bit string to be .
In the last line above we have invoked equation 9. Now we note that sampling the bit string such that is the same as doing a random restriction of the type and then randomly picking a bit string say . So we can rewrite the last inequality as,
So after putting back the values of and the largest scaling of that we can have (from equation 3), the upperbound on the above probability becomes,
Thus the probability is upperbounded by as long as
Stated as a lowerbound we have that if a LTF-of-ReLU has to match the bit Andreev function on more than fraction of the inputs for for some (asymptotically this is like having a constant ) then the LTF-of-ReLU needs to be of size . Now we define such that and that gives the form of the almost linear lowerbound as stated in the theorem. ∎
4 Smaller upper bounds on the sign-rank of LTF-of-(ReLU) with weight restrictions only on the bottom most layer (Proof of Theorem 2.5)
For a LTF-of-ReLU circuit with any given weights on the network the inputs to the threshold function of the top LTF gate are some set of real numbers (one for each input). Over all these inputs let be the distance from of the largest negative number on which the LTF gate ever gets evaluated. Then by increasing the bias at this last LTF gate by a quantity less then we can ensure that no input to this LTF gate is while the entire circuit still computes the same Boolean function as originally. So we can assume without loss of generality that the input to the threshold function at the top LTF gate is never . We also recall that the weights at the bottom most layer are constrained to be integers of magnitude at most .
Let this depth LTF-of-(ReLU) circuit map . Let be the widths of the ReLU layers at depths indexed by increasing with increasing distance from the input. Thus, the output LTF gate gets inputs; the -th input, for , is the output of a circuit of depth composed of only ReLU gates. Let be the pseudo-Boolean function implemented by . Thus the output of the overall LTF-of-(ReLU) circuit is,
Let and be natural numbers. Consider a circuit with inputs and a single output, consisting of only ReLU gates of depth with ReLU gates at each depth, with corresponding to the layer closest to the input (note that single output ReLU gate is not counted here). We restrict the inputs to , so the circuit implements a pseudo-Boolean function .
Assume that the weights of the ReLU gates in the layer closest to the input are restricted as per Definition 2. Define the matrix whose rows and columns are indexed by as
Then has a block structure, where the rows and columns can be partitioned contiguously into blocks (thus, has blocks), and within each block is constant valued.
Before we prove the Lemma, let us see why it implies Theorem 2.5. Let be the matrix obtained from the ReLU circuit outputs from (10), and let be the matrix obtained from . Let be the matrix of all ones. Then
where the first inequality follows from the definition of sign-rank, the second inequality follows from the subadditivty of rank and the last inequality is a consequence of Lemma 4.1. Indeed, a matrix with block structure as in the conclusion of Lemma 4.1 has rank at most by expressing it as a sum of these many matrices of rank one and using subaddivity of rank.
Now we recall that the Arkadev-Nikhil function (which is linear sized depth LTF) on bits has sign-rank . It follows that and for any constant s.t for large enough we would have, . From the above upper bound on the sign-rank of our bottom layer weight restricted LTF-of-(ReLU) with widths it follows that for this to represent this Arkadev-Nikhil function it would need, . Hence it follows that the size () required for such LTF-of-(ReLU) circuits to represent the Arkadev-Nikhil function is .
The statement about LTF circuits is a straightforward consequence of the above result and Claim 5 in Appendix B which says that any LTF gate can be simulated by 2 ReLU gates.
We now prove Lemma 4.1.
Proof of Lemma 4.1.
We will prove this Lemma by induction on .
The base case of the induction : A single ReLU gate.
A single ReLU gate’s output is given by , where and . Since the entries of and are assumed to be integers bounded by , the terms and can each take at most different values, since . So we can arrange the rows and columns in increasing order of and and then partition the rows and columns contiguously according to these values, and the base case is proved.
The induction step.
We first make a simple claim about the sum of matrices which are block wise constant.
Let be fixed natural numbers. Let be any matrices such that for each the rows and columns can be partitioned contiguously into blocks (not necessarily equal in size), such that is constant valued within each of the blocks. Then is an matrix whose rows and columns can be partitioned contiguously into blocks such that is constant valued within each block defined by this partition of the rows and columns.
The partition of the rows of into contiguous blocks is equivalent to a choice of lines out of lines. When we sum the matrices, the refined partition in the sum is a selection of lines out of lines, giving us contiguous blocks. The same argument holds for the columns. ∎
To complete the induction step, we observe that a ReLU circuit with depth layers can be seen as computing where is the output of a ReLU circuit of depth . Thus, the corresponding matrices satisfy , where is the matrix of all ones, and the “max” is taken entrywise. the induction hypothesis tells us that the rows and columns each matrix can be partitioned contiguously into such that is constant valued within each block. Thus, by Claim 4, the rows and columns of the matrix can be partitioned into contiguous blocks. ∎
We would like to thank Aurko Roy (Google Brain, San Francisco Bay Area) for extensive discussions on the methods used and the questions addressed in this work. We also thank Nikhil Mande (TIFR), Piyush Srivastava (TIFR) and Xin Li (JHU) for helpful conversations on circuit complexity. Amitabh Basu and Anirbit Mukherjee gratefully acknowledge support from the NSF grant CMMI1452820.
-  A. E. Andreev. About one method of obtaining more than quadratic effective lower bounds of complexity of pi-schemes, 1987.
-  R. Arora, A. Basu, P. Mianjy, and A. Mukherjee. Understanding deep neural networks with rectified linear units. https://eccc.weizmann.ac.il/report/2017/098/, 2016.
-  H. Buhrman, N. Vereshchagin, and R. de Wolf. On computation and communication with small bias. In Computational Complexity, 2007. CCC’07. Twenty-Second Annual IEEE Conference on, pages 24–32. IEEE, 2007.
-  M. Bun and J. Thaler. Improved bounds on the sign-rank of ac^ 0. In LIPIcs-Leibniz International Proceedings in Informatics, volume 55. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2016.
-  A. Chattopadhyay and N. S. Mande. Weights at the bottom matter when the top is heavy. arXiv preprint arXiv:1705.02397, 2017.
-  R. Chen, R. Santhanam, and S. Srinivasan. Average-case lower bounds and satisfiability algorithms for small threshold circuits. In LIPIcs-Leibniz International Proceedings in Informatics, volume 50. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik, 2016.
-  A. Daniely. Depth separation for neural networks. arXiv preprint arXiv:1702.08489, 2017.
-  R. Eldan and O. Shamir. The power of depth for feedforward neural networks. In Conference on Learning Theory, pages 907–940, 2016.
-  J. Forster. A linear lower bound on the unbounded error probabilistic communication complexity. Journal of Computer and System Sciences, 65(4):612–625, 2002.
-  J. Forster, M. Krause, S. V. Lokam, R. Mubarakzjanov, N. Schmitt, and H. U. Simon. Relations between communication complexity, linear arrangements, and computational complexity. In International Conference on Foundations of Software Technology and Theoretical Computer Science, pages 171–182. Springer, 2001.
-  A. Hajnal, W. Maass, P. Pudlák, M. Szegedy, and G. Turan. Threshold circuits of bounded depth. In Foundations of Computer Science, 1987., 28th Annual Symposium on, pages 99–110. IEEE, 1987.
-  B. Hanin. Universal function approximation by deep neural nets with bounded width and relu activations. arXiv preprint arXiv:1708.02691, 2017.
-  J. Hastad. Almost optimal lower bounds for small depth circuits. In Proceedings of the eighteenth annual ACM symposium on Theory of computing, pages 6–20. ACM, 1986.
-  R. Impagliazzo, R. Meka, and D. Zuckerman. Pseudorandomness from shrinkage. In Foundations of Computer Science (FOCS), 2012 IEEE 53rd Annual Symposium on, pages 111–119. IEEE, 2012.
-  R. Impagliazzo and M. Naor. Decision trees and downward closures. In Structure in Complexity Theory Conference, 1988. Proceedings., Third Annual, pages 29–38. IEEE, 1988.
-  R. Impagliazzo, R. Paturi, and M. E. Saks. Size–depth tradeoffs for threshold circuits. SIAM Journal on Computing, 26(3):693–707, 1997.
-  V. Kabanets, D. Kane, and Z. Lu. A polynomial restriction lemma with applications. In Electronic Colloquium on Computational Complexity (ECCC), volum