Risk Bounds for Highdimensional Ridge Function Combinations Including Neural Networks
Abstract
Let be a function on with an assumption of a spectral norm . For various noise settings, we show that , where is the sample size and is either a penalized least squares estimator or a greedily obtained version of such using linear combinations of sinusoidal, sigmoidal, ramp, rampsquared or other smooth ridge functions. The candidate fits may be chosen from a continuum of functions, thus avoiding the rigidity of discretizations of the parameter space. On the other hand, if the candidate fits are chosen from a discretization, we show that .
This work bridges nonlinear and nonparametric function estimation and includes singlehidden layer nets. Unlike past theory for such settings, our bound shows that the risk is small even when the input dimension of an infinitedimensional parameterized dictionary is much larger than the available sample size. When the dimension is larger than the cube root of the sample size, this quantity is seen to improve the more familiar risk bound of , also investigated here.
eq(LABEL:#1) \newrefformatthmTheorem LABEL:#1 \newrefformatsecSection LABEL:#1 \newrefformatalgoAlgorithm LABEL:#1 \newrefformatfigFig. LABEL:#1 \newrefformattabTable LABEL:#1 \newrefformatrmkRemark LABEL:#1 \newrefformatdefDefinition LABEL:#1 \newrefformatcorCorollary LABEL:#1 \newrefformatlmmLemma LABEL:#1 \newrefformatpropProposition LABEL:#1 \newrefformatappAppendix LABEL:#1 \newrefformatexExample LABEL:#1 \newrefformatassAssumption LABEL:#1
Contents
1 Introduction
Functions in are approximated using linear combinations of ridge functions with one layer of nonlinearities. These approximations are employed via functions of the form
(1) 
which is parameterized by the vector , consisting of in , and in for , where is the number of nonlinear terms. Models of this type arise with considerable freedom in the choice of the activation function , ranging from general smooth functions of projection pursuit regression [17] to the unit step sigmoid and ramp functions of singlehidden layer neural nets [8, 6, 7, 13, 27].
Our focus in this paper is on the case that is a fixed Lipschitz function (such as a sigmoid or ramp or sinusoidal function), though some of our conclusions apply more generally. For these activation functions, we will obtain statistical risk conclusions using a penalized least squares criterion. We obtain generalization error bounds for these by balancing the approximation error and descriptive complexity. The most general form of our bounds hold for quite general nonlinear infinite dictionaries. A hallmark of our conclusions is to lay bare how favorable risk behavior can be obtained as long as the logarithm of the number of parameters relative to sample size is small. This entails a slower rate of convergence through a rate that is smaller than what is cemented in traditional cases, but leads to better results than these earlier bounds would permit in certain very highdimensional situations. From an applied perspective, good empirical performance of neural net (and neural net like) models has been reported as in [26] even when is much larger than , though theoretical understanding has been lacking. Returning to the case of a single layer of nonlinearly parameterized function, it is useful to view the representation \prettyrefeq:general as
(2) 
where the are a selection of functions from the infinite library of functions of the form for real vector and the are coefficients of linear combination of in the library. These representations are single hiddenlayer networks. Deep network approximations are not very well understood. Nevertheless our results generalize provided some of our arguments are slightly modified.
We can reduce \prettyrefeq:general to \prettyrefeq:simple_general as follows. Suppose the library is symmetric and contains the zero function. Without loss of generality, we may assume that the or are nonnegative by replacing the associated with , that by assumption also belongs to . One can assume the internal parameterization take the form by appending a coordinate of constant value to and a coordinate of value to the vector . Note that now and are dimensional.
We will take advantage of smoothness of activation function (assumption that either is Lipschitz or that its first derivative is Lipschitz). Suppose is an arbitrary probability measure on . Let be the norm induced by the inner product . For a symmetric collection of dictionary elements containing the zero function, we let be the linear span of .
The variation of with respect to (or the atomic norm of with respect to ) is defined by
where . For functions in , this variation picks out the smallest among representations . In the particular case that , we have . For functions in the closure of the linear span of , the variation is the smallest limit of such norms among functions approaching the target. The subspace of functions with finite is denoted . Such variation control provides for approximation (opportunity) for dimension independent rates of order with an term approximation.
It is fruitful to discuss spectral conditions for finite variation for various choices of . To this end, define , for . If has a bounded domain in and a Fourier representation with , it is possible to use approximating functions of the form \prettyrefeq:general with a single activation function . Such activation functions can be be general bounded monotone functions. We use for vectors in and for scalars such as . As we have said, to obtain risk bounds in later sections, we will assume that either is bounded Lipschitz or that, additionally, its derivative is Lipschitz. These two assumptions are made precise in the following statements.
Assumption 1.
The activation function has norm at most one and satisfies
for all in and for some positive constant .
Assumption 2.
The activation function has norm at most one and satisfies
and
for all in and for some positive constants and .
In particular, \prettyrefass:ass2 implies that
for all in .
A result from [7] provides a useful starting point for approximating general functions by linear combinations of such objects. Suppose is finite. Then by [7] the function has finite variation with respect to step functions and, consequently, there exists an artificial neural network of the form \prettyrefeq:general with , , and such that, if a suitable constant correction is subtracted from , then
In particular, minus a constant correction has variation less than .
If has right at left limits and , respectively, the fact that as allows one to use somewhat arbitrary activation functions as basis elements. For our results, it in undesirable to have unbounded weights. Accordingly, it is natural to impose a restriction on the size of the internal parameters and to also enjoy a certain degree of smoothness not offered by step functions. Although, it should be mentioned that classical empirical process theory allows one to obtain covering numbers for indicators of halfspaces (which are scale invariant in the size of the weights) by taking advantage of their combinatorial structure [3]. Nevertheless, we adopt the more modern approach of working with smoothly parameterized dictionaries. In this direction, we consider the result in [13], which allows one to approximate by linear combinations of ramp ridge functions (also known as first order ridge splines or hinging hyperplanes) , with , .
The ramp activation function (also called a lowerrectified linear unit or ReLU) is currently one of the most popular form of artificial neural network activation functions, particularly because it is continuous and Lipschitz. In particular, it satisfies the conditions of \prettyrefass:ass1 with depending on the size of its domain. In \prettyrefthm:splineapprox, we refine a result from [13]. For an arbitrary target function with finite has finite variation with respect to the ramp functions and, consequently, there exists an approximation of the form \prettyrefeq:general activated by ridge ramp functions with and such that if a suitable linear correction is subtracted from , then
(3) 
In particular, minus a linear correction has variation less than .
The second order spline , which may also be called rampsquared, satisfies the conditions of \prettyrefass:ass2 with constants and depending on the size of its domain. Likewise, in \prettyrefthm:splineapprox, we show that for an arbitrary target function with finite a quadratically corrected has finite variation with respect to the second order splines, there exists an approximation of the form \prettyrefeq:general activated by second order ridge splines with and such that, if a suitable quadratic correction is subtracted from , then
(4) 
In particular, minus a quadratic correction has variation less than .
For integer , we define the infinite dictionary
We then set to be the linear span of . With this notation, .
The condition ensures that (corrected by a th degree ridge polynomial) belongs to and . Functions with moderate variation are particularly closely approximated. Nevertheless, even when is infinite, we express the tradeoffs in approximation accuracy for consistently estimating functions in the closure of the linear span of .
In what follows, we assume that the internal parameters have norm at most . Likewise, we assume that so that . This control on the size of the internal parameters will be featured prominently throughout. In the case of spline activation functions, we are content with the assumption . Note that if one restricts the size of the domain and internal parameters (say, to handle polynomials), the functions are still bounded and Lipschitz but with possibly considerably worse constants.
Suppose data are independently drawn from the distribution of . To produce predictions of the realvalued response from its input , the target regression function is to be estimated. The function is assumed to be bounded in magnitude by a positive constant . We assume the noise has moments (conditioned on ) that satisfy a Bernstein condition with parameter . That is, we assume
where . This assumption is equivalent to requiring that is uniformly bounded in for some . A stricter assumption is that is uniformly bounded in , which corresponds to an error distribution with subGaussian tails. These two noise settings will give rise to different risk bounds, as we will see.
Because is bounded in magnitude by , it is useful to truncate an estimator at a level at least . Depending on the nature of the noise , we will see that will need to be at least plus a term of order or . We define the truncation operator that acts on function in by . Associated with the truncation operator is a tail quantity that appears in the following analysis and our risk bounds have a term, but this will be seen to be negligible when compared to the main terms. The behavior of is studied in \prettyreflmm:truncate_bound.
The empirical mean squared error of a function as a candidate fit to the observed data is . Given the collection of functions , a penalty , , and data, a penalized least squares estimator arises by optimizing or approximately optimizing
(5) 
Our method of risk analysis proceeds as follows. Given a collection of candidate functions, we show that there is a countable approximating set of representations , variabledistortion, variablecomplexity cover of , and a complexity function , with the property that for each in , there is an in such that is not less than a constant multiple of , where is a constant (depending on , , , and ) and is given as a suitable empirical measure of distortion (based on sums of squared errors). The variabledistortion, variablecomplexity terminology has its origins in [10, 11, 15]. The task is to determine penalties such that an estimator approximately achieving the minimum of satisfies
(6) 
for some universal . Valid penalties take different forms depending on the size of the effective dimension relative to the sample size and smoothness assumption of .

When is large compared to and if satisfies \prettyrefass:ass1, a valid penalty divided by sample size is at least
(7) 
When the noise is zero and is large compared to and if satisfies \prettyrefass:ass1, a valid penalty divided by sample size is at least
(8) 
When is large compared to and if satisfies \prettyrefass:ass2, a valid penalty divided by sample size is at least of order
(9) 
When is small compared to and if satisfies \prettyrefass:ass1, a valid penalty divided by sample size is at least
(10)
Here and for some and .
Accordingly, if belongs to , then is not more than a constant multiple of the above penalties with replaced by .
In the singlehidden layer case, we have the previously indicated quantification of the error of approximation . Nevertheless, the general result \prettyrefeq:risk allows us to likewise say that the risk for multilayer networks will be at least as good as the deep network approximation capability will permit. The quantity
is an index of resolvability of by functions with sample size . We shall take particular advantage of such risk bounds in the case that does not depend on . Our restriction of to is one way to allow the construction of such penalties.
The following table expresses the heart of our results in the case of penalty based on the norm of the outer layer coefficients of onehidden layer networks expressible through (subject to constraints on the inner layer coefficients). These penalties also provide risk bounds for moderate and highdimensional situations.
Activation  

I  \prettyrefass:ass1  
II  \prettyrefass:ass2  
III  \prettyrefass:ass1 
\prettyrefthm:main_result over continuum of candidate fits
Activation  

A  \prettyrefass:ass1  
B  \prettyrefass:ass2  
C  \prettyrefass:ass1 
\prettyrefthm:main_result over discretization of candidate fits
The results we wish to highlight are contained in the first two rows of \prettyreftab:T1. The penalties as stated are valid up to modest universal constants and negligible terms that do not depend on the candidate fit. The quantity is of order in the subexponential noise case, order in the subGaussian noise case and of constant order in the zero noise case. This (as defined in \prettyreflmm:truncate_bound) depends on the variance bound , Bernstein parameter , the upper bound of , and the noise tail level of the indicated order.
When belongs to , a resulting valid risk bound is a constant multiple of or , according to the indicated cases. In this way the expression provides a rate of convergence. Thus the columns of \prettyreftab:T1 provide valid risk bounds for these settings.
The classical risk bounds for mean squared error, involving to some power, are only useful when the sample size is much larger than the dimension. Here, in contrast, in the first two lines of \prettyreftab:T1, we see the dependence on dimension is logarithmic, permitting much smaller sample sizes. These results are akin to those obtained in [31] (where the role of the dimension there is the size of the dictionary) for highdimensional linear regression. However, there is an important difference. Our dictionary of nonlinear parameterized functions is infinite dimensional. For us, the role of is the input dimension, not the size of the dictionary. The richness of is largely determined by the sizes of and and more flexibly represents a larger class of functions.
The price we pay for the smaller dependence on input dimension is a deteriorated rate with exponent in general and under slightly stronger smoothness assumptions on , rather than the familiar exponents of .
The rate in the last row improves upon the familiar exponent of to . Note that when is large, this enhancement in the exponent is negligible. The rate in the first row is better than the third approximately for , the second is better than the third row approximately for , and both of these first two rows have risk tending to zero as long as .
For functions in , an upper bound of for the squared error loss is obtained in [8]. The squared error minimax rates for functions in [33], was determined to be between
and
Using the truncated penalized least squares estimator \prettyrefeq:risk, we obtain an improved rate of order , where is logarithmic in , using techniques that originate in [20] and [19], with some corrections here.
2 How far from optimal?
For positive , let
(11) 
be the dictionary of all such inner layer ridge functions with parameter restricted to the ball of size and variables restricted to the cube . The choice of the norm on the inner parameters is natural as it corresponds to for . Let be the closure of the set of all linear combinations of functions in with norm of outer coefficients not more than . For any class of functions on , the minimax risk is
(12) 
Consider the model for , where and . It was determined in [25], that for , roughly corresponding to ,
(13) 
and for ,
(14) 
These lower bounds are similar in form to the risk upper bounds that are implied from the penalties in \prettyreftab:T2. These quantities have the attractive feature that the rate (the power of ) remains at least as good as or even as the dimension grows. However, rates determined by \prettyrefeq:lowerlowdim and the last line in \prettyreftab:T2 are only useful provided is small. In high dimensional settings, the available sample size might not be large enough to ensure this condition.
These results are all based on obtaining covering numbers for the library . If satisfies a Lipschitz condition, these numbers are equivalent to covering numbers of the internal parameters or of the Euclidean inner product of the data and the internal parameters. The factor of multiplying the reciprocal of the sample size is produced from the order log cardinality of the standard covering of the library . What enables us to circumvent this polynomial dependence on is to use an alternative cover of that has log cardinality of order . Misclassification errors for neural networks with bounded internal parameters have been analyzed in [12, 3, 27] (via VapnikChervonenkis dimension and its implications for covering numbers). Unlike the setup considered here, past work [8, 5, 33, 20, 9, 11, 27, 14, 4, 34, 24] has not investigated the role of such restricted parameterized classes in the determination of suitable penalized least squares criterion for nonparametric function estimation. After submission of the original form of this work, our results have been put to use in [32] to give risk statements about multilayer (deep) networks activated by ramp functions.
3 Computational aspects
From a computational point of view, the empirical risk minimization problem \prettyrefeq:least_squares is highly nonconvex, and it is unclear why existing algorithms like gradient descent or back propagation are empirically successful at learning the representation \prettyrefeq:general. There are relatively few rigorous results that guarantee learning for regression models with latent variables, while keeping both the sampling and computational complexities polynomial in and . Here we catalogue some papers that make progress toward developing a provably good, computationally feasible estimation procedure. Most of them deal with parameter recovery and assume that has exactly the form \prettyrefeq:general. Using a theory of tensor decompositions from [1], [22] apply the method of moments via tensor factorization techniques to learn mixtures of sigmoids, but they require a special nondegeneracy condition on the activation function. It is assumed that the input distribution is known apriori. In [37], the authors use tensor initialization and resampling to learn the parameters in a representation of the form \prettyrefeq:general with smooth that has sample complexity and computation complexity .
In [21], the authors estimate the gradient of the regression function (where is Gaussian and is the logistic sigmoid) at a set of random points, and then cluster the estimated gradients. They prove that the estimated gradients concentrate around the internal parameter vectors. However, unless the weights of the outer layer are positive and sum to , the complexity is exponential in . In [2], it was shown that for a randomly initialized neural network with sufficiently many hidden units, the generic gradient descent algorithm learns any low degree polynomial. Learning nonlinear networks through multiple rounds of random initialization followed by arbitrary optimization steps was proposed in [35]. In [36], an efficiently learned kernel based estimator was shown to perform just as well as a class of deep neural networks. However, its ability to wellapproximate general conditional mean regression functions is unclear.
The next section discusses an iterative procedure that reduces the complexity of finding the penalized least squares estimator \prettyrefeq:least_squares.
4 Greedy algorithm
The main difficulty with constructing an estimator that satisfies \prettyrefeq:risk is that it involves a dimensional optimization. Here, we outline a greedy approach that reduces the problem to performing dimensional optimizations. This construction is based on the penalized greedy pursuit (LPGP) in [20], with the modification that the penalty can be a convex function of the candidate function complexity. Greedy strategies for approximating functions in the closure of the linear span of a subset of a Hilbert space has its origins in [23] and many of its statistical implications were studied in [9] and [20].
Let be a function, not necessarily in . Initialize . For iteratively, given the terms of as and the coefficients of it as , we proceed as follows. Let , with the term in chosen to come within a constant factor of the maximum inner product with the residual ; that is
Define . Associated with this representation of is the norm of its coefficients . The coefficients and are chosen to minimize .
In the empirical setting, with , the highdimensional optimization task is to find such that
The fact that one does not need to find the exact maximizer of the above empirical inner product, but only come to within a constant multiple of it, has important consequences. For example, in adaptive annealing, one begins by sampling from an initial distribution and then iteratively samples from a distribution proportional to , evolving according to , where satisfies . The mean of is at least for sufficiently large .
Theorem 1.
Suppose is a realvalued nonnegative convex function. If is chosen according to the greedy scheme described previously, then
(15) 
where . Furthermore, for all ,
(16) 
and hence with ,
Proof.
Fix any in the linear span , with the form , with nonnegative and set
From the definition of and as minimizers of for each , and the convexity of ,
Now is equal to . Expanding this quantity leads to
Next we add to this expression to obtain
(17) 
The expression in brackets in \prettyrefeq:norm is equal to and hence the entire quantity is further upper bounded by
Consider a random variable that equals with probability having mean . Since a maximum is at least an average, the choice of implies that is at least . This shows that is no less than . Expanding the squares in and using the CauchySchwarz inequality yields the bound . Since and , we find that is at most . Hence we have shown that
and