The Randomized Dependence Coefficient
Abstract
We introduce the Randomized Dependence Coefficient (RDC), a measure of nonlinear dependence between random variables of arbitrary dimension based on the HirschfeldGebeleinRényi Maximum Correlation Coefficient. RDC is defined in terms of correlation of random nonlinear copula projections; it is invariant with respect to marginal distribution transformations, has low computational cost and is easy to implement: just five lines of R code, included at the end of the paper.
The Randomized Dependence Coefficient
David LopezPaz, Philipp Hennig, Bernhard Schölkopf Max Planck Institute for Intelligent Systems Spemannstraße 38, Tübingen, Germany {dlopez,phennig,bs}@tue.mpg.de
1 Introduction
Measuring statistical dependence between random variables is a fundamental problem in statistics. Commonly used measures of dependence, Pearson’s rho, Spearman’s rank or Kendall’s tau are computationally efficient and theoretically well understood, but consider only a limited class of association patterns, like linear or monotonically increasing functions. The development of nonlinear dependence measures is challenging because of the radically larger amount of possible association patterns.
Despite these difficulties, many nonlinear statistical dependence measures have been developed recently. Examples include the Alternating Conditional Expectations or backfitting algorithm (ACE) [2, 9], Kernel Canonical Correlation Analysis (KCCA) [1], (Copula) Maximum Mean Discrepancy (MMD, CMMD in their HSIC formulations) [6, 5, 15], Distance or Brownian Correlation (dCor) [24, 23] and the Maximal Information Coefficient (MIC) [18]. However, these methods exhibit high computational demands (at least quadratic costs in the number of samples for KCCA, MMD, CMMD, dCor or MIC), are limited to measuring dependencies between scalar random variables (ACE, MIC), show poor performance under the existence of additive noise (MIC) or can be difficult to implement (ACE, MIC).
This paper develops the Randomized Dependence Coefficient (RDC), an estimator of the HirschfeldGebeleinRényi Maximum Correlation Coefficient (HGR) addressing the issues listed above. RDC defines dependence between two random variables as the largest canonical correlation between random nonlinear projections of their respective empirical copulatransformations. RDC is invariant to monotonically increasing transformations, operates on random variables of arbitrary dimension, and has computational cost of with respect to the sample size. Moreover, it is easy to implement: just five lines of R code, included in Appendix A.
The following Section reviews the classic work of Alfréd Rényi [17], who proposed seven desirable fundamental properties of dependence measures, proved to be satisfied by the HirschfeldGebeleinRényi’s Maximum Correlation Coefficient (HGR). Section 3 introduces the Randomized Dependence Coefficient as an estimator designed in the spirit of HGR, since HGR itself is computationally intractable. Properties of RDC and its relationship to other nonlinear dependence measures are analysed in Section 4. Section 5 validates the empirical performance of RDC on a series of numerical experiments on both synthetic and realworld data.
2 HirschfeldGebeleinRényi’s Maximum Correlation Coefficient
In 1959 [17], Alfréd Rényi argued that a measure of dependence between random variables and should satisfy seven fundamental properties:

is defined for any pair of nonconstant random variables and .



iff and are statistically independent.

For bijective Borelmeasurable functions , .

if for Borelmeasurable functions or , or .

If , then , where is the correlation coefficient.
Rényi also showed the HirschfeldGebeleinRényi Maximum Correlation Coefficient (HGR) [3, 17] to satisfy all these properties. HGR was defined by Gebelein in 1941 [3] as the supremum of Pearson’s correlation coefficient over all Borelmeasurable functions of finite variance:
(1) 
Since the supremum in (1) is over an infinitedimensional space, HGR is not computable. It is an abstract concept, not a practical dependence measure. In the following we propose a scalable estimator with the same structure as HGR: the Randomized Dependence Coefficient.
3 Randomized Dependence Coefficient
The Randomized Dependence Coefficient (RDC) measures the dependence between random samples and as the largest canonical correlation between randomly chosen nonlinear projections of their copula transformations. Before Section 3.4 defines this concept formally, we describe the three necessary steps to construct the RDC statistic: copulatransformation of each of the two random samples (Section 3.1), projection of the copulas through randomly chosen nonlinear maps (Section 3.2) and computation of the largest canonical correlation between the two sets of nonlinear random projections (Section 3.3). Figure 1 offers a sketch of this process.
3.1 Estimation of CopulaTransformations
To achieve invariance with respect to transformations on marginal distributions (such as shifts or rescalings), we operate on the empirical copula transformation of the data [14, 15]. Consider a random vector with continuous marginal cumulative distribution functions (cdfs) , . Then the vector , known as the copula transformation, has uniform marginals:
Theorem 1.
(Probability Integral Transform [14]) For a random variable with cdf , the random variable is uniformly distributed on .
The random variables are known as the observation ranks of . Crucially, preserves the dependence structure of the original random vector , but ignores each of its marginal forms [14]. The joint distribution of is known as the copula of :
Theorem 2.
(Sklar [20]) Let the random vector have continuous marginal cdfs , . Then, the joint cumulative distribution of is uniquely expressed as:
(2) 
where the distribution is known as the copula of .
A practical estimator of the univariate cdfs is the empirical cdf:
(3) 
which gives rise to the empirical copula transformations of a multivariate sample:
(4) 
The MassartDvoretzkyKieferWolfowitz inequality [13] can be used to show that empirical copula transformations converge fast to the true transformation as the sample size increases:
Theorem 3.
(Convergence of the empirical copula, [15, Lemma 7]) Let be an i.i.d. sample from a probability distribution over with marginal cdf’s . Let be the copula transformation and the empirical copula transformation. Then, for any :
(5) 
Computing involves sorting the marginals of , thus operations.
3.2 Generation of Random NonLinear Projections
The second step of the RDC computation is to augment the empirical copula transformations with nonlinear projections, so that linear methods can subsequently be used to capture nonlinear dependencies on the original data. This is a classic idea also used in other areas, particularly in regression. In an elegant result, Rahimi and Brecht [16] proved that linear regression on random, nonlinear projections of the original feature space can generate highperformance regressors:
Theorem 4.
(RahimiBrecht) Let be a distribution on and . Let . Draw iid from . Further let , and be some Lipschitz loss function, and consider data drawn iid from some arbitrary . The for which minimizes the empirical risk has a distance from the optimal estimator in bounded by
(6) 
with probability at least .
Intuitively, Theorem 4 states that randomly selecting in instead of optimising them causes only bounded error.
The choice of the nonlinearities is the main, unavoidable assumption in RDC. This choice is a wellknown problem common to all nonlinear regression methods and has been studied extensively in the theory of regression as the selection of reproducing kernel Hilbert space [19, §3.13]. The choice of the family (space) of features, and of probability distributions over it, is unlimited. The only way to favour one such family and distribution over another is to use prior assumptions about which kind of distributions the method will typically have to analyse.
Features popular in parts of the literature are sigmoids, parabolas, radial basis functions, complex sinusoids, sines or cosines. In our experiments, we will use sine and cosine projections, . Arguments favouring this choice are that shiftinvariant kernels are approximated with these features when using the appropriate random parameter sampling distribution [16],[4, p. 208] [22, p. 24], and that functions with absolutely integrable Fourier transforms are approximated with error below by of these features [10].
Let the random parameters , . Choosing to be Normal is analogous to the use of the Gaussian kernel for MMD, CMMD or KCCA [16]. Tuning is analogous to selecting the kernel width, that is, to regularize the nonlinearity of the random projections.
Given a data collection , we will denote by
(7) 
the th order random nonlinear projection from to . The computational complexity of computing using naive matrix multiplications is . However, recent techniques [11] allow computing these feature expansions within a computational cost of using storage.
3.3 Computation of Canonical Correlations
The final step of RDC is to compute the linear combinations of the augmented empirical copula transformations that have maximal correlation. Canonical Correlation Analysis (CCA, [7]) is the calculation of pairs of basis vectors such that the projections and of two random samples and are maximally correlated. The correlations between the projected (or canonical) random samples are referred to as canonical correlations. There exist up to of them. Canonical correlations are the solutions to the eigenproblem:
(8) 
where and the matrices and are assumed to be invertible. Therefore, the largest canonical correlation between and is the supremum of the correlation coefficients over their linear projections, that is:
When , the cost of CCA is dominated by the estimation of the matrices , and , hence being for two random variables of dimensions and , respectively.
3.4 Formal Definition or RDC
Given the random samples and and the parameters and , the Randomized Dependence Coefficient between and is defined as:
(9) 
4 Properties of RDC
Computational complexity:
In the typical setup (very large , large and , small ) the computational complexity of RDC is dominated by the calculation of the copulatransformations. Hence, we achieve a cost in terms of the sample size of .
Ease of implementation:
An implementation of RDC in R is included in the Appendix A.
Relationship to the HGR coefficient:
It is tempting to wonder whether RDC is a consistent, or even an efficient estimator of the HGR coefficient. However, a simple experiment shows that it is not desirable to approximate HGR exactly on finite datasets: Consider which is independent, thus, by both Rényi’s 4th and 7th properties, has . However, for finitely many samples from , almost surely, values in both and are pairwise different and separated by a finite difference. So there exist continuous (thus Borel measurable) functions and mapping both and to the sorting ranks of , i.e. . Therefore, the finitesample version of Equation (1) is constant and equal to “1” for continuous random variables. Meaningful measures of dependence from finite samples thus must rely on some form of regularization. RDC achieves this by approximating the space of Borel measurable functions with the restricted function class from Theorem 4:
Assume the optimal transformations and (Equation 1) to belong to the Reproducing Kernel Hilbert Space (Theorem 4), with associated shiftinvariant, positive semidefinite kernel function . Then, with probability greater than :
(10) 
where and , denote the sample size and number of random features. The bound (10) is the sum of two errors. The error is due to the convergence of CCA’s largest eigenvalue in the finite sample size regime. This result [8, Theorem 6] is originally obtained by posing CCA as a least squares regression on the product space induced by the feature map . Because of approximating with random features, an additional error is introduced in the least squares regression [16, Lemma 3]. Therefore, an equivalence between RDC and KCCA is established if RDC uses an infinite number of sine/cosine features, the random sampling distribution is set to the inverse Fourier transform of the shiftinvariant kernel used by KCCA and the copulatransformations are discarded. However, when regularization is needed to avoid spurious perfect correlations, as discussed above.
Relationship to other estimators:
Table 1 summarizes several stateoftheart dependence measures showing, for each measure, whether it allows for general nonlinear dependence estimation, handles multidimensional random variables, is invariant with respect to changes in marginal distributions, returns a statistic in , satisfy Rényi’s properties (Section 2), and how many parameters it requires. As parameters, we here count the kernel function for kernel methods, the basis function and number of random features for RDC, the stopping tolerance for ACE and the searchgrid size for MIC, respectively. Finally, the table lists computational complexities with respect to sample size.
Testing for independence with RDC:
Consider the hypothesis “the two sets of nonlinear projections are mutually uncorrelated”. Under normality assumptions (or large sample sizes), Bartlett’s approximation [12] can be used to show:
(11) 
where are the canonical correlations between the two sets of nonlinear projections and . Alternatively, nonparametric asymptotic distributions can be obtained from the spectrum of the inner products of the nonlinear random projection matrices [25, Theorem 3].
5 Experimental Results
We performed numerical experiments on both synthetic and realworld data to validate the empirical performance of RDC versus the nonlinear dependence measures listed in Table 1. In some experiments, we don’t compare against to KCCA due its prohibitive running times (see Table 2).
Parameter selection:
The number of random features for RDC was set to symmetrically for both random samples, since no significant improvements were observed for larger values. However, this parameter can be set to the largest value that fits within the available computational budget. The random sampling parameters were set independently for each of the two random samples, equal to their squared euclidean distance empirical median [5]. Competing kernel methods make use of Gaussian RBF kernels of the form for the random variable and analogously for the random variable . For the MIC statistic, the searchgrid size is set to , as recommended by the authors [18]. The stopping tolerance for ACE is set to , the default value in the R package acepack^{1}^{1}1http://cran.rproject.org/web/packages/acepack/index.html.
5.1 Synthetic Data
Resistance to additive noise:
We define the power of a dependence measure as its ability to discern between dependent and independent samples that share equal marginal forms. In the spirit of Simon and Tibshirani^{2}^{2}2http://wwwstat.stanford.edu/~tibs/reshef/comment.pdf, we conducted experiments to estimate the power of RDC as a measure of nonlinear dependence. We chose 8 bivariate association patterns, depicted inside little boxes in Figure 3. For each of the 8 association patterns, 500 repetitions of 500 samples were generated, in which the input variable was uniformly distributed on the unit interval. Next, we regenerated the input variable randomly, to generate independent versions of each sample with equal marginals. Figure 3 shows the power for the discussed nonlinear dependence measures as the variance of some zeromean Gaussian additive noise increases from to . RDC shows worse performance in the linear association pattern due to noise overfitting and in the stepfunction due to the smoothness prior induced by the use of sine/cosine basis functions, but has good performance in nonfunctional association patterns (such as the circle and the mixture of sinusoidal waves).
Running times:
Table 2 summarizes running times (in seconds) for the considered nonlinear dependence measures on scalar, uniformly distributed, independent samples of sizes when averaging over 100 runs. Single runs above ten minutes were cancelled (empty cells in table). In this comparison, Pearson’s , ACE, dCor and MIC are using compiled C code, while RDC, along with MMD, CMMD and KCCA are implemented as interpreted R code.
sample size  Pearson’s  RDC  ACE  dCor  MMD  CMMD  MIC  KCCA 

1,000  0.0001  0.0047  0.0080  0.3417  0.3103  0.3501  1.0983  166.29 
10,000  0.0002  0.0557  0.0782  59.587  27.630  29.522  —  — 
100,000  0.0071  0.3991  0.5101  —  —  —  —  — 
1,000,000  0.0914  4.6253  5.3830  —  —  —  —  — 
Value of statistic in :
Figure 4 shows RDC, ACE, dCor, MIC, Pearson’s , Spearman’s rank and Kendall’s dependence estimates for 14 different associations of two scalar random variables. RDC scores values close to one on all the proposed dependent associations, whilst scoring values close to zero for the independent association, depicted last. When the associations are Gaussian (first row), RDC scores values close to the Pearson’s correlation coefficient, as suggested in the seventh property of Rényi (Section 2).
5.2 Feature Selection in RealWorld Data
We performed greedy feature selection via dependence maximization [21] on eight realworld datasets. More specifically, we attempted to construct the subset of features that minimizes the normalized mean squared regression error (NMSE) of a Gaussian process regressor. We do so by selecting the feature maximizing dependence between the feature set and the target variable at each iteration , such that and .
We considered 12 heterogeneous datasets, obtained from the UCI dataset repository^{3}^{3}3http://www.ics.uci.edu/~mlearn, the Gaussian process web site Data^{4}^{4}4http://www.gaussianprocess.org/gpml/data/ and the Machine Learning data set repostitory^{5}^{5}5http://www.mldata.org. Random training/test partitions are computed to be disjoint and equal sized.
Since can be multidimensional, we compare RDC to the nonlinear methods dCor, MMD and CMMD. Given their quadratic computational demands, dCor, MMD and CMMD use up to points when measuring dependence; this constraint only applied on the sarcos and calcensus datasets. Results are averages of random training/test partitions.
Figure 2 summarizes the results for all datasets and algorithms as the number of selected features increases. RDC performs best in most datasets, with much lower running time than its contenders.
6 Conclusion
We have presented the randomized dependence coefficient, a lightweight nonlinear measure of dependence between multivariate random samples. Constructed as a finitedimensional estimator in the spirit of the HirschfeldGebeleinRényi maximum correlation coefficient, RDC performs well empirically, is scalable to very large datasets, and is easy to adapt to concrete problems.
Appendix A R Source Code
rdc < function(x,y,k,s) { x < cbind(apply(as.matrix(x),2,function(u) ecdf(u)(u)),1) y < cbind(apply(as.matrix(y),2,function(u) ecdf(u)(u)),1) wx < matrix(rnorm(ncol(x)*k,0,s),ncol(x),k) wy < matrix(rnorm(ncol(y)*k,0,s),ncol(y),k) cancor(cbind(cos(x%*%wx),sin(x%*%wx)), cbind(cos(y%*%wy),sin(y%*%wy)))$cor[1] }
References
 [1] F. R. Bach and M. I. Jordan. Kernel independent component analysis. JMLR, 3:1–48, 2002.
 [2] L. Breiman and J. H. Friedman. Estimating Optimal Transformations for Multiple Regression and Correlation. Journal of the American Statistical Association, 80(391):580–598, 1985.
 [3] H. Gebelein. Das statistische Problem der Korrelation als Variations und Eigenwertproblem und sein Zusammenhang mit der Ausgleichsrechnung. Zeitschrift für Angewandte Mathematik und Mechanik, 21(6):364–379, 1941.
 [4] I.I. Gihman and A.V. Skorohod. The Theory of Stochastic Processes, volume 1. Springer, 1974s.
 [5] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel twosample test. JMLR, 13:723–773, 2012.
 [6] A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf. Measuring statistical dependence with HilbertSchmidt norms. In Proceedings of the 16th international conference on Algorithmic Learning Theory, pages 63–77. SpringerVerlag, 2005.
 [7] W. K. Härdle and L. Simar. Applied Multivariate Statistical Analysis. Springer, 2nd edition, 2007.
 [8] D. Hardoon and J. ShaweTaylor. Convergence analysis of kernel canonical correlation analysis: theory and practice. Machine Learning, 74(1):23–38, 2009.
 [9] T. Hastie and R. Tibshirani. Generalized additive models. Statistical Science, 1:297–310, 1986.
 [10] L. K. Jones. A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training. Annals of Statistics, 20(1):608–613, 1992.
 [11] Q. Le, T. Sarlos, and A. Smola. Fastfood – Approximating kernel expansions in loglinear time. In ICML, 2013.
 [12] K. V. Mardia, J. T. Kent, and J. M. Bibby. Multivariate Analysis. Academic Press, 1979.
 [13] P. Massart. The tight constant in the DvoretzkyKieferwolfowitz inequality. The Annals of Probability, 18(3), 1990.
 [14] R. Nelsen. An Introduction to Copulas. Springer Series in Statistics, 2nd edition, 2006.
 [15] B. Poczos, Z. Ghahramani, and J. Schneider. Copulabased kernel dependency measures. In ICML, 2012.
 [16] A. Rahimi and B. Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. NIPS, 2008.
 [17] A. Rényi. On measures of dependence. Acta Mathematica Academiae Scientiarum Hungaricae, 10:441–451, 1959.
 [18] D. N. Reshef, Y. A. Reshef, H. K. Finucane, S. R. Grossman, G. McVean, P. J. Turnbaugh, E. S. Lander, M. Mitzenmacher, and P. C. Sabeti. Detecting novel associations in large data sets. Science, 334(6062):1518–1524, 2011.
 [19] B. Schölkopf and A.J. Smola. Learning with Kernels. MIT Press, 2002.
 [20] A. Sklar. Fonctions de repartition à dimension set leurs marges. Publ. Inst. Statis. Univ. Paris, 8(1):229–231, 1959.
 [21] L. Song, A. Smola, A. Gretton, J. Bedo, and K. Borgwardt. Feature selection via dependence maximization. JMLR, 13:1393–1434, June 2012.
 [22] M.L. Stein. Interpolation of Spatial Data. Springer, 1999.
 [23] G. J. Székely and M. L. Rizzo. Rejoinder: Brownian distance covariance. Annals of Applied Statistics, 3(4):1303–1308, 2009.
 [24] G. J. Székely, M. L. Rizzo, and N. K. Bakirov. Measuring and testing dependence by correlation of distances. Annals of Statistics, 35(6), 2007.
 [25] K. Zhang, J. Peters, D. Janzing, and B.Schölkopf. Kernelbased conditional independence test and application in causal discovery. CoRR, abs/1202.3775, 2012.