The Randomized Dependence Coefficient

The Randomized Dependence Coefficient

David Lopez-Paz, Philipp Hennig, Bernhard Schölkopf
Max Planck Institute for Intelligent Systems
Spemannstraße 38, Tübingen, Germany
{dlopez,phennig,bs}@tue.mpg.de
Abstract

We introduce the Randomized Dependence Coefficient (RDC), a measure of non-linear dependence between random variables of arbitrary dimension based on the Hirschfeld-Gebelein-Rényi Maximum Correlation Coefficient. RDC is defined in terms of correlation of random non-linear copula projections; it is invariant with respect to marginal distribution transformations, has low computational cost and is easy to implement: just five lines of R code, included at the end of the paper.

 

The Randomized Dependence Coefficient


  David Lopez-Paz, Philipp Hennig, Bernhard Schölkopf Max Planck Institute for Intelligent Systems Spemannstraße 38, Tübingen, Germany {dlopez,phennig,bs}@tue.mpg.de

1 Introduction

Measuring statistical dependence between random variables is a fundamental problem in statistics. Commonly used measures of dependence, Pearson’s rho, Spearman’s rank or Kendall’s tau are computationally efficient and theoretically well understood, but consider only a limited class of association patterns, like linear or monotonically increasing functions. The development of non-linear dependence measures is challenging because of the radically larger amount of possible association patterns.

Despite these difficulties, many non-linear statistical dependence measures have been developed recently. Examples include the Alternating Conditional Expectations or backfitting algorithm (ACE) [2, 9], Kernel Canonical Correlation Analysis (KCCA) [1], (Copula) Maximum Mean Discrepancy (MMD, CMMD in their HSIC formulations) [6, 5, 15], Distance or Brownian Correlation (dCor) [24, 23] and the Maximal Information Coefficient (MIC) [18]. However, these methods exhibit high computational demands (at least quadratic costs in the number of samples for KCCA, MMD, CMMD, dCor or MIC), are limited to measuring dependencies between scalar random variables (ACE, MIC), show poor performance under the existence of additive noise (MIC) or can be difficult to implement (ACE, MIC).

This paper develops the Randomized Dependence Coefficient (RDC), an estimator of the Hirschfeld-Gebelein-Rényi Maximum Correlation Coefficient (HGR) addressing the issues listed above. RDC defines dependence between two random variables as the largest canonical correlation between random non-linear projections of their respective empirical copula-transformations. RDC is invariant to monotonically increasing transformations, operates on random variables of arbitrary dimension, and has computational cost of with respect to the sample size. Moreover, it is easy to implement: just five lines of R code, included in Appendix A.

The following Section reviews the classic work of Alfréd Rényi [17], who proposed seven desirable fundamental properties of dependence measures, proved to be satisfied by the Hirschfeld-Gebelein-Rényi’s Maximum Correlation Coefficient (HGR). Section 3 introduces the Randomized Dependence Coefficient as an estimator designed in the spirit of HGR, since HGR itself is computationally intractable. Properties of RDC and its relationship to other non-linear dependence measures are analysed in Section 4. Section 5 validates the empirical performance of RDC on a series of numerical experiments on both synthetic and real-world data.

2 Hirschfeld-Gebelein-Rényi’s Maximum Correlation Coefficient

In 1959 [17], Alfréd Rényi argued that a measure of dependence between random variables and should satisfy seven fundamental properties:

  1. is defined for any pair of non-constant random variables and .

  2. iff and are statistically independent.

  3. For bijective Borel-measurable functions , .

  4. if for Borel-measurable functions or , or .

  5. If , then , where is the correlation coefficient.

Rényi also showed the Hirschfeld-Gebelein-Rényi Maximum Correlation Coefficient (HGR) [3, 17] to satisfy all these properties. HGR was defined by Gebelein in 1941 [3] as the supremum of Pearson’s correlation coefficient over all Borel-measurable functions of finite variance:

(1)

Since the supremum in (1) is over an infinite-dimensional space, HGR is not computable. It is an abstract concept, not a practical dependence measure. In the following we propose a scalable estimator with the same structure as HGR: the Randomized Dependence Coefficient.

3 Randomized Dependence Coefficient

The Randomized Dependence Coefficient (RDC) measures the dependence between random samples and as the largest canonical correlation between randomly chosen non-linear projections of their copula transformations. Before Section 3.4 defines this concept formally, we describe the three necessary steps to construct the RDC statistic: copula-transformation of each of the two random samples (Section 3.1), projection of the copulas through randomly chosen non-linear maps (Section 3.2) and computation of the largest canonical correlation between the two sets of non-linear random projections (Section 3.3). Figure 1 offers a sketch of this process.

Figure 1: RDC computation for a simple set of samples drawn from a noisy circular pattern: The samples are used to estimate the copula, then mapped with randomly drawn non-linear functions. The RDC is the largest canonical correlation between these non-linear projections.

3.1 Estimation of Copula-Transformations

To achieve invariance with respect to transformations on marginal distributions (such as shifts or rescalings), we operate on the empirical copula transformation of the data [14, 15]. Consider a random vector with continuous marginal cumulative distribution functions (cdfs) , . Then the vector , known as the copula transformation, has uniform marginals:

Theorem 1.

(Probability Integral Transform [14]) For a random variable with cdf , the random variable is uniformly distributed on .

The random variables are known as the observation ranks of . Crucially, preserves the dependence structure of the original random vector , but ignores each of its marginal forms [14]. The joint distribution of is known as the copula of :

Theorem 2.

(Sklar [20]) Let the random vector have continuous marginal cdfs , . Then, the joint cumulative distribution of is uniquely expressed as:

(2)

where the distribution is known as the copula of .

A practical estimator of the univariate cdfs is the empirical cdf:

(3)

which gives rise to the empirical copula transformations of a multivariate sample:

(4)

The Massart-Dvoretzky-Kiefer-Wolfowitz inequality [13] can be used to show that empirical copula transformations converge fast to the true transformation as the sample size increases:

Theorem 3.

(Convergence of the empirical copula, [15, Lemma 7]) Let be an i.i.d. sample from a probability distribution over with marginal cdf’s . Let be the copula transformation and the empirical copula transformation. Then, for any :

(5)

Computing involves sorting the marginals of , thus operations.

3.2 Generation of Random Non-Linear Projections

The second step of the RDC computation is to augment the empirical copula transformations with non-linear projections, so that linear methods can subsequently be used to capture non-linear dependencies on the original data. This is a classic idea also used in other areas, particularly in regression. In an elegant result, Rahimi and Brecht [16] proved that linear regression on random, non-linear projections of the original feature space can generate high-performance regressors:

Theorem 4.

(Rahimi-Brecht) Let be a distribution on and . Let . Draw iid from . Further let , and be some -Lipschitz loss function, and consider data drawn iid from some arbitrary . The for which minimizes the empirical risk has a distance from the -optimal estimator in bounded by

(6)

with probability at least .

Intuitively, Theorem 4 states that randomly selecting in instead of optimising them causes only bounded error.

The choice of the non-linearities is the main, unavoidable assumption in RDC. This choice is a well-known problem common to all non-linear regression methods and has been studied extensively in the theory of regression as the selection of reproducing kernel Hilbert space [19, §3.13]. The choice of the family (space) of features, and of probability distributions over it, is unlimited. The only way to favour one such family and distribution over another is to use prior assumptions about which kind of distributions the method will typically have to analyse.

Features popular in parts of the literature are sigmoids, parabolas, radial basis functions, complex sinusoids, sines or cosines. In our experiments, we will use sine and cosine projections, . Arguments favouring this choice are that shift-invariant kernels are approximated with these features when using the appropriate random parameter sampling distribution [16],[4, p. 208] [22, p. 24], and that functions with absolutely integrable Fourier transforms are approximated with error below by of these features [10].

Let the random parameters , . Choosing to be Normal is analogous to the use of the Gaussian kernel for MMD, CMMD or KCCA [16]. Tuning is analogous to selecting the kernel width, that is, to regularize the non-linearity of the random projections.

Given a data collection , we will denote by

(7)

the th order random non-linear projection from to . The computational complexity of computing using naive matrix multiplications is . However, recent techniques [11] allow computing these feature expansions within a computational cost of using storage.

3.3 Computation of Canonical Correlations

The final step of RDC is to compute the linear combinations of the augmented empirical copula transformations that have maximal correlation. Canonical Correlation Analysis (CCA, [7]) is the calculation of pairs of basis vectors such that the projections and of two random samples and are maximally correlated. The correlations between the projected (or canonical) random samples are referred to as canonical correlations. There exist up to of them. Canonical correlations are the solutions to the eigenproblem:

(8)

where and the matrices and are assumed to be invertible. Therefore, the largest canonical correlation between and is the supremum of the correlation coefficients over their linear projections, that is:

When , the cost of CCA is dominated by the estimation of the matrices , and , hence being for two random variables of dimensions and , respectively.

3.4 Formal Definition or RDC

Given the random samples and and the parameters and , the Randomized Dependence Coefficient between and is defined as:

(9)

4 Properties of RDC

Computational complexity:

In the typical setup (very large , large and , small ) the computational complexity of RDC is dominated by the calculation of the copula-transformations. Hence, we achieve a cost in terms of the sample size of .

Ease of implementation:

An implementation of RDC in R is included in the Appendix A.

Relationship to the HGR coefficient:

It is tempting to wonder whether RDC is a consistent, or even an efficient estimator of the HGR coefficient. However, a simple experiment shows that it is not desirable to approximate HGR exactly on finite datasets: Consider which is independent, thus, by both Rényi’s 4th and 7th properties, has . However, for finitely many samples from , almost surely, values in both and are pairwise different and separated by a finite difference. So there exist continuous (thus Borel measurable) functions and mapping both and to the sorting ranks of , i.e. . Therefore, the finite-sample version of Equation (1) is constant and equal to “1” for continuous random variables. Meaningful measures of dependence from finite samples thus must rely on some form of regularization. RDC achieves this by approximating the space of Borel measurable functions with the restricted function class from Theorem 4:

Assume the optimal transformations and (Equation 1) to belong to the Reproducing Kernel Hilbert Space (Theorem 4), with associated shift-invariant, positive semi-definite kernel function . Then, with probability greater than :

(10)

where and , denote the sample size and number of random features. The bound (10) is the sum of two errors. The error is due to the convergence of CCA’s largest eigenvalue in the finite sample size regime. This result [8, Theorem 6] is originally obtained by posing CCA as a least squares regression on the product space induced by the feature map . Because of approximating with random features, an additional error is introduced in the least squares regression [16, Lemma 3]. Therefore, an equivalence between RDC and KCCA is established if RDC uses an infinite number of sine/cosine features, the random sampling distribution is set to the inverse Fourier transform of the shift-invariant kernel used by KCCA and the copula-transformations are discarded. However, when regularization is needed to avoid spurious perfect correlations, as discussed above.

Relationship to other estimators:

Table 1 summarizes several state-of-the-art dependence measures showing, for each measure, whether it allows for general non-linear dependence estimation, handles multidimensional random variables, is invariant with respect to changes in marginal distributions, returns a statistic in , satisfy Rényi’s properties (Section 2), and how many parameters it requires. As parameters, we here count the kernel function for kernel methods, the basis function and number of random features for RDC, the stopping tolerance for ACE and the search-grid size for MIC, respectively. Finally, the table lists computational complexities with respect to sample size.

Name of Coeff. Non-Linear Vector Inputs Marginal Invariant Renyi’s Properties Coeff. # Par. Comp. Cost
Pearson’s 0
Spearman’s 0
Kendall’s 0
CCA 0
KCCA [1] 1
ACE [2] 1
MIC [18] 1
dCor [24] 1
MMD [5] 1
CMMD [15] 1
RDC 2
Table 1: Comparison between non-linear dependence measures.

Testing for independence with RDC:

Consider the hypothesis “the two sets of non-linear projections are mutually uncorrelated”. Under normality assumptions (or large sample sizes), Bartlett’s approximation [12] can be used to show:

(11)

where are the canonical correlations between the two sets of non-linear projections and . Alternatively, non-parametric asymptotic distributions can be obtained from the spectrum of the inner products of the non-linear random projection matrices [25, Theorem 3].

5 Experimental Results

We performed numerical experiments on both synthetic and real-world data to validate the empirical performance of RDC versus the non-linear dependence measures listed in Table 1. In some experiments, we don’t compare against to KCCA due its prohibitive running times (see Table 2).

Parameter selection:

The number of random features for RDC was set to symmetrically for both random samples, since no significant improvements were observed for larger values. However, this parameter can be set to the largest value that fits within the available computational budget. The random sampling parameters were set independently for each of the two random samples, equal to their squared euclidean distance empirical median [5]. Competing kernel methods make use of Gaussian RBF kernels of the form for the random variable and analogously for the random variable . For the MIC statistic, the search-grid size is set to , as recommended by the authors [18]. The stopping tolerance for ACE is set to , the default value in the R package acepack111http://cran.r-project.org/web/packages/acepack/index.html.

5.1 Synthetic Data

Resistance to additive noise:

We define the power of a dependence measure as its ability to discern between dependent and independent samples that share equal marginal forms. In the spirit of Simon and Tibshirani222http://www-stat.stanford.edu/~tibs/reshef/comment.pdf, we conducted experiments to estimate the power of RDC as a measure of non-linear dependence. We chose 8 bivariate association patterns, depicted inside little boxes in Figure 3. For each of the 8 association patterns, 500 repetitions of 500 samples were generated, in which the input variable was uniformly distributed on the unit interval. Next, we regenerated the input variable randomly, to generate independent versions of each sample with equal marginals. Figure 3 shows the power for the discussed non-linear dependence measures as the variance of some zero-mean Gaussian additive noise increases from to . RDC shows worse performance in the linear association pattern due to noise overfitting and in the step-function due to the smoothness prior induced by the use of sine/cosine basis functions, but has good performance in non-functional association patterns (such as the circle and the mixture of sinusoidal waves).

Running times:

Table 2 summarizes running times (in seconds) for the considered non-linear dependence measures on scalar, uniformly distributed, independent samples of sizes when averaging over 100 runs. Single runs above ten minutes were cancelled (empty cells in table). In this comparison, Pearson’s , ACE, dCor and MIC are using compiled C code, while RDC, along with MMD, CMMD and KCCA are implemented as interpreted R code.

sample size Pearson’s RDC ACE dCor MMD CMMD MIC KCCA
1,000 0.0001 0.0047 0.0080 0.3417 0.3103 0.3501 1.0983 166.29
10,000 0.0002 0.0557 0.0782 59.587 27.630 29.522
100,000 0.0071 0.3991 0.5101
1,000,000 0.0914 4.6253 5.3830
Table 2: Average running times (in seconds) for dependence measures on versus sample sizes.

Value of statistic in :

Figure 4 shows RDC, ACE, dCor, MIC, Pearson’s , Spearman’s rank and Kendall’s dependence estimates for 14 different associations of two scalar random variables. RDC scores values close to one on all the proposed dependent associations, whilst scoring values close to zero for the independent association, depicted last. When the associations are Gaussian (first row), RDC scores values close to the Pearson’s correlation coefficient, as suggested in the seventh property of Rényi (Section 2).

5.2 Feature Selection in Real-World Data

We performed greedy feature selection via dependence maximization [21] on eight real-world datasets. More specifically, we attempted to construct the subset of features that minimizes the normalized mean squared regression error (NMSE) of a Gaussian process regressor. We do so by selecting the feature maximizing dependence between the feature set and the target variable at each iteration , such that and .

We considered 12 heterogeneous datasets, obtained from the UCI dataset repository333http://www.ics.uci.edu/~mlearn, the Gaussian process web site Data444http://www.gaussianprocess.org/gpml/data/ and the Machine Learning data set repostitory555http://www.mldata.org. Random training/test partitions are computed to be disjoint and equal sized.

Since can be multi-dimensional, we compare RDC to the non-linear methods dCor, MMD and CMMD. Given their quadratic computational demands, dCor, MMD and CMMD use up to points when measuring dependence; this constraint only applied on the sarcos and calcensus datasets. Results are averages of random training/test partitions.

Figure 2: Feature selection experiments on real-world datasets.

Figure 2 summarizes the results for all datasets and algorithms as the number of selected features increases. RDC performs best in most datasets, with much lower running time than its contenders.

6 Conclusion

We have presented the randomized dependence coefficient, a lightweight non-linear measure of dependence between multivariate random samples. Constructed as a finite-dimensional estimator in the spirit of the Hirschfeld-Gebelein-Rényi maximum correlation coefficient, RDC performs well empirically, is scalable to very large datasets, and is easy to adapt to concrete problems.

Figure 3: Power of discussed measures on several bivariate association patterns as noise increases. Insets show the noise-free form of each association pattern.
Figure 4: RDC, ACE, dCor, MINE, Pearson’s , Spearman’s rank and Kendall’s estimates (numbers in tables above plots, in that order) for several bivariate association patterns.

Appendix A R Source Code

rdc <- function(x,y,k,s) {
  x  <- cbind(apply(as.matrix(x),2,function(u) ecdf(u)(u)),1)
  y  <- cbind(apply(as.matrix(y),2,function(u) ecdf(u)(u)),1)
  wx <- matrix(rnorm(ncol(x)*k,0,s),ncol(x),k)
  wy <- matrix(rnorm(ncol(y)*k,0,s),ncol(y),k)
  cancor(cbind(cos(x%*%wx),sin(x%*%wx)), cbind(cos(y%*%wy),sin(y%*%wy)))$cor[1]
}

References

  • [1] F. R. Bach and M. I. Jordan. Kernel independent component analysis. JMLR, 3:1–48, 2002.
  • [2] L. Breiman and J. H. Friedman. Estimating Optimal Transformations for Multiple Regression and Correlation. Journal of the American Statistical Association, 80(391):580–598, 1985.
  • [3] H. Gebelein. Das statistische Problem der Korrelation als Variations- und Eigenwertproblem und sein Zusammenhang mit der Ausgleichsrechnung. Zeitschrift für Angewandte Mathematik und Mechanik, 21(6):364–379, 1941.
  • [4] I.I. Gihman and A.V. Skorohod. The Theory of Stochastic Processes, volume 1. Springer, 1974s.
  • [5] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Schölkopf, and A. Smola. A kernel two-sample test. JMLR, 13:723–773, 2012.
  • [6] A. Gretton, O. Bousquet, A. Smola, and B. Schölkopf. Measuring statistical dependence with Hilbert-Schmidt norms. In Proceedings of the 16th international conference on Algorithmic Learning Theory, pages 63–77. Springer-Verlag, 2005.
  • [7] W. K. Härdle and L. Simar. Applied Multivariate Statistical Analysis. Springer, 2nd edition, 2007.
  • [8] D. Hardoon and J. Shawe-Taylor. Convergence analysis of kernel canonical correlation analysis: theory and practice. Machine Learning, 74(1):23–38, 2009.
  • [9] T. Hastie and R. Tibshirani. Generalized additive models. Statistical Science, 1:297–310, 1986.
  • [10] L. K. Jones. A simple lemma on greedy approximation in Hilbert space and convergence rates for projection pursuit regression and neural network training. Annals of Statistics, 20(1):608–613, 1992.
  • [11] Q. Le, T. Sarlos, and A. Smola. Fastfood – Approximating kernel expansions in loglinear time. In ICML, 2013.
  • [12] K. V. Mardia, J. T. Kent, and J. M. Bibby. Multivariate Analysis. Academic Press, 1979.
  • [13] P. Massart. The tight constant in the Dvoretzky-Kiefer-wolfowitz inequality. The Annals of Probability, 18(3), 1990.
  • [14] R. Nelsen. An Introduction to Copulas. Springer Series in Statistics, 2nd edition, 2006.
  • [15] B. Poczos, Z. Ghahramani, and J. Schneider. Copula-based kernel dependency measures. In ICML, 2012.
  • [16] A. Rahimi and B. Recht. Weighted sums of random kitchen sinks: Replacing minimization with randomization in learning. NIPS, 2008.
  • [17] A. Rényi. On measures of dependence. Acta Mathematica Academiae Scientiarum Hungaricae, 10:441–451, 1959.
  • [18] D. N. Reshef, Y. A. Reshef, H. K. Finucane, S. R. Grossman, G. McVean, P. J. Turnbaugh, E. S. Lander, M. Mitzenmacher, and P. C. Sabeti. Detecting novel associations in large data sets. Science, 334(6062):1518–1524, 2011.
  • [19] B. Schölkopf and A.J. Smola. Learning with Kernels. MIT Press, 2002.
  • [20] A. Sklar. Fonctions de repartition à dimension set leurs marges. Publ. Inst. Statis. Univ. Paris, 8(1):229–231, 1959.
  • [21] L. Song, A. Smola, A. Gretton, J. Bedo, and K. Borgwardt. Feature selection via dependence maximization. JMLR, 13:1393–1434, June 2012.
  • [22] M.L. Stein. Interpolation of Spatial Data. Springer, 1999.
  • [23] G. J. Székely and M. L. Rizzo. Rejoinder: Brownian distance covariance. Annals of Applied Statistics, 3(4):1303–1308, 2009.
  • [24] G. J. Székely, M. L. Rizzo, and N. K. Bakirov. Measuring and testing dependence by correlation of distances. Annals of Statistics, 35(6), 2007.
  • [25] K. Zhang, J. Peters, D. Janzing, and B.Schölkopf. Kernel-based conditional independence test and application in causal discovery. CoRR, abs/1202.3775, 2012.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
""
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
   
Add comment
Cancel
Loading ...
355611
This is a comment super asjknd jkasnjk adsnkj
Upvote
Downvote
""
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters
Submit
Cancel

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test
Test description