With the huge influx of various data nowadays, extracting knowledge from them has become an interesting but tedious task among data scientists, particularly when the data come in heterogeneous form and have missing information. Many data completion techniques had been introduced, especially in the advent of kernel methods. However, among the many data completion techniques available in the literature, studies about mutually completing several incomplete kernel matrices have not been given much attention yet. In this paper, we present a new method, called Mutual Kernel Matrix Completion (MKMC) algorithm, that tackles this problem of mutually inferring the missing entries of multiple kernel matrices by combining the notions of data fusion and kernel matrix completion, applied on biological data sets to be used for classification task. We first introduced an objective function that will be minimized by exploiting the EM algorithm, which in turn results to an estimate of the missing entries of the kernel matrices involved. The completed kernel matrices are then combined to produce a model matrix that can be used to further improve the obtained estimates. An interesting result of our study is that the E-step and the M-step are given in closed form, which makes our algorithm efficient in terms of time and memory. After completion, the (completed) kernel matrices are then used to train an SVM classifier to test how well the relationships among the entries are preserved. Our empirical results show that the proposed algorithm bested the traditional completion techniques in preserving the relationships among the data points, and in accurately recovering the missing kernel matrix entries. By far, MKMC offers a promising solution to the problem of mutual estimation of a number of relevant incomplete kernel matrices.
Mutual Kernel Matrix Completion
Tsuyoshi Kato, Rachelle Rivero
|Faculty of Science and Engineering, Gunma University, Kiryu-shi, Gunma, 326–0338, Japan.|
|Center for Informational Biology, Ochanomizu University, Bunkyo-ku, Tokyo, 112–8610, Japan.|
Analysis of biological data has been an interesting topic among researchers in the field, due to the challenges presented by biology being a data-rich science: the data are represented in various forms, and are painstakingly obtained. Most of the time, the data obtained are noisy and have missing information. The simplest technique in dealing with missing data is to delete the data points with missing entries; however, it reduces the data size and its statistical power . For subsequent vector-based analysis, some missing data are imputed with zero (zero-imputation) or mean (mean-imputation) in advance. However, doing so dilutes the relationship among variables, which leads to underestimation of the variance [15, 12]. On the other hand, some methods estimate with a regression model or EM algorithm. The EM algorithm was introduced by Dempster et al. as a general approach to compute the maximum likelihood estimates iteratively for incomplete data sets . Each iteration consists of an expectation step (E-step) and a maximization step (M-step), hence the term EM algorithm.
Aside from missing information, another challenge faced by biological data is its heterogeneous representation. Biological data may come as strings (such as protein and genome sequences), microarrays (such as gene expression data), protein-protein interaction, or metabolic pathways. Simultaneous analysis of such data is difficult, unless they are converted into a single form. In Lanckriet et al. , each data set is represented by a generalized similarity relationship between pairs of genes or proteins. This similarity relationship is defined by a kernel function; and data sets represented via a kernel function can be combined directly. The resulting matrix whose entries are the relationships defined by a kernel function is called a kernel matrix. A huge literature about kernel methods and matrices is available: [5, 16, 19, 3].
Lanckriet et al.  utilized the simultaneous analysis of kernel matrices in solving the problem of protein classification. They optimally combined multiple kernel representations by formulating the problem as a convex optimization problem that can be solved using semidefinite programming (SDP) techniques [9, 10, 11]. They had shown that a statistical learning algorithm performs better if trained from the integrated data than from a single data alone.
Integrated kernel matrices can also be used in the analysis of a data set with missing entries. For example, a supervised network inference among proteins can be formulated as a kernel matrix completion problem, whose missing entries can be inferred from multiple biological data types such as gene expression, phylogenetic profiles, and amino acid sequences . Moreover, when a data set has missing entries, its corresponding kernel matrix will be incomplete as well. Inferring the missing entries of the corresponding matrix rather than the data matrix itself has several advantages : there are far fewer missing entries in the kernel matrix than in the actual data; and machine learning methods like support vector machines (SVMs) work with kernel matrices, to name a few. The completed kernel matrix, as well as the other relevant kernel matrices, can now be used in protein classification or protein network inference problems. Several kernel matrix completion methods that depend on other complete kernel matrices had been introduced: [18, 6, 7]. However, it is not always the case that only one data set is incomplete. To date, simultaneous analysis and mutual completion of kernel matrices have not been extensively studied, but is now becoming an increasing interest among data scientists. Kumar et al.  considered the problem of deriving a single consistent similarity kernel matrix (with binary entries) from a set of self-consistent but incomplete similarity kernel matrices. They solved this problem by simultaneously completing the kernel matrices and finding their best linear combination via SVM. However, their algorithm involves an iterative optimization of each of the kernel matrices and weights via semidefinite program (SDP) solver, making their method very time- and memory-consuming. Another multiple kernel completion technique is introduced in . In their work, they only have one set of objects to be observed but some relationships between pairs of these objects are missing, generating a kernel matrix with missing rows and columns. They then generated multiple rearrangements of the said incomplete kernel matrix, thus generating multiple incomplete kernel matrices (multi-view setting) and posed the problem as multi-view completion problem. Their method then completes a kernel matrix with the help from the other views of that matrix, until all of the views are completed.
Our study is more similar to the works of [18, 9, 11, 6], where different descriptions of yeast proteins are utilized (such as primary protein sequences, protein-protein interaction data, and mRNA expression data) for protein classification later on. Each description constitutes a data set. In contrast with their work, our study assumes that some or all of the descriptions (or data set) have missing rows and columns, where the set of proteins with missing information in each data set may not be the same. This is illustrated by the different positions of white rows and columns of the kernel matrices in Fig. 1 (c). We can also see from Fig. 1 how our study (c) differs from those of Tsuda et al. (a) and Kato et al. (b). The problem is now translated to multiple kernel matrix completion problem, which our study aims to solve.
In this paper, we present a solution to multiple kernel matrix completion problem by introducing an algorithm, which we call Mutual Kernel Matrix Completion (MKMC) algorithm, that exploits the EM algorithm to minimize the Kullback-Leibler (KL) divergences among the kernel matrices. The E-step and M-step are given in closed form (the derivations of which are given in Appendix A), making our method time- and memory-efficient.
The rest of the paper is organized as follows: The next section describes the setting for our proposed method; Sect. 3 describes the theory behind the proposed algorithm and how to implement it; Sect. 4 reviews the EM framework to show that the MKMC algorithm is an EM algorithm; Sect. 5 expounds the data used and experimental design; Sect. 5.2 discusses the empirical results; and Sect. 6 concludes.
2 Problem Setting
Suppose we wish to analyze objects and we have available relevant data sources for the same set of objects. The relationships among these objects in each of the data sources are given as entries in the following symmetric kernel matrices: . Suppose also that some or all of the available data sources have missing entries, as can be seen in Fig. 1 (c). The rows and columns with missing entries in each of the kernel matrices are rearranged such that the information is available only for the first samples in , and unavailable for the remaining samples. Then, the resultant reordered matrix, denoted by , can be partitioned as
where the symmetric submatrix have visible entries, while the submatrix and submatrix have all entries missing.
The goal of this study is to utilize the given information to infer the missing information and , thus completing the kernel matrices.
3 Mutual Kernel Matrix Completion (MKMC) Method
To determine the difference between two kernel matrices, we will make use of the KL divergence between the corresponding probability distributions of the kernel matrices. To define the KL divergence, let us relate each of the kernel matrices to the covariance of a zero-mean Gaussian distribution as follows:
where the ’s are called the empirical distributions. Also, let us consider a model matrix and relate it to the covariance of a zero-mean Gaussian distribution as called the model distribution. Then for , the KL divergence of probability distribution from is given by
and the KL divergence of the model matrix from is given by
where the size of and is . To infer the missing entries, we minimize the following objective function that takes the sum of the KL divergences:
where is the set of submatrices with missing entries, and is the model matrix.
Minimization of the KL divergence can be done by first fixing and minimizing the objective function with respect to and , then fixing the obtained solutions to update . Iteratively doing these steps monotonically decreases the KL divergence of from , for each . These steps correspond to E-step and M-step in an EM algorithm, as detailed in the next section.
Succinctly, the following two steps are repeated until convergence:
E-step. Fix the model matrix to obtain .
M-step. Use to update .
This algorithm is guaranteed to decrease the objective function monotonically. An outline of the proposed algorithm is given in Algorithm 1. In Step 7 of the algorithm, the model matrix is reordered and partitioned in the same way as the current . Since we are taking KL divergences from to each of s, the reordering of as well as the size of its submatrices will depend on that of for which it is being compared to at the moment. We denote the reordered and partitioned with respect to as
We then denote the sizes for the submatrices as for and ; for and ; and for and , where . and denote the matrix transpose of the submatrices and , respectively.
4 Connection to EM Algorithm
In this section, we review the EM framework and then show that MKMC is an EM algorithm.
4.1 EM Algorithm Revisited
The EM method is a numerical optimization algorithm for statistical inference of model parameters such as maximum likelihood estimation. From observed data , the maximum likelihood estimation finds the value of that maximizes the log-likelihood function defined as
where is the model. A penalizing term is often added to the log-likelihood function . The penalized log-likelihood function is given as
where is a predefined prior distribution of the model parameters. With an empirical distribution , a slightly generalized form can be considered:
Given the empirical distribution , the EM framework iteratively maximizes with respect to by repeating E-step and M-step alternately. In the EM framework, unobserved data are considered in addition to the observed data , and it is supposed that the joint distribution, , is in the exponential family . Namely, there exist vector-valued or matrix-valued functions, and , such that
where is known as the cumulant function
The pair of and is called the complete data . The model distribution of the observed data is the marginal distribution of the joint distribution:
In the E-step of -th iterate, the expected sufficient statistics
is computed where is the value determined in the previous iterate, and we denote . Then, in the M-step, the so-called Q-function  is maximized with respect to to get the new value , where the Q-function is defined as
If the penalized log-likelihood function is bounded from above, the convergence of the EM algorithm is ensured from the following property:
The EM algorithm produces a non-decreasing series of the penalized log-likelihood:
(Proof is given in Appendix C.1). The proposed algorithm, MKMC, makes the solution converge to a local optimum, and so does the EM algorithm.
4.2 MKMC is an EM Algorithm
We are now ready to establish a connection between MKMC and the EM framework. In a setting described below, we can show that MKMC is an EM algorithm. We first describe the setting of the empirical distribution, followed by the definition of the model distribution. Now let us suppose the setting in which the complete data is expressed as a set of -dimensional vectors where each entry in corresponds to a row or a column of . For , let and be the sub-vectors of , where the entries in correspond to available data in -th kernel matrix, and the entries in correspond to missing data in -th kernel matrix. In the context of EM framework, the observed data is and the unobserved data is . The second order moment of the empirical distribution is supposed to be given as
Notice that an example that satisfies the assumption in (12) is
Next, we define the model distribution. In the model, and the joint densities of the complete data are given by
Note that this model distribution is in the exponential family by setting
the derivations of which are given in Appendix C.3. The prior distribution of is defined as the following Wishart distribution:
We now apply the EM algorithm to this setting. In the E-step, the expectation of the sufficient statistics
has to be computed. The symmetric matrices have the following sub-matrices: , , and . The matrix does not depend on the model parameters . Letting
it can be shown that the remaining two expected matrices are given by
In the M-step, the Q-function is maximized with respect to . Substituting all the settings to (11), the Q-function is expressed as
which is maximized at
It has turned out that the update rules of the E-step and the M-step are same as those of MKMC algorithm. In conclusion, MKMC can be interpreted as an EM algorithm.
5 Experimental Results
5.1 Experimental Settings
The MKMC algorithm is applied to seven kernel matrices derived from three different types of data: four from primary protein sequence, two from protein-protein interaction data, and one from mRNA expression data. These are the data used in  to test their statistical framework for genomic data fusion and yeast protein classification. They have shown in their experiment that the knowledge obtained from all of the aforementioned data improved protein classification, as compared to classifying proteins using only a single type of data. In order to utilize the heterogeneous data for protein classification task, the data are reduced into the common format of kernel matrices, allowing them to combine . These kernel matrices are square symmetric matrices whose entries represent similarities among pairs of yeast proteins. Each gene is annotated as membrane (), non-membrane (), or unknown (). In this study we only considered proteins with known annotations, leaving 2,318 yeast proteins out of 6,112.
In our experiment, for each object, one of seven kernel matrices is picked and the corresponding row and column from that matrix are “removed”, i.e., replaced with undetermined values without changing the size of the matrix. We considered varying the ratio of the missing entries from 10% up to 90% missing entries. First, we randomly chose 10% of the entries to be missed, where completion methods were to be performed. Next, we randomly chose another 10% entries to be missed, in addition to the previous 10% — comprising the now 20% missing entries. This was repeatedly done until we reached the 90% missing entries. For comparison, other matrix completion techniques were considered such as zero-imputation and mean-imputation methods. Zero-imputation completes a kernel matrix by imputing zeros in its missing entries. This was also how we initialized the kernel matrices in the MKMC algorithm. On the other hand, imputing the mean of the remaining entries to complete a kernel matrix is called completion by mean-imputation. Appendix B describes how each of the kernel matrices was completed using this method.
The above-mentioned matrix completion techniques were individually performed on each of the seven kernel matrices. We then randomly picked data points to be used in training the SVM classifier for protein classification task afterwards.
In this study, 200 and 1,000 training data points were considered. We first made the division of training and testing datasets when 200 data are used for training, and then made the division when 1,000 training data are used. For the 200 training data, we randomly picked 200 out of 2,318 data points, and used them in training SVM classifiers in each of the kernel matrices. The remaining 2,118 data points served as testing data. The 1,000 training data points were obtained by retaining the first 200 training data points, and adding to it 800 more randomly-picked data points. This time, there were 1,318 testing data.
As regards to the completion methods, the MKMC algorithm gives us the completed kernel matrices , where is the number of kernel matrices used. SVM classifiers were then trained on these matrices, and on the combination of these kernel matrices, for the task of classifying a protein as membrane or non-membrane. Meanwhile, after completion by zero-imputation and mean-imputation, the kernel matrices were also combined to obtain the model matrix as in (22), where the SVM classifier was also trained alongside with the seven completed kernel matrices. The zero-imputation and mean-imputation methods, applied with SVM, were then called zero-SVM and mean-SVM, respectively.
In this section, the classification performance and accuracy of the completion methods used were evaluated. The methods’ classification performances were determined in different set-up: varying number of training points and varying ratio of missing entries. On the other hand, how accurate the methods were able to recover the missing data was evaluated by taking the mean of the correlation between the original matrices and the estimated matrices.
5.2.1 Classification Performance
The classification performance of the three completion methods were measured via the ROC scores. The ROC score, or area under the ROC curve, tells us how well a binary classifier can discriminate the data. Better classifiers achieve higher ROC scores. The ROC is also invariant of class label imbalance, which makes it the best tool in measuring the classification performance in our experiment since our data set is heavily imbalanced: the positively-labeled proteins are only 497 out of 2,318 (only ). In this experiment, we can determine how well a completion method preserves the relationship among pairs of proteins by measuring how accurate an SVM classifier can discriminate proteins as membrane or non-membrane.
The ROC scores of the three completion methods on each kernel matrix, for 200 and 1,000 training points, are shown in Fig. 2. Here, about of the entries in each of the kernel matrices were missing, and had been estimated by the three methods. Our empirical results show that as the number of training data increases, the SVM classification performance improves. We can also see from the plots that the model matrix , which is the combination of the completed kernel matrices, obtained the highest ROC score; this matches the claim by Lanckriet et al.  that a learning algorithm performs better if trained from the integrated data than from a single data alone. Most importantly, our proposed method obtained the highest ROC scores in all of the classification performance tests, with significant differences compared to zero-SVM and mean-SVM. The statistical differences were assessed using a two-sample -test. For 200 training data points, MKMC obtained an average ROC score of 0.779 for the model matrix , as compared to those of zero-SVM and mean-SVM: 0.721 (P-value ) and 0.745 (P-value ), respectively. On the other hand, when the number of training data is 1,000, MKMC achieved an average ROC score of 0.848 for as compared to zero-SVM: 0.836 and mean-SVM: 0.832 (P-values and , respectively).
In addition to the above tests, the effect of the amount of missing values in the kernel matrices on the methods’ performances were also determined. The ratio of the missing entries was artificially changed from (no missing entries, as baseline) to . Fig. 3 demonstrates the performances of MKMC, zero-SVM, and mean-SVM in this set-up. Here, although the classification performance decreased as the number of missing entries was increased, MKMC remained having the highest ROC scores among the three methods; the gap in the ROC scores being more prominent when there are to missing entries.
The experiments show that among the matrix completion techniques used, MKMC best preserves the relationships among the data points.
5.2.2 Completion Accuracy
To test how accurate the missing values were recovered by the three methods, we computed the mean of the correlation distance between the estimated matrices and their corresponding complete kernel matrices as follows:
where is the Frobenius norm, and and are the “true” and the estimated kernel matrices, respectively; the computation was done for each ratio of missing entries. Lower correlation matrix distance means better correlation between the matrices. As with the previous experiments, the ratio of the missing entries was artificially changed from to . Fig. 4 shows how MKMC recovered the missing values more accurately than the two imputation methods, with the exception when about at least of the entries are missing. In this case, however, the gap between MKMC and mean-imputation is not very large.
In this study, we introduced a new algorithm, called MKMC, to solve the problem of mutually inferring the missing entries of similarity matrices (or kernel matrices). MKMC works by combining relevant data sources, although some or all are incomplete, and using the combined sources (called the model matrix) to estimate the missing entries of the kernel matrices. The inference is done by minimizing the KL divergence of the model matrix from each of the kernel matrices. The new kernel matrices are recombined, thus updating the model matrix which can be reused to improve the estimates of the entries. The optimal solutions given by MKMC in the E- and M-steps are given in closed forms, making MKMC efficient in terms of time and memory. In addition, results show that MKMC has good to excellent classification performance as the amount of training data and missing entries were varied— a good indication that MKMC preserves relationships among pairs of data points. We have also shown that when portions of the kernel matrices were artificially missed, MKMC recovers the missing entries more accurately than the other two methods.
This work was supported by JSPS KAKENHI Grant Number 26249075, 40401236.
-  Shun-Ichi Amari. Information geometry of the EM and em algorithms for neural networks. Neural Netw., 8(9):1379–1408, December 1995.
-  Sahely Bhadra, Samuel Kaski, and Juho Rousu. Multi-view kernel completion, February 2016.
-  Christopher M. Bishop. Pattern recognition and machine learning. Springer, 2006.
-  A. P. Dempster, N. M. Laird, and D. B. Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the Royal Statistical Society. Series B (Methodological), 39(1):1–38, 1977.
-  Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning – Data Mining, Inference, and Prediction. Springer, 2nd edition.
-  Tsuyoshi Kato, Koji Tsuda, and Kiyoshi Asai. Selective integration of multiple biological data for supervised network inference. Bioinformatics, 21(10):2488–2495, February 2005.
-  Taishin Kin, Tsuyoshi Kato, and Koji Tsuda. Protein classification via kernel matrix completion. In Kernel Methods in Computational Biology, chapter 3, pages 261–274. The MIT Press, 2004. In B. Schölkopf, K. Tsuda and J.P. Vert (eds).
-  Ritwik Kumar, Ting Chen, Moritz Hardt, David Beymer, Karen Brannon, and Tanveer Syeda-Mahmood. Multiple kernel completion and its application to cardiac disease discrimination, April 2013. IEEE 10th International Symposium on Biomedical Imaging.
-  Gert R. G. Lanckriet, Tijl De Bie, Nello Cristianini, Michael I. Jordan, and William Stafford Noble. A statistical framework for genomic data fusion. Bioinformatics, 20(16):2626–2635, 2004.
-  Gert R. G. Lanckriet, Nello Christianini, Michael I. Jordan, and William Stafford Noble. Kernel-based integration of genomic data using semidefinite programming. In Kernel Methods in Computational Biology, pages 231–259. The MIT Press, 2004. In B. Schölkopf, K. Tsuda and J.P. Vert (eds).
-  Gert R. G. Lanckriet, M. Deng, Nello Christianini, Michael I. Jordan, and William Stafford Noble. Kernel-based data fusion and its application to protein function prediction in yeast. To appear in Proceedings of the Pacific Symposium on Biocomputing, 2004.
-  Roderick J. A. Little and Donald B. Rubin. Statistical Analysis with Missing Data. John Wiley & Sons, Inc., Hoboken, New Jersey, 2nd edition, 2002. ISBN 0-471-18386-5.
-  Geoffrey J. McLachlan and Thriyambakam Krishnan. The EM algorithm and extensions, 2nd Edition. Wiley series in probability and statistics. Wiley, Hoboken, NJ, 2008.
-  Kevin P. Murphy. Machine Learning: A Probabilistic Perspective. The MIT Press, 2012.
-  Philip L. Roth, James E. Campion, and Steven D. Jones. The impact of four missing data techniques on validity estimates in human resource management. Journal of Business and Psychology, 11(1), 1996.
-  Bernhard Schölkopf and Alexander J. Smola. Learning with Kernels. The MIT Press, Cambridge, Massachusetts, 2000.
-  J. Schur. Über potenzreihen, die im innern des einheitskreises beschränkt sind. Journal für die reine und angewandte Mathematik, 147:205–232, 1917. ISSN: 0075-4102; 1435-5345/e.
-  Koji Tsuda, Shotaro Akaho, and Kiyoshi Asai. The em algorithm for kernel matrix completion with auxiliary data. Journal of Machine Learning Research, 4:67–81, 2003. Chris Williams (eds).
-  Vladimir N. Vapnik. The Nature of Statistical Learning Theory. Springer, 2nd edition, 2000.
-  Martin J. Wainwright and Michael I. Jordan. Graphical models, exponential families, and variational inference. Found. Trends Mach. Learn., 1(1-2):1–305, January 2008.
Appendix A Derivation of the Closed-Form Solutions
This section shows us how the closed-form solutions to the MKMC algorithm were obtained.
To begin with, let us introduce an objective function that takes the sum of the KL divergences:
where are the data sources represented as symmetric kernel matrices in which some or all may be incomplete; is the model matrix; and is a small positive scalar.
In this step, we fix and minimize with respect to .
and the determinant of is given by
Similarly for each in ,
Let us now represent the inverse of as
for . Equating the above equation to 0 and solving for , we get
Solving for and taking its transpose, we obtain
which we will substitute to (27) to get
which gives us the closed-form solutions for the E-step.
In this step, is fixed and we define
is minimized when , that is, when
Solving for , we get
where denotes the set of positive-definite symmetric matrices. Since matrix inverse is one-to-one, we can express as , thus, from (30),
minimizes , giving us the closed-form solution for the M-step.
Appendix B The Mean-Imputation Method
This section describes how each of the kernel matrices are completed using mean-imputation method.
Let us again partition the kernel matrices (for ) as shown in (1), with the corresponding sizes for the submatrices. Let us also denote the entries for as where denotes the inner product. The entries for the submatrix will then be given by
where , and denotes the index set of the visible entries . On the other hand, the entries for the submatrix is given by
while the submatrix is the matrix transpose of . We have now obtained the values that complete the kernel matrices using mean-imputation method.
Appendix C Remaining Proofs and Derivations
c.1 Proof of Proposition 4.1
We shall use the inequality:
which is immediately derived from the non-negativity of the KL divergence:
Under the assumption that , the equality
holds for any value of unobserved data . Taking the expectation according to and and adding the penalizing term , the LHS becomes . Hence, we get
From the update rule of the M-step, we have . With help of this and (31), we get
In the setting of (13), note that
from the definition of . The empirical distribution is expressed as the product of those Gaussians:
From this, we have