Composite Kernel Local Angular Discriminant Analysis for Multi-Sensor Geospatial Image Analysis
With the emergence of passive and active optical sensors available for geospatial imaging, information fusion across sensors is becoming ever more important. An important aspect of single (or multiple) sensor geospatial image analysis is feature extraction — the process of finding “optimal” lower dimensional subspaces that adequately characterize class-specific information for subsequent analysis tasks, such as classification, change and anomaly detection etc. In recent work, we proposed and developed an angle-based discriminant analysis approach that projected data onto subspaces with maximal “angular” separability in the input (raw) feature space and Reproducible Kernel Hilbert Space (RKHS). We also developed an angular locality preserving variant of this algorithm. In this letter, we advance this work and make it suitable for information fusion — we propose and validate a composite kernel local angular discriminant analysis projection, that can operate on an ensemble of feature sources (e.g. from different sources), and project the data onto a unified space through composite kernels where the data are maximally separated in an angular sense. We validate this method with the multi-sensor University of Houston hyperspectral and LiDAR dataset, and demonstrate that the proposed method significantly outperforms other composite kernel approaches to sensor (information) fusion.
Optical remote sensing has made significant advances in recent years. Among these has been the deployment and wide-spread use of hyperspectral imagery on a variety of platforms (including manned and unmanned aircraft and satellites) for a wide variety of applications, ranging from environmental monitoring, ecological forecasting, disaster relief to applications pertaining to national security. With rapid advancements in sensor technology, and the resulting reduction of size, weight and power requirements of the imagers, it is also now common to deploy multiple sensors on the same platform for multi-sensor imaging. As a specific example, it is appealing for a variety of remote sensing applications to acquire hyperspectral imagery and Light Detection and Ranging (LiDAR) data simultaneously — hyperspectral imagery offers a rich characterization of object specific properties, while LiDAR provides topographic information that complements Hyperspectral imagery . Modern LiDAR systems provide the ability to record entire waveforms for every return signal as opposed to providing just the point cloud. This enables a richer representation of surface topography.
While feature reduction is an important preprocessing to analysis of single-sensor high dimensional passive optical imagery (particularly hyperspectral imagery), it becomes particularly important with multi-sensor data where each sensor contributes to high dimensional raw features. A variety of feature projection approaches have been used for feature reduction, including classical approaches such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA) and their many variants, manifold learning approaches such as Supervised and Unsupervised Locality Preserving Projections . Several of these methods are implemented in both the input (raw) feature space and the Reproducible Kernel Hilbert Space (RKHS) for data that are nonlinearly separable. Further, most traditional approaches to feature extraction are designed for single-sensor data — a unique problem with multi-sensor data is that feature spaces corresponding to each sensor often have different statistical properties, and a single feature projection may hence be sub-optimal. It is hence desired to have a projection for feature reduction that preserves the underlying information from each sensor in a lower dimensional subspace.
More recently, we developed a feature projection approach, referred to as Angular Discriminant Analysis (ADA) , that was optimized for hyperspectral imagery and demonstrated robustness to spectral variability. Specifically, the approach sought a lower dimensional subspace where classes were maximally separated in an angular sense, preserving important spectral shape related characteristics. We also developed a local variant of the algorithm (LADA) that preserved angular locality in the subspace. In this paper, we propose a composite kernel implementation of this framework and demonstrate for the purpose of feature projection in multi-sensor frameworks. Specifically, by utilizing a composite kernel (a dedicated kernel for each sensor), and ADA (or LADA) for each sensor, the resulting projection is highly suitable for classification. The proposed approach serves as a very effective feature reduction algorithm for sensor fusion — it optimally fuses multi-sensor data and projects it to a lower dimensional subspace. A traditional classifier can be employed following this, for supervised learning. We validate the method with the University of Houston multi-sensor dataset comprising of Hyperspectral and LiDAR data and show that the proposed method significantly outperforms other approaches to feature fusion.
The outline of the remainder of this paper is as follows. In Section 2, we review related work. In Section 3, we describe the proposed approach for multi-sensor feature extraction. In Section 4, we describe the experimental setup and present results with the proposed method, comparing it to several state-of-the-art techniques to feature fusion.
Traditional approaches to feature projection based dimensionality reduction such as PCA, LDA and their variants largely rely on Euclidean measures. Manifold learning approaches  also seek to preserve manifold structures and neighborhood locality through projections that preserve such structures. Other projection based approaches to feature reduction, such as Locality Preserving Projections (LPP), Local Fisher’s Discriminant Analysis (LFDA)  etc. integrate ideas of local neighborhoods through affinity matrices, into classical projection based analysis approaches such as PCA, LDA etc. As a general feature extraction approach, Euclidean distance is a reasonable choice, including for remotely sensed image analysis. However, by noting the well understood benefits of spectral angle for hyperspectral image analysis, in previous work, we developed an alternate feature projection paradigm that worked with angular distance measures instead of euclidean distance measures  — we demonstrated that when projecting hyperspectral data through this class of transformations, the resulting subspaces were very effective for downstream classification and significantly outperformed their Euclidean distance counterparts. In addition to benefits with classification, we also demonstrated other benefits of this class of methods, including robustness to illumination differences — something that is very important for remote sensing. In other previous work, it has been shown that a reproducible kernel Hilbert space (RKHS) generated by composite kernels (a weighted linear combination of basis kernels) is very effective for multi-source fusion . Here, we briefly review the developments related to angular discriminant analysis. This will provide a context and motivation for the proposed work in this paper that seeks to demonstrate the benefits of composite kernel angular discriminant analysis for multi-source image analysis.
2.1Angular Discriminant Analysis
Here, we briefly review Angular Discriminant Analysis (ADA) and its locality preserving counterpart, Local Angular Discriminant Analysis (LADA). Consider a -dimensional feature space (e.g. hyperspectral imagery with spectral channels). Let be the -th training sample with an associated class label , where is the number of classes. The total number of training samples in the library is , where denotes the number of training samples from class . Let be the desired projection matrix, where denotes the reduced dimensionality. We also denote symbols having unit norm with a tilde — this will be useful where we normalize the data to a unit norm to focus on angular separability.
Traditional LDA seeks to find a subspace that maximizes between-class scatter while minimizing within-class scatter, where the scatter is measured using Euclidean distances. While similar in philosophy, ADA is an entirely new approach to subspace learning that is based on angular scatter — it seeks a subspace where within-class angular scatter is maximized, and the between-class angular scatter is maximized. Just like LDA, the ADA optimization problem can be posed as a generalized eigenvalue problem. Specifically, ADA seeks to find a projection where the ratio of between-class inner product to within-class inner product of data samples is minimized. The within-class outer product matrix and between-class outer product matrix are defined as
where is the normalized mean of -th class samples, and is defined as the normalized total mean.
It was shown in  that the projection matrix of ADA can be approximated as the solution to the following trace ratio problem
The projection matrix can be obtained by solving the generalized eigenvalue problem involving and .
Similar to LDA, ADA is a “global” projection in that it does not specifically promote preservation of local (neighborhood) angular relationships under the projection. We hence developed LADA in , which is a local variant of ADA. The within and between-class outer product matrices of LADA are obtained as follows
where the normalized weight matrices are defined as
The normalized affinity matrix between and is defined as
where denotes the local angular scaling of data samples in the angular neighborhood of , and is the K-th nearest neighbors of .
Similar to ADA, the projection matrix of LADA can be defined as
3Composite Kernel Angular Discriminant Analysis for Image Fusion
In this section, we develop and describe the proposed approach to multi-source feature extraction — composite kernel angular discriminant analysis (CKADA) and its locality preserving counterpart (CKLADA). Our underlying hypothesis with this work is that even when angular information is important for optical image analysis, in a multi-source (e.g. multi-sensor scenario), having dedicated kernels (specific to each source) would result in a superior projection that addresses source-specific nonlinearities. With that goal, we extend our previous work with angular discriminant analysis by implementing it in a composite kernel reproducible kernel Hilbert space and demonstrate for a specific application of multi-sensor image analysis that the resulting subspace is highly discriminative and outperforms other subspace learning approaches.
Consider a nonlinear mapping from the input space to a RKHS as follows:
and a kernel function defined as:
where is the inner product of two vectors. Consider next a set of co-registered multi-source images resulting in the following -Tuple of feature vectors from co-registered images for every geolocation (co-registered pixels): , where . Associated with every pixel (geolocation) for which ground truth is available, there is a class label . A composite kernel RKHS can then be constructed as
where is a basis kernel for the ’th source, formed by any valid Mercer’s kernel. To implement Composite Kernel ADA (CKADA), note that and can be reformulated as
where is given as
and is given as
ADA can hence be re-expressed as the solution to the following generalized eigenvalue problem
Since can be represented as a linear combination of columns of , it can be formulated using a vector as
where is a symmetric kernel (Gram) matrix. Here represents a simple inner product kernel, but can be replaced by by utilizing the kernel trick. Multiplying on both sides of , results in the following generalized eigenvalue problem.
Let be the generalized eigenvectors associated with the smallest eigenvalues . A test sample can be embedded in via
where is a vector. Composite Kernel Local ADA (CKLADA) can likewise be implemented by replacing the weight matrices ( and ) above with their local counterparts defined in and .
We note that in the proposed approach, the empirical kernel (Gram) matrix from that is formed as a weighted linear combination over all sources is used in the generalized eigenvalue problem for CKLADA . The algorithm projects the data from sources onto a unified RKHS through a bank of kernels individually optimized for each source. The final embedding seeks to optimally separate (in an angular sense) data in the RKHS. The linear mixture of kernel enables us to optimize each kernel (for example the kernel parameters) for each source instead of applying a single kernel for all sources, and to specify source importance (via mixing weights) to the overall analysis task at hand.
We note the following free parameters in the overall embedding that affect the subspace that is generated: Embedding dimension, , mixture weights used in the composite kernel, , choice of kernel and related kernel parameters. We note that unlike some other embedding techniques such as LDA and its variants where the embedding dimension is upper bounded due to rank deficiency of the between class scatter, with composite kernel local ADA, the between class angular scatter is not rank limited, and as a result, the projection matrix resulting from the solution to the generalized eigenvalue problem does not enforce an upper bound on the embedding dimension. Hence, is a free parameter that represents the unified subspace generated by all sources. The choice of should hence be governed by the information content (as quantified for example in the eigenspectra of the decomposition). The choice of weights can be made through cross validation or techniques such as kernel alignment — in our experience, there is often a very wide plateau over a range of values of the weights, and hence we chose to use simple cross validation to learn weights from our training data. We utilized a standard radial basis function (RBF) kernel for each source (), but the kernel parameter (width of the RBF kernel) is optimized for each source individually via cross validation.
We note that following a CKLADA projection, a simple classifier can be utilized for down-stream analysis. This follows from the observation that applying kernel projections while simultaneously ensuring preservation of angular locality will result in subspaces where class-specific data are compactly clustered. We validate and measure the efficacy of subspaces resulting from CKLADA by utilizing the following classifiers: (1) A K Nearest neighbor (KNN) classifier, (2) A Gaussian maximum-likelihood (ML) classifier, and a (3) sparse representation based classifier (SRC) . While the choice of KNN and ML are obvious for subspaces formed by Kernel projections as noted in , we make a remark on choice of SRC as an additional classifier to measure efficacy of subspaces — this choice is motivated not only by the observation that SRC has emerged as a powerful classification approach for high dimensional remote sensing data  and that it exploits the inherent sparsity when representing samples using training dictionaries, but also because popular solvers used (e.g. Orthogonal Matching Pursuit, OMP) are driven by inner products to learn the sparse representation and hence they essentially exploit angular information. They implicitly seek a representation where a test sample is represented sparsely in a dictionary of training data such that the atoms that eventually contribute (have non-zero, significant representation coefficients) to the representation are angularly similar to the test data samples. We hence contend that CKLADA is particularly well suited for SRC and its variants.
4Experimental Setup and Results
The dataset we utilize represents a sensor fusion scenario, comprising of LiDAR pseudo-waveforms, and a hyperspectral image cube, and is popular in the remote sensing community as a benchmark. The data were acquired over the University of Houston campus and the neighboring urban area. All the images are at the same spatial resolution of 2.5 m and have the same spatial size of . The hyperspectral image was acquired with the ITRES CASI sensor, containing 144 spectral bands, ranging from 380 nm to 1050 nm. The LiDAR DSM data was acquired by an Optech Gemini sensor and then co-registered to the hyperspectral image. The laser pulse wavelength and repetition rate were 1064 nm and 167 kHz, respectively. The instrument can make up to 4 range measurements. The total number of ground reference samples is 2832, covering 15 classes of interest, with approximately 200 samples for each class — these were determined by photo-interpretation of high resolution optical imagery.. The groundtruth map is overlaid with the gray scale image showing one channel of the hyperspectral image in Figure 1.
From the dense LiDAR point cloud, a pseudo-waveform was generated for each geolocation, that is co-registered with the hyperspectral image. The pseudo-waveform was generated by quantizing the elevation into uniform sized bins, and determining the average intensity of points as a function of elevation bins. This provides us with a co-registered cube of waveform-like LiDAR data that is coregistered with our hyperspectral image. We note that like spectral reflectance profiles that have unique shapes depending on the material in the pixel, shapes of pseudo-waveform also correlate with the material and topographic properties in the image. Hence, angular measures (such as provided by CKLADA) would be appropriate for such analysis compared to Euclidean measures.
To validate the efficacy of the subspaces generated by CKLADA and CKADA, we setup classification experiments using the University of Houston multi-sensor dataset. We used popular and commonly employed embeddings as baselines to compare against, including CKLFDA and KPCA. Each of these embeddings was used with 3 classifiers: KNN, ML and SRC. We note that CKLFDA is a composite kernel counterpart of LFDA based on Euclidean distance measures, and is the best possible multi-source embedding that can be compared with CKLADA — a comparison of CKLFDA vs. CKLADA provides a direct understanding of the benefits of angular information for multi-source embeddings, and of the resulting algorithmic framework proposed in Section 3.
With CKLFDA, CKADA and CKLADA, we treat each sensor (hyperspectral imagery and pseudo-waveform LiDAR) as a source, each getting its dedicated base kernel. With a single kernel KPCA, we stack features from the two sensors and project them via a single transformation based on these methods. In all cases, we use RBF kernels as our base kernels, and the width of the kernel is determined via cross validation. Other free parameters including sparsity level used in SRC, number of nearest neighbors in are also determined empirically from the training data via cross-validation.
4.3Visualization of Embeddings
To provide a visual demonstration on the power of composite kernel angular discriminant analysis for geospatial image analysis, we provide visualization of composite kernel projections CKLADA, CKADA (both angular discriminant analysis) and CKLFDA. These results are depicted in fig. ?. The figure depicts false color images generated by projecting the multi-sensor data onto the first three most significant eigenvectors learned from CKLADA, CKADA and CKLFDA respectively. It can be clearly seen that CKLADA (and CKADA to some degree) preserve object specific properties throughout the image (for example, the highly textured objects such as urban vegetation, residential areas etc. have their spatial context significantly preserved in the lower dimensional subspace). On the contrary, CKLFDA, which can be considered as the closest benchmark/baseline competitor does not perform as well. Further, towards the right corner of the image, we point the reader to the substantial benefit of CKLADA under cloud shadows - spatial structures under cloud shadows are visible under CKLADA (and CKADA to some extent), but not when using CKLFDA.
Experimental results comparing performance of CKADA and CKLADA with baseline embeddings are provided. As mentioned previously, free parameters were determined empirically via cross-validation. Tab. ? depicts overall accuracy as a function of training sample size, ranging from a small to a sufficiently large value for the proposed and baseline embeddings with various classifiers. We notice that the proposed composite kernel angular discriminant analysis approaches (CKADA and CKLADA) provide anywhere from improvement in performance compared to state of the art (CKLFDA), and provide even higher accuracies compared to a traditional single-kernel baseline, KPCA. We note that even when using very limited training data, CKLADA is able to substantially outperform other composite kernel and single-kernel methods (using just 10 samples per class, we obtain as much as a gain in performance with CKLADA). Even when using a sufficiently large training sample set (e.g. 50 samples per class), CKLADA and CKADA outperform other methods. In Tab. , Tab. , and Tab. , we depict class specific accuracies, overall and average accuracies using the proposed methods and baselines using SRC, ML and KNN classifiers respectively. Once again, it is clear that CKADA and CKLADA consistently provide robust classification, particularly for the “difficult” classes (such as residential buildings, commercial buildings, roads, parking lots etc.). We also provide classification maps in fig. ? using the proposed method (CKLADA) and its closest competitor, CKLFDA, using the SRC classifier. We note that CKLADA results in a map with very little noise and misclassifications, and is particularly robust under the very challenging area in the right corner of the image that is under a cloud shadow (for e.g., when using CKLFDA, the area under a cloud shadow get systematically misclassified as water — something that is visibly remedied by CKLADA). The improvements to misclassifications occurring over difficult classes is even more apparent with these ground cover classification maps.
|Parking Lot 1||30||162||75.25.7||74.26.3||67.86.4||62.35.0|
|Parking Lot 2||30||154||65.54.7||74.54.5||58.46.8||50.75.7|
|Parking Lot 1||30||162||71.78.7||49.811.6||64.57.6||71.810.2|
|Parking Lot 2||30||154||92.23.3||96.71.5||84.86.7||92.54.3|
|Parking Lot 1||30||162||78.85.9||76.66.4||70.76.5||67.86.5|
|Parking Lot 2||30||154||65.45.0||69.64.1||54.24.9||41.04.7|
We presented a composite kernel variant of angular discriminant analysis and local angular discriminant analysis. Angular discriminant analysis was previously shown to be very beneficial for high dimensional hyperspectral classification. In this paper, we expanded those developments via a composite kernel and demonstrated that this paradigm can be a very useful feature embedding algorithm in multi-source scenarios, such as when fusing multiple geospatial images. We validated our results with a popular multi-sensor benchmark and demonstrated that composite kernel angular discriminant analysis consistently outperforms other feature embeddings.
- M. Dalponte, L. Bruzzone, and D. Gianelle, “Fusion of hyperspectral and lidar remote sensing data for classification of complex forest areas,” IEEE Transactions on Geoscience and Remote Sensing, vol. 46, no. 5, pp. 1416–1427, 2008.
- Y. Zhang and S. Prasad, “Locality preserving composite kernel feature extraction for multi-source geospatial image analysis,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 8, no. 3, pp. 1385–1392, March 2015.
- R. Brennan and T. Webster, “Object-oriented land cover classification of lidar-derived surfaces,” Canadian Journal of Remote Sensing, vol. 32, no. 2, pp. 162–172, 2006.
- M. Shimoni, G. Tolt, C. Perneel, and J. Ahlberg, “Detection of vehicles in shadow areas using combined hyperspectral and lidar data,” in Geoscience and Remote Sensing Symposium (IGARSS).1em plus 0.5em minus 0.4emIEEE, 2011, pp. 4427–4430.
- M. Pedergnana, P. R. Marpu, M. Dalla Mura, J. A. Benediktsson, and L. Bruzzone, “Fusion of hyperspectral and lidar data using morphological attribute profiles,” in SPIE Remote Sensing.1em plus 0.5em minus 0.4emInternational Society for Optics and Photonics, 2011, pp. 81 801G–81 801G.
- D. Lunga, S. Prasad, M. Crawford, and O. Ersoy, “Manifold-learning-based feature extraction for classification of hyperspectral data: A review of advances in manifold learning,” Signal Processing Magazine, IEEE, vol. 31, no. 1, pp. 55–66, 2014.
- M. Cui and S. Prasad, “Angular discriminant analysis for hyperspectral image classification,” Selected Topics in Signal Processing, IEEE Journal of, vol. 9, no. 6, pp. 1003–1015, 2015.
- S. Prasad and M. Cui, “Sparse representations for classification of high dimensional multi-sensor geospatial data,” in Proceedings of the 2013 Asilomar Conference on Signals, Systems and Computers., November 2013, pp. 811–815.
- M. Cui and S. Prasad, “Sparsity promoting dimensionality reduction for classification of high dimensional hyperspectral images,” in 38th IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), May 2013, pp. 2154–2158.
- S. Prasad, M. Cui, W. Li, , and J. Fowler, “Segmented mixture of gaussian classification for robust sub-pixel hyperspectral ATR,” IEEE Geoscience and Remote Sensing Letters, vol. 11, no. 1, pp. 138–142, January 2014.
- M. Cui, S. Prasad, W. Li, and L. Bruce, “Locality preserving genetic algorithms for spatial-spectral hyperspectral image classification,” Selected Topics in Applied Earth Observations and Remote Sensing, IEEE Journal of, vol. 6, no. 3, pp. 1688–1697, 2013.
- Z. Wang and X. Sun, “Multiple kernel local fisher discriminant analysis for face recognition,” Signal Processing, vol. 93, no. 6, pp. 1496–1509, 2013.
- D. Tuia, F. Ratle, A. Pozdnoukhov, and G. Camps-Valls, “Multisource composite kernels for urban-image classification,” IEEE Geoscience and Remote Sensing Letters, vol. 7, no. 1, pp. 88–92, 2010.
- G. Camps-Valls, L. Gomez-Chova, J. Muñoz-Marí, J. Vila-Francés, and J. Calpe-Maravilla, “Composite kernels for hyperspectral image classification,” IEEE Geoscience and Remote Sensing Letters, vol. 3, no. 1, pp. 93–97, 2006.
- J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma, “Robust face recognition via sparse representation,” Pattern Analysis and Machine Intelligence, IEEE Transactions on, vol. 31, no. 2, pp. 210–227, 2009.
- M. Cui and S. Prasad, “Class dependent sparse representation classifier for robust hyperspectral image classification,” IEEE Transactions on Geosciences and Remote Sensing, vol. 53, no. 5, pp. 2683–2695, May 2015.
- X.-T. Yuan, X. Liu, and S. Yan, “Visual classification with multitask joint sparse representation,” Image Processing, IEEE Transactions on, vol. 21, no. 10, pp. 4349–4360, 2012.
- Y. Chen, N. M. Nasrabadi, and T. D. Tran, “Hyperspectral image classification using dictionary-based sparse representation,” Geoscience and Remote Sensing, IEEE Transactions on, vol. 49, no. 10, pp. 3973–3985, 2011.
- Y. Chen, N. Nasrabadi, and T. Tran, “Sparse representation for target detection in hyperspectral imagery,” Selected Topics in Signal Processing, IEEE Journal of, vol. 5, no. 3, pp. 629–640, 2011.
- L. Zhang, M. Yang, and X. Feng, “Sparse representation or collaborative representation: Which helps face recognition?” in Computer Vision (ICCV), 2011 IEEE International Conference on.1em plus 0.5em minus 0.4emIEEE, 2011, pp. 471–478.