A clustering approach to heterogeneous change detection

A clustering approach to heterogeneous change detection

Luigi Tommaso Luppino luigi.t.luppino@uit.noMachine Learning Group, Dept. of Physics and Technology, University of Tromsø, Norway    Stian Normann Anfinsen Machine Learning Group, Dept. of Physics and Technology, University of Tromsø, Norway    Gabriele Moser Machine Learning Group, Dept. of Physics and Technology, University of Tromsø, Norway    Robert Jenssen Machine Learning Group, Dept. of Physics and Technology, University of Tromsø, Norway    Filippo Maria Bianchi Machine Learning Group, Dept. of Physics and Technology, University of Tromsø, Norway    Sebastiano Serpico Machine Learning Group, Dept. of Physics and Technology, University of Tromsø, Norway    Gregoire Mercier Machine Learning Group, Dept. of Physics and Technology, University of Tromsø, Norway

Change detection in heterogeneous multitemporal satellite images is a challenging and still not much studied topic in remote sensing and earth observation. This paper focuses on comparison of image pairs covering the same geographical area and acquired by two different sensors, one optical radiometer and one synthetic aperture radar, at two different times. We propose a clustering-based technique to detect changes, identified as clusters that split or merge in the different images. To evaluate potentials and limitations of our method, we perform experiments on real data. Preliminary results confirm the relationship between splits and merges of clusters and the occurrence of changes. However, it becomes evident that it is necessary to incorporate prior, ancillary, or application-specific information to improve the interpretation of clustering results and to identify unambiguously the areas of change.

Domain adaptation; heterogeneous image sources; change detection; clustering.

Luppino et al.

1 Introduction

Change detection systems provide crucial information for damage assessment after natural disasters such as floodings, earthquakes, landslides, or to detect long-term trends in land usage, urban development, glacier dynamics, deforestation, and desertification [1, 2, 3, 4, 5, 6, 7]. In the last years, thanks to the development of heterogeneous or multimodal change detection methods, it was possible to relax the assumption of homogeneous and co-calibrated measurements. However, despite its undeniable potential, there is still a limited amount of research on heterogeneous change detection in the fields of computer vision, pattern recognition and machine learning. In [8], copula theory is exploited to build local models of dependence between unchanged areas in heterogeneous images and to link their statistical distributions. In [9], joint distributions of heterogeneous images are obtained by transforming their marginal densities in meta-Gaussian distributions, which provide simple and efficient models of multitemporal correlations. In [10, 11], a method based on evidence theory is proposed, which fuses clustering maps of the individual heterogeneous images and then detects ”change” and ”no-change” classes, from the transition probabilities between clusters. In [12], the physical properties of the considered sensors and, especially, the associated measurement noise models and local joint distributions are exploited to define a ”no-change” manifold.

The capability of processing data from heterogeneous sources in the same application opens for usage of a much larger amount of information. With respect to time series, the temporal resolution can be increased and the overall time window can be extended. Nonetheless, new issues arise. Different sensors are sensitive to distinct physical conditions and comparing their measurements may produce false detections, due to inconsistencies in sensor behaviour rather than actual changes in the monitored entities. As the complexity of the fused data set increases, there could be a requirement for more flexible and complicated statistical models, which are harder to fit on data, they may be characterized by larger uncertainty in the parameter estimation and a higher computational cost. Finally, detecting and characterizing changes in heterogeneous images is not as trivial as in the homogeneous case, where a change corresponds simply to a difference in the signal values.

In this work, we propose a novel cluster-based approach for change detection in heterogeneous data. We design an unsupervised method to be as general as possible, i.e. application-independent. The proposed method processes pairs of images, acquired at different times from different sensors. In particular, one image comes from an optical sensor, whereas the second is a synthetic aperture radar (SAR) image. The images must be co-registered by a pre-processing step, to avoid that spatial misalignment of the images is misclassified as a change. Moreover, a third type of images is considered, whose elements are obtained by stacking optical and SAR images. A clustering method is executed independently on each of the three data sets. Then, the clusters identified in the first two data sets are matched against the ones from the third data set, in order to determine if the clusters from the first image split or merge in the second image. We associate changes to the occurrence of such modifications.

In this preliminary study, the problem has been defined, a possible solution has been suggested and experiments have been performed to assess the capability of the proposed methodology. Making the whole process automatic is the following step, which will be treated in a further extension of this work.

2 Background

This work leverages on the information delivered by distance-based clustering analysis on image data. To select the proper distance measures, we first need to identify the correct statistical models to represent the data. Since we process optical and SAR images, we consider only models commonly used when dealing with these two specific data.

A simple probability distribution that describes well the optical images is the Gaussian distribution [13, 14]. Specifically, a sensor with channels yields feature vectors , which are modelled by a multivariate Gaussian probability density function (pdf)

which compactly reads as . Here and are the mean vector and the covariance matrix associated to cluster , respectively.

Concerning SAR images in single polarisation, using the gamma distribution is a simplistic, yet effective option [15]:

This is denoted by . is the gamma function, while and are the shape and the scale parameter, respectively. Since (the number of looks) is the same for all the clusters, these can be fully characterised by their mean .

The log-normal distribution is an alternative to the gamma pdf. It fits data reasonably well under most circumstances and, contrarily to the gamma pdf, it allows to model heavy-tailed SAR intensity data [15]. A positive-valued random variable follows a log-normal distribution if . The pdf reads

denoted by . The first two moments of random variables and are related according to

To conclude, if the statistical behaviour of a SAR image can be described by log-normal distributions, then a logarithmically transformed image can be modelled by a Gaussian distribution. This property will be useful to process the stacked data , which combines all features of the optical and the SAR image into one stacked feature vector, associated to each pixel.

As distance measures, we use Mahalanobis distance [16] for multivariate Gaussian distributed data and Hellinger distance [17] for gamma distributed data. A notorious drawback in cluster methods is the dependence of their results to initial conditions, such as initialization of cluster centers and ordering of the data. Additionally, the desired number of clusters or the scale parameter (used in methods such as hierarchical or density-based clustering) is often unknown. Ensemble clustering methods tackle these issues, by providing more stable results at the cost of higher computational complexity [18, 19, 20]. Ensemble methods can identify clusters of nontrivial shape and with different densities, handle noise and outliers, and they provide an estimate to the optimal number of clusters. In our case, such a number is unknown and, therefore, we perform cluster analysis with an ensemble approach based on Fuzzy C-Means (FCM) [21, 22, 23]. The ensemble procedure consists in repeating several times the FCM initialized with different number of clusters, which each time is drawn from a uniform discrete distribution. FCM is implemented with the distance measures mentioned above. The FCM algorithm represents an iterative approach, where at each iteration a partition matrix is returned as output. The membership values contained in are exploited to evaluate the covariance matrix of each cluster as:

When multivariate Gaussian distributed data are involved, the Mahalanobis distance computed in the following iteration employs these updated covariance matrices. In a possible future development, we plan to examine the partition matrix to identify the most reliable clustering results, in order to improve the post-clustering analysis.

3 Recognition of cluster splits and merges

Figure 1: First step of the proposed methodology: obtainment of the stacked image and of the three partitionings.

Given two heterogeneous images of the same geographical area captured respectively at times and , we want to detect if a change occurred during the time lapse. Each image is clustered by using the distance measures that captures its statistical properties. The clustering ensemble procedure on each image provides the partitions

where and are the number of clusters in each partition. Then, if the SAR data are assumed to follow a log-normal distribution, the logarithm of their intensities can be modelled by Gaussian pdfs. Since also the optical data are modelled by Gaussian pdfs, the stacked vector could be thought of a realization of a multivariate Gaussian random variable. Accordingly, we compute a third partition on the stacked data, as shown in Fig. 1. To determine the number of clusters to be considered by the ensemble procedure on stacked images, useful information can be extracted from the clustering results obtained on optical and SAR images. In fact, the allowed number of clusters provided to every instance in the ensemble procedure for the stacked image are drawn from the uniform discrete pdf .

Once the three partitions are obtained, we check whether a cluster from the image at time splits into two or more clusters in the stacked image, or whether two or more clusters from the stacked image may merge into one cluster of the image at time . Instead of comparing directly the clusters from time and time , with our method we leverage the information contained in the covariance matrix of the stacked image, which captures the cross-correlation between the original images. Moreover it may provide a regularization that filters out the effect of the speckle noise on the clustering results. The proposed methodology is depicted in Fig.2. In Fig. 2(a), a region in the optical image at time is fully contained in a cluster . The same region, is divided in two clusters, and , in the stacked image. This means that in the SAR image at the region is split in two clusters as well, and . This denotes that a change occurred in the time lapse . In Fig. 2(a) instead, we can see the region that in the stacked image corresponds to two clusters and , merges into a single cluster in the SAR image at time . This indicates another type of change from , where the region is characterized by two clusters and in the optical image.

Figure 2: Proposed methodology: it is possible to recognize changes as splits (a) and merges (b) by the comparison with the partitioning of the stacked image.

4 Experiments and results

In this section the proposed approach is applied, showing the potential and limitations of ensemble clustering and of an analysis of splits and merges.

(a) Sept. the 5th, 1999
(b) Oct. the 21st, 2000
(c) Ground truth
Figure 3: Gloucester before – optical image (a) – and after a flooding the event – SAR image (b). In (c), the ground truth of the change.

The images in Fig. 3 represent the countryside at the periphery of Gloucester, Gloucestershire, United Kingdom, before and after a flood. Since the speckle noise affecting the latter was too strong, a 7-by-7 enhanced Lee filter [24] was applied to attenuate the noise, while preserving the details contained in heterogeneous areas. The analysis is carried out on the presented images by dividing them into smaller and non-overlapping windows of pixels, and then by looking for changes inside them separately. In this way, it can be reasonably thought that pixels can be grouped into a limited number of clusters, making the clustering process easier and more accurate regardless of the spatial nonstationarity of the image data. Processing smaller windows also reduces the computational cost, which scales quadratically with the windows size and the number of clusters. The FCM algorithm has been iterated 20 times, drawing a different number of clusters each time from the uniform probability mass function .

4.1 First experiment

The region selected for the first experiment is shown in Fig. 4. It contains some agricultural fields and a river in the lower part of it. As seen in Fig. 4(d), clusters relative to different parts of the image are very well separated.

(a) Time
(b) Time
(c) Ground truth mask
(d) Clustering
(e) Clustering
(f) Clustering
Figure 4: First experiment: (a) window of the optical image, (b) window of the SAR image, (c) window of the ground truth mask, (d) clustering result on image a, (e) clustering result on image b, (f) clustering result on the stacked image.

The true number of clusters is unknown and it must be properly evaluated for a correct separation of the objects. In fact, an excessive (or insufficient) number of clusters will lead to oversegmentation (or undersegmentation) of the image. Concerning the SAR acquisition, from Fig. 4(b) we observe that the flooded area covers the majority of the window. Such area is correctly identified by the large black cluster in Fig. 4(e). Comparing the three clustering results, two clear examples of clusters merging and clusters splitting are spotted. The big cluster representing some fields, that from the upper part of the optical image goes down to the right, has split into two different clusters in the SAR image (the light grey one and the black one), and this is highlighted by the presence of the light grey and the dark grey clusters in Fig. 4(f). Then, the dominant cluster of Fig. 4(e) is the result of the merging of some clusters of the optical image, i.e. the white cluster (the river), the dark grey cluster (some fields close to the river), a good percentage of the black cluster (the boundaries around the river and the fields) and one part of the above mentioned big cluster which has split. All these clusters are visible in the result obtained with the stacked data, and they are respectively: the dark grey cluster (the river), the grey cluster close to it (the fields close to the river and the boundaries) and the light grey cluster (the part of the splitting).

4.2 Second experiment

(a) Time
(b) Time
(c) Ground truth mask
(d) Clustering
(e) Clustering
(f) Clustering
Figure 5: Second experiment: (a) window of the optical image, (b) window of the SAR image, (c) window of the ground truth mask, (d) clustering result on image a, (e) clustering result on image b, (f) clustering result on the stacked image.

The region selected for the second experiment is displayed in Fig. 5. In this case, the different areas are not well separated (Fig. 5(d)), especially in the center and in the lower right corner of the window, mainly because these parts of the image present miscellaneous ground covers. For example, some of the central pixels in Fig. 5(a) look darker, so the clustering algorithm erroneously cluster them together with the ones belonging to the river, as it happened in the first experiment. Instead, the bare soil field presents some brighter pixels close to the river and some darker pixels far from it, and these two groups are divided. Moving on to the image in Fig. 5(b), it can be seen how it looks still noisy and muddled, even after being filtered. Consequently, the clustering in Fig. 5(e) does not yields the same quality of the first experiment. Making a comparison with Fig. 5(c), more accurate delineation of changed areas would have been emphasized if the grey and black classes were grouped together and, most importantly, some of the agricultural fields in the lower right corner were grouped differently. But this is not a fault of the ensemble clustering, as these last areas are very similar to the flooded portion of the region, due to the characteristics of the specific kind of field and its SAR signature. Recognizing the flooded area in Fig. 5(b) by visual inspection and without prior knowledge, is also very difficult. It is worth noting that the available ground truth itself is only partially accurate, because for example the sharp edge on the right side is unlikely, but it still gives an idea about the location of the affected areas.

As expected according to the aforementioned characteristics of the SAR scene, Fig. 5(f) shows how the output of the ensemble clustering is irregular. The quality of the partitioning is heavily influenced by the speckle noise, which is a fundamental issue in the field of SAR data analysis. Under these conditions, it is not trivial to recognise splits and, most of all, merges, due to the amount of noise in the SAR image at time . This case study highlights that an approach for change detection from an optical and a SAR image based on cluster splits and merges is limited by the clustering results. These latter are affected, in turn, by the characteristics of the input data (noise ratio, contrast, etc.), by the adopted clustering algorithm, and by the selection of its hyperparameters.

5 Conclusions and future works

In this paper, we studied the challenging problem of change detection in multitemporal and heterogeneous images, by means of a cluster-based techniques. Our study focused on the case of image pairs, relative to the same area, captured by heterogeneous sensors at different times. We designed an unsupervised method, conceived to be application independent, which tackles a multitemporal image analysis problem, which is challenging due to the difficulty detecting changes from heterogeneous signals with no prior information. We evaluated to which extent a completely unsupervised approach could be successful in addressing change detection from heterogeneous image sources. Since there exist good models to capture the statistics of the kind of images considered (i.e. satellite optical and SAR images), the proposed method is based on distance measures that account for those properties. The proposed idea is that changes on the ground can be related to clusters of the image at time splitting and/or merging into the clusters of the image at time . The possibility to model SAR intensity as log-normally distributed and optical data as Gaussian allowed us to apply a multivariate Gaussian model in the joint domain of the optical channels and of the log-transformed SAR data. This allowed us to also apply the clustering algorithm on a stack of the images, to improve the chances of identify splits and merges. Furthermore, a clustering algorithm has been developed that combines an extension of the fuzzy C-means algorithm with adaptive intra-cluster Mahalanobis metric and an ensemble approach aimed at minimizing dependence on the initialization. Experimental results were obtained on real satellite heterogeneous images. The data set is relative to a flooded area and it contains pre-event and post-event images collected by optical and SAR spaceborne sensors. Our experiments demonstrated the relationship between the cluster splits and merges with the changed and unchanged areas. This confirmed the potential of the clustering approach with respect to the problem of change detection from heterogeneous sources and suggested the effectiveness of the ensemble clustering approach. However, the experiments also highlighted the limitations of this unsupervised approach. In particular, the relationship between cluster splits/merges and changed/unchanged areas does not always hold. This limitation can be addressed if prior, ancillary, or application-specific information is used to constrain the relationship between cluster splits/merges and changed/unchanged areas. For example, possible improvements might result from:

  • providing some a priori information to the system, such as the most probable changing parts according to their position

  • introducing hypothesis that the changing parts are the majority or the minority of the image, according to the particular application

  • to indicate the particular class that represents the sought changed areas, e.g. water for floods, bare soil for forest fires, etc.

The detection of cluster splits and merges carried out in this work is based on visual inspection and human interpretation, but it could be automated. A possible solution would be to overlap the mask of each cluster from the image at time to the stacked image, in order to identify the areas where splits occur. Accordingly, this procedure could be applied to the clusters of the image at time to recognise merges. Once splits and merges are identified, one could rely on prior information (if available) to improve the accuracy of change detection. For example, if the location of the river in the image is provided, one could focus the search for flooded areas with in the clusters close to its position. Alternatively, one could leverage on the statistical characteristics of water in SAR images to identify the areas of interest. On one hand, this approach would provide an automatic tool to improve clusters interpretation and to identify relevant splits and merge, associated with changes of interest. On the other hand, the necessity of prior information confirms the extreme difficulty of performing automatic and unsupervised change detection in heterogeneous data.

A significant improvement is also expected if polarimetric SAR images are used instead of SAR acquisitions with only one polarisation. This is because they bring a lot more intrinsic information which would enhance the capability of clustering results to identify natural classes in feature spaces associated with SAR observations. Obviously, this would force to consider different and more complicated models and distance measures. The parallelisation of the proposed approach, which is favored by its window-based formulation and would benefit of current cluster or GPU-based architectures, represents another possible and interesting future development. Last, but not least, the research can be extended to the multitemporal case in which more than two images are considered, exploiting the proposed method for the analysis of long term trends such as deforestation, glacier dynamics, desertification, land use change and urban development.


  • [1] Sebastiano B Serpico, Silvana Dellepiane, Giorgio Boni, Gabriele Moser, Elena Angiati, and Roberto Rudari. Information extraction from remote sensing images for flood monitoring and damage evaluation. Proceedings of the IEEE, 100(10):2946–2970, 2012.
  • [2] Fabio Dell’Acqua and Paolo Gamba. Remote sensing and earthquake damage assessment: Experiences, limits, and perspectives. Proceedings of the IEEE, 100(10):2876–2890, 2012.
  • [3] Guan Ningning, Yin Jingyuan, Li Chengfan, Lei Ming, and Zhang Ming. Landslide recognition in remote sensing image based on fuzzy support vector machine. In Computer and Information Technology (CIT), 2012 IEEE 12th International Conference on, pages 1103–1108. IEEE, 2012.
  • [4] Mingyi Du and Guoyin Cai. Spatial and temporal dynamics of land cover in beijing. In Geoscience and Remote Sensing Symposium (IGARSS), 2012 IEEE International, pages 6333–6336. IEEE, 2012.
  • [5] Xuejiao Wu, Anxin Lu, Lihong Wang, Jianchen Pu, Huawei Zhang, and Haigang Tong. Glacier change along wusun road in chinese tien shan during 1973–2007 monitored by remote sensing. In Remote Sensing, Environment and Transportation Engineering (RSETE), 2011 International Conference on, pages 1252–1256. IEEE, 2011.
  • [6] Zulkiflee Abd Latif, Hud Mohamad Zaqwan, Mohamed Saufi, Nor Aizam Adnan, and Hamdan Omar. Deforestation and carbon loss estimation at tropical forest using multispectral remote sensing: Case study of besul tambahan permanent forest reserve. In Space Science and Communication (IconSpace), 2015 International Conference on, pages 348–351. IEEE, 2015.
  • [7] Wu Jianjun, Xia Hong, and Liu Yani. Theory and methodology on monitoring and assessment of desertification by remote sensing. In Geoscience and Remote Sensing Symposium, 2004. IGARSS’04. Proceedings. 2004 IEEE International, volume 4, pages 2302–2305. IEEE, 2004.
  • [8] Gregoire Mercier, Gabriele Moser, and Sebastiano Serpico. Conditional copula for change detection on heterogeneous sar data. In Geoscience and Remote Sensing Symposium, 2007. IGARSS 2007. IEEE International, pages 2394–2397. IEEE, 2007.
  • [9] Bård Storvik, Geir Storvik, and Roger Fjortoft. On the combination of multisensor data using meta-gaussian distributions. IEEE Transactions on Geoscience and Remote Sensing, 47(7):2372–2379, 2009.
  • [10] Zhun-ga Liu, Jean Dezert, Grégoire Mercier, and Quan Pan. Dynamic evidential reasoning for change detection in remote sensing images. IEEE Transactions on geoscience and remote sensing, 50(5):1955–1967, 2012.
  • [11] Zhun-ga Liu, Gregoire Mercier, Jean Dezert, and Quan Pan. Change detection in heterogeneous remote sensing images based on multidimensional evidential reasoning. IEEE Geoscience and Remote Sensing Letters, 11(1):168–172, 2014.
  • [12] Jorge Prendes, Marie Chabert, Frédéric Pascal, Alain Giros, and Jean-Yves Tourneret. A new multivariate statistical model for change detection in images acquired by homogeneous and heterogeneous sensors. IEEE Transactions on Image Processing, 24(3):799–812, 2015.
  • [13] Francesca Bovolo and Lorenzo Bruzzone. The time variable in data fusion: A change detection perspective. IEEE Geoscience and Remote Sensing Magazine, 3(3):8–26, 2015.
  • [14] François Goudail and Philippe Réfrégier. Contrast definition for optical coherent polarimetric images. IEEE transactions on pattern analysis and machine intelligence, 26(7):947–951, 2004.
  • [15] Chris Oliver and Shaun Quegan. Understanding synthetic aperture radar images. SciTech Publishing, 2004.
  • [16] Michele Basseville. Distance measures for signal processing and pattern recognition. Signal processing, 18(4):349–369, 1989.
  • [17] Alejandro C Frery, Abraao DC Nascimento, and Renato J Cintra. Analytic expressions for stochastic distances between relaxed complex wishart distributions. IEEE Transactions on Geoscience and Remote Sensing, 52(2):1213–1226, 2014.
  • [18] Alexander Strehl and Joydeep Ghosh. Cluster ensembles—a knowledge reuse framework for combining multiple partitions. Journal of machine learning research, 3(Dec):583–617, 2002.
  • [19] Ana LN Fred and Anil K Jain. Combining multiple clusterings using evidence accumulation. IEEE transactions on pattern analysis and machine intelligence, 27(6):835–850, 2005.
  • [20] Joydeep Ghosh and Ayan Acharya. Cluster ensembles. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 1(4):305–315, 2011.
  • [21] Susmita Ghosh, Niladri Shekhar Mishra, and Ashish Ghosh. Unsupervised change detection of remotely sensed images using fuzzy clustering. In Advances in Pattern Recognition, 2009. ICAPR’09. Seventh International Conference on, pages 385–388. IEEE, 2009.
  • [22] Krishna Kant Singh, Akansha Mehrotra, MJ Nigam, and Kirat Pal. Unsupervised change detection from remote sensing images using hybrid genetic fcm. In Engineering and Systems (SCES), 2013 Students Conference on, pages 1–5. IEEE, 2013.
  • [23] James M Keller, Michael R Gray, and James A Givens. A fuzzy k-nearest neighbor algorithm. IEEE transactions on systems, man, and cybernetics, (4):580–585, 1985.
  • [24] Armand Lopes, Ridha Touzi, and E Nezry. Adaptive speckle filters and scene heterogeneity. IEEE transactions on Geoscience and Remote Sensing, 28(6):992–1000, 1990.
Comments 0
Request Comment
You are adding the first comment!
How to quickly get a good reply:
  • Give credit where it’s due by listing out the positive aspects of a paper before getting into which changes should be made.
  • Be specific in your critique, and provide supporting evidence with appropriate references to substantiate general statements.
  • Your comment should inspire ideas to flow and help the author improves the paper.

The better we are at sharing our knowledge with each other, the faster we move forward.
The feedback must be of minimum 40 characters and the title a minimum of 5 characters
Add comment
Loading ...
This is a comment super asjknd jkasnjk adsnkj
The feedback must be of minumum 40 characters
The feedback must be of minumum 40 characters

You are asking your first question!
How to quickly get a good answer:
  • Keep your question short and to the point
  • Check for grammar or spelling errors.
  • Phrase it like a question
Test description