Automatic Change Detection Using Unsupervised Saliency Guided Method with UAV and Aerial Images

Abstract

In this paper, an unsupervised saliency guided change detection method using UAV and aerial imagery is proposed. Regions that are more different from other areas are salient, which make them more distinct. The existence of the substantial difference between two images makes saliency proper for guiding the change detection process. Change Vector Analysis (CVA), which has the capability of extracting of overall magnitude and direction of change from multi-spectral and temporal remote sensing data, is used for generating an initial difference image. Combined with an unsupervised CVA and the saliency, Principal Component Analysis(PCA), which is possible to implemented as the guide for change detection method, is proposed for UAV and aerial images. By implementing the saliency generation on the difference map extracted via the CVA, potentially changed areas obtained, and by thresholding the saliency map, most of the interest areas correctly extracted. Finally, the PCA method is implemented to extract features, and K-means clustering is applied to detect changed and unchanged map on the extracted areas. This proposed method is applied to the image sets over the flooded and typhoon-damaged area and is resulted in 95 percent better than the PCA approach compared with manually extracted ground truth for all the data sets. Finally, we compared our approach with the PCA K-means method to show the effectiveness of the method.

1. Introduction

Saliency detection algorithms identify regions that a human eye focus on at first (Itti, 2001a; Itti, 1998b). Visual saliency detection is effective for several tasks, such as object recognition (Rutishauser et al., 2004), image segmentation (Rother et al., 2004), detection of a single dominant object from the input image (Hou, 2007), and specific applications such as autofocus.

Implementation of saliency detection has been widely applied in remote sensing application. Zhang et al. (2014) developed a saliency detection algorithm by implementing unsupervised feature learning and dropout regularization method to classify high-resolution satellite imagery. For extracting salient areas from SAR polarimetric images, a scale-invariant method is proposed (Jager, 2005). Zhang et al. (2016) introduced a method based on the two-layer visual saliency analysis, and support vector machines for detecting airports and aircraft from high-resolution remote sensing images. For the extraction of residential areas, a global and local model for saliency analysis from remote sensing images is utilized (Zhang, 2014).

Saliency is a feasible guide for change detection procedures in remote sensing. Tian et al. (2007) performed change detection by modifying the Itti’s saliency approach with combination of texture instead of color, intensity, and orientation. Feng et al. (2018) integrated visual saliency and random forest method for change detection from high-resolution images. Zheng et al. (2017) applied unsupervised saliency- guided SAR image change detection based on Principal Component Analysis (PCA).

Previous saliency-based approaches are mostly generating difference map for SAR and satellite images in which log-ratio works well enough for them. But same approach is not good enough for extremely high- resolution UAV and moderate high-resolution aerial image, especially in flooded and landslide areas. Change Vector Analysis (CVA) was basically formulated for images with only two spectral dimensions in two different times, brightness and greenness, both from the tasseled cap transform (Malila, 1980; Crist, 1984). In CVA, a change vector characterized by an angle and a magnitude which is easily possible to implemented in order to estimate the changing area from different kinds of multi-temporal satellite and aerial image (Solano- Correa et al., 2018).

Saliency guided change detection and CVA are pixel-wise, which derive good performance because the pixel is the smallest statistical unit. The main problem for these methods is the existence of noises in the input difference images. For the purpose of overcoming noise influence in the saliency guided change detection result, feature extraction with using thresholding one way to solve the problem (Zheng et al., 2017).

In this research, we proposed an efficient change detection method using UAV and aerial orthoimages over the damaged areas caused by Typhoon Mitag during 2019 in Korea. Change detection challenge is dealt with as a saliency map detection issue and CVA method is utilized to extract the difference image as input for saliency map generation step. Then, PCA has been implemented for the extraction of features. Finally, K-means approach is applied for the clustering of the features into changed and unchanged regions by selecting K=2.

2. Method

The methodology for this research is presented in this section. The schematic diagram of the proposed unsupervised change detection method is shown in Fig. 1. The input data for the method is aerial and UAV images that are taken before and after the natural disaster happened over the research area. Because CVA performs well for extracting maximum information of the overall magnitude and direction of change is applied for generating difference map. Then, by implementing the saliency generation on the difference map from the previous method, potentially changed areas obtained via thresholding on the saliency map. For feature extraction step PCA approach is implemented. Finally, K-means is applied for clustering of the features into changed and unchanged regions by selecting K=2. A detailed description of the proposed method is presented in the following subsections.

Fig. 1. Flowchart of Proposed Method.

1) Change vector analysis (CVA)

CVA is one of the well-known unsupervised change detection methods, which, instead of classification, extracts land cover changed area by comparison of pixel-wise radiometric value (Chen et al., 2003). In CVA, a change vector is obtained by an angle and a magnitude of change from two different times. If the subtraction of the digital number of a pixel in two images are illustrated by U=(u₁, u₂, ..., u_n)^T and A=(a₁, a₂, ..., a_n)^T, and n shows the number of bands, then a change vector is defined as:

\(\Delta=U-A=\left\{\begin{array}{c} u_{1}-a_{1} \\ u_{2}-a_{2} \\ \cdots \\ u_{n}-a_{n} \end{array}\right\}\)       (1)

where, U and A are UAV and Aerial image, respectively. Since there are three bands for color RGB image, the magnitude of change is calculated by:

\(\|\Delta\|=\sqrt{\left(u_{1}-a_{1}\right)^{2}+\left(u_{2}-a_{2}\right)^{2}+\left(u_{3}-a_{3}\right)^{2}}\)        (2)

||Δ|| indicates the total difference between the two images. The larger ||Δ|| is, the greater the change possibility. When the magnitude of change exceeds a particular threshold, it could be considered as a changed pixel. If the angles between Δ and each band are (θ₁, θ₂, …, θ_n), then its direction is described by cosine angles as:

\(\cos \theta_{1}=\frac{\left(u_{1}-a_{1}\right)}{|\Delta|}, \cos \theta_{2}=\frac{\left(u_{2}-a_{2}\right)}{|\Delta|}, \cos \theta_{3}=\frac{\left(u_{3}-a_{3}\right)}{|\Delta|}\)        (3)

A change vector direction is demonstrated as a unique point by the new vector defined as V(cosθ1, cosθ2, cosθ3) in the direction cosine space. In this space for every change pixel there are corresponding points. Now, the extraction of change becomes the category of classification issue of points inside the direction cosine space.

2) Saliency Map Generation and Thresholding

Saliency detection is mainly utilized to detect the distinct area which are much different from other areas. The initial difference image generated by the CVA has areas which are significantly distinguished from other areas. It is possible to illustrate that there is shape similarity between ground truth data and the extracted saliency areas. Therefore, the saliency map is feasible to be utilized as a good initial help to the change detection of UAV and aerial images. A context-aware saliency detection method has been implemented by Goferman et al. (2011). In this method both locally and globally saliency is unified by computing the similarity of each patch with other image patches extracted in the entire image, then two patches dissimilarity is defined as:

\(d\left(m_{i}, n_{j}\right)=\frac{d_{c}\left(m_{i}, n_{j}\right)}{1+c \cdot d_{p}\left(m_{i}, n_{j}\right)}\)       (4)

where m_i and n_j are two patches extracted from difference image, and d_c(m_i, n_j) is the Euclidean distance between the positions of m_i and n_j. d_p(m_i, n_j) is normalized of Euclidean distance between the positions of m_i and n_j. cis 3 in this implementation.

Background pixels (patches) have more similar patches at multiple scales. Therefore, a combination of multiple scales is applied to obtain more contrast for salient and non-salient regions. The saliency value of patch miat scale rwith considering the saliency of an image patch in multiple scales calculated as:

\(S_{i}^{r}=1-\exp \left\{-\frac{1}{K} \sum_{K}^{K=1} d\left(m_{i}^{r}, n_{j}^{r}\right)\right\}\)       (5)

Considering the saliency maps in multiple scales and that the nearest patches around the interested point of saliency should be applied, the saliency value for patch m_i calculated as:

\(\hat{s}_{i}=\frac{1}{M} \sum_{r \in R} S_{i}^{r}\left(1-d_{f o c i}^{r}(i)\right)\)       (6)

where, Mis the number of scales. \(d_{f o c i}^{r}(i)\) is the Euclidean positional distance between pixel i and the closest focus of attention pixel at scale r, normalized to the range [0, 1]. The parameter settings have been followed as in Goferman et al. (2011), in which Kis 64, the patch size for partition 7 × 7 with 50% pixel overlapping, and Mis 4.

As the saliency map has been generated from the initial difference map, the final shape and location of change map can be determined. Thus, for the identification of the potential change locations in unsupervised approach, a saliency map has been utilized to lead change detection procedure. Thresholding is applied to extract and preserve pixels that have a higher value than the given threshold (α); otherwise, the pixels are considered as non-change areas. The thresholding is as the following equation:

\(S_{t}=\left\{\begin{array}{l} 1, S>\alpha \\ 0, \text { otherwise } \end{array}\right.\)       (7)

S_t is considered as a thresholding map, where 1 stands for the pixel preserved in the extracted region, and 0 is for rejection of pixel. By applying thresholding on salinity map S, the regions have more possibility of change will be preserved, and regions that had noises could be removed. Therefore, the parts equivalent to 1 in the saliency map has been extracted for generating a new difference image.

3) PCA and K-means clustering

Principal component analysis (PCA) and K-means are simple, but feasible for automatic change detection and extracting difference map of aerial and remote sensing images. Eigenvectors of the difference image are generated by using PCA for the areas without overlapping. By projecting h×h neighborhood information onto eigenvector space feature vector is generated for each pixel of difference image. The feature vector space is clustered into two change and unchanged using the K-means (Gonzalez, 2006). Finally, the change detection process is done by checking the minimum Euclidean distance between the feature vector and the mean feature vector of each pixel to cluster them in change and unchanged classes. Fig. 2 illustrates PCA and K-means clustering procedure step by step (Celik, 2009).

Fig. 2. PCA K-means clustering.

3. Experimental Results and Discussion

1) Study Area and Dataset

Buildings and structures were damaged, and many places were affected by landslides after a typhoon in South Korea during early October 2019. It was said that at least six people had been killed in typhoon Mitag. The location of the study areas, southern part of Korea, Uljin-gun, is shown in Fig. 3. Post-typhoon UAV images were captured in February 2020 using the DJI- INSPIRE over Yeonji-ri, Geummae, and Giyang at the Uljin-gun. UAV images with 3 cm GSD are acquired, and orthoimages are generated by using Pix4D software by using on-site surveying GCPs, which are collected by the RTK-GNSS surveying. Aerial images are taken before the disaster in 2019. Orthoimages are compiled from a traditional photogrammetric method using aerial images, with a final ground resolution of 25 cm provided by NGII (National Geographic Information Institute).

Fig. 3. (a) Yeonji-ri’s aerial image, (b) Yeonji-ri’s UAV image, (c) Geummae’s aerial image, (d) Geummae’s UAV image, (e) Giyang ‘s aerial image, (f) Giyang’s UAV image.

The first dataset in Fig. 3(a) and (b) is the Yeonji-ri dataset, which has two images with an aerial image with a size of 1398 × 1350 pixels and a UAV image with a size of 13052 × 12007 pixels; this area is affected by the landslide. The second dataset (Fig. 3(c), (d)) is acquired from Geummae, which has an aerial image with a size of 1517 × 888 pixels and a UAV image with a size of 12112 × 13734 pixels. The third dataset (Fig. 3(e), (f)) is the Giyang dataset, in this area, structures are collapsed after the typhoon, an aerial image with a spatial size of 1398 × 1350 pixels and a UAV image with a spatial size of 29759 × 15743 pixels.

2) Results and Discussion

The results after the saliency maps with the proposed method are shown in Fig. 4. It shows the efficiency of the saliency extraction on the change detection of UAV and aerial images. As we can see, the most concentrated area is where the changes happened, and there are fewer isolated pixels. The reason for that is because saliency map extraction omits the non-salient areas and also excludes some noise derived as a changing area in the difference image.

Fig. 4. Proposed Method Saliency Map (a) Yeonji-ri, (b) Geummae, (c) Giyang.

Results for PCA, the proposed method, and the manually created ground truth data is shown in Fig. 5. The difference between PCA and the proposed method is that difference map by CVA and saliency extraction did not implement in PCA and changes are extracted directly from UAV and aerial images. On the other hand, the proposed method is utilizing three steps before PCA which are explained in the method section. The comparison of results shows the effectiveness of the saliency extraction on UAV and aerial images change detection. It is clearly noticeable from Fig. 5, with the extraction of saliency map, there are fewer isolated pixels in the final change maps. It is mostly because of the ignorance of non-salient areas by thresholding of saliency maps, which helps to keep distinct and salient features. Because K-means clustering is so sensitive to outliers, ignoring the non-salient areas can improve clustering results.

Fig. 5. Change detection results with different method for Yeonji-ri, Geummae, Giyang in order.

Five indexes are applied to evaluate the accuracy of proposed method and compare the result with those of PCA (Hou et al., 2016).

1) False Alarm (FA): the number of unchanged pixels that are incorrectly detected as changed ones.

False Alarm Rate is expressed as:

\(\frac{F A}{\text { Number of unchancged pixels }} \times 100\)       (8)

2) Missed Alarm (MA): the number of changed pixels that are incorrectly detected as unchanged ones.

Missed Alarm Rate is:

\(\frac{M A}{\text { Number of changed pixels }} \times 100\)        (9)

3) Overall Alarm (OA): the total number by (FA) and (MA);

the Overall alarm rate is: 

\(\frac{F A+M A}{\text { Number of unchanged pixels }+\text { Number of changed picels }} \times 100 \)       (10)

4) Kappa coefficient (Kappa): the consistency between results and the reference data; Kappa is calculated as: \(\frac{P_0 - P_c}{1 - P_c} \times 100, P_0\)  indicates real consistency and P_c stands for the theoretical consistency.

The Kappa coefficient is a method of measuring of the actual agreement and chance agreement. It measures how the classification performs as compared to the reference data. In order to have the same classification when we are describing the relative strength of agreement with Kappa statistics, the following labels in Table 1 will be considered to the ranges of Kappa (Hou et al., 2016; Landis, 1984).

Table 1. Strength of agreement label with Kappa statistics

5) Real consistency is calculated as:

\((TP + TN) / (TP + MA + TN + FA)\)        (11)

(TP): Is the number of changed pixels correctly detected as changed.

(TN): Is the number of unchanged pixels correctly detected as unchanged.

Experimental results and statistical analysis of the proposed method and PCA are listed in Table 2. By comparing OA and Kappa between our proposed method and PCA method, we can notice considerable improvement in our method, which indicates the efficiency of saliency feature extraction from UAV and aerial imagery change detection. Since changes in Uljin dataset are within a small region, the threshold α is set to a large value of 0.7 as indicated in equation 7.

Yeonji-ri dataset, without using saliency feature extraction and just implementing PCA, shows that the OA and Kappa are 6.22 and 0.11, respectively. By implementing of saliency extraction method with initial difference images by CVA, the OA and Kappa of the same area are 0.41 and 0.62 which got substantial results based on Table 1. This indicates that our approach shows improvement of 5.81 in OA and 0.5 in Kappa compared with PCA. Geummae dataset, also shows the effectiveness of our saliency feature extraction; it shows results with clear edges and reduced number of isolated pixels compare to PCA, with the enhancement of 10.01 in OA and 0.286 in Kappa and with comparing strength of agreement with Kappa it obtained fair results.

For Giyang dataset, there are significant improvements similar to pervious datasets. Without using saliency feature extraction and just implementing PCA, the OA and Kappa are 8.65 and 0.31, respectively. But by adopting the proposed saliency area extraction, the OA and Kappa of the same area are 4.35 and 0.40, respectively and it shows that results improved from poor to moderated based on Table 2. Because the spillway wall collapsed on changed parts, which had the same material as the wall, the algorithms could not identify those areas as changed pixels.

Table 2. Compression of detection results for datasets.

CVA has been applied to generates a difference map, but we speculate that using a fused difference map will further improve our approach. The limitation in our approach is for small change pixels. For example, changes in small landslide happened with only a few pixels; they will not appear in the final map. The reason is that in the process of extraction of the salience map, the small changes corresponding to the difference image are not in the focused region, then they will be ignored. For this problem, it is necessary to apply accurate statistics and methods to keep small point targets.

4. Conclusion

In this research, we proposed an unsupervised change detection method for UAV and aerial images over the flooded area which contains landslides and structure collapses, and by applying proposed change detection method on the datasets the following results is obtained.

First, PCA is working good in some cases but overall our proposed method performs better in terms Kappa and OA, because we utilized CVA and saliency map generation to minimize the effect of isolated pixel and noises.

Second, by comparing our proposed method with PCA only to the pixels correctly detected as changed, performance is 15% better. Third, real consistency using five indices our proposed method gives about 95% compared with manually extracted ground truth outperforming the PCA. Experiments on the UAV and aerial images have showed the effectiveness of the proposed model.

There are still more works to do for improving the performance of the proposed method. Using a fused difference map as an initial difference image need to be improved, and choosing the parameters of change detection automatically will be one of future tasks to enhance the results.