Automatic Registration between EO and IR Images of KOMPSAT-3A Using Block-based Image Matching

Abstract

This paper focuses on automatic image registration between EO (Electro-Optical) and IR (InfraRed) satellite images with different spectral properties using block-based approach and simple preprocessing technique to enhance the performance of feature matching. If unpreprocessed EO and IR images from Kompsat-3A satellite were applied to local feature matching algorithms(Scale Invariant Feature Transform, Speed-Up Robust Feature, etc.), image registration algorithm generally failed because of few detected feature points or mismatched pairs despite of many detected feature points. In this paper, we proposed a new image registration method which improved the performance of feature matching with block-based registration process on 9-divided image and pre-processing technique based on adaptive histogram equalization. The proposed method showed better performance than without our proposed technique on visual inspection and I-RMSE. This study can be used for automatic image registration between various images acquired from different sensors.

1. Introduction

Image registration is the process of aligning objective image geometrically to the reference image with the same scene taken from different orientations or different sensors. Image registration is important on the image processing because its result can be used for various fields of image processing such as change detection, time-series analysis, image fusion, pansharpening, image mosaic, etc.(Bentoutou et al., 2005).

In the previous researches, the image registration method is generally classified into two types: the area-based method and the feature-based method. The area-based approach extracts small windows from the input images and quantifies the similarity between two images using with the pixel value or statistical value to apply the image matching process. There are several previous researches using various statistics in the area-based approach, for example, cross-correlation (Li and Zhou, 1995; Li, 1995), sum of squared differences (Irani and Anandan, 1998), and mutual information (Chen et al., 2003). On the other hand, the feature based method detects feature points such as corner and blob in two images. Then, it generates feature descriptors and measures the similarity of descriptors to calculate the transformation matrix between two images(Zheng, 1993; Kim et al., 2007; Kim, 2015; Wu et al., 2015; Zotiva and Flusser, 2003). These two approaches have a problem of mismatching when input images have high noise levels or image properties are different. Accordingly, various approaches to solve these problems have been conducted. Hong and Zhang (2007) proposed a method to improve image registration performance by combining an area-based method with normalized cross-correlation and a feature-based wavelet method. SIFT (Lowe, 1999) is known as shown a good performance in the matching process (Oh and Lee, 2011; Nag, 2017).

Recently, several methods of modifying the SIFT algorithm were proposed, such as ASIFT (Morel and Yu, 2009), PCA-SIFT (Ke and Sukthankar, 2004), and there were other methods such as SURF (Bay et al., 2008) and modified SURF descriptor (Bouchiha and Besbes, 2013). On the other hand, multi-modal studies, which is matching method for the images from different types of sensors, matching images obtained from different types of sensors, have been conducted and used them for image registration. Byun et al. (2017) proposed an image registration method between EO images and synthetic aperture radar (SAR) images using mutual information. Also, image registration using a simple pre-processing technique was proposed for the three types of IR images (LWIR, MWIR, SWIR) by Kim (2017).

KOMPSAT-3, 3A, 5 satellites are currently operating in Korea, and the target images of this study are KOMPSAT-3AEO images and IR images. It provides multi-spectral(EO), MWIR, and panchromatic images and the resolution of each image is 2.2 m, 5.5 m, 0.55 m, respectively (Lee et al., 2019). Because of each detector position in the platform is different, the geometry of each image is not coincide with each other, so image registration process is necessary. Previous image registration methods do not work well because EO images and IR images have different spectral characteristics. Several researches using satellite image from KOMPSAT series except 3A have been conducted (Lee and Oh, 2020; Oh and Lee, 2019; Seo and Eo, 2019). Since KOMPSAT-3A satellite has recently started its mission among the KOMPSAT series currently in operation, few studies on image registration between the EO and IR images using the images obtained in KOMPSAT-3A have been conducted.

The purpose of this research is to propose an automatic image registration between the EO and IR images of KOMPSAT-3A satellite using the feature-based method. Our approach attempted to transform the two images with different spectral properties into similar shapes to enhance the performance of feature matching algorithms(Kim, 2017). In addition, we also proposed the block-based approach to improve the accuracy of image registration for a wide area and analyzed the results ofthe block-based method through the experiments. The key points of this research are simple preprocessing, which is applied for the EO and IR images, and block-based method to improve the accuracy of image registration.

Chapter 2 describes the detailed image registration method of the research, and chapter 3 summarizes the experimental results using actual satellite images. Based on the experiments, chapter 4 summarizes the results of the research.

2. Image Registration

The images used in this study are EO images in the visible region and IR images (Fig. 1). EO image is composed of reflection information in a visible light wavelength, and IR image is composed of temperature information. Since the positions of each sensor on the platform are different, their actual ground location can be different. It can occur to align images at the same area which are taken at different time. However, due to differences ofthe image formation process, it is difficult to obtain good registration results for the different sensorimages by simply applying the feature matching algorithms (Kim, 2017). To solve the above problem, automatic image registration between EO and IRimage was performed as follows: ① image preprocessing, ② block division, ③ feature point extraction and feature matching, ④ removing mismatching and image registration. Details of each process are described in the relevant sections.

Fig. 1. The spectrum of visible and IR wavelength (Courtesy of InfraTec).

1) Image preprocessing

In this step, two images with different spectral characteristics are preprocessed to make similar characteristics of images. Dark pixels are the object with high temperature due to low reflectivity, and light pixels are the object with low temperature due to high reflectivity (Wu et al., 2015). Thus, the phenomenon of contrastreversalmay occurin a part ofthe EO image and the IR image. Contrast reversal decreases the performance of feature matching algorithm due to changing the direction of the gradient. In order to overcome above problem, a simple preprocessing method for the input images was used for feature matching between EO and IR images. Preprocessing was performed as the following equation (Eq. (1)) to maintain local contrast and suppress noise in images over a wide area, such as Kim (2017).

I_pp = abs (ahisteq (I_all) – median (I_all))       (1)

Where I_all means whole image, abs, ahisteq, median means absolute value, adaptive histogram equalization (Pizer et al., 1986) and median value respectively. Adaptive histogram equalization was applied to the images so that the two images had similar brightness values as a whole, and that the contrast wasreversed by taking an absolute distance from the median intensity value. Using this preprocessing, the area where contrast reversal occurred was processed (Fig. 2).

Fig. 2. Correction of contrast reversal using pre-processing (Up : EO, Down : IR).

2) Block division

When the entire image is realigned by a single transformation matrix, it is difficult to match exactly for the whole image because of the height difference among matching areas and non-uniformly distributed matching points. Fig. 3 shows the result of image registration with manually selected ground control points. Ground control points are uniformly distributed in the whole image and the image was warped using 2-D affine transformation.

Fig. 3. Dislocated road of manual processing.

EO and IR images are shown as the form of a chessboard to check visibly the connection status of the straight road, by this process, 2-pixel dislocation can be found. For this reason, if the wide area image registration is performed, inaccurate results may be occurred.

Therefore, we proposed a dividing process by dividing the EO and IR images into nine sub-images. On dividing the image into sub-images, 50-pixel margin was placed at the boundary of the each divided image to remove a blank area due to difference of transformation matrix as shown in Fig. 4. The method of generating each sub-image is as Eq. (2).

Fig. 4. Position of divided images from Eq. (2).

\(\begin{array}{l} \mathrm{I}_{\left(b_{x}, b_{y}\right)}=I&\left(\max \left(1,\left(b_{y}-1\right) * \frac{n_{\text {column }}}{3}-50\right)\right. \\ &\min \left(b_{y} * \frac{n_{\text {column }}}{3}-50, n_{\text {column }}\right) \\ &\max \left(1,\left(b_{x}-1\right) * \frac{n_{\text {row }}}{3}-50\right) \\ &\min \left(b_{x} * \frac{n_{\text {row }}}{3}-50, n_{\text {row }}\right) \end{array}\)       (2)

where n_column, n_row means the number of horizontal and vertical pixels of the input image and b_x, b_y∈{1, 2, 3}.

3) Feature extraction and matching

Since this study is a block-based method, image registration is possible for the entire image only if true matching pairs exist in all sub-images. As a result of comparing the three methods of SIFT (Lowe, 2004), SURF (Bay et al., 2008), and BRISK (Leutenegger et al., 2011), true matching pairs were produced stably fromall blocks when we used SIFTalgorithm. Because the EO image and the IR image have different resolution, SIFT algorithm known for scale invariant has been stably produced the true matching pair in all blocks (Mistry and Banarjee, 2017).

The SIFT algorithm is used for generating descriptors by extracting feature points that are invariant to scale and rotation of an image. It shows the robustness of scale change, illumination change, translation,rotation, and some occlusion. The sequence of SIFT algorithm consists offoursteps: ① Scale-spaceExtremaDetection, ② Keypoint Localization, ③ OrientationAssignment, and ④ Keypoint Descriptor.In the process ①, extreme points are detected from DoG (Difference of Gaussian) asthe candidates offeature point afterGaussian pyramid calculation. In the process ②, incorrect candidates are excluded, and the orientation of the detected feature points is assigned in the step ③. If the positions of the feature points determined through the previous step, 128-dimensional SIFT feature point descriptors are generated in the step ④. Comparing the feature descriptor of one image with that of the other image, matching pairs are generated. In this research, we use the spectral angle instead of the Euclidean distance when comparing vector values, because Euclidean distance increases in computational complexity as the vector dimension increases.

4) Removal of mismatched pair and image registration

Mismatched pairs exist among the matching pairs resulting from the above process, and it is necessary to remove them to calculate more accurate transformation matrix. Mismatched pairs can be removed by RANSAC, as the outlier removal method. RANSAC algorithm is a prediction method that repeatedly finds the optimal solution by randomly sampling the minimum data to determine the coefficients of model among all data. In this study, mismatched pairs were more robustly removed using MSAC (Torr et al., 2000), the variation of RANSAC algorithm. The positions of feature points in the two images are input variables, 1000 iterations and 2-pixel of matching threshold are set to remove mismatching pairs.

There are forms oftranslation,rotation,scale, affine, and projective for the transformation of 2D image. In this research, it was assumed that the number of matching pairs existed sufficiently, and since the position difference of the sensors on the platform was not large, image matching was performed by calculating the affine transformation matrix between the two images as the following equation (Eq. (3)).

\(\left[\begin{array}{l} x^{\prime} \\ y^{\prime} \\ 1 \end{array}\right]=\left[\begin{array}{lll} a & b & t_{x} \\ c & d & t_{x} \\ 0 & 0 & 1 \end{array}\right]\left[\begin{array}{l} x \\ y \\ 1 \end{array}\right]\)       (3)

Two-dimensional affine transformation matrix is represented using homogeneous coordinate system. Where a, b, c and d are coefficients including scaling, rotation, shearing, t_x and t_y express parallel movement of horizontal and vertical direction. When three or more matching pairs are detected in order to calculate six unknowns, coefficients of the affine transformation matrix can be calculated using least square method.

3. Experiments

1) Experiment Image

The images used in the experiment are the EO and IR satellite images of KOMPSAT-3A, the Greater Hobby Area in Houston, USA. The EO image has spatial resolution of 2.2 m and the IR images of 5.5 m. The number of pixels of the EO image is 5,850 × 6,090, and the number of pixels of the IR image is 1,840 × 1,810. IR image was transformed with crop and 30 degree rotation to test proposed method on experiments. Although the experiments were performed using satellite images with original resolution, printed IR images were downsampled to 33 m resolution for the security reason.

Since the EO image contains the reflection information of the visible region of the object and the IR image containstemperature information, the image properties are different, the preprocessing was performed so that the two images have similar characteristics. The preprocessed images are divided into 9 blocks having 100-pixel overlapped area at 1/3 and 2/3 of whole image (Fig. 5).

Fig. 5. Division result of preprocessed satellite images (Left : EO, Right : IR).

2) Feature point extraction and matching pair connection

For each block of the input image, feature points were extracted using the SIFT algorithm and feature were matched. The most of feature points extracted from one block is 13,279, which is about 2.8% of the pixels. The matching threshold, which indicates the ratio when finding similarfeature points, was 0.75. The matching pairs contain incorrect matchingsin position because the vector values of the feature descriptors are only compared. Thus, the pixel locations of the matching pair in the two images were assumed 2- dimension affine transform and compared before and after applying MSAC (M-estimator SAmple and Consensus) algorithm. Fig. 6 is the matching result of the (2, 2) center block of whole image and Fig. 7 is the matching result of the (1, 1) edge block of whole image. There are several mismatching pairs before MSAC, but mismatching pairs can be removed after MSAC. Table 1 shows the number of feature points detected on each block and the number of matching pairs before and after MSAC.

Fig. 6. Matching results on block (2,2) (Upper : without MSAC, Lower : with MSAC).

Fig. 7. Matching results on block (1,1) (Upper : without MSAC, Lower : with MSAC).

Table 1. Result of SIFT feature and MSAC algorithm

3) Image registration and accuracy comparison

Image registration was finally performed using the matching pair obtained as a result ofthe previoussteps. By calculating the affine transformationmatrix for each block, the IR image was warped to align to the EO image.I-RMSE (InverseRoot Mean Square Error) was calculated to analyze quantitatively the accuracy of image registration (Table 1).

The accuracy using by the proposed block-based method was compared with the result produced by manually selected ground control points in Fig. 3 because image matching with SIFT algorithm didn’t work due to few detected feature points andmismatched pairs.I-RMSE value of Fig. 3 is about 1.4983 pixel and I-RMSE value of our block-based method is about 0.6389 pixel.

While the affine transformation model cannotreflect some factorssuch asregional altitude difference when image registration is performed on large area asshown in Fig. 3, the proposed block-based algorithm performed on relatively small areas and produced small I-RMSE value.

On the other hand, there is a methods for visually analyzing the results of image registration (Liu and Seipel, 2015; Kim, 2017). Result images are shown as the form of a chessboard for visual inspection if the roads are properly connected or not. By comparing Fig. 8 with Fig. 3, the proposed method hasrepresented more accurate registration result by visual analyzing.

Fig. 8. Result of our image registration algorithm.

4. Conclusion

This paper is a research on the method of automatic image registration of the KOMPSAT-3A EO and IR images. We proposed a feature based method which produced similar properties for the two input images through the preprocessing and the division of 9 subimages for the purpose of applying SIFT algorithm on them.

Because matching points were uniformly distributed on the images, image registration could be performed accurately by the block-basedmethod.The experiments showed performance improvement ofimage registration using with lower I-RMSE value and visual inspection. The proposed approach on this paper could be expanded to utilized for image registration between not only satellite EO and IR images but also among other heterogeneous images.

On the other hand, if matching points are not detected sufficiently in a block or both blocks of the same number do not have common area, the proposed method may have a problem to complete registration as a whole because the block would not be matched accurately. Further research should be continued to improve the versatility ofthe approach by suggesting a solution of the above problem.