Fast and Efficient Satellite Imagery Fusion Using DT-CWT Proportional and Wavelet Zero-Padding

Abstract

Among the various image fusion or pan-sharpening methods, those wavelet-based methods provide superior radiometric quality. However, the fusion processing is not only simple but also flexible, since many low- and high-frequency sub-bands are often produced in the wavelet domain. To address this issue, a novel DT-CWT (Dual-Tree Complex Wavelet Transform) proportional to the fusion method by a WZP (Wavelet Zero-Padding) is proposed. The proposed method produces a single high-frequency image in the spatial domain that is injected into the LRM (Low-Resolution Multispectral) image. Thus, a wavelet domain fusion can be simplified to spatial domain fusion. In addition, in the proposed DT-CWTP (DT-CWT Proportional) fusion method, it is unnecessary to decompose the LRM image by adopting WZP. The comparison indicates that the proposed fusion method is nearly five times faster than the DT-CWT with SW (Substitute-Wavelet) fusion method, meanwhile simultaneously maintaining the radiometric quality. The conducted experiments with WorldView-2 satellite images demonstrated promising results with the computation efficiency and fused image quality.

1. Introduction

The satellite sensors typically supply LRM image and HRP (High-Resolution Panchromatic) image separately because of physical and technological constraints (Thomas et al., 2008; Wang et al., 2005); which of that implies that the design of the HRM (High-Resolution Multispectral) sensor is limited (Kim et al., 2011a; Zhang, 2004). Those constraints on the signal-to-noise ratio impose with the lower spatial resolution, but the desired spectral resolution is larger. Conversely, if and only if the highest spatial resolution is obtained whenever no spectral diversity is required (Chen, 2012), the image fusion or the pan-sharpening aims to improve the geometric information of the original LRM image. However, we cannot obtain an ideal HRM image due to the abovementioned limitation (Choi et al., 2013; Wang et al., 2005). Thus, to use an HRM image in various geospatial fields, an image fusion is a requisite with the efficient method in remote sensing.

Among the many image fusion methods that are currently available, the DWT (Discrete Wavelet Transform) methods provide superior fused images that maintain the radiometric information of the LRM image better than CS (Component-Substitution) fusion methods, such as the intensity-hue-saturation transform and principal component analysis (Wang et al., 2005). Although the DWT-based methods are spectrally consistent, there are two problems; firstly, DWT is a shift-variant, and exhibits artifacts due to the aliasing in fused image (Ioannidou and Karathanassi, 2007; Thomas et al., 2008); secondly, these fusion methods have a high computation cost and complexity compared with CS fusion methods (Pradhan et al., 2006). In particular, the latest satellite sensors for Earth observation are currently producing a nearly continual stream data. The explosion in the amount of collected information has required the development of computationally efficient techniques for transforming the massive amount of remote sensing data into scientific understanding.

The first problem can be overcome by the DT-CWT, which is nearly shift-invariant and directionally selective in two or more dimensions of making these methods particularly suitable for image fusion (Ioannidou and Karathanassi, 2007; Selesnick et al., 2005). In a practice, the second problem results in additional computational power and a longer wait time for the fused imagery. Thus, to address this issue, this paper proposes the DT-CWTP fusion method to solve the second problem - i.e., the high computation cost and complexity of the fusion method. In other words, the aim of the study is to present a fusion technique enabling an easier implementation of improved wavelet-based image fusion.

There are many studies on the DT-CWT in the remote sensing fields. Those superresolution imaging (Celik and Tjahjadi, 2010) and despeckling of synthetic aperture radar data were proposed (Ranjani and Thiruvengadan, 2010). In image fusion, the DT-CWT fusion method with SW method (Ioannidou and Karathanassi, 2007) and a DT-CWT fusion method with an injection model have been proposed (Renza et al., 2011). However, the study focused on the SW method rather than AW (Additive-Wavelet) method. In the image fusion community, the AW-based fusion method is preferred because the preservation of the LRM image is critical (Kim et al., 2011b; Núnez et al., 1999; Thomas et al., 2008). Additionally, these studies overlook the convenience of the fusion method in terms of algorithm flexibility.

In this study, we improve upon the DT-CWT fusion method to reduce the computation complexity compared to conventional methods and increase the flexibility of fusion method. To increase the radiometric quality of the fused image, the proposed method considers the WZP for conciseness and adopts the proportional injection model. To verify the efficiency and quality of the proposed method, experimental evaluations are conducted on the WorldView-2 data, and compared with the wavelet-based fusion methods. This paper organized as follows; in section 2, the DT-CWT image fusion and WZP briefly introduced, while the section3 proposed DT-CWTP fusion method; in section 4, those results and discussion are presented, when conclusions are drawn in section 5.

 

2. DT-CWT and Image Fusion

This section reviews the DT-CWT and related fusion methods with briefly discussions the characteristics of these methods. In addition, we focused on fusion schemes rather than mathematical formulations. It is assumed that those LRM and HRP images are a priori geometrically registered and superimposed. We enlarged the LRM image size to the HRP image size using bicubic interpolation.

Kingsbury (2001) introduced a new type of wavelet transform known as the DT-CWT, which exhibits a shift-invariant property and improves directional resolution compared with the DWT. The DT-CWT also yields the perfect reconstruction using two parallel decimated trees with real-valued coefficients generated at each tree (Celik and Tjahjadi, 2010), so as provides a time frequency analysis of the signal by measuring its frequency contents at different times. The algorithm decomposed the input signal into two signals, or sub-bands, that represent the low- and high-frequency components, respectively. The DT-CWT with the complex wavelet function and complex scaling function decomposes an image into one complex scaling subband and six complex wavelet sub-bands at each decomposition level. The wavelet sub-bands are oriented in six dimensions and provide the directionality of the complex wavelet function (Renza et al., 2011). More detailed theoretical and application parts of DT-CWT can be seen in Selesnick et al. (2005).

2.1 Perfect reconstruction with WZP

An image can be decomposed into a series of low- and high-frequency sub-bands using the specific wavelet transform. PR (Perfect Reconstruction) means that the synthesized final image is the same as the original input image. Among the many wavelet transforms, the DT-CWT supports the PR condition. This condition is important in signal and image processing because the recovered signal or image should not take the errors within the precision of the specific arithmetic implementation. This paper adapts the PR condition to image fusion.

The WZP was used to produce the super-resolved imagery by filling the unknown high-frequency sub-bands with zeros (Temizel and Vlachos, 2005b). In other words, an approximation to the unknown high-resolution image is generated using WZP. Using the available low-resolution image x of size m × n, the unknown high-resolution image is reconstructed by the zero-padding of high-frequency sub-band following to the inverse wavelet transform (IWT) as follows:

where 0m,n is an all-zero of size m×n (Temizel and Vlachos, 2005a). The WZP method can be conversely applied in image fusion. That is, the known low-frequency sub-bands can be filled with zeros. By performing the IWT, a single high-frequency image in the spatial domain is produced. Conversely, the known high-frequency sub-bands can be filled with zeros. By performing the IWT, a single low-frequency image in the spatial domain is produced, which approach can be applied to the DT-CWT. The low frequency or high frequency of the original image can be easily discarded. The PR conditions and the WZP method of DT-CWT are shown in Fig. 1, which can be simply implemented as an AW method. This method can be naturally expanded for any wavelet transform that supports the PR condition.

Fig. 1.PR condition and WZP of the DT-CWT

2.2 DT-CWT image fusion and general image fusion framework

The primary objective of DT-CWT fusion method is the reduction of aliasing in DWT-based fusion methods and obtaining in the analytic transform to minimize shift dependence, which is possible to use complex wavelets for a complex-type filter bank. However, the currently proposed DT-CWT fusion methods are based on the SW method (Ioannidou and Karathanassi, 2007; Renza et al., 2011). The primary disadvantage of these methods is the requisite decomposition of the LRM image and the subsequent loss of the high frequency of the LRM image. The first problem increases the computation cost, and the second problem results in the loss of the radiometric and geometric information of the LRM image (Kim et al., 2011b). The information that is visible in the LRM image can be missing in the HRP image and vice versa (Thomas et al., 2008). That is, SW-based methods could eliminate both the geometric and radiometric information of the LRM image. Thus, the AW-based fusion methods are preferable. Also, we have to develop an effective fusion method that is based on the AW method using DT-CWT with a fast computation speed. This task requires to maintain the LRM image during the fusion process, and inject the high-frequency, which is not included in the LRM image, by using a simple and effective approach that is the main concept of the proposed method.

Aiazzi et al. (2009) demonstrated a general fusion framework that can be defined as follows:

where HRMi is the ith HRM image, LRMi is the resized ith LRM image, LRP is the low-resolution panchromatic image or intensity image, and ωn is the global/local fusion parameter. Also, Wang et al. (2005) present a similar general image fusion method, in which the mathematical model is as follows:

where α is the modulation coefficient and ω is the signal difference between the HRP and LRP images. Parameter ω represents the geometric information between the high- and low-resolution levels. Thus, an important point in image fusion is the development of a modulation coefficient to mitigate the radiometric distortion and generation of geometric information to inject into the LRM image.

 

3. Proposed DT-CWTP Fusion Method

Firstly, we developed the DT-CWT fusion method as an AW method in the spatial domain by adopting the PR condition and WZP. After that, the proposed DT-CWTP is presented.

The procedure of proposed DT-CWT with AW fusion method can be summarized as follows:

1) Perform histogram matching between the HRP image and intensity image. The intensity image is obtained by averaging the LRM image.

2) Decompose only the histogram-matched HRP image.

3) The low-frequency sub-bands of the decomposed HRP image are filled with zeros.

4) An IDT-CWT (Inverse DT-CWT) is conducted to produce the WP (Wavelet Plane). The WP is the sum of the high frequencies and contains the detailed geometric information of the HRP image.

5) The WP is added to the resized LRM image, and finally, the fused HRM image is generated. Mathematically, the WP can be defined as follows:

where LFHRP is the low frequency of the HRP image and HFHRP is the high frequency of the HRP image. Zeros is an all-zero matrix of the LFHRP image size. Thus, the AW fusion method can be simplified into single WP injection method in the spatial domain as follows:

where the WP has the same size as the resized LRM and HRP images. This fusion method can produce the same type of fused imagery produced by the conventional AW methods. However, the proposed method is simpler and more flexible in that this method fuses in the spatial domain, which illustrated in Fig. 2. Those previous studies used the AW method as the addition approach in the wavelet domain (Amolins et al., 2007).

Fig. 2.DT-CWT fusion method with the AW method

However, the proposed fusion method injects the same geometric information into each LRM image because this method does not yet possess the modulation coefficient of the general image fusion framework. The radiometric signature of each HRM image is not preserved, which implies that the fixed injection method can produce unnecessary or redundant information; the redundant information means that the geometric and radiometric information is not presented in the LRM image, but is presented in the fused image, and vice versa. Thus, some further improvements can be achieved by injecting the detailed geometric information using the modulation coefficient. Consequently, to preserve the geometric and radiometric information in the fused data, the proposed fusion method can be rewritten as follows:

where L is the number of LRM bands. This method proportionally injects the geometric information into every pixel while considering the relative radiometric signatures of the LRM bands in the manner of the AWLP (AdditiveWavelet Luminance Proportional) (Otazu et al., 2005) and IAWP (Improved Additive-Wavelet Proportional) fusion methods (Kim et al., 2011b). To obtain this weighting factor, the ratio between the LRM image and the mean value of all LRM images was calculated. This approach facilitates the injection of geometric details into the LRM image in a manner proportional to their original values. Thus, the fused image can preserve the radiometric angle between the original LRM and fused HRM images. This method is termed the DT-CWTP method. Importantly, there is no need to decompose the LRM image, however, only a simple addition operation was used. Thus, the proposed fusion method is more capable; and it has better computational efficiency than the conventional SW-based fusion methods. In addition, this fusion method can apply the local fusion parameter in the spatial domain rather than the wavelet domain (Choi et al., 2013). Moreover, the trade-off between radiometric and geometric information could be controlled using a modulation coefficient, as in LilloSaavedra and Gonzalo (2006).

 

4. Results and Discussion

In this study, we used the WorldView-2 satellite imagery to verify our fusion method. In our experiments, the LRM image size was 512 × 512 pixels, while the HRP image size was 2048 × 2048 pixels. A Fig. 3 shows the WorldView-2 data, whereas study area covers a large of agricultural fields, urban areas, and river. We compared the proposed fusion method with DT-CWT with the SW method (Ioannidou and Karathanassi, 2007). Additionally, the à trous wavelet-based fusion methods presented in Otazu et al., (2005) and Kim et al., (2011b) were compared with our method. Also, the fusion processing time has compared based on the computation cost. The reported times were measured in the MATLAB 2014b environment and in a quad-core 2.5 GHz personal computer platform by using the average elapsed time of 20 trials.

Fig. 3.WorldView-2 data

Fig. 4.Fused images comparison

The Table 1 demonstrates the elapsed time of each fusion method. The AWLP method was the fastest method among those compared methods. The IAWP method was two times slower than AWLP method, since the IAWP method must generate the LRP image by filtering with a Gaussian low-pass filter that of frequency response matches the shape of the modulation transfer function. The DT-CWT with the SW method was the slowest method because of the decomposition of the LRM image and the substitution process in the wavelet domain.

Table 1.Elapsed time comparison

The DT-CWT with the AW method and the proposed DT-CWTP method exhibit similar elapsed times that resemble those of the IAWP method. Above all, the proposed method is nearly five times faster than the DT-CWT with the SW method (Ioannidou and Karathanassi, 2007). If the histogram matching were performed with each LRM image rather than intensity image, the elapsed time would be increased in proportion to the total number of bands in all of the fusion methods. On the other hand, the computation complexity or cost is affected by the image size, data read and write speed, pre- and post-processing, and speed of internal memory. Thus, further studies using more accurate mathematical model will be addressed in our future work. Nevertheless, the proposed fusion method is contributes to image fusion community and other image processing fields.

Objectively evaluate these fusion methods, the QNR (Quality with No Reference) index, as proposed by Alparone et al. (2008), was used. The radiometric distortion is referred to as Dλ, and the geometric distortion index is referred to as Ds. The highest value of the QNR is one and obtained when the radiometric and geometric distortion are both zero. The QNR index is defined as

A higher QNR value indicates that most of the radiometric and geometric information of the LRM and HRP images are incorporated during fusion processing. The main advantage of the QNR index is that, in spite of the lack of a reference data, the global quality of a fusion method can be assessed at the full scale of HRP image. Table 2 shows the performance comparisons of the fused images. The results indicate that the proposed DT-CWTP method provides a less distorted fused image compared with other fusion methods. Thus, as demonstrated by the elapsed time and QNR index, the proposed fusion method has low computational time requirements and is superior in fused image quality.

Table 2.QNR indices

In addition, we compare the subset WPs of the AWLP, IAWP, and DT-CWT fusion methods in Fig. 5. The WP is the sum of high frequencies and the zero-mean image. The WP of AWLP method resembles the primary high frequency in Choi et al.(2013); that was clearly demonstrated that the primary high-frequency information maximizes the geometric clarity of fused image, while maintaining the radiometric distortion caused by injecting the excessive high-frequency information. The WP of IAWP and DT-CWT is similar but displays a subtle distinction in visual analysis. The WP of the DT-CWT accentuates the dominant edge information of the feature compared with that in the AWLP and IAWP methods.

Fig. 5.Comparison of subset WPs

 

5. Conclusions

The critical issue in image fusion is how much radiometric information is preserved while simultaneously increasing the geometric information. Additionally, if the fused data from specific fusion method are nearly identical with other fused data from a more computationally efficient method, the latter method is ideal for a large quantity of remote sensing data. To address these problems, we proposed the DT-CWTP fusion method, which is considered an improvement upon the AW and SW methods of DT-CWT fusion in the sense. The LRM image is not decomposed and injects the geometric information as a proportional method. In the experimental results using WorldView-2 data, the proposed method demonstrated the higher speed and the higher quality. The comparison indicates that the proposed fusion method is nearly five times faster than the DT-CWT with SW fusion method. The most significantly, the proposed fusion method increases flexibility because the wavelet domain fusion is simplified to the spatial domain by adopting the WZP method. This approach facilitates the general and less computationally intensive fusion method using the DT-CWT.