1. Introduction
Spatial resolution of image and photos can be increased by the super-resolution algorithms. In the image processing context, image super-resolution commonly refers to the process of using a set of cross-correlated coarse spatial resolution images of the same scene to obtain a single higher spatial resolution image. There are numerous studies on such super-resolution mapping such as Elad and Feuer (1999), Tipping and Bishop (2003). Although widely applied in image processing, these approaches are hardly applicable for superresolution of remotely sensed multispectral (MS) imagery because of the lack of a sequence of images of the scene at the same or similar times. The only feasible application of the super-resolution approaches using image sequences is for hyperspectral imagery (Akgun et al. 2005). For other common multispectral remotely sensed imagery, only few methods for increasing the spatial resolution to sub-pixel level have been proposed, such as a Point Spread Functionderived convolution filter (Pinilla Ruiz and Ariza Popez 2002), segmentation technique (Schneider and Steinwendner 1999), and geostatistical method (Kyriakidis and Yoo 2005).
Sub-pixel spatial resolution land cover maps can be predicted using super-resolution mapping techniques. The input data for super-resolution mapping are commonly the land cover proportions estimated by soft-classification (Foody 2002a; Foody 2002b). There is a list of super-resolution mapping techniques which have been introduced including spatial dependence maximisation (Atkinson 1997), linear optimisation techniques (Verhoeye and Wulf 2002), Hopfield neural network (HNN) optimisation (Tatem et al. 2002), two-point histogram optimisation (Atkinson et al. 2008), genetic algorithms (Merterns et al. 2003) and feed-forward neural networks (Merterns et al. 2003). The supplementary data are also supplied to HNN to produce more accurate sub-pixel land cover maps such as multiple sub-pixel shifted image (Ling et al. 2010), fused and panchromatic (PAN) imagery (Nguyen et al. 2006; Nguyen et al. 2011). These latter approaches produce a synthetic MS or PAN image as an intermediate step for super-resolution mapping based on a forward model and then these images are compared with the predicted and observed MS or PAN images to produce an accurate sub-pixel image classification.
The creation of the predicted MS and then PAN image by a forward model suggested a possibility to implement a super-resolution for the MS image. A method for increasing the spatial resolution of the original MS image is introduced by Nguyen et al. (2009). The new model is based on the HNN super-resolution mapping technique from unsupervised soft-classification combined with a forward model using local end-member spectra (Nguyen et al. 2006; Nguyen et al. 2011). The method is examined with a degraded remote sensing image and both visual and statistical evaluations shown a good result. However, there still exist some concerns about the feasibility of the model because it is not tested in a more complicated landscape with different kinds of land cover features which are varying in sizes and shapes as well as spectral characteristics. This paper, therefore, is to implement the test of the algorithm in a complicated landscape.
2. General Model
The proposed model is an extension of the super-resolution mapping approach based on HNN optimisation. The prediction of a MS image at the sub-pixel spatial resolution is based on a forward model with local spectra as was used in Nguyen et al. 2006. In addition to the goal functions and the proportion constraint of the HNN for super-resolution mapping, a reflectance constraint is used to retain the brightness values of the original MS image.
Fig. 1 presents the HNN sub-pixel MS image prediction algorithm. The procedure is as follows: From the MS images at the original MS spatial resolution, land cover area proportion images are predicted using a soft-classifier. A set of local end-member spectra values is calculated based on the estimated land cover proportions and the original MS image. Land cover proportions are then used to constrain the HNN for super-resolution mapping with a zoom factor z to produce the land cover map at the sub-pixel spatial resolution. From the super-resolution land cover map at the first iteration, an estimated MS image (at the sub-pixel spatial resolution) is then produced using a forward model and the estimated local end-member spectra. The estimated MS image is then convolved spatially to create a synthetic MS image at the coarse spatial resolution of the original image. Following a comparison of the observed and synthetic MS images, an error value is produced to retain the brightness value of the pixels of the original MS image. The process is repeated until the energy function of the HNN is minimised and the synthetic MS image is generated.
Fig. 1.General model for super-resolution MS imagery prediction
A demonstration of the algorithm for an image of 2×2 pixels can be described in Fig. 2. Firstly, the soft-classification predicts land cover proportion as in Fig. 2b from the MS spectral image as in Fig. 2a. There are two land covers in this image called Class A and Class B. From the land cover proportions in Fig. 2b, the land cover classes at sub-pixel level are predicted as in Fig. 2c where a pixel is divided into 4×4 sub-pixels and the 2×2 pixel image is super-resolved to 16×16 pixel land cover image of Class A and Class B. The brightness of the new 16×16 pixels image is predicted using end-member spectra (standard brightness for the Class A and B in this area of the image). For example, the brightness of the pure pixel of Class A is 35 and Class B is 50 and it is possible to produce a new spectral image by assigning all the sub-pixels belonging to Class A the brightness value of 35 and the sub-pixels of Class B the brightness value of 50 as in Fig. 2d.
Fig. 2.Creation of 16×16 pixels image from 2×2 pixels imag
2.1. Soft-classification for super-resolution mapping of MS imagery
Soft-classification is an intermediate step in the subpixel MS image prediction process. The prediction of the MS image based on super-resolution mapping requires land cover proportions which are obtained from soft-classification as input data. Conventionally, there must be a set of training data for the soft-classifier. Accordingly, it is necessary to have some prior information about the spectral distribution of land cover classes in the MS bands, although training data are not always available for the image. And sometimes, there is a requirement of increasing the spatial resolution of the image without concerning the land cover classes in the image scene. In these cases, the algorithm can be implemented with unsupervised soft-classified land cover proportions such as fuzzy c-means classification (Bezdek et al. 1999).
Supervised soft-classifiers could also be used, such as Bayesian, neural network or k-NN classifiers. However, the training data for these soft-classification techniques should be obtained from the unsupervised classifications. In the research implemented by Nguyen et al. (2009), a test for algorithm was implemented with a set of degraded MS image and soft-classified land cover proportions was obtained using k-NN classification using a training data set extracted from unsupervised Interactive Self-Organising Data (ISODATA) classifier. In this case, training data were clustered in the reference image. In this experiment, a fuzzy c-means classification is used to predict land cover proportions of a real SPOT image to produce super-resolved spectral image of different spatial resolutions to evaluate the algorithm.
2.2. Forward model and end-member spectra
After the first iteration of the HNN algorithm, once the sub-pixel classification is obtained, a forward model is used to produce a sub-pixel MS image from the sub-pixel land cover classes. The brightness value (e.g., reflectance, radiance, digital number) of a sub-pixel (m, n) of a spectral band s can be predicted as
where Ve is the output neuron of the class e and Ss,e is the end-member spectra of the land cover class e for a spectral band s. As presented in Nguyen et al. 2006, the end-member spectra vector Ss (Ss = [Ss,1,..,Ss,e,..,Ss,c]) of the original pixel (x, y) of the spectral band s can be estimated locally using the predicted land cover class proportions and the MS image at the original coarse spatial resolution as:
where P is a maxtrix of land cover robportions :
and with is digital number of pixel (x, y).
3. Experiment Condition
3.1. Data
The experiment in Nguyen et al. (2009) is conducted in an area having many large objects with linear boundaries. It may lead to a concern that the algorithm proposed in this paper is able to work well only with some specific landscapes. Therefore, a second data set is used for testing the algorithm in a more complicated landscape. This image was obtained in Bac Giang Province, Vietnam.
The SPOT 5 image used in this test was acquired in August 2011 with the spatial resolution of 10m and four spectral bands (Fig. 3). The test image is registered to WGS- 1984 UTM map projection in Zone 48N and the location is at 21°17'53.65"N, 106°11'7.64"E. The test image covers 1 square kilometre area of 102×102 pixels. To evaluate the results of increasing the spatial resolution algorithm, a 2.5m spatial resolution panchromatic image acquired at the same time was used (Fig. 4b).
Fig. 3.SPOT 5 image in Bac Giang Province, Vietnam: (a) Band 1, (b) Band 2, (c) Band 3 and (d) Band 4
Fig. 4.(a) Original image, (b) Panchromatic image at 2.5m spatial resolution and land cover proportions from fuzzy c-means classification: (c) 5 land cover classes and (d) 6 land cover classes
3.2. Soft-classification
In this exprement, the land cover proportions are estimated from 10m spatial resolution SPOT 5 image using fuzzy c-means classifiers therefore it is not necessary to have training samples for the area. The soft-classified land cover proportions of five land cover classes and six land cover classes are obtained as in Fig. 4c and Fig. 4d, respectively.
4. Results and Discussions
4.1. Results
The method of increasing the MS image using HNN and forward model was applied to super-resolve the 10m SPOT 5 MS images to predict MS image at spatial resolutions of 5m (zoom factor of 2), 3.3m (zoom factor of 3) and 2.5m (zoom factor of 4). The predicted soft-classified proportions were used to constrain the HNN with weighting factors of k1 = 100, k2 =100, k3 = 100 and k4 = 100 to predict the sub-pixel land cover and then the MS image. The false colours compositions using Band 1, Band 2 and Band 4 as Red, Green and Blue are shown in Fig. 5.
Fig. 5.Super-resolution of 10m SPOT 5 multispectral image: (a) 5m super-resolved image using 5 land cover classes, (b) 3.3m super-resolution image using 5 land cover classes, (c) 2.5m super-resolution image using 5 land cover classes, (d) 5m super-resolution image using 6 land cover classes, (e) 3.3m super-resolution image using 6 land cover classes, and (f) 2.5m super-resolution image of 6 land cover classes
4.2 Evaluation
The visual comparison of the super-resolved image from the real SPOT 5 data with the panchromatic image (Fig. 4b) also shows an improvement in sharpness of the results. The objects in Fig. 5(a-f) are sharpened and look clearer that of original image (Fig. 4a). Although the landscape of image area is complicated with small and linear features such as houses and roads, the improvement of the algorithm can be seen in the boundaries of between the objects. Fig. 6 shows the improvement of the algorithm for increasing the spatial resolution of MS image using HNN with a forward model to the original image. The boundaries of ponds in the centre of the original MS image (Fig. 6a) are blurred and fragmented because of the mixing of the land categories in these boundary pixels. In the predicted MS image using HNN and a forward model (Fig. 6b), these boundaries are clear and look more similar to the real ponds in panchromatic image (Fig. 6c).
Fig. 6.Some land cover features in (a) original MS image (false colour composite), (b) increased resolution MS image to 2.5m spatial resolution from 5 land cover class proportions (false colour composite) and (c) panchromatic image
To illustrate the advantage of the new downscaled multispectral image to the original one, the small objects such as houses and roads are considered. There are few objects which is not clearly seen in the original image can be recognised in the super-resolved image. In Fig. 7a (composite image using Band 1, Band 2 and Band 4 of the original MS image), the road is difficult to recognised because it is fragmented due to the mixed pixels effect. Using the HNN super resolution mapping and then the forward model as in Fig. 7b (composite image using Band 1, Band 2 and Band 4 of the 2.5m spatial resolution increased image), it is possible to recreate the road similar to the shape of the real road shown in the panchromatic image Fig. 7c.
Fig. 7.Some land cover features such as roads and houses in (a) original image, (b)spatial resolution increased image (false colour composite) and (c) panchromatic image
For the small features such as a group of houses in Fig. 7c, the performance of the algorithm is not as good as that for the road. However, the newly proposed algorithm still shows some improvement in defining clear boundaries of these features. This may be because of the soft-classifier cannot define the houses as a separate class. This problem may be resolved by increasing the number of classes for fuzzy c-means classifier or using prior information on these classes in supervised soft-classifiers.
4.3 Discussions
The effect of zoom factor to spatial resolution increasing algorithm can be seen in Fig. 8. Comparing the image created by HNN using zoom factor of 2 (Fig. 5a), with the image created with zoom factor of 3 (Fig. 5b) and 4 (Fig. 5c), it is possible to see that when the zoom factor increases, the boundaries between the features are smoother. The boundaries between the ponds and the surrounding features are fragmented in Fig. 8a and Fig. 8b, and look smoother and clearer in Fig. 8c.
Fig. 8.Effect of zoom factor to spatial resolution increasing algorithm: (a) zoom factor of 2, (b) zoom factor of 3 and (c) zoom factor of 4
Inspite of increasing the spatial resolution of the remotely sensed MS images, the proposed method has a problem with pixels that belong to the same class (referred to as pure pixels in this paper). The problem can be partly resolved by increasing the number of classes so some pure pixels will be identified as mixed pixels. Another solution is to divide a class into sub-classes. However, in those cases there still exist pure pixels in the image. The effect of the number of land cover classes can be seen in Fig. 9. The ponds in Fig. 9a apparently smaller than those in Fig. 9b due to some “pure pixels” in the boundaries were re-classified as mixed pixels when the number of classes increased from 5 to 6. These mixed pixels are then super-resolved to produce different boundaries between the same features in the two images.
Fig. 9.(a) Result from HNN using 5 land cover classes, and (b) result from HNN using 6 land cover classes
Because the HNN super-resolution method works only on mixed pixels, which are usually located across the border between different classes, it is suggested that the method is suitable for the super-resolution of images of large objects, for example the agricultural scenes. In these images, spatial variation is homogeneous within the land parcels and superresolution based on the spatial clustering goal functions of the HNN can work particularly well for the field boundaries or increas the sharpness of linear features such as roads or canals.
As mentioned above, the use of unsupervised classification can reduce the errors in land cover class proportion prediction. Furthermore, the use of unsupervised classification facilitates the automation of the spatial resolution increasing process because the class proportions can be obtained without training data and without a training step. Through the choice of the number of classes, the user can control the effect of the super-resolution algorithm on the resulting sub-pixel MS images. When the number of classes is changed, the number of mixed pixels may be changed as a result.
A drawback of the HNN super-resolution procedure is the subjective choice of the parameters for the goal functions, the proportion constraint and the multi-class constraint (Nguyen et al. 2006). By empirical investigation, the values of the parameters should retain an equal effect between the constraints and the goal functions in the optimisation process. For example, the empirical investigation in this paper shows that the values of these parameters in this paper were similar and around the value of 100.The finding is also obtained from the Nguyen et al. (2009).
5. Conclusions
The approach for increasing the spatial resolution MS imagery utilised the HNN super-resolution mapping technique combined with a forward model is tested with 10 SPOT 5 remotely sensed data. In this research, the softclassified land cover proportions were estimated using a fuzzy c-means classifier. The feasibility of the method was evaluated based on visual comparison of the resulted image with panchromatic image acquired at the same time with original image. The comparison showed that the proposed method can generate MS images with more detail features.
The super-resolved image was apparently sharper than the original coarse spatial resolution image. In addition, the evaluation also demonstrated that when the zoom factor increased, the resulting sub-pixel images were closer to the reference image. Based on this mechanism, the future research will focus on using this algorithm for increasing and automatically defining the spatial objects from remotely sensed imagery.
References
- Akgun, T., Y., A., and Mersereau, R., ( 2005), Superresolution reconstruction of hyperspectral images, IEEE Transactions on Image Processing, 14, pp. 1860-1875. https://doi.org/10.1109/TIP.2005.854479
- Atkinson, P. M., (1997), Mapping sub-pixel boundaries from remotely sensed images. In Z. K. (Ed.) (Ed.), Innovation in GIS 4 (pp. 166-180), London: Taylor and Francis.
- Atkinson, P., Pardo-Iguzquiza, E., and Chica-Olmo, M., (2008), Downscaling Cokriging for Super-Resolution Mapping of Continua in Remotely Sensed Images, IEEE Transactions on Geoscience and Remote Sensing, 46(2), pp. 573 - 580. https://doi.org/10.1109/TGRS.2007.909952
- Bezdek, J. C., Keller, J. M., Krishnapuram, R., and and Pal, N. R., (1999), Fuzzy Models and Algorithms for Pattern Recognition and Image Processing, Boston: Kluwer.
- Elad, M., and Feuer, A., (1999), Super-resolution reconstruction of image sequences, IEEE Transactions on Pattern Analysis and Machine Intelligence, 21, pp. 817-834. https://doi.org/10.1109/34.790425
- Foody, G. M., (2002a), Hard and soft classifications by a neural network with a nonexhaustively defined set of classes, International Journal of Remote Sensing, 23, pp. 3853-3864. https://doi.org/10.1080/01431160110109570
- Foody, G. M., (2002b), The role of soft classification techniques in the refinement of estimates of Ground Control Point location, Photogrammetric Engineering and Remote Sensing, 68, pp. 897-903.
- Kyriakidis, P. C., and Yoo, E. H., (2005), Geostatistica prediction and simulation of point values from the areal data, Geographical Analysis, 37, pp. 124-152. https://doi.org/10.1111/j.1538-4632.2005.00633.x
- Ling, F., Du, Y., Xiao, F., Xue, H., and Wu, S. J., (2010), Superresolution land-cover mapping using multiple sub-pixel shifted remotely sensed images, International Journal of Remote Sensing, 31(19), pp. 5023-5040. https://doi.org/10.1080/01431160903252350
- Merterns, K. C., Verbeke, L. D., and De Wulf, R., (2003), Using genetic algorithms in sub-pixel mapping, International Journal of Remote Sensing, 24, pp. 4241-4247. https://doi.org/10.1080/01431160310001595073
- Merterns, K. C., Verbeke, L., Westra, T., and De Wulf, R., ( 2004), Sub-pixel mapping and sub-pixel sharpening using neural network predicted wavelet coefficients, Remote Sensing of Environment, 91(2), pp. 225-236. https://doi.org/10.1016/j.rse.2004.03.003
- Nguyen, Q. M., Atkinson, P. M., and Lewis, H. G., (2006), Super-resolution mapping using Hopfield neural netwok with fused images, IEEE Transactions on Geoscience and Remote Sensing, 44(3), pp. 736-749. https://doi.org/10.1109/TGRS.2005.861752
- Nguyen, Q. M., Atkinson, P. M., and Lewis, H. G., (2011), Super-resolution mapping using Hopfield Neural Network with panchromatic imagery, International Journal of Remote Sensing, 32(21), pp. 6149-6176. https://doi.org/10.1080/01431161.2010.507797
- Nguyen, Q. M., Do, V. D., Atkinson, P., and H.G.Lewis., (2009), Downscaling Multispectral Imagery Based on the HNN Using Forward Model, 7th FIG Regional Conference on Spatial Data Serving People: Land Governance and the Environment-Building the Capacity, Hanoi.
- Pinilla Ruiz, C., and Ariza Popez F.J., (2002), Restoring SPOT images using PSF-derived deconvolution filters, International Journal of Remote Sensing, 23, pp. 2379-2391. https://doi.org/10.1080/01431160110075857
- Schneider, W., and Steinwendner, J., (1999), Land cover mapping by interrelated segmentation and classification of satellite images, International Archives of Photogrammetry and Remote Sensing, 32, part 7-4-3.
- Tatem, A. J., Lewis, H. G., Atkinson, P. M., and Nixon, M. S., (2002), Super-resolution land cover pattern prediction using a Hopfield neural network. In F. G. M., and A. P. M. (Eds.), Uncertainty in Remote Sensing and GIS (pp. 59-76.). London: John Wiley and Sons.
- Tipping, M. E., and Bishop, C. M., (2003), Bayesian image super-resolution in Advances in Neural Information Processing Systems, Boston: The MIT Press.
- Verhoeye, J., and De Wulf, R., (2002), Land cover mapping at sub-pixel scales using linear optimisation techniques, Remote Sensing of Environment, 79, pp. 96-104. https://doi.org/10.1016/S0034-4257(01)00242-5