A Study on the Land Cover Classification and Cross Validation of AI-based Aerial Photograph

Abstract

The purpose of this study is to evaluate the classification performance and applicability when land cover datasets constructed for AI training are cross validation to other areas. For study areas, Gyeongsang-do and Jeolla-do in South Korea were selected as cross validation areas, and training datasets were obtained from AI-Hub. The obtained datasets were applied to the U-Net algorithm, a semantic segmentation algorithm, for each region, and the accuracy was evaluated by applying them to the same and other test areas. There was a difference of about 13-15% in overall classification accuracy between the same and other areas. For rice field, fields and buildings, higher accuracy was shown in the Jeolla-do test areas. For roads, higher accuracy was shown in the Gyeongsang-do test areas. In terms of the difference in accuracy by weight, the result of applying the weights of Gyeongsang-do showed high accuracy for forests, while that of applying the weights of Jeolla-do showed high accuracy for dry fields. The result of land cover classification, it was found that there is a difference in classification performance of existing datasets depending on area. When constructing land cover map for AI training, it is expected that higher quality datasets can be constructed by reflecting the characteristics of various areas. This study is highly scalable from two perspectives. First, it is to apply satellite images to AI study and to the field of land cover. Second, it is expanded based on satellite images and it is possible to use a large scale area and difficult to access.

1. Introduction

Land cover is an important variable in predicting various physical environments related to humans, including global warming, through changes in land conditions (Foody, 2002). Such land covers are classified using remote sensing methods including “hyperspectral imaging (HSI),” “aerial imaging”, and “radar imaging,” through various remote sensing sensors (Gislason et al., 2005). From the 2000s to the 2010s, studies on land cover classification were actively conducted using AI- based machine learning algorithms including support vector machine (SVM) models (McIver and Friedl, 2001; Huang et al., 2002; Qian et al., 2015; Vatsaval et al., 2011; Rodriquez-Galiano et al., 2012). Since then, deep learning algorithms were developed to improve the raw data processing capabilities of existing machine learning algorithms. In particular, convolution neural network (CNN) algorithms in the image recognition field showed very good performance (LeCun et al., 2015; Gu et al., 2018). A number of studies were also conducted applying these CNN algorithms to land cover until the mid-2010s, which showed high performance in various remote sensing data and regions (Scott et al., 2017; Hu et al., 2018; Mahdianpari et al., 2018; Kampffmeyer et al., 2018). Recently, research and algorithms applying fully convolutional network (FCN)-based algorithms have been reported, which are optimized for object segmentation in existing CNN algorithms (Guo et al., 2018; Wang et al., 2018). In the land cover classification and mapping fields, studies applying semantic segmentation algorithms were conducted, and the land covers classified and the performance evaluated using various images including multispectral satellite images, high-resolution aerial images, and SAR images (Saralioglu and Gungor, 2020; Chen et al., 2021; Mohammadimanesh et al., 2019).

Fig. 1. Flow chart in this study.

Large-scale artificial intelligence datasets are being built to apply various AI algorithms to land covers (Xia et al., 2018). The EuroSAT dataset is a dataset built based on Sentienl-2 satellite images, and contains 27,000 images with 13 spectral bands and 10 classes labeled (Helber et al., 2019). DOTA is a dataset that classifies 2,806 images into 14 classes using Google Earth images. The Agriculture-vision dataset classifies images with resolutions of 10 m, 15 m and 20 m into 9 classes including trees for US forested areas. Lancover.ai is a dataset constructed with images with resolutions of 25 cm and 30 cm for the Polish region (Xia et al., 2018; Chiu et al., 2020; Boguszewski et al., 2021). Of such datasets, as land cover datasets constructed for regions within a specific country, there are the Vaihingen and Postdam datasets, which are the 2D semantic segmentation datasets of remote sensed imagery provided by the International Society for Photogrammetry and Remote Sensing (ISPRS). Each of these datasets is classified into 5 categories including water for specific German regions; Postdam and Vaihigen datasets consist of images with resolutions of 5 cm and 9 cm, respectively (ISPRS website: https://www.isprs.org). In addition, a remote sensing land-cover dataset for domain adaptive semantic segmentation (LoveDA) was constructed as an urban and rural land cover dataset for three regions due to differences in urban and rural images by region (Wang et al., 2022). There were also studies comparing the accuracy of datasets constructed for two or more regions; a difference of about 4% in accuracy was also found between data of similar classification items in adjacent regions (Wang et al., 2022). However, there are still insufficient studies to evaluate the difference in classification accuracy when existing datasets constructed for certain areas are applied to other areas. The purpose of this study is to evaluate the performance and cross validation of existing datasets when such datasets are applied to other regions. To this end, first, Gyeongsang-do and Jeolla-do were selected as study areas, and datasets were constructed and acquired for the regions for a cross-analysis. Second, the datasets for the two regions were applied to semantic segmentation algorithms, respectively. After training, there weights of the two regions were calculated. Finally, the results were derived by applying the training weights to the test areas (TAs) extracted from the two regions. Classification accuracy and results were evaluated in consideration of the surface area of dataset items (Fig. 1).

Fig. 2. Land cover map aerial image, annotation data and json file contents in AI-Hub

2. Methods

1) AI-Hub

AI-Hub (https://www.aihub.or.kr) was created by the Ministry of Science and ICT (MSIT) and the National Information Society Agency (NIA) to solve various social problems using artificial intelligence and expand its use since 2010. In early 2018, AI-Hub was focused on the function of providing data from AI training conducted by the MSIT and the NIA. Around 2020, it was being evolved into an integrated platform that integrates and provides AI infrastructure. It is necessary for developing AI technologies, products, and services, including increased diversity and physical quantity in building AI Training Data (TD), support algorithms and service models to expand the use of AI, and AI computing support.

Currently, AI-Hub provides 93 types of data related to the Korean language, 78 types of video image data, 67 types of healthcare data, 59 types of environmental data for Disaster and Safety Management (DSM), 41 types of agricultural, livestock and fishery data, and 46 types of traffic and logistics data. The land cover map constructed using aerial and satellite images, the target of this study, is a sub-dataset of the environmental data for DSM. Aerial and satellite image data of the land cover map were constructed annually by dividing Korea by region. Specifically, the image data of Jeolla- do, Gyeongsang-do, and Jeju-do were built through a dataset construction project for AI training in 2021. The aerial and satellite image data for land cover maps of the three regions were obtained from the National Geographic Information Institute (NGII), and the European Space Agency (ESA) Sentinel-2, respectively, which constitutes a dataset for AI training that is classified into a total of 8 objects including buildings, parking lots, roads, roadside trees, rice fields, dry fields, forests, and bare land. The main use of the data was to develop an AI algorithm model for automatic segmentation of land cover and detection of changed areas in order to efficiently detect various environmental changes in the regions (Fig.2).

2) Study Area

In this study, we utilized the AI TD built by the NIA in 2020 and 2021, which is publicly available on AI-Hub. Based on the data, we selected Gyeongsangdo and Jeolla-do as study areas, excluding Jeju-do (AI-Hub). Jeju-do was excluded because it was not suitable for regional comparison because there was a difference from Gyeongsang-do and Jeolla-do in the annotation category, which is the criterion for classifying land cover. Gyeongsang-do is the combined name of Gyeongsangnam-do and Gyeongsangbuk-do. Gyeongsangnam-do is geographically located at the southeastern tip of the Republic of Korea and has an area of about 10,541 km²(Gyeongsangnam-do website: https://www.gyeongnam.go.kr). Gyeongsangbuk-do has an area of about 19,043 km², which is connected to Gyeongsangnam-do and Ulsan to the south, and has many mountainous areas and a coastline of 335 km (Gyeongsangbuk-do website: https://www.gb.go.kr). Jeolla-do is the combined name of Jeollanam-do and Jeollabuk-do. Jeollanam-do is located in the southwestern part of Korea with an area of about 12,348 km², and contains 2,165 islands, which account for 65% of the South Korean islands (Jeollanam-do website: https:// www.jeonnam.go.kr). Jeollabuk-do has an area of about 8,069 km², and forest land occupies about 56% of the total area (Jeollabuk-do website: https://www. jeonbuk.go.kr). In Gyeongsang-do and Jeolla-do regions, mostly urbanized areas were selected for AI training and prediction in the components of land cover, and the target areas are shown in Fig. 3. Based on a 1:5000 scale digital map, 40 map sheets were used for Jeolla- do, and 43 map sheets were used for Gyeongsang-do. Total area of objects in Gyeongsang-do map sheets and Jeolla-do map sheets are 2390.29 km2, respectively, which is the same (Fig. 3).

Table 1 shows the results of analysis on the status of area and differences in sub-classified land cover on 1:5000 scale map sheets for Gyeongsang-do and Jeollado used in this study. Of the sub-classified land cover items for Gyeongsang-do, the object that occupies the largest area is urban (103.665 km²), followed by readjusted cropland (68.639 km²), other grasslands (23.915 km²), and detached forest (22.915 km²). Of the sub-classified land cover items for Jeolla-do, the object occupying the largest area is readjusted cropland (120.902 km²), followed by urban (47.312 km²), grassland (25.568 km²), and detached housing forest (16.099 km²). Comparing the difference in the area of objects in Jeolla-do based on Gyeongsang-do (Table 1), the objects showing the largest difference in area are urban (56.352 km²), detached forest (6.876 km²), and bareland (4.052 km²). Compared to Gyeongsang-do, objects occupying a larger area in Jeolla-do include readjusted cropland (–52.264 km²), grassland (–1.652 km²), and wetland (–1.220 km²).

Table 1. Based on the land cover map of the Ministry of Environment, Gyeongsang-do, and Jeolla-do

Fig. 3. Study area map. (a) is Jeolla Province and (b) is Gyeongsang Province

3) Status of AI training data construction

In this study, of the aerial images of land cover maps provided by NIA via AIHub, those of Gyeongsang-do and Jeolla-do were used. First, for aerial images, images – taken in 2019 2020 with a spatial resolution of 25 cm provided by NGII were used. The aerial images from NGII were not processed separately, and those were only adjusted to 512 x 512 of pixel size with an overlap rate of 25%.

Annotation data, which comprise a set with a video image, was constructed by annotating images into 8 classes of buildings, parking lots, roads, street trees, rice fields, fields, forests, and bare fields based on refined aerial and satellite images. In this study, finely annotated data was used which precisely distinguishes the boundaries of objects to be classified. As for objects subject to be annotated, those with a linear width of 12 m or more were commonly annotated; objects with an area of 100 m2or more for facilities such as buildings were classified and annotated; objects with an area of 500 m² or more were classified and annotated for parking lots, rice fields, dry field/bare lands; and objects with an area of 2,500 m2or more were classified and annotated for forest. In addition, objects that are clearly distinguished were classified even if they were smaller than these criteria. On the other hand, of buildings in construction areas, objects whose roof or rooftop were not completed were classified as non-target sites (Table 2).

Table 2. The NIA training dataset classification entry codes and names

Fig. 4. Example of a built AI dataset. (a) is an aerial image and (b) is 8-bit grayscale annotation data.

For the finely annotated datasets, data inspection was performed on a map sheet basis in consideration of the data quality inspection for and data overlapping of adjacent areas. After the data quality inspection, the datasets were adjusted to 512 × 512 of pixel size with an overlap rate of 25% and converted to 8-bit grayscale (Fig. 4).

A total of 42,000 image data was used for the datasets in this study with 21,000 images each for Gyeongsang-do and Jeolla-do. In terms of the composition of the training datasets, the objects occupying the largest area for the Gyeongsang-do dataset were buildings (36%), rice fields (30.4%), and roads (10.3%), while objects including roadside trees, parking lots, and bare lands accounted for a small area of less than 10% compared to others. The object occupying the largest area for the Jeolla-do dataset was rice fields (55%), followed by dry fields (19.8%) and buildings (12%), while objects including roadside trees, parking lots, and bare lands occupied a small area compared to others. Comparing the difference in object area between the Gyeongsangdo and Jeolla-do training datasets, based on a 1:5000 scale digital map sheet, a large area of the Gyeongsangdo dataset for objects was occupied by buildings, parking lots, roads, roadside trees, and bare lands, while a large area of the Jeolla-do dataset for objects was taken up by rice fields, dry fields, and forests (Table 3).

Table 3. Areas and difference in area for each data set class in Gyeongsang-do and Jeolla-do used in this study

Fig. 5. An example of U-Net architecture introduced by Ronneberger et al. (2015).

4) Algorithm

The U-Net algorithm is a semantic segmentation algorithm published by Ronneberger et al. (2015), and it is characterized in a way that the convolution layer and the deconvolution layer are U-shaped (Ronnegerger et al., 2015). Like fully convolutional networks (FCN), – it has an encoder decoder structure as an effective algorithm for recovering object details through skip connection, which was originally developed for medical image segmentation (Zhou et al., 2020). It has been widely used in land cover classification studies using remote sensing, and its performance has been also recognized (Rakhlin et al., 2018; Giang et al., 2020; Ulmas and Liiv, 2020). Compared to the Pyramid Scene Parsing Network (PSPNet) published in 2017 and the Deeplabv3+ algorithm published in 2018, the U-Net algorithm is a relatively old algorithm (Roonberger et al., 2015; Zhao et al., 2017; Chen et al., 2018). However, it shows similar or better performance in land cover classification and segmentation compared to relatively recently published algorithms including the DeeplabV3+ algorithm (Zhang et al., 2021; Chiu et al., 2020). Therefore, based on the high reliability and stable performance of the algorithm as well as various applications in many previous studies, the classification results were derived and analyzed by applying the U-Net algorithm in this study. (Fig. 5).

Table 4. Dataset distribution and hyperparameters for training U-Net algorithms

Fig. 6. Algorithm learning result graph. (a) is the Gyeongsang-do learning dataset validation result, and (b) is the Jeollado learning dataset validation result.

3. Results

1) Training environment and result

Of the 21,000 data usable for TD on Gyeongsangdo and Jeolla-do, 19,000 data were used for datasets of the training and validation process. The remaining 2,000 data were used to organize datasets for prediction. Datasets for prediction were not involved in training and validation and configured identically for both regions. As for the training environment, trainings were conducted for Gyeongsang-do and Jeolla-do, respectively. For the objective evaluation of the dataset, of hyper parameters, the number of Epochs was set at a minimum of 100 to a maximum of 300, and the batch size was set at 48. Three Tesla V100 GPUs and finely annotated data were used for training (Table 4).

The verification result of the training process is shown in Fig. 6 below; (a) shows the pixel accuracy and loss values of the Gyeongsang-do dataset, and (b) shows those of the Jeolla-do dataset. In the case of learning verification accuracy, the Gyeongsang-do region accuracy is about 86% and the Jeolla-do region accuracy is about 88%. Both regions showed accuracy of 85% or higher, and the Jeolla region yielded an accuracy of about 2% higher (Fig. 6).

2) Comparison result

After training on the Gyeongsang-do and Jeolla-do datasets, the calculated weights were equally applied to the TAs of Gyeongsang-do and Jeolla-do to calculate the accuracy. First, when the weights of Gyeongsangdo were applied to areas other than Gyeongsang-do training areas, and to Jeolla-do, respectively, the accuracy of the U-Net algorithm is shown in Table 5 below.

Table 5. Accuracyby class after applyingGyeongsang-do learning weights to Gyeongsang-do and Jeollado test areas

When the weights of Gyeongsang-do were applied to the Gyeongsang-do TAs, the overall accuracy was about 85.45%. This indicates that the accuracy was about 13.6% higher than that derived when the weights of Gyeongsang-do were cross validation to the Jeollado test area. Comparing the accuracy of each object item, it was found that most items including parking lots, roads, rice fields, dry fields, and bare lands were calculated with high accuracy in the Gyeongsang-do TAs. However, for buildings and forests, the Jeolla-do TAs where the weights were cross validation showed – about 5 8% higher accuracy. The highest difference of about 42% in accuracy was shown for bare lands, and a difference of more than 20% for roads.

Second, when the weights of Jeolla-do were applied to areas other than the Jeolla-do training areas, and to Gyeongsang-do, respectively, the accuracy of the U- Net algorithm are shown in Table 6 below.

When the training weights of Jeolla-do were applied to the Jeolla-do TAs, high classification accuracy was shown in the areas, but the accuracy in the cross validation Gyeongsang-do areas were analyzed to be relatively low. About 15.4% higher accuracy was derived from the Jeolla-do TAs, which is the same area as the training areas, compared to the Gyeongsang-do TAs. That is, as for the accuracy of each object item, buildings, parking lots, rice fields, and forests showed higher accuracy for Jeolla-do, and roads and bare lands for Gyeongsang-do. Particularly, for forests, there was a difference of about 31% in accuracy, and unlike the case where the weights of Gyeongsang-do were applied, there was a difference of about 25% even for buildings.

Table 6. AccuracybyclassafterapplyingJeolla-dolearning weights to Gyeongsang-do and Jeolla-do test areas

Table 7. Accuracy and accuracy difference for each class applying two weights for each test area

Fig. 7. Image of Gyeongsang-do test area classification result. (a) is the original aerial image image, (b) is the classification result to which Gyeongsang-do weight is applied, and (c) is the classification result to which Jeolla-do weight is applied.

Third, when weights were cross validation, classification accuracy was organized and compared for each object item. Table 7 below shows the difference in the accuracy and results of applying different weights to the same areas.

Comparing the difference in accuracy with other TAs, it was found that applying the weights of TD to the same TAs generally showed a smaller difference of about 10% in accuracy. Basically, object items which occupied a large area in the training datasets, were found to show an accuracy of 50% or more. For classification accuracy of buildings among object items in both regions, although the building item of the Gyeongsangdo dataset occupied a larger area that the Jeolla-do dataset, its average accuracy was calculated to be higher in the Jeolla-do TAs. For forests, a larger area of the item was learned from the Jeolla-do TD than from the Gyeongsang-do training dataset, but it showed higher test accuracy in the Gyeongsang-do TAs.

Fig. 8. Image of the Jeolla-do test area classification result. (a) is the original aerial image image, (b) is the classification result to which Gyeongsang-do weight is applied, and (c) is the classification result to which Jeolla-do weight is applied.

Fig. 7 below shows the images of classification results in the Gyeongsang-do TAs; (a) is the raw image, (b) is the image to which the weights of the Gyeongsangdo TD are applied, (c) is the image to which the weights of the Jeolla-do TD are applied. Based on the images, it was found that for forests, the image to which the weights of Gyeongsang-do TD were applied was more precisely classified rather than that to which the weights of Jeolla-do TD were applied. Whereas for dry fields, it was found that weights of Jeolla-do TD displayed higher classification performance.

Fig. 8 below shows the images of classification results in the Jeolla-do TAs; (a) is the original aerial image, (b) is the image to which the weights of Gyeongsangdo TD were applied, (c) is the image to which the weights of Jeolla-go TD were applied. Even in the Gyeongsang-do TAs, like the Jeolla-do TAs, it was found that the weights of Gyeongsang-do TD displayed higher classification performance for forests, and that the weights of Jeolla-do TD showed higher performance for dry fields.

4. Discussion and Conclusions

The purpose of this study is to evaluate the classification accuracy and performance when a previously constructed dataset for the application of AI algorithms is applied to other regions. For the analysis, datasets constructed for Gyeongsang-do and Jeolla-do were used, and weights were calculated by training with the same hyperparameters and algorithm. The calculated weight was applied to the Gyeongsang-do and Jeolla-do TAs to calculate the accuracy. Comparing the case where the weights of the same region were applied and the case where the weights of other regions were applied, it was found that there was a difference – of about 13 15% in accuracy. This result showed the possibility of application to other regions, although there were somewhat differences in the performance of the existing datasets depending on the characteristics of the region where the data was built. In addition, with regard to forests, although they had a larger area in the Jeolla-do training dataset, the classification accuracy was higher when the weights of Gyeongsang-do were applied. This indicates that the regional characteristics of TD are reflected in the accuracy for some items. Based on this, it is predicted that, if large-scale AI training datasets are built focused on a small area, the land cover classification performance of such datasets may deteriorate when applied to other areas such as cities and rural areas (Wang et al., 2022). However, in the future, it will be likely necessary to analyze the possibility that factors such as the shooting timing and the effect of correction of high-resolution images taken locally have affected the performance of the dataset. When constructing a large-scale land cover dataset for AI training, it is expected that a high-quality dataset can be constructed by considering the characteristics of various areas based on the findings of this study. The approach of this study can be expanded to other areas with diverse land cover types such as agricultural land, forest, urban areas and buildings using satellite images in an effective manner. Further it should be conducted the feasibility test of training data to other areas with difficult access including North Korea.