Weather Recognition Based on 3C-CNN

Abstract

Human activities are often affected by weather conditions. Automatic weather recognition is meaningful to traffic alerting, driving assistance, and intelligent traffic. With the boost of deep learning and AI, deep convolutional neural networks (CNN) are utilized to identify weather situations. In this paper, a three-channel convolutional neural network (3C-CNN) model is proposed on the basis of ResNet50.The model extracts global weather features from the whole image through the ResNet50 branch, and extracts the sky and ground features from the top and bottom regions by two CNN5 branches. Then the global features and the local features are merged by the Concat function. Finally, the weather image is classified by Softmax classifier and the identification result is output. In addition, a medium-scale dataset containing 6,185 outdoor weather images named WeatherDataset-6 is established. 3C-CNN is used to train and test both on the Two-class Weather Images and WeatherDataset-6. The experimental results show that 3C-CNN achieves best on both datasets, with the average recognition accuracy up to 94.35% and 95.81% respectively, which is superior to other classic convolutional neural networks such as AlexNet, VGG16, and ResNet50. It is prospected that our method can also work well for images taken at night with further improvement.

1. Introduction

Human activities are often affected by weather situations [1]. Traditional methods of identifying weather phenomena use weather sensors, which are often expensive and require professional maintenance [2]. Another option is to use computer vision technology to identify weather situations from images [3]. Weather recognition from images is a comparatively novel subject in computer vision that has great social influence [4]-[5]. Weather images carry most of weather information for weather recognition, and they also have the characteristics of large number and low cost. However, identifying weather from weather images is not easy. The challenge is that the global and local characteristics are disabled in weather recognition, which is crucial in recognition due to the particularity of weather condition. [6]. Different from the theory of scene recognition and semantic segmentation, weather recognition requires understanding more complex phenomena such as illumination and reflection in the scene of weather images, and needs to analyze the changes of scene under different weather situations, rather than the structure of scene itself [7]-[10]. Moreover, there is almost no research on intelligent traffic management based on weather situations and other information. Therefore, weather recognition has broad research value and application prospect, especially in traffic scenes [11].

In modern transportation, bad weather will have a significant influence on the traffic situations. Heavy weather including rain, fog and snow will make the traffic road slippery and reduce the visibility, resulting in traffic congestion and severe traffic accidents, which has potential danger. Through recognizing weather situations and analyzing traffic information in real time, the traffic signals can be controlled according to weather changes, such as properly increasing the green light time and yellow light time in bad weather situations. It can improve travel efficiency and prevent severe traffic accidents effectually. Some driving assistance systems can promote security with weather recognition by limiting driving speed, turning on wipers automatically and maintaining safe distance in bad weather situations [12-13]. In conclusion, the automatic identification of weather situations is very meaningful to traffic alerting, driving assistance, and intelligent traffic management system [14]-[16]. Therefore, it is of great significance to identify weather situations from weather images.

In view of above research, a new three-channel convolutional neural network (3C-CNN) model is proposed to identify weather situations. The rest of the paper is framed as follows: In section 2, we retrospect the previous work on weather recognition. Section 3 introduces the two weather image datasets and describes the architecture of the three-channel convolutional neural network (3C-CNN) for weather identification. Section 4 evaluates the experimental results of 3C-CNN on two datasets and discusses the performance compared with other methods. Finally, the conclusions are summarized in section 5.

2. Related Work

The traditional method of weather identification is to deploy the weather sensors, which can realize the induction to rainfall and snowfall. By combining the data from multiple sensors and images, weather recognition results are obtained. If one of the sensors does not work, the accuracy of the recognition results will be greatly affected [17]. The advantage of our method lies in its simplicity, facility and maturity of technology. However, the installation and maintenance of sensors are very expensive, and it is difficult to identify weather situations in any space and time due to the complexity of weather phenomena. Therefore, it is obviously not suitable for the occasion of public transportation that require large-scale deployment.

Benefit from the development of intelligent transportation system in the last decade, more cameras have been fixed along the roads, so weather identification have gradually turned to image processing and computer vision. Li et al. employed support vector machine (SVM) and decision tree to identify weather situations after extracting visual characteristics such as saturation, noise, contrast, and power spectrum slope. The method was evaluated on the Wild Dataset including hundreds of outdoor images, but the recognition performance of rainy days is too poor to satisfy the requirements of practical application [18]-[20]. Moreover, the Wild Dataset lacks authority and generalization because the images were taken in fixed scene and the size is too small. Lu et al. collected a dataset with different backgrounds named Two-class Weather Images (TWI) which contains 10 thousand weather images and divided into sunny and cloudy days only. A collaborative learning approach was proposed to divide the weather images into sunny and cloudy days [21]. In conclusion, these machine learning-based methods are very cumbersome and deficient in universality and generalization due to the complex engineering features.

With the boost of artificial intelligence and deep learning, classic convolutional neural networks (CNN) such as AlexNet, VGGNet, GoogLeNet and ResNet have achieved excellent results in the area of machine vision, e.g. object tracking, tag identification, image recognition, image retrieval, natural language processing, and semantic segmentation [22]-[31]. CNN can extract abstract, deep and rich semantic information from outdoor images, they perform better than traditional methods in many ways. Elhoseiny et al. fine-tuned the pre-trained AlexNet on the Two-class Weather Images (TWI) [21], and achieved good results, with an average recognition accuracy of 82.2% [32]. Lu et al. combined the manually extracted weather features with those extracted by CNN to further improve the performance in accuracy, thus make the recognition accuracy reach 91.4% [33]. Zhu et al. established a large-scale dataset including various extreme weather images and fine-tuned the GoogLeNet pre-trained on ILSVRC-2012 dataset. Later, they simplified the classic GoogLeNet model and train it through a novel double-fine-tuning method. The model size is reduced to one third of the original GoogLeNet model, and the accuracy is increased from 94.5% to 95.46% [34]-[36]. In summary, although these methods based on deep learning are totally better than traditional methods, they can only run efficiently on GPU with high configuration, and need big data supporting, which makes weather identification costly.

The main contributions of this paper are as below. Firstly, deep convolutional neural networks (CNN) are adopted to identify weather situations from outdoor weather images. Moreover, we established a medium-scale dataset including 6185 outdoor weather images called WeatherDataset-6. Finally, a new three-channel convolutional neural network (3C-CNN) model is proposed on the basic of ResNet50 for the task of weather recognition. 3C-CNN is used to train and test both on the Two-class Weather Images (TWI) and WeatherDataset-6, and desirable recognition results are obtained. Compared with previous work, WeatherDataset-6 contains more weather categories and 3C-CNN achieves good performance in terms of weather recognition.

3. Materials and Methods

3.1 Dataset

The availability of sufficient expertly labeled datasets is the key point in allowing CNNS to work efficiently. As a new subject in machine vision, the application of weather recognition is relatively less. Therefore, the public datasets for weather recognition are inadequate in size and quantity. The large-scale dataset ILSVRC-2012 contains 1.2 million images covering 1000 categories, whereas the small-scale dataset named Wild Dataset above consists of only a few hundred outdoor images taken in fixed scene. The medium-scale datasets for weather recognition in between are as follows. Two-class Weather Images (TWI) [21] is provided by Lu et al., while WeatherDataset-6 is established by us.

3.1.1 Two-class Weather Images (TWI)

Lu et al. collected the Two-class Weather Images Images (TWI) from Sun Dataset, Labelme Dataset, and Flickr [37]-[38]. After calculating the color histogram distance of all image pairs, identical or highly similar images were deleted. Finally, the images were labeled by two annotators. The Two-class Weather Images (TWI) contains 5000 sunny and 5000 cloudy images, of which 8000 are used for training and 2000 are used for testing, as shown in Table 1.

Table 1. Details of Two-class Weather Images (TWI)

3.1.2 WeatherDataset-6

Considering the lack of available public weather datasets, we collected a medium-scale dataset including 6,185 outdoor weather images from the Internet and divided them into six categories: cloudy days, foggy days, rainy days, sandy days, snowy days and sunny days. Details of this dataset are presented in Table 2, and the images of this dataset are randomly divided into train set and test set by the ratio of 8:2 roughly. The images are from aerial photography, video recording, traffic accidents, news reports, automobile data recorder, etc. Various weather images in WeatherDataset-6 are presented in Fig. 1. Our dataset has more generalization and universality because the size of which is relatively large and the images are taken from different positions that contain a variety of complex scenes.

Table 2. Component of WeatherDataset-6

Fig. 1. Six categories of sample weather images from WeatherDataset-6

3.1.3 Data Augmentation

In order to suppress over fitting in the process of model learning, we take a series of measures to enhance the training images in WeatherDataset-6. For example, we randomly crop the original images from 256×256 to 224×224. Then, we rotate, flip, translate, cut and enlarge the image block. Fig. 2 presents a series of different images after data augmentation. Specifically, we adopted ImageDataGenerator function offered in the Keras API and defined parameters such as shear range, width offset and rotation range to generate images after random transformation in each epoch [39]. The model will not receive any two same images in the learning process through adoption of this method. This is beneficial to restrain over fitting and increase the robustness and generalization ability of the model. In order to demonstrate the significance of data augmentation for weather recognition, we conducted a series of comparative experiments with or without data augmentation as shown in section 4.2.

Fig. 2. Results of data augmentation for training images

3.2 Architecture of 3C-CNN

To carry out image classification and object detection in the 2015 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [36], He et al. adopted the deep residual network (ResNet) and won the championship [23]. ResNet can effectually solve the problem of gradient vanishing in the learning process of deep convolution neural network, which makes it feasible to train deeper network model. Without adding additional parameters, the deeper residual structure can improve the classification accuracy and accelerate the convergence speed. Considering the excellent performance of ResNet, ResNet50 can be used for automatic identification of weather situations. ResNet50 consists of a convolution layer, a max pooling layer and a series of residual modules. Two basic stacking modes of residual modules are presented in Fig. 3. One mode is identity block (IB), whose input and output have the same shape, so they can be directly appended in series. The other is conv block (CB), whose input and output have different shape, so they cannot be appended in series, and the 1×1 convolution layer is adopted to unify the dimension of feature vector. ResNet50 extracted deep-level weather features through the continuous stacking of 4 groups of Conv Block and 12 groups of Identity Block. Finally, the weather recognition results were output through Average pooling layer with size of 7 × 7 and Softmax classifiers.

Fig. 3. Two basic stacking modes of residual modules

The background information in the weather image is very complex. It is roughly divided into sky features and ground features. Sometimes different areas in the same weather picture show different weather situations. Here we divide the input image into the top and bottom regions, and extract the sky features and ground features from these two areas respectively. Inspired by the structure of Diverse Region-Based CNN (DR-CNN) proposed by Zhang et al. [40], a three-channel convolutional neural network (3C-CNN) model is proposed based on ResNet50 for automatic recognition of weather situations, the architecture of which is shown in Fig. 4. This model contains three CNN branches, each of which processes input images from different regions. The image blocks of the top and bottom regions are input into a CNN5 branch composed of 5 layers of convolution layer and 1 layer of Max pooling layer to extract sky features and ground features respectively, while the whole image is input into the branch of residual network ResNet50 to extract global features. The local features and global features extracted by the three branches are merged by Concat function, and then the key weather features are down-sampled through the Average pooling layer. Finally, the weather recognition results are output through Softmax classifier. The composition of 3C-CNN model is shown in Table 3.

Fig. 4. Architecture of 3C-CNN

Table 3. Structure composition of 3C-CNN

4. Experimental Results and Discussion

In this paper, a 3C-CNN model is built based on Keras framework, and this model is used to train and test both on the Two-class Weather Images (TWI) and WeatherDataset-6. After data augmentation for each image in train set, the images and labels are fed in model by the ImageDataGenerator function for training. The weight of our model is updated and optimized in each epoch through the Stochastic Gradient Descent (SGD) optimizer, with the momentum setting to 0.9 and the weight decay to 0.0002. Moreover, the initial learning rate is set to 0.001 and can be adjusted adaptively through the callback function of ReduceLROnPlateau. Finally, the model is evaluated on test set after training. The experiments are operated on the server with NVIDIA GeForce GTX 2080 Ti GPU.

4.1 Experimental Results

In order to demonstrate the significance of the proposed 3C-CNN model, we implement this algorithm based on the Keras framework. Firstly, the training images were fed into the 3C-CNN model for training. Moreover, three CNN branches were trained separately for comparison, and all models after training were saved. Then, the trained models were evaluated on test set and the recognition results were output. Finally, the average accuracy of 3C-CNN model and three CNN branch models were calculated respectively both on the Two-class Weather Images (TWI) and WeatherDataset-6 according to the recognition results of test images, as presented in Table 4. The average accuracy of the ResNet50 branch model after training the whole images of Two-class Weather Images (TWI) and WeatherDataset-6 was 92.30% and 90.32% respectively. With the addition of two CNN branches, the average accuracy of 3C-CNN was increased to 94.35% and 95.81% respectively. The experimental results indicate that 3C-CNN model performs well in weather recognition tasks both on the Two-class Weather Images (TWI) and WeatherDataset-6 and it is indeed superior to the single-branch CNN model.

Table 4. Recognition results of different channels in 3C-CNN on two datasets

4.2 Comparison with or without Data Augmentation

To demonstrate the effectiveness of data augmentation on weather recognition tasks, the training set images of Two-class Weather Images (TWI) and WeatherDataset-6 with or without data augmentation were input into the 3C-CNN model for training, and the test set images were used for evaluation. The experimental results are presented in Table 5. The training set images of Two-class Weather Images (TWI) and WeatherDataset-6 without data augmentation were directly fed into the 3C-CNN model for training and the recognition accuracy was 88.90% and 84.62%. After data augmentation, the recognition accuracy of 3C-CNN was increased to 94.35% and 95.81% respectively. The accuracy curves of the validation set in Two-class Weather Images (TWI) and WeatherDataset-6 are shown in Fig. 5(a) and Fig. 5(b) respectively, in which the dotted curve indicates the recognition accuracy without data augmentation. As can be seen in the graph, the precision curve began to converge and the model began to fit after about 5 training epochs. In comparison, data augmentation for train images can effectively suppress over fitting, drastically improve the recognition accuracy, and reinforce the robustness and generalization of those models. Moreover, WeatherDataset-6 established by ourselves is smaller than Two-class Weather Images (TWI) in scale, while data augmentation improves the accuracy on our dataset even more. A conclusion can be drawn that data augmentation is more effective for image recognition on smaller datasets.

Table 5. Recognition results with or without data augmentation on two datasets

Fig. 5. Comparison of validation curves with or without data augmentation on two datasets

(a) Accuracy curves on TWI

(b) Accuracy curves on WeatherDataset-6

4.3 Comparison with Other Methods

In order to evaluate the performance of 3C-CNN model for weather identification, the average recognition accuracy of 3C-CNN both on the Two-class Weather Images (TWI) and WeatherDataset-6 is compared with other classic CNN such as AlexNet [21], VGG16 [22], and ResNet50 [23]. The accuracy curves of different methods on two datasets are presented in Fig. 6, and different colored curves indicate various methods. According to the average accuracy of different methods, the histogram comparison is drawn in Fig. 7. Among them, 3C-CNN model performs best in weather recognition tasks both on the Two-class Weather Images (TWI) and WeatherDataset-6, with the average recognition accuracy up to 94.35% and 95.81% respectively, followed by ResNet50 [23], AlexNet [21] and VGG16 [22] successively. The confusion matrices of various models on the Two-class Weather Images (TWI) and WeatherDataset-6 are presented in Fig. 8 and Fig. 9 respectively. The values on the diagonal of the confusion matrix indicate the recognition accuracy of each category. As can be seen from Fig. 8, the 2,000 test images of TWI are divided into cloudy and sunny weather, and the recognition accuracy of 3C-CNN model for cloudy days is 93.10%, for sunny days is 95.60%. As can be seen from Fig. 9, the 1385 test images of WeatherDataset-6 are divided into cloudy, foggy, rainy, sandy, snowy and sunny weather. The recognition accuracy of sandy weather was always the highest among these six weather situations. This was because the whole background features in most sandy weather images were yellow and easy to distinguish and only a few sandy weather images in light colors were misidentified as foggy weather. The recognition accuracy of cloudy, rainy and snowy weather was relatively low, because the sky features of these weather images were basically the same, and they were distinguished only by the smoothness of the ground features. Therefore, with the addition of two CNN branches to extract sky and ground features separately, the recognition accuracy of these weather images can be greatly improved.

Fig. 6. Comparison of validation curves of different methods on two datasets

(a) Accuracy curves on TWI

(b) Accuracy curves on WeatherDataset-6

Fig. 7. Histogram comparison of different methods on two datasets

(b) Average accuracy on WeatherDataset-6

Fig. 8. Confusion matrixes of different methods on TWI

(a) AlexNet

(b) VGG16

(c) ResNet50

(d) 3C-CNN​​​​​​​

Fig. 9. Confusion matrixes of different methods on WeatherDataset-6

(a) AlexNet

(b) VGG16

(c) ResNet50

(d) 3C-CNN​​​​​​​

5. Conclusion

In this paper, a novel three-channel convolutional neural network (3C-CNN) model based on ResNet50 is proposed for weather recognition, and a medium-scale dataset including 6,185 outdoor weather images called WeatherDataset-6 is established. A great number of experiments demonstrate that 3C-CNN model achieves a great result both on the Two-class Weather Images (TWI) and WeatherDataset-6, with the average recognition accuracy up to 94.35% and 95.81% respectively. Compared with previous work, WeatherDataset-6 contains more weather categories and 3C-CNN achieves better performance in terms of weather recognition than other classic network models such as ResNet50. At present, our model can only be utilized to identify the weather situations in the daytime. It is prospected that our method can also work well for images taken at night with further improvement.