Blind Quality Metric via Measurement of Contrast, Texture, and Colour in Night-Time Scenario

Abstract

Night-time image quality evaluation is an urgent requirement in visual inspection. The lighting environment of night-time results in low brightness, low contrast, loss of detailed information, and colour dissonance of image, which remains a daunting task of delicately evaluating the image quality at night. A new blind quality assessment metric is presented for realistic night-time scenario through a comprehensive consideration of contrast, texture, and colour in this article. To be specific, image blocks' color-gray-difference (CGD) histogram that represents contrast features is computed at first. Next, texture features that are measured by the mean subtracted contrast normalized (MSCN)-weighted local binary pattern (LBP) histogram are calculated. Then statistical features in Lαβ colour space are detected. Finally, the quality prediction model is conducted by the support vector regression (SVR) based on extracted contrast, texture, and colour features. Experiments conducted on NNID, CCRIQ, LIVE-CH, and CID2013 databases indicate that the proposed metric is superior to the compared BIQA metrics.

1. Introduction

Visual inspection is good at measuring and judging image by means of the input texture, brightness and colour of the images, and attracts tremendous attention in industry, medicine, transportation, education, etc. fields at present [1]. For example, in autonomous driving [2], it is necessary to use visual inspection technology to automatically detect lane lines, road markings, and obstacles. Unfortunately, lots of images may be not satisfactory due to the distortion introduced during image capture, processing, and transmission, etc.. Especially, at night-time, the poor illumination condition results in a low-quality image with low brightness, low contrast, inconspicuous texture feature, invisible details and inharmonious colour, which may cause a visual inspection error. In this situation, the quality assessment of night-time image can be employed to identify the low-quality image before visual inspection process [3]. Therefore, identifying and quantifying the quality of night-time image is a crucial requirement, which is helpful for the development of visual inspection technology.

The topic of image quality assessment (IQA) has witnessed a tremendous attention in visual inspection and image processing in recent years. IQA metrics include two classes, subjective IQA that evaluates the image by human beings and objective IQA that predict image quality by image characteristics [4,5]. More specifically, the objective IQA metrics are reclassified in to three types according to the reference information, full-reference (FR) metric that needs all the information of the original image as a reference [6,7], reduced-reference (RR) metric that requires partial knowledge about the original image [8,9] and no-reference (NR) metric that does not refer to the original image [10,11]. In practice, it is infeasible to provide a reference image, and thus the NR IQA, which go by the name of blind image quality assessment (BIQA) method attracts attentions of many researches.

However, pre-existing BIQA metrics are jointly designed with the simulated-distorted image databases [12, 13, 14] and take no account of the night-time scenario. Consequently, when measure the night-time images, these BIQA metrics generate apparent performance loss due to neglecting the case of low brightness, loss of contrast, texture feature inconspicuous, invisible details, etc.. So far, only a handful of works have been done for quantitatively judging the images captured in night-time scenario. [15] first established a night-time image database (NNID) that contained more than 2000 realistic images, together with the MOS values, and developed a BIQA metric, called BNBT, to evaluate the image that captured at night. BNBT first extracted brightness and texture features of the image over two scales. At each scale, 9 features were extracted. After features extracted, SVR was used to map the extracted brightness and texture features to the MOS of the tested image in Night-Time Scenario. Although the BNBT has achieved notable performance, there still is a gap for performance improvement. First, BNBT only extracted two types of night-time time features, including brightness and texture, which can’t fully represent the characteristics of night-time image because realistic night-time image often contains mixtures of multiple distortions. Second, BNBT, only extracted global features of night-time image, which didn’t consider local features of night-time image. However, extracted local and global features are demonstrated to well characterize the perceptual effects of some critical image attributes which are typically involved in the human perception of image. Third, the performance evaluations of BNBT need to be further improved. For example, the SRCC value tested on the entire NNID database in BNBT is 0.8697, but the SRCC value of excellent IQA metric is generally above 0.95. This inspires us to extract features from more aspects to improve the performance of IQA metric for night-time image.

In this paper, we analyze the parameters that are correlated with the quality of night-time images based on the BIQA model. The analyses signifies that the quality of night-time image highly depends on the contrast that measures visual distortion, texture that measures the definition and colour that measures colour dissonance, which is different from the parameters in BNBT. And thus, an enhanced BIQA algorithm is presented to achieve a performance improvement in contrast to the conventional BNBT by considering the contrast, texture, and colour features simultaneously. In the proposed algorithm, the contrast features are calculated by the image blocks’ color-gray-difference (CGD) histogram, texture features are measured by the MSCN-weighted LBP histogram and the colour features are obtained from Lαβ colour space. Whereafter, the night-time image quality prediction model is conducted by the SVR based on extracted contrast, texture, and colour features.

The main contributions of this paper are summarized as follows.

• Analyze the characteristics that are correlated with the quality of the night-time images.

• Design a new BIQA metric for realistic night-times scenario, which considers multiple features of night-time images including contrast, texture, and colour.

• The proposed algorithm shows a considerable performance improvement compared to the conventional BNBT.

The structure of this paper is organized as follows. The related NR-IQA methods are briefly reviewed in Section II. Section III presents the proposed night-time image quality assessment method. In Section IV, the simulation results of experiment using the proposed method on four benchmark IQA databases compared with existing NR-IQA methods are presented and analyzed. Finally, Section V summarizes the whole paper.

2. Related work

In this section, the pre-existing related BIQA metrics are reviewed. BIQA metric includes general BIQA that copes with the situation of unknowing the type of distortion and specific BIQA that processes the images with specific distortion.

2.1 General BIQA metrics

General BIQA metrics can evaluate the image with an assumption of unknowing the distortion type. BLIINDS-II [16] was a BIQA metric using the natural scene statistics (NSS) model and extracting image characters from the discrete cosine transform (DTC) domain, which involved 24 features over three scales. At each scale, 8 features were extracted. After given the extracted features, the image quality scores were predicted relied on a simple Bayesian inference model. BRISQUE [17] was also a BIQA approach based on NSS model. But different from BLIINDS-II, this method no longer used DTC coefficients, but quantified the image in the spatial domain, which extracted mean subtracted contrast normalized (MSCN) coefficients fitted as asymmetric generalized Gaussian distribution (AGGD) to analyze the image distortion. Unlike NSS-based BIQA metric, GM-LOG [18] first attempted to combine the gradient magnitude (GM) and Laplacian of Gaussian (LOG) characters to assess the natural images. MSGF [19] extracted the image features from two domains including the spatial-frequency and spatial, then employed the piecewise regression to further improve the accuracy of quality assessment. For another branch of general BIQA, the image was evaluated based on the free energy model that was described as the summation of a relative entropy and mean likelihood of image data, and the representative schemes included NEFQM (firstly used free-energy model) that considered 3 types of image characters [5], and NFERM that considered a total of 23 image characters [20]. The above methods require to use traditional machine learning algorithm SVR or random forest (RF) model to map the extracted visual features to the subjective rating scores of the tested image. Due to the powerful feature representation ability of deep learning, many BIQA models based on deep learning were developed recently [21, 22, 23, 24, 25]. Different from the traditional BIQA model of manual feature extraction, the model based on deep learning can learn the mapping relationship between image and image quality based on end-to-end Network. [22] proposed an end-to-end depth convolution neural network model (MEON) based on multi-task optimization for BIQA. In MEON, the first stage trained the distortion type identification sub-network, and the second stage trained the parameters of the quality prediction task based on the pre-trained shared convolution layer parameters and the parameters of the first sub-network. Zhang et.al developed DB-CNN based BIQA metric [23], which constituted two streams of deep convolutional neural networks for synthetic distortions and authentic distortion respectively. Unlike the aforementioned methods, NIQE [26] and IL-NIQE [27] were completely blind IQA, which didn’t require the subjective score of the original image. NIQE proposed in [26] constructed the multivariate Gaussian (MVG) model based on the portion of image patches and evaluated the image by the distance between the pre-established MVG model and the characteristic model of the tested image, and thus the subjective scores was unnecessary. Integrated Local NIQE (IL-NIQE) model presented in [27] constructed MVG model based on the statistical characteristics of structure, multiscale direction, frequency statistical feature, colour statistical feature of each image patch, and used the Bhattacharyya distance like NIQE to measure the quality score.

2.2 Specific BIQA metrics

Specific BIQA approaches cope with the images that suffer given type of distortion, in other words, prior knowledge of the distortions type should be required at processor, such as noise [28] or blurring distortion [29, 30, 31, 32], JPEG compression distortion [33,34], contrast distortion [35, 36, 37, 38], etc.. To be specific, [31] and [32] handled the blur distortion by means of sparse representation and discrete Tchebichef Kernel, respectively. The authors in [33] employed the gradient domain discrete Fourier transform to analyze the periodic peaks of the signal so that the effect of JPEG compression distortion to quality assessment was reduced. The NR-CDIQA [39] was developed for the situation of contrast distortion, which constructed the moment and entropy characters based on NSS. NIQMC presented in [40] signified that, the local character that the entropy of salient regions was calculated based on semi-parametric model and global character that the symmetric Kullbace- Leibler divergence was calculated between the distributions of image histogram and the uniformly distributed histogram were highly dependent with the accuracy of quality assessment in the situation of contrast distortion. Moreover, HEFCS [38] employed the local image histograms that delicately described the luminance and contrast of image, and was good at addressing the contrast distortion.

3. Proposed night-time image quality metric

Because night-time images are usually acquired in poor and varying lighting and illumination condition, night-time images typically contain multiple distortions, specifically, low brightness, low contrast, loss of detailed information, and colour dissonance. Through the analysis of the NNID database [15] and our knowledge of the human visual system (HVS), we find that contrast, texture and colour are determining factors in the quality assessment of the night-time images. Contrast variation has great influence on image evaluation, especially for the limitation of acquisition and poor lighting of night-time images. Moreover, texture information containing the details of the visual contents dominate the quality assessment of images captured at night. Given the fact that colour information reflects the plentiful degree of color scenes and it has been experimented that in the HVS, in the first 20s, the colour account for about 80% in perception the image quality, thus it is meaningful to extract the color features for the night-time images. These observations inspire us to put forward a BIQA algorithm, which combine the contrast, texture and colour features to measure night-time images quality. Fig. 1 illustrates the basic framework of the proposed metric for night-time images.

Fig. 1. The framework of the proposed method

The proposed method works as follows. In order to extract contrast, texture and colour features, we process the night-time image by three related modules simultaneously. For the contrast feature extraction, we randomly select P blocks of the whole image and calculate the CGD histograms of each block to obtain histogram eigen-features by performing the principle component analysis (PCA). For texture feature extraction, we calculate the MSCN coefficient of the image after down-sampling to obtain the weighted LBP histogram. For colour feature extraction, we convert the input RGB color images into the Lαβ space and calculate the statistical distribution of Green-Blue (GB) channel and Blue-Yellow (BY) channel coefficient, which is modeled as AGGD. Finally, SVR is applied to compute quality scores of night-time image by mapping the extracted quality features to the MOS measured by human observers. The details are presented in the following subsections.

3.1 Contrast Feature Extraction

Compared with light-time images, night-time images usually suffered with low dynamic range of brightness, which may be caused by poor and varying lighting and illumination condition. Fig. 2 presents five natural night-time images with size 512×512 from NNID database [15]. From Fig. 2, we can observe that the MOS of Fig. 2 (a) which has high contrast is obviously higher than that of (e) which has low contrast. Intuitively, night-time images captured in low-illumination environment are often suffered with low contrast, in other words, contrast variation is a significant element that determines the quality assessment of the night-time images.

Fig. 2. Five images sampled NNID database. The MOS is displayed at the bottom

As indicated by the study [38], the image histograms can reflect variations of luminance and contrast information about the image appearance, and hence provide a great deal of information about the overall appearance of the image. Therefore, we first employ image histograms to evaluate contrast variation.

We denote the color night-time image as Iand its gray scale image as I _Gray . The gray histogram is H _Gray, which is to count the occurrence frequency of all pixels in the image according to the gray value. The gray histogram with gray value i is formulated as:

\(H_{\text {Gray }}(i)=\sum_{(x, y) \in X} \delta\left(i-I_{\text {Gray }}(x, y)\right)\)       (1)

where X denoted the set of image pixels, δ is a binary function, and \(\delta_{i j}=\left\{\begin{array}{l} 1, \text { if } i=j \\ 0, \text { if } i \neq j \end{array}\right.\).

The night-time image I is a superposition of the red, green, and blue component images Let I_red, I_Green and I_Blue represent the red, green, and blue component image. H_Red, H_Green and H_Blue are the corresponding color histogram of I_red, I_Green and I_Blue. H_Red, H_Green and H_Blue are given by:

\(H_{\text {Red }}(i)=\sum_{(x, y) \in X} \delta\left(i-I_{\text {Red }}(x, y)\right)\)       (2)

\(H_{\text {Green }}(i)=\sum_{(x, y) \in X} \delta\left(i-I_{\text {Green }}(x, y)\right)\)       (3)

\(H_{\text {Blue }}(i)=\sum_{(x, y) \in X} \delta\left(i-I_{\text {Blue }}(x, y)\right)\)       (4)

H_Red, H_Green and H_Blue indicate the distribution of red, green, and blue components of Image I , respectively. So the total RGB histogram of image I is given by:

\(H_{R G B}(i)=H_{R e d}(i)+H_{G r e e n}(i)+H_{B \text { lue }}(i)\)        (5)

When H_Gary is subtracted from H_RGB, Color Gray Difference (CGD) histogram H_CGD is remained. That is,

\(H_{C G D}(i)=H_{R G B}(i)-H_{G r a y}(i)\)      (6)

Contrast refers to the measurement of the different levels of brightness between brightest white and darkest black in the light and dark areas of an image. The larger the difference range, the greater the contrast, and the smaller the difference range, the smaller the contrast. Histogram provides the information about the contrast of the image. It is known that the shape of histogram can indicate the global characteristics of the image: dark, bright, low contrast and high contrast. Narrow histograms reflect less contrast and may appear dull or washed out gray, whereas broad histograms reflect a scene with significant contrast. In addition, high contrast images usually have pure and intense colors, and thus the difference between the gray and RGB histogram (i.e. CGD) is high.

Fig. 3 shows the comparisons of CGD histogram of images with various MOS values. (a)-(e) are five original images from the NNID database [15] and (f)-(j) are the corresponding normalized CGD histogram. From (f)-(j), we have the following observations. First, the CGD histogram provides the brightness and contrast information of the image. Besides, by comparing (f)-(j), we can observe that the peak position of CGD histogram of different night-time images with different luminance is also different. For example, compared with (f), the CGD histogram of (e) has higher peak position. Moreover, the shape of the CGD histogram reflects the contrast of the night-time image. For example, low-contrast (e) has narrower CGD histogram, while high-contrast (a) has wider CGD histogram, which can be seen from (f) and (j). Therefore, the following conclusion can be obtained, the shape of the CGD histogram and the contrast of the night-time image are directly related. By tracking the changes of the peak value of the CGD histogram, night-time image contrast distortion can be evaluated.

Fig. 3. Comparisons of CGD histogram of images with various MOS values. (The MOS values from left to right are 0.7914, 0.5771, 0.3914, 0.2471 and 0.105 7, respectively.)

Whereas, the CGD histogram of the overall image includes global contrast measure, which can’t fully describe the local contrast changes. When human beings observe the image, their visual attention is random [41], so we randomly select a group of P blocks in the overall image for purpose of obtaining the local contrast-changed of the image. Fig. 4 present an example of selecting P blocks in the whole image, where the number of patch P= 200 .

Fig. 4. Illustration of Pblocks in the night-time image

The CGD histogram set collects the CGD histogram local image blocks, which is given by:

\(H_{C G D}=\left[\begin{array}{ccc} h_{0,0} & \cdots & h_{0, P-1} \\ \vdots & \vdots & \vdots \\ h_{255,0} & \cdots & h_{255, P-1} \end{array}\right]\)       (7)

where P is the number of blocks. Let matrix \(H_{C G D}=\left[H_{0}, H_{1}, \cdots, H_{P-1}\right]\), in which \(H_{i}=\left\{h_{0, i}, h_{1, i}, \cdots, h_{255, i}\right\}^{T}\) represents the CGD histogram of the i^th block. The matrix H_CGD contains 256 rows and P columns. When P becomes larger, the dimension of the matrix H_CGD becomes higher.

Principal component analysis (PCA) [42] is a linear dimensionality reduction method. In PCA, the importance of information is expressed by variance. Therefore, the basic idea of PCA is to construct a series of linear combinations of original features to form low-dimensional features to remove the correlation of the data, and to make the reduced data maintain the variance information of the original high-dimensional data to the greatest degree. Therefore, with considering of the computational cost, PCA approach is adopted, and the detailed process is shown in the following.

Input: CGD histogram H_CGD, reduced data dimension m .

1) Calculate the average CGD histogram \(\overline{H_{C G D}}\) which is defined by \(\overline{H_{C G D}}=\frac{1}{P} \sum_{i=0}^{P-1} H_{i}^{T}\).

2) Calculate \(\tilde{H}\), which is expressed by \(\tilde{H}=\left[H_{0}-\overline{H_{C G D}}, H_{1}-\overline{H_{C G D}}, \cdots H_{P-1}-\overline{H_{C G D}}\right]\).

3) Calculate covariance matrix C , where \(C=\tilde{H} \tilde{H}^{T}\) Obviously, the dimension of C is P × P, where P is the number of histogram blocks.

4) Calculate eigenvectors \(W=\left\{w_{1}, w_{2}, \cdots, w_{d}\right\}\) and eigenvalues \(\lambda=\operatorname{diag}\left\{\lambda_{1}, \lambda_{2}, \cdots, \lambda_{d}\right\}\) of , then sort the eigenvalues in descending order: \(\lambda_{1} \geq \lambda_{2} \geq \cdots \lambda_{d} \geq 0\), in which d is the rank of C.

5) Select the largest m of the eigenvalues, and then the corresponding m eigenvectors are used to form the eigenvector-histogram.

Output: image patches eigen-histograms \(\overline{\bar{H}}\).

3.2 Texture Feature Extraction 

Night-time images are usually acquired in low-light environments, so the details of night-time image may be blurred, which will directly impact on night-time image perception. Here, we measure the blurred details via texture features, which is an indispensable factor to perceive image quality in the HVS. Texture features extraction on night-time image composes of two main modules, which are down-sampling, and texture features computation. Down-sampling [43] is used to generate night-time images with different resolutions so as to line with the characteristics of the HVS. Texture character computation works as follows. Firstly, MSCN coefficients are obtained by applying MSCN to the luminance of the distorted night-time image. Then, the statistical parameters and weighted LBP histogram features are extracted from the MSCN coefficients, where the statistical parameters are obtained from the generalized Gaussian model and the weight is the magnitude of MSCN coefficients.

3.2.1 Statistical feature of MSCN coefficients

Performing nonlinear operations on image brightness can eliminate the correlation between pixels. Mittal et al. [17] extracted the statistical features of MSCN coefficients to predict perceived image quality.

The MSCN coefficient is given by:

\(\hat{I}=\frac{I(i, j)-\mu(i, j)}{\sigma(i, j)+C}\)       (8)

\(\mu(i, j)=\sum_{h=-H}^{H} \sum_{l=-L}^{L} w_{k, l} I_{k, l}(i, j)\)       (9)

\(\sigma(i, j)=\sqrt{\sum_{h=-H}^{H} \sum_{l=-L}^{L} w_{k, l}\left[I_{k, l}(i, j)-\mu(i, j)\right]^{2}}\)       (10)

where I(i , j)is the pixel value of image I at location (i , j) , µ(i , j) and σ(i , j) represent the mean and standard variance values of local region that the center is (i , j) , the length is 2H , and the width is 2L , respectively. w is Gaussian filter.

MSCN coefficient can be modeled by general Gaussian distribution (GGD) [17], which is given by:

\(f\left(x ; \alpha, \sigma^{2}\right)=\frac{\alpha}{2 \beta \Gamma(1 / \alpha)} \exp \left[-(|x| / \beta)^{\alpha}\right]\)       (11)

where \(\beta=\alpha \sqrt{\Gamma(1 / \alpha) / 3 / \alpha}\) and \(\Gamma(x)=\int_{0}^{\infty} t^{x-1} e^{-t} d t(x>0)\). α is the general shape of the distribution and σ is the variance.

3.2.2 MSCN-weighted LBP histogram features

LBP [44] is an efficient local texture descriptor, which is widely used in face recognition, texture classification, and image quality assessment. Therefore, we adopt LBP, to describe the texture features of night-time image.

The classic uniform rotation invariant LBPs procedure is introduced [44], as:

\(L B P\left(x_{i}\right)= \begin{cases}\sum_{i=0}^{P-1} s\left(g_{i}-g_{c}\right) & \text { if } U\left(L B P_{P, R}\right) \leq 2 \\ P+1 & \text { else }\end{cases}\)       (12)

\(U\left(L B P_{P, R}\right)=\left|s\left(g_{P-1}-g_{c}\right)-s\left(g_{0}-g_{c}\right)\right|+\sum_{i=1}^{P-1}\left|s\left(g_{i}-g_{c}\right)-s\left(g_{i-1}-g_{c}\right)\right|\)       (13)

\(s\left(g_{i}-g_{c}\right)= \begin{cases}1, & g_{i}-g_{c} \geq 0 \\ 0 & g_{i}-g_{c}<0\end{cases}\)       (14)

where g_c is the intensity of the central pixel x_c , g_i denotes the intensity of neighbor pixel x_i , R is the radius of the circle, P represents the number of neighbors evenly distributed around the circle.

LBP features are usually directly extracted from the original image [44]. Different from these work, the MSCN coefficients processed by formula (1) are selected for LBP feature extraction in the proposed method. In addition, different from the original statistical binary mode using frequency times to obtain the histogram, we use the MSCN coefficient amplitude as the weighted statistical normalized histogram. The weighted statistical normalized histogram is given by:

\(h^{\prime}(k)=\sum_{i=1}^{M} \sum_{j=1}^{N} w_{i, j} \delta(L B P-k)\)       (15)

\(\delta(x)=\left\{\begin{array}{l} 1, x=0 \\ 0, \text { other wise } \end{array}\right.\)       (16)

where w_i,j, is the amplitude value of MSCN coefficient, δ(x) is the impulse function.

Fig. 5 shows the comparisons of MSCN coefficients and MSCN-weighted LBP histogram of images with various MOS values. (a)-(e) are five original images from the NNID database [15] and (f)-(j) are the corresponding MSCN coefficients. (k)-(o) are the corresponding MSCN-weighted LBP histogram. It is clear that night-time images with different MOS results in MSCN coefficients distribution changing. Fig. 5(a) with high MOS exhibits a Gaussian like appearance, while (e) with low MOS deviates from Gaussian distribution. The following conclusions can be drawn from (k) to (o): MSCN-weighted LBP histogram can better reflect the image structure information, and MSCN-weighted LBP histogram for different night-time images with different MOS are also different.

3.3 Colour Feature Extraction

The features of texture explained in the previous section are extracted from grayscale night-time image. Considering that night-time images usually fail to truly reflect the true color of the image, and the color space is more in line with the characteristics of HVS, the statistical features of the color space is used to enhance the perception of night-time image quality. However, the three channels of RGB space have a strong correlation, which is not conducive to the separation of color and brightness features. We use the Lαβ color space proposed by Ruderman [45], which has the characteristics of the least correlation between channels for images.

Fig. 5. Comparisons of MSCN coefficients and MSCN-weighted LBP histogram of images with various MOS values (The MOS values from left to right are 0.7914, 0.5771, 0.2471, and 0.1057, respectively. (f)-(j) are the corresponding MSCN coefficients. (k)-(o) are the corresponding normalized CGD histogram.)

Considering that Lαβ is the transformation of LMS, formula (17) is firstly used to convert from RGB to LMS according to [49,50].

\(\left\{\begin{array}{l} L=0.3811 \cdot R+0.5783 \cdot G+0.0402 \cdot B \\ M=0.1967 \cdot R+0.7244 \cdot G+0.0782 \cdot B \\ S=0.0241 \cdot R+0.1288 \cdot G+0.844 \cdot B \end{array}\right.\)       (17)

Since the data in LMS is relatively scattered, we further convert LMS to logarithmic LMS space via (18) to make the data distribution more convergent, which is more in accord with human subjective perception of colors.

\(\left\{\begin{array}{l} L^{\prime}=\log L \\ M^{\prime}=\log M \\ S^{\prime}=\log S \end{array}\right.\)       (18)

For the logarithmic space components \(L^{\prime}, M^{\prime} \text { and } S^{\prime}\), we use (8) to normalize the contrast to get the components \(\widehat{L}, \hat{M} \text { and } \widehat{S}\), and then use the following equation to transform from LMS to Lαβ.

\(\left\{\begin{array}{l} \hat{l}=\frac{1}{\sqrt{3}}(\widehat{L}+\widehat{M}+\widehat{S}) \\ \hat{\alpha}=\frac{1}{\sqrt{6}}(\widehat{L}+\hat{M}-2 \widehat{S}) \\ \hat{\beta}=\frac{1}{\sqrt{2}}(\widehat{L}-\widehat{M}) \end{array}\right.\)       (19)

where \(\hat{l}\) reflects brightness information, \(\hat{\alpha} \text { and } \hat{\beta}\) reflect the information of the blue-yellow channel and red-green channel, respectively. Fitting the AGGD [17] to \(\hat{\alpha} \text { and } \hat{\beta}\), the expression is:

\(f\left(x ; \alpha, \sigma_{l}^{2}, \sigma_{r}^{2}\right)=\left\{\begin{array}{l} \frac{\alpha}{\left(\beta_{r}+\beta_{l}\right) \Gamma(1 / \alpha)} \exp \left(-\left(|x| / \beta_{l}\right)^{\alpha}\right), x<0 \\ \frac{v}{\left(\beta_{r}+\beta_{l}\right) \Gamma(1 / \alpha)} \exp \left(-\left(|x| / \beta_{r}\right)^{\alpha}\right), x \geq 0 \end{array},\right.\)       (20)

where \(\beta_{l}=\sigma_{l} \sqrt{\Gamma(1 / \alpha) / \Gamma(3 / \alpha)} \text { and } \beta_{r}=\sigma_{r} \sqrt{\Gamma(1 / \alpha) / \Gamma(3 / \alpha)}\).

Fig. 6 shows the comparisons of BY opponent coefficients and RG opponent coefficients of images with various MOS values. (a)-(e) are five original images from the NNID database [15] and (f)-(j) are the corresponding BY opponent coefficients. (k)-(o) are the corresponding RG opponent coefficients. As shown in Fig. 6, both the distributions of BY opponent coefficients and RG opponent coefficients of (a) to (e) vary with respect to their subjective quality scores (by comparing with their MOS scores). (a) with high MOS exhibits an AGGD like appearance, while (e) with low MOS deviates from AGGD. That is, statistical features in color space have the ability to perceive night-time image quality.

Fig. 6. Comparisons of BY opponent coefficients and RG opponent coefficients of images with various MOS values (The MOS values from left to right are 0.7914, 0.5771, 0.2471, and 0.1057, respectively. (f)-(j) are the corresponding BY opponent coefficients. (k)-(o) are the corresponding RG opponent coefficients.)

3.4 Quality Regression

After the above operations, we can extract three types of quality-aware features, including contrast, texture and colour features. Given a night-time image, we can obtain 106 features. The classic SVR model [46] is adopted to establish the mapping between the quality-aware features and MOS of the image.

Assuming that the training sample set is \(\left\{\left(x_{1}, y_{1}\right), \ldots\left(x_{k}, y_{k}\right)\right\}, x_{i} \in R^{n}\) is the extracted feature, y_i is the corresponding MOS, parameter C > 0 , ε > 0 , the standard expression of SVR is:

\(\begin{gathered} \min _{w, b, \xi, \xi^{*}} \frac{1}{2} w^{T} w+C\left\{\sum_{i=1}^{k} \xi_{i}+\sum_{i=1}^{k} \xi_{i}^{*}\right\} \\ \text { st : } w^{T} \phi\left(x_{i}\right)+b-y_{i} \leq \varepsilon+\xi_{i} \\ y_{i}-w^{T} \phi\left(x_{i}\right)-b \leq \varepsilon+\xi_{i}^{*} \\ \xi_{i}, \xi_{i}^{*} \geq 0, i=1, \cdots k \end{gathered}\)       (21)

where \(K\left(x_{i}, y_{i}\right)=\phi\left(x_{i}\right)^{T} \phi\left(x_{j}\right)\) is the kernel function, this paper utilizes the radial basis (RBF) kernel function given as \(K\left(x_{i}, y_{i}\right)=\exp \left(-\gamma\left\|x_{i}-x_{j}\right\|^{2}\right)\).

4. Experimental Results

4.1 Databases and Protocols

The experiments are conducted based on NNID [15]. NNID database contains 2240 night-time image with 448 groups, each of which contains five images with different exposure, aperture, shutter, ISO settings, shutter and the same shooting position, and etc.. Among them, 280 groups of images are captured by a digital camera (Nikon D5300), 128 groups of images are captured by a mobile phone (iPhone 8plus), and the remaining 40 groups were captured by a tablet (iPadmini2). Classified by resolution, images with resolutions of512 × 512 , 1024 × 1024 and 2048 × 2048 are 268, 90, and 90 groups, respectively. All images are saved in JPEG format, together with the MOS values in the range of [0,1] . 1400 images captured by Nikon D5300, 640 images captured by iPhone 8plus and 200 images captured by iPad mini2. Table 1 summarizes the general information of the NNID databases.

Table 1. NNID database

In addition, the subsets of CCRIQ [47], LIVE-CH [48] and CID2013 [35] databases are considered for comprehensively testing, and contain 44, 303, 79 night-time images (luminance value <=0.4), respectively.

Pearson Linear Correlation Coefficient (PLCC), Spearman Rank order Correlation Coefficient (SRCC), Kendall Rand order Correlation Coefficient (KRCC) and Root Mean Square Error (RMSE) are considered as performance evaluations. PLCC and RMSE indicate the prediction accuracy [31], while SRCC and KRCC represent the prediction monotonicity [32]. The range of PLCC, SRCC and KRCC is [0,1] , while the range of RMSE is[0, ∞]. Generally speaking, SRCC, KRCC and PLCC are as high as possible, and RMSE is as low as possible.

Before calculating the above performance criteria, we employ a five parameters nonlinear logistic function suggested by [28] to complete the mapping from objective scores to human rating scores.

4.2 Performance Evaluation

\(f(x)=\tau_{1}\left(\frac{1}{2}-\frac{1}{1+e^{\tau_{2}\left(x-\tau_{3}\right)}}\right)+\tau_{4} x+\tau_{5}\)      (22)

where x is the objective score, f(x) represents the mapped objective score, \(\tau_{1}, \tau_{2} \cdots \tau_{5}\) are the parameters to be fitted, the values of which is realized by using the nonlinear least squares function, nlinfit provided in Matlab. The final fitting parameters are essentially a parameter combination that minimizes the sum of squared errors between x and f(x) . For the function nlinfit, its input includes subjective score, corresponding objective score, non-linear function, while the output includes fitting parameters, residuals and Jacobian matrix.

4.2 Performance Evaluation

In this subsection, the proposed method is compared with the ten general BIQA methods (BRISQUE [17], BLINDS-II [16], NFERM [20], NIQE [26], IL-NIQE [27], GM-LOG [18], MSGF [19], DIIVINE [11], MEON [21] and DB-CNN [22]), four special BIQA methods that are designed for contrast distorted (MDM [36], NR-SPL [37], NR-CDIQA [39], and NIQMC[40]) and an approach that is specially designed for night-time images (BNBT [15]) on NNID, CCRIQ [47], LIVE-CH [48] and CID2013 [35]. All the results except BNBT [15] are calculated by the source code released by authors. Note that BNBT is implemented according to the [15].

4.2.1 Comparison with NR-IQA methods

Fifteen IQA metrics and proposed method are tested on the entire NNID as shown in left part of Table 2. The experiment results are also conducted on the subsets of NNID that include images captured by three devices (Nikon D5300, iPhone 8plus and iPadmini2) as shown in right part of Table 2 and Table 3. Table 4 and Table 5 further present the experimental results of the fifteen IQA metrics on the subset of CCRIQ [47], LIVE-CH [48] and CID2013 [35] databases. In the experiments, NIQE [26], IL-NIQE [27], and NIQMC [40] are the training-free metrics while the other twelve methods are training-based quality metrics. For the training-based methods, the database is randomly divided into two parts, the model is trained based on 80% of the data, and the test is conducted by other 20% data. In order to ensure the fairness of the division, the above random division process is experimented 1000 times, and the final predicted result takes the median values. For the training-free methods, all the images in the database are used to obtain the results.

From Table 2 ~ Table 3, the following observation can be obtained. (1)The PLCC, SRCC and KRCC values of proposed algorithm are the highest among overall listed algorithms, while RMSE value is the lowest on NNID [15] database and the subsets of NNID [15]. Therefore, the proposed method achieves higher prediction accuracy (PLCC and RMSE) and prediction monotonicity (SRCC and KRCC) in contrast to the listed methods. In other word, the evaluation results of the propose method are more in line with human rating scores. This shows that the proposed method can predict the quality of night-time image more accurately than the related IQA metrics listed in Table 2. (2)The performance of BNBT [15], which is an IQA method specially designed for night-time images, is slightly worse than of the proposed method. This phenomenon indicates that the features extracted by the proposed method can better characterize the characteristics of the night-time images than those extracted by BNBT.

In addition, from Table 2 ~ Table 5, the following three observations can also be obtained. (1) For the ten general BIQA methods, the performance of training-based algorithms is generally better than that of training-free algorithms. Especially, DB-CNN [22] that employs deep learning, shows excellent performance in the general BIQA benchmark comparison. This phenomenon suggests that supervised learning brings better algorithm performance than unsupervised learning. (2) From Table 4 and Table 5, we can see that the majority of quality metrics listed in Table 4 perform worse at SRCC, KRCC, PLCC and RMSE than the proposed method on CCRIQ [47], LIVE-CH [48] and CID2013 [35] databases, which verifies that our proposed method can accurately evaluate the quality of image which is captured not only in night-time scenario but also in the low illumination conditions. (3) By comparing the performance of four contrast-distorted algorithms and the ten general BIQA methods, it is found that contrast-distorted methods have superior performance than most general algorithms in measuring the quality of night-time images and the images captured in low lighting conditions. However, compared with BNBT [15] and our proposed method, the performance of the considered contrast-distorted methods is slightly worse, which shows that when designing the night-time image quality metric, not only the contrast distortion should be considered, but also the burred details, colour, low visibility and other features should be considered.

Table 2. Performance comparison on NNID database

Table 3. Performance comparison on NNID database

Table 4. Performance comparison on subsets of CCRIQ and LIVE-CH databases

Table 5. Performance comparison on subsets of CID2013 databases

4.2.2 Performance Analysis on Individual Features

Three types of features, including contrast feature, texture feature and colour feature, are employed in our proposed approach to jointly measure the distortion of the realistic night-time images. Next, we further use ablation experiments to conduct contribution of each individual character to the proposed method on the entire NNID database [15]. In the experiment, the contrast, texture and color features are separately used for SVR training and quality prediction on the entire NNID database [15]. The experimental results are listed in Table 6.

Some findings are observed from Table 6: (1) Compared with Table 2, the proposed metric with separate contrast, texture and color feature on the NNID database [15] still has obvious advantages over most considered IQA methods. This proves the effectiveness of the three type features proposed in this paper. (2) The contrast feature achieves 0.8623 SRCC and 0.8635 PLCC. By contrast, the SRCC and PLCC values of texture features are 0.8436 and 0.8469, respectively, while those of colour features are 0.6325 and 0.6391. These results demonstrate that the contrast features contribute more to the proposed method in measuring the night-time images than the texture or color features. (3) The performance of the method after combining the three type features is much better than that of using each feature alone. These results show the rationality and necessity of combining the three type characters in the proposed method.

Table 6. Performance of Individual Features on NNID Database

4.2.3 Complexity Analysis

In practical application, the computational complexity is an important factor affecting the image processing system. Consequently, the computational complexity that is required for each method are listed in Table 2. Experiments are performed on a desktop computer configured with Intel Core-i7-8700CPU@3.2GHz and 8GB RAM. The simulation software is MATLAB R2016b, which is used under Windows 10 operating system. Fig. 7 summarizes the running time that is for processing 1000 images with resolution 512 × 512 randomly selected from the NNID database. Fig. 7 signifies that the proposed metric needs moderate running time and performs statistically the best compared to other existing methods.

Fig. 7. Running Time Comparison of Proposed method and Involved IQA Methods

5. Conclusion

This paper has introduced an efficient NR-IQA metric for realistic night-time scenario. Three types of features including contrast, texture, and colour, which are correlated with the quality of the night-time images, are extracted from the image simultaneously. More specifically, CGD histogram information was captured to quantify contrast; MSCN-weighted LBP histogram information was estimated to infer texture; statistical features in LMS colour space were extracted to represent image colour distortions. After feature extraction, SVR is adopted to train the quality prediction model. Experiments are conducted on NNID, CCRIQ, LIVE-CH, and CID2013 databases, and the proposed method shown to be best compared to the existing NR-IQA metrics at SRCC, KRCC, PLCC and RMSE. Moreover, the processing speed of the proposed method is acceptable. As future work, we would like to employ deep learning to improve the performance of BIQA method for night-time image.