Edge-preserving demosaicing method for digital cameras with Bayer-like W-RGB color filter array

Abstract

A demosaicing method for a Bayer-like W-RGB color filter array (CFA) is proposed. When reproducing images from a W-RGB CFA, conventional color separation methods for W-RGB CFA are likely to cause blurring near the edges due to rough averaging using a color ratio of neighboring pixels. Moreover, these methods cannot be applied to real-life digital cameras with W-RGB CFA because the methods were proposed under an ideal situation, W=R+G+B, not a real-life situation, $W{\neq}R+G+B$. To improve edge performance, we propose a method of constant color difference assumption with inversed weight, which uses information from all edge directions for interpolating all missing color channels. The proposed method calculates the correlation between W, R, G, and B to enable its application to real-life digital cameras with W-RGB CFA. Simulations were performed to evaluate the proposed method using images captured from a real-life digital camera with W-RGB CFA. Simulation results shows that we can demosaic by using the proposed algorithm compared with the conventional one in about +34.79% SNR, +11.43% PSNR, +1.54% SSIM and 14.02% S-CIELAB error. Thus, the proposed method demosaics better than the conventional methods.

1. Introduction

while the resolution of consumer digital cameras is increasing, the size of complementary metal-oxide semiconductor (CMOS) image sensors remains limited. As a result of this trend, CMOS image sensors now contain a huge number of pixels. However, the size of each pixel on the sensor is becoming increasingly smaller, and the amount of light absorbed in the same period by each unit pixel is therefore significantly decreased. This degradation of light sensitivity by each unit pixel can lead to deterioration in image quality, especially in low-light conditions.

Images taken by digital cameras that use the above-mentioned CMOS sensors in low-light conditions can have significant blurring and noise because a long exposure time is required for the aperture to gather enough light to produce acceptable images. To overcome this defect, a sensor with a higher sensitivity is required. To address this need, a White-RGB (W-RGB) color filter array (CFA), which has a greater sensitivity than a Bayer color filter array[1], was developed [2–6].

Examples of CFA patterns are shown in Fig. 1. Examples of the Bayer CFA pattern and a Bayer-like W-RGB CFA pattern are shown in Figs. 1(a) and 1(b), respectively. A Bayer-like W-RGB CFA substitutes one green pixel on every 2×2 matrix of the Bayer CFA with a white pixel (W2x,x) which is made by transparent resin films. To increase sensitivity, the white pixel can be penetrated by all visible light with wavelengths of 450 to 700 nm. Unlike the Bayer CFA, W pixels on the W-RGB CFA enable penetration of all color strengths, while other pixels do not. Therefore, sensitivity can be improved because more information exists in the W pixel locations of the W-RGB CFA than in G pixel locations of the Bayer CFA. Consequently, the existence of the W pixel itself improves sensitivity of the W-RGB sensors.

Fig. 1.Examples of CFA patterns: (a) Bayer pattern, and (b) Bayer-like W-RGB pattern.

A demosaicing method for W-RGB CFA is required to reproduce full-color images from a W-RGB CFA. Therefore a new method that performs demosaicing from Bayer-like W-RGB CFA to full-color image is proposed in this paper.

 

2. Background

Two conventional methods were proposed by Hiroto Honda et al. [4,5]. Their methods assume an ideal situation, W2,1=R2,2+G1,2+B1,1, which is referred to in Fig. 1.(b). These methods estimate R, G, and B values at the pixel location W based on the color ratio of neighboring pixels using Eq. 1.

Raverage, Gaverage, and Baverage represent the average values of R, G, and B located near the target pixel W. RW, GW and BW represent the estimated R, G, and B values located at target pixel W. The method assumes an ideal situation (W2,1=R2,2+G1,2+B1,1), where R, G, and B have the same weight, 1. Their second method [5] considers edge directions for avoiding the blur problem of the method [4] by estimating RW, GW, and BW values only if an edge is not detected at pixel location W. If it is not detected, only GW at location W is estimated using Eq. 1. Performance near edges is poor even though the method [5] considers edge existence.

Ideally, W pixels on the W-RGB CFA can be penetrated by all R, G, and B elements of light; therefore, the method assumes the equation W=R+G+B is satisfied. However, the equation of W=R+G+B is not satisfied in a real-life situation. Consequently, the methods of [4] and [5] cannot be properly applied to real-life digital cameras with W-RGB CFA. In this paper, we propose a method that improves edge performance of conventional methods through a constant color difference assumption with inversed weight, which uses information from all edge directions for interpolating all missing color channels. The proposed method calculates the correlation between W, R, G, and B to enable its application to real-life digital cameras with W-RGB CFA.

 

3. Demosaicing method for Bayer-like W-RGB CFA

Demosaicing method interpolates missing color channels for reproducing full-color images from mosaiced images using the information of reference pixels near target pixels. It is obvious that averaging reference pixels which lie across an edge structure can cause serious blur defect. Therefore, in order to preserve edge structure well when estimating missing color channels, it is important to avoid averaging between reference pixels which lie across the edge structures. In other words, it is important to utilize the information of edge structure properly for preserving edge structure when estimating missing color channels. To that end, the proposed demosaicing method starts with extracting the information of edge structure at every pixel location.

The proposed method for demosaicing the W-RGB CFA consists of four steps: (1) building an edge direction map, (2) estimating correlations among W, R, G, and B, (3) estimating missing color channels using the constant color difference assumption, and (4) updating the missing color channels previously estimated in an adaptive way. As with the conventional methods mentioned above [4,5], the proposed method assumes a real-life situation (W2,1≠ R2,2+G1,2+B1,1), which estimates the value of G within a 5×5 reference matrix.

3.1 Edge Direction Map Building

Sharpness near the object edge is a key factor for the human visual experience. To prevent the appearance of blurring near the edge, it is important to avoid averaging across it. To that end, edge information should be calculated in advance. A commonly used method of expressing edge information utilizes the differences in luminance between adjacent pixels. This method, however, is not reasonable for use in the W-RGB CFA case where only one of three color channels is available for each pixel, including the W pixel; therefore a mosaiced image which is shown in Fig. 1.(b) does not have complete luminance information at every pixel. Therefore, an edge strength filter [7] for the mosaiced image, Eq. 2, is adopted for expressing edge information.

By using an edge strength filter, an edge direction map can be built using Eq. 3, where Si,j is the approximate edge strength at pixel location (i, j), and D_Xi,j is the total cost toward X degree at pixel location (i, j). By comparing the adjacent edge strength costs themselves, the target pixel (i, j) is labeled 45° or 90° if D_45i,j or D_90i,j is larger than D_135i,j or D_180i,j and vice versa. Two edge direction maps are made by this method. One map has information for 45° or 135° while the other has information for 90° or 180°.

3.2 Estimation of Pixel Correlations

The conventional method [4,5] assumes the ideal situation, W=R+G+B; however, the ideal situation and the real-life situation are different. We assume W=αR+βG+γB to estimate the correlation among W, R, G, and B in a 2×2 matrix. To find the best matched solution of α, β and γ, we use the method of least squares (LS) [8] in a non-singular situation. We can write the non-singular LS expression as Eq. 4:

In other words, we can express Eq. 4 as follows:

where

After further mathematical manipulation of Eqs. 5 and 6, we can obtain a solution as follows:

where inv(∙) indicates the function of inverse calculation. Candidates for calculating correlation information among W, R, G, and B using Eq. 4 should not be saturated for precise calculation. Namely the candidate values of the W, R, G, and B should be in the range of 0 to 255. Each time we take a picture, circumstances (scene, amount of light absorbed, etc.) change; therefore, correlation coefficients α, β, and γ are renewed when each picture is taken.

3.3 Estimation of Missing Color Channels

In order to interpolate missing color channels from W-RGB mosaiced input image, utilization of the existing raw pixels near target locations is the most important part for preserving edge structure. Therefore, the W locations are the best target locations for interpolating missing G channel first because the G pixels lie on all diagonal directions from the target W locations in the center, referred in Fig. 1 (b), raw Bayer-like W-RGB image. Estimation of missing color channels consists of two steps: (1) green color channel estimation at pixel location W, and (2) missing color channel estimation at every pixel location.

3.3.1 Green Color Channel Estimation at Pixel Location W

Using the edge direction maps and correlation coefficients previously obtained, the W pixel values are estimated into G values using the weighted color ratio with correlation coefficients, Eq. 8, and the constant color difference assumption [9], Eq. 9:

where is the temporal directional estimation result of the edge labeled X in degrees for constant color difference assumption and is the resulting value of G at location W using the constant color difference assumption. Using Eqs. 8 and 9, G color channels on pixel location W can be estimated As a result, a W-RGB patterned image which is shown in Fig. 1.(b) will be converted into an image like standard Bayer pattern which is shown in Fig. 1.(a). With the temporal result from the section 3.3.1, the conventional demosaicing method for Bayer CFA can be adopted for reproducing full-color images; however, we will interpolate missing color channels keeping the same principles, constant color difference assumption with edge information, for preserving edge structure.

3.3.2 Missing Color Channel Estimation at Every Pixel Location

The approach to estimating the missing color channels of R, G, and B at every pixel location differs somewhat from those outlined in section 3.3.1. In the case of green channel estimation at W pixel locations, a weighted color ratio with pre-calculated correlation coefficients and a constant color difference assumption are used to separate W pixel values into R, G, and B pixel values. However, in the case of estimation at R, G, and B, other kinds of directional estimation and constant color difference assumption are used. Eqs. 10 and 11 are for estimation at B pixel values at R pixel locations.

and are temporal directional estimations used for the constant color difference assumption and is the estimation result of the blue pixel value using the constant color difference assumption. Eqs. 10 and 11 use the information of edge direction toward 45 and 135 degrees because R pixels and B pixels are situated diagonally, like R2,2, B1,1, B1,3, B3,1 and B3,3 that are referred to in Fig. 1.(a). Missing pixels that lie diagonally can be estimated by replacing variables of Eqs. 10 and 11; however missing pixels lying vertically and horizontally can be estimated by using Eqs. 12 and 13 which are modified versions of Eqs. 10 and 11, respectively.

3.4 Update of Estimated Missing Color Channels

The next step is a color update using the correlation weight for more precise estimation near the edge. Two pixels located across a strong edge influence each other less. Therefore, the difference in edge strength will be large if the pixels are located across a strong edge. That means that pixels are inversely correlated; therefore, we utilize an inverse weight to improve performance near an edge. The final value of G at pixel location W can be obtained with Eq. 14 using pre-estimated values and inverse weights:

where is the updated final G pixel values at W pixel locations in an adaptive way using inversed weights, and Dx and Wx are edge strength toward x in degrees and inversed weights of x in degrees, respectively. All estimated missing colors at 3.3.1 and 3.3.2 can be updated by replacing the variables and directional suffixes. With the proposed final equation, Eq. 14, all missing channels of R, G, and B can be effectively estimated while avoiding blur; that is, images from W-RGB CFA can be demosaiced to full-color images with good results.

 

4. Simulation Results

To simulate the proposed method, images captured by a real-life digital camera with W-RGB CFA were used as input images as shown in Fig. 2. The ground truth images are shown in Fig. 3, and the simulation results of the conventional method and the proposed method are shown in Figs. 4 and 5, respectively. We can recognize the subjective edge performance by comparing the three sets of figures, Fig. 3, 4 and 5. Two types of objective simulations were performed to evaluate the accuracy of the proposed method.

Fig. 2.Bayer-like W-RGB patterned images used for the performance evaluation.

Fig. 3.The set of ground truth images for comparison.

Fig. 4.Resulting images from the conventional method [5].

Fig. 5.Resulting images from the proposed method.

The first type of simulation evaluates the accuracy of the demosaicing method by comparing the resulting images of the proposed method with ground truth images. The input images of the proposed method and the ground truth images are of the same scene captured under the same circumstances. SNR, PSNR, and SSIM were measured against ground truth images; the results are shown in Tables 1, 2 and 3, respectively. To prove the performance of the proposed method, measuring SNR, PSNR, and SSIM is insufficient; therefore, we adopted the method of the S-CIELAB color difference [10] metric for the second simulations.

Table 1.Comparisons of SNR for different methods.

Table 2.Comparisons of PSNR for different methods.

Table 3.Comparisons of SSIM for different methods.

The S-CIELAB color difference between two colors becomes smaller when the two colors have similar elements in L*a*b* space. Namely, the smaller the S-CIELAB color difference becomes, the better the result of the proposed method. The S-CIELAB color difference metric returns 1 for any just-noticeable error; 10 for very high error. The proposed method has the better overall averages of SNR, PSNR, SSIM and S-CIELAB color difference than those of the method [5], outperforming the conventional method by 4.652 dB, 4.061 dB, 0.015 and −0.045, respectively. Also, we can also figure out that the performance of the proposed method is improved by 34.787%, 11.432%, 1.543% and 14.024% on average for SNR, PSNR, SSIM and S-CIELAB color difference measuring factors, respectively.

Table 4.Comparisons of average S-CIELAB distance for different methods.

In addition, we measure the percentage of pixels exceeding 3, 5, and 10 of the S-CIELAB color differences, as shown in Table 5. With data from Table 5, we illustrate the spatial distribution of the errors in Fig. 6 using white indicator when the S-CIELAB color difference exceeds 1 or larger, and green indicator when it exceeds 10 or larger. Each subfigure in Fig. 6 shows the distribution of the errors of the conventional method (left part) and the proposed method (right part), respectively. For every subfigures in Fig. 6, we can see that the left images have more green and white indicators than the right images; therefore, we can confirm that the proposed method is superior to the conventional one in the aspect of S-CIELAB color difference.

Table 5.Subjective simulation results of S-CIELAB color difference.

Fig. 6.Spatial distribution of the errors by using S-CIELAB color difference.

As shown in all simulation results based on measuring SNR, PSNR, SSIM and S-CIELAB difference, the proposed method outperforms the conventional method.

 

5. Conclusion

In this paper, we proposed a new method of demosaicing for W-RGB CFA. The proposed method can be applied effectively to real-life devices which have CMOS image sensors with W-RGB CFA using information of edge structure, the correlation information among W, R, G, and B, the constant color difference assumption, and adaptive weighted estimation. With the proposed method, images captured by a real-life digital camera with Bayer-like W-RGB CFA can be demosaiced to full-color images with less blur than conventional methods.