I. INTRODUCTION
Three-dimensional (3D) information includes the longitudinal distance to an object. Extraction of distance information has been the subject of intensive research since 3D information cannot be acquired with a conventional camera [1-3]. Integral imaging (II) was proposed for 3D display [4-7] and has been researched for 3D information processing such as distance extraction and object recognition [8-14]. The II recording process generates a set of elemental images with an array of pinholes or microlenses, thus an elemental image has different perspectives of a 3D object. Unlike stereo imaging, multiple-perspective imaging of an object can yield a compact 3D sensing system, as only a single exposure is required without any active illumination, unlike holography or light detection and ranging [15-17].
The depth-extraction technique using II was studied in [9-14]. The depth is extracted by means of one-dimensional elemental image modification and a correlation-based multibaseline stereo algorithm [9]. In [10], the most focused reconstruction plane was extracted using two adjacent reconstructed images. The multibaseline method was modified to extract the depth from unidirectional II [11]. The depth was estimated when the sum of the standard deviations of the corresponding pixel’s intensity was minimized [12]. A novel depth-extraction method using the windowing technique was proposed in [13]. Points of segmented features were matched to improve the depth accuracy in [14].
Photon-counting sensing has been researched for low-light imaging applications [18-25]. Advanced photon-counting imaging technology can register a single photo-event at each pixel generating a binary dotted image. Therefore, little power is consumed for imaging, and a fast algorithm can be implemented to process the binary dotted image such as linear and nonlinear matched filters for object recognition [21-23]. The nonlinear correlation between photon-counting images was proposed in [24], but only stereoscopic images were considered, and no theoretical analysis was conducted. In [25], multiple reconstructions of photon-counting scenes were performed to obtain the most focused depth level.
In this paper, a distance-extraction method is proposed using the nonlinear correlation of photon-counting II. The distance is calculated from the pixel disparity, which is obtained using the nonlinear correlation between photon-counting elemental images. The nonlinearity of the correlation is defined by the power of the sum of photocounts in the denominator. The first- and second-order statistical properties of photon-counting nonlinear correlation are investigated theoretically and experimentally. In [25], multiple reconstructions could increase the computational load significantly when a large number of elemental images were involved in the reconstruction process. However, the proposed method can be fast since it utilizes the sum of the photocounts without the reconstruction process.
In the experiments, the distance is extracted by varying the mean number of photocounts. The statistical properties of the nonlinear correlation are verified through the experimental observations. These experimental results confirm that the proposed method extracts distance information under low-light-level conditions. To the best of the author’s knowledge, this is the first report on distance extraction using the nonlinear correlation of photon-counting II.
The rest of the paper is organized as follows: The II recording process with a photon-counting detection model is described in Section 2. The distance-extraction method is proposed in Section 3, and the statistical properties of the nonlinear correlation are described in Section 4. The experimental results and conclusion are presented in Sections 5 and 6 respectively.
II. INTEGRAL IMAGING PICK-UP PROCESS WITH PHOTON-COUNTING DETECTION MODEL
The II pick-up (recording) process generates an array of elemental images using an array of pinholes or microlenses. A microlens array is composed of a number of small, convex lenslets. The elemental images bear different perspectives of the object since each lenslet corresponds to a different elemental image. Figure 1 shows the II pickup process where eij is the elemental image located in the i-th column and j-th row in the x and y directions respectively. A conventional imaging sensor like a CCD camera is located at the imaging plane.
FIG. 1.Pickup diagram of the II recording system.
A photon-counting detector can be located at the imaging plane generating a photon-counting elemental image array as shown in Fig. 2: pij is a photon-counting elemental image located in the i-th column and j-th row, and q(i+li)j is a photon-counting elemental image located in the (i+li)th column and j-th row.
FIG. 2.Distance extraction using the pixel disparity of a photon-counting elemental image pair.
The photon-counting detector registers a photo-event at a low light level. The probability of a photo-event is assumed to follow the Poisson distribution under low-light-level conditions [26]:
where pij(m,n) is the number of photocounts detected at the pixel located in the m-th column and n-th row; Np is the mean number of photocounts in the image array, and eij(m, n) is the normalized intensity such that where I and J are, respectively, the numbers of elemental images in the x and y directions in the array, and M and N are, respectively, the pixel numbers in the x and y directions in an elemental image. Therefore, it can be seen that, . In the experiments, the photon-counting elemental images are generated using a pseudorandom number generator on a computer.
III. DISTANCE EXTRACTION WITH PHOTON-COUNTING INTEGRAL IMAGING
In this section, the method for extracting longitudinal distance with photon-counting II is described. The pixel disparity between a pair of photon-counting elemental images is obtained by means of the nonlinear correlation as follows
It is noted that Cij(Δ, li; v1, v2) is the nonlinear correlation normalized by the powers v1 and v2 of the sum of the photocounts.
Figure 2 shows the stereoscopic configuration with the elemental image pair pij and q(i+li)j. In this paper, two elemental images in the same row are considered a stereoscopic image pair, thus the baseline distance bli for stereo matching is the same as φ ·li, where φ is the pitch of the microlenslet. It can be seen that φ =M·p and bli = M·p·li, where p is the pixel pitch.
The distance is extracted using the photon-counting elemental image pair as follows
where f is the focal length of the microlenslet. The distance estimate based on the photon-counting elemental image array is defined as the average of dij(li;v1,v2) over i, j, and li as follows
where Ip and Jp are the numbers of elemental image pairs in the x and y directions respectively, and li is set to vary from Si to Ei.
IV. THEORETICAL DERIVATION OF THE MEAN AND VARIANCE OF THE NONLINEAR CORRELATION
In this section, the first- and second-order statistical properties of Cij(0,0;v1,v2) are derived for the cases v1 = v2 = 0 and v1 = v2 = 1. These statistical properties of the nonlinear correlation are verified experimentally in the next section. The mean of Cij(0,0;0,0) is derived as
since pij(m, n) and qij(m, n) are statistically independent, and E[pij (m,n)] E[qij (m,n)]= Npeij (m,n)|.
The variance of Cij(0,0;0,0) is derived as
since Var[pij (m,n)] Var[qij (m,n)]= Npeij (m,n).
In order to derive the mean and the variance of Cij(0,0;1,1), and are defined for simplicity as
The mean of Cij(0,0;1,1) becomes
since and are statistically independent, and [23].
The variance of Cij(0,0;1,1) is derived as
since [23]. It is noted that the mean value of the nonlinear correlation is constant as in Eq. (10), while the variance is inversely proportional to as in Eq. (11).
V. EXPERIMENTAL RESULTS
5.1. Distance Extraction Based on Pixel Disparity
The II recording system is composed of a microlens array and a pickup camera. The pitch of each lenslet is 1.09 mm; the focal length of the lenslet is around 3.0 mm. Figure 3(a) shows the elemental image array of a toy car [25]. The array is composed of 6×6 elemental images, and the size of the array is 390×390 pixels, thus the size of each elemental image is 65×65 pixels. Figure 3(b) shows the elemental image e13, which is located in the 1st column and 3rd row in the x and y directions respectively. Figures 4(a)-4(e) show the samples of the photon-counting elemental image arrays, when Np varies as 1×105 , 5×105, 1×106, 5×106, and 1×107; in each case 1,000 photon-counting elemental image arrays are generated on a computer.
FIG. 3.(a) Elemental image array, (b) one elemental image.
FIG. 4.Photon-counting elemental image array samples when Np is (a) 1× 105, (b) 5× 105, (c) 1× 106, (d) 5× 106, (e) 1× 107.
The pixel disparity in Eq. (3) is extracted using 30 elemental image pairs and the corresponding distance in Eq. (5) as well; Ip and Jp in Eq. (6) are set to 3 and 6 respectively; S1, S2, and S3 are all set to 3, and both E1 and E2 are set to 4, but E3 is set to 3. Table 1 shows the average of dij(li;v1,v2) over 1,000 photon-counting elemental image arrays when v1 and v2 are set to 1 and Np is set to 5×105. Table 2 shows the average, standard deviation, and root-mean-square error (RMSE) of over 1,000 random experiments with varying Np when v1 and v2 are set to 1. As the photon number increases, the standard deviation and RMSE in Table 2 decrease showing that the precision and accuracy are proportional to the photon number. The RMSE of is obtained assuming that the true distance is extracted from the intensity information, which is obtained by replacing Pijand q(i+li)j with eij and e(i+li)j respectively in Eq. (4). Since this distance is deterministic, the standard deviation and RMSE in the rightmost column of Table 2 are zero.
TABLE 1.Average distance extracted from the photon-counting elemental image pairs
TABLE 2.Average, standard deviation, and RMSE of the distance estimate for varying Np
5.2. Experimental Verification of the Nonlinear Correlation
The elemental image shown in Fig . 3(b) is employed to verify the theoretical proofs in Section IV. Therefore, the photon-counting elemental image numbers I and J from Section II are both equal to one. Figure 5 shows the sample photon-counting images of Fig. 3(b). The mean photon number Np varies as 1×102, 1×103, 1×104, 1×105, 1×106, and 1×107. In each case 10,000 experiments are run to calculate the mean and standard deviation of the nonlinear correlation.
FIG. 5.Photon-counting elemental image samples when Np is (a) 1× 102, (b) 1× 103, (c) 1× 104, (d) 1× 105, (e) 1× 106, (f) 1× 107.
Figures 6(a)-6(d) show the means and variances of the nonlinear correlation as derived in Eqs. (7), (8), (10), and (11). The graphs in Figs. 6(a)-6(d) are shown on a log scale. The experimental result in Fig. 6(d) deviates as the photon number increases, since the theoretical proof for the variance in [23] was derived with the assumption of a small number of photocounts.
FIG. 6.Mean and variance of nonlinear correlation results, (a) mean (v1 = v2 = 0), (b) variance (v1 = v2 = 0), (c) mean (v1 = v2 = 1), (d) variance (v1 = v2 = 1).
VI. CONCLUSION
In this study, the longitudinal distance to an object was obtained by means of the nonlinear correlation of photon-counting II. The first- and second-order statistical properties of the nonlinear correlation were theoretically derived and experimentally verified. The method is based on a compact system that requires only a single exposure under passive lighting to obtain 3D information. A fast algorithm has been developed without a reconstruction process. The experimental results confirm that the proposed method can extract the information about the distance to an object at a low light level. This distance is consistent with the results of previous studies [25]. Further investigation of the extraction of distance information for multiple objects under low-light-level conditions remains for future study.
참고문헌
-
Y. Y. Schechner and N. Kiryati, “Depth from defocus vs. stereo: how different really are they?,” in
Proc. 14th International Conf. on Pattern Recognition (Brisbane, Australia, Aug. 1998), pp. 1784-1786 -
S.-W. Lee and N. Kim, “A method for precise depth detection in stereoscopic display,” J. Opt. Soc. Korea
10 , 37-41 (2006). https://doi.org/10.3807/JOSK.2006.10.1.037 -
A. K. Dalmia and M. Trivedi, “Depth extraction using a single moving camera: an integration of depth from motion and depth from stereo,” Mach. Vis. Appl.
9 , 43-55 (1996). https://doi.org/10.1007/BF01214359 -
G. Lippmann, “La photographie integrale,” C. R Acad. Sci.
146 , 446-451 (1908). -
X. Xia, B. Javidi, M. Martinez-Coral, and A. Stern, “Advances in three-dimensional integral imaging: sensing, display, and applications,” App. Opt.
52 , 546-560 (2013). https://doi.org/10.1364/AO.52.000546 -
S.-Gi. Park, B.-S. Song, and S.-W. Min, “Analysis of image visibility in projection-type integral imaging system without diffuser,” J. Opt. Soc. Korea
14 , 121-126 (2010). https://doi.org/10.3807/JOSK.2010.14.2.121 -
M. Cho and D. Shin, “3D Integral imaging display using axially recorded multiple images,” J. Opt. Soc. Korea
17 , 410-414 (2013). https://doi.org/10.3807/JOSK.2013.17.5.410 -
J.-H. Park, J. Kim, and B. Lee, “Three-dimensional optical correlator using a sub-image array,” Opt. Express
13 , 5116-5126 (2005). https://doi.org/10.1364/OPEX.13.005116 -
J.-H. Park, S. Jung, H. Choi, Y. Kim, and B. Lee, “Depth extraction by use of a rectangular lens array and one-dimensional elemental image modification,” Appl. Opt.
43 , 4882-4895 (2004). https://doi.org/10.1364/AO.43.004882 -
D.-C. Hwang, D.-H. Shin, S.-C. Kim, and E.-S. Kim, “Depth extraction of three-dimensional objects in space by the computational integral imaging reconstruction technique,” Appl. Opt.
47 , D128-D135 (2008). https://doi.org/10.1364/AO.47.00D128 -
C. Wu, M. McCormick, A. Aggoun, and S. Y. Kung, “Depth mapping of integral images through viewpoint image extraction with a hybrid disparity analysis algorithm,” J. Display Technol.
4 , 101-108 (2008). https://doi.org/10.1109/JDT.2007.904360 -
S. Yeom, D. Lee, J. Son, and S. H. Kim, “Three-dimensional object reconstruction and recognition using computational integral imaging and statistical pattern analysis,” Jpn. J. Appl. Phys.
48 , 09LB05-1~4 (2009). -
H. Yoo, “Depth extraction for 3D objects via windowing technique in computational integral imaging with a lenslet array,” Opt. Laser Eng.
51 , 912-915 (2015). -
Y. Chen, X. Wang, and Q. Zhang, “Depth extraction method based on the regional feature points in integral imaging,” Optik
127 , 763-768 (2016). https://doi.org/10.1016/j.ijleo.2015.10.171 -
J.-Y. Son, V. V. Saveljev, Y.-J Choi, J.-E. Bahn, S.-K Kim, and H. Choi, “Parameters for designing autostereoscopic imaging systems based on lenticular, parallax barrier, and integral photography plates,” Opt. Eng.
42 , 3326-3333 (2003). https://doi.org/10.1117/1.1615259 -
N. T. Shaked, B. Katz, and J. Rosen, “Review of three-dimensional holographic imaging by multiple-viewpoint-projection based methods,” Appl. Opt.
48 , H120-H136 (2009). https://doi.org/10.1364/AO.48.00H120 -
J. J. Degnan, “Photon counting lidars for airborne and spaceborne topographic mapping,” in
Proc. Applications of Lasers for Sensing and Free Space Communications (LSC) 2010 (San Diego, CA, USA, Feb. 2010), LSWA4. -
E. Hecht,
Optics, 4th ed. , (Addison Wesley, San Francisco, USA, 2001). -
E. Pérez-Cabré, M. Cho, and B. Javidi, “Information authentication using photon-counting double-random-phase encrypted images,” Opt. Lett.
36 , 22-24 (2011). https://doi.org/10.1364/OL.36.000022 -
I. Moon, I. Muniraj, and Bahram Javidi, “3D visualization at low light levels using multispectral photon counting integral imaging,” J. Display Technol.
9 , 51-55 (2013). https://doi.org/10.1109/JDT.2012.2227938 -
G. M. Morris, “Scene matching using photon-limited images,” J. Opt. Soc. Am. A
1 , 482-488 (1984). https://doi.org/10.1364/JOSAA.1.000482 -
E. A. Watson and G. M. Morris, “Comparison of infrared upconversion methods for photon-limited imaging,” J. Appl. Phys.
67 , 6075-6084 (1990). https://doi.org/10.1063/1.345167 -
S. Yeom, B. Javidi, and E. Watson, “Photon counting passive 3D image sensing for automatic target recognition,” Opt. Express
13 , 9310-9330 (2005). https://doi.org/10.1364/OPEX.13.009310 -
S. Yeom, “Stereoscopic photon counting passive sensing for extraction of distance information,” 3D Res.
02 , 03003 (2011). -
S. Yeom, Y. Woo, and W. Baek, “Distance extraction by means of photon-counting passive sensing combined with integral imaging,” J. Opt. Soc. Korea
15 , 357-361 (2011). https://doi.org/10.3807/JOSK.2011.15.4.357 -
J. W. Goodman,
Statistical optics , (John Wiley & Sons, Inc., Boston, USA, 1985).