Journal of Information Processing Systems

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

Infrared and Visible Image Fusion Based on NSCT and Deep Learning

  • Feng, Xin (College of Mechanical Engineering and Key Laboratory of Manufacturing Equipment Mechanism Design and Control, Chongqing Technology and Business University)
  • Received : 2017.07.12
  • Accepted : 2017.10.23
  • Published : 2018.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

An image fusion method is proposed on the basis of depth model segmentation to overcome the shortcomings of noise interference and artifacts caused by infrared and visible image fusion. Firstly, the deep Boltzmann machine is used to perform the priori learning of infrared and visible target and background contour, and the depth segmentation model of the contour is constructed. The Split Bregman iterative algorithm is employed to gain the optimal energy segmentation of infrared and visible image contours. Then, the nonsubsampled contourlet transform (NSCT) transform is taken to decompose the source image, and the corresponding rules are used to integrate the coefficients in the light of the segmented background contour. Finally, the NSCT inverse transform is used to reconstruct the fused image. The simulation results of MATLAB indicates that the proposed algorithm can obtain the fusion result of both target and background contours effectively, with a high contrast and noise suppression in subjective evaluation as well as great merits in objective quantitative indicators.

Keywords

E1JBB0_2018_v14n6_1405_f0001.png 이미지

Fig. 1. Flow diagram of this paper.

E1JBB0_2018_v14n6_1405_f0002.png 이미지

Fig. 2. Ordinary BM and deep BM models: (a) ordinary BM model and (b) three-layer of DBM model is usedin this paper.

E1JBB0_2018_v14n6_1405_f0003.png 이미지

Fig. 3. Iterative process of segmentation algorithm.

E1JBB0_2018_v14n6_1405_f0004.png 이미지

Fig. 4. Results of image region segmentation. (a) Infrared image 1, (b) regional segmentation result, (c) infrared image 2, (d) regional segmentation result, (e) infrared image 3, and (f) regional segmentation result.

E1JBB0_2018_v14n6_1405_f0005.png 이미지

Fig. 5. Comparison of several fusion methods (Equinox faces image). (a) Visible image, (b) infrared image, (c) NSCT method, (d) shearlet method, (e) DBM-DWT method, and (f) proposed method in this paper.

E1JBB0_2018_v14n6_1405_f0006.png 이미지

Fig. 6. Comparison of various fusion methods’ effect (TNO UN Camp image). (a) Visible image, (b) infrared image, (c) NSCT method, (d) shearlet method, (e) DBM-DWT method, and (f) proposed method in this paper.

E1JBB0_2018_v14n6_1405_f0007.png 이미지

Fig. 7. Comparison of various fusion methods’ effect (Bristol Queen’s Road image). (a) Infrared image, (b) visible image, (c) adding noise infrared image, (d) adding noise visible image, (e) NSCT method, (f) shearlet method, (g) DBM-DWT method, and (h) proposed method in this paper.

E1JBB0_2018_v14n6_1405_f0008.png 이미지

Fig. 8. Fusion effect comparison of local area. (a) NSCT method, (b) shearlet method, (c) DBM-DWT method, and (d) proposed method in this paper.

Table 1. Fusion evaluation index

E1JBB0_2018_v14n6_1405_t0001.png 이미지

References

  1. Z. Wang, Y. Ma, and J. Gu, "Multi-focus image fusion using PCNN," Pattern Recognition, vol. 43, no. 6, pp. 2003-2016, 2010. https://doi.org/10.1016/j.patcog.2010.01.011
  2. D. P. Bavirisetti and R. Dhuli, "Fusion of infrared and visible sensor images based on anisotropic diffusion and Karhunen-Loeve transform," IEEE Sensors Journal, vol. 16, no. 1, pp. 203-209, 2016. https://doi.org/10.1109/JSEN.2015.2478655
  3. V. Bhateja, H. Patel, A. Krishn, A. Sahu, and A. Lay-Ekuakille, "Multimodal medical image sensor fusion framework using cascade of wavelet and contourlet transform domains," IEEE Sensors Journal, vol. 15, no. 12, pp. 6783-6790, 2015. https://doi.org/10.1109/JSEN.2015.2465935
  4. Y. Yang, S. Tong, S. Huang, and P. Lin, "Multifocus image fusion based on NSCT and focused area detection," IEEE Sensors Journal, vol. 15, no. 5, pp. 2824-2838, 2015. https://doi.org/10.1109/JSEN.2014.2380153
  5. H. Zheng, C. Zheng, X. Yan, and H. Chen, "Visible and infrared image fusion algorithm based on shearlet transform," Chinese Journal of Scientific Instrument, vol. 33, no. 7, pp. 1613-1619, 2012.
  6. J. Wang, J. Peng, and G. He, X. Feng, and K. Yan, "Fusion method for visible and infrared images based on non-subsampled contourlet transform and sparse representation," Acta Armamentarii, vol. 34, no. 7, pp. 815-820, 2013. https://doi.org/10.3969/j.issn.1000-1093.2013.07.003
  7. T. Gan, S. T. Feng, S. P. Nie, and Z. Q. Zhu, "Image fusion algorithm based on block-DCT in wavelet domain," Acta Physica Sinica, vol. 60, no. 11, article no. 114205, 2011.
  8. W. Kong, L. Zhang, and Y. Lei, "Novel fusion method for visible light and infrared images based on NSST-SF-PCNN," Infrared Physics & Technology, vol. 65, pp. 103-112, 2014. https://doi.org/10.1016/j.infrared.2014.04.003
  9. Y. Shen, X. Feng, and Y. Hou, "Infrared and visible images fusion based on Tetrolet transform," Spectroscopy and Spectral Analysis, vol. 33, no. 6, pp. 1506-1511, 2013. https://doi.org/10.3964/j.issn.1000-0593(2013)06-1506-06
  10. G. Dahl, A. R. Mohamed, and G. E. Hinton, "Phone recognition with the mean-covariance restricted Boltzmann machine," Advances in Neural Information Processing Systems, vol. 23, pp. 469-477, 2010.
  11. R. Salakhutdinov and H. Larochelle, "An efficient learning procedure for deep Boltzmann machines," Neural Computation, vol. 24, pp. 1967-2006, 2012. https://doi.org/10.1162/NECO_a_00311
  12. S. M. Ali Eslami, N. Heess, and J. Winn, "The shape Boltzmann machine: a strong model of object shape," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, 2012, pp. 406-413.
  13. L. J. Latecki, R. Lakamper, and T. Eckhardt, "Shape descriptors for non-rigid shapes with a single closed contour," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Hilton Head Island, SC, 2000, pp. 424-429.
  14. W. Li, Q. Li, W. Gong, and S. Tang, "Total variation blind deconvolution employing split Bregman iteration," Journal of Visual Communication and Image Representation, vol. 23, no. 3, pp. 409-417, 2012. https://doi.org/10.1016/j.jvcir.2011.12.003
  15. A. L. Da Cunha, J. Zhou, and M. N. Do, "The nonsubsampled contourlet transform: theory, design, and applications," IEEE Transactions on Image Processing, vol. 15, no. 10, pp. 3089-3101, 2006. https://doi.org/10.1109/TIP.2006.877507
  16. D. Brunet, E. R. Vrscay, and Z. Wang, "On the mathematical properties of the structural similarity index," IEEE Transactions on Image Processing, vol. 21, no. 4, pp. 1488-1499, 2012 https://doi.org/10.1109/TIP.2011.2173206
  17. Q. Zhang and B. L. Guo, "Fusion of infrared and visible light images based on nonsubsampled contourlet transform," Journal of Infrared and Millimeter Waves (Chinese Edition), vol. 26, no. 6, pp. 476-480. 2007. https://doi.org/10.3321/j.issn:1001-9014.2007.06.018
  18. TNO UN Camp image [Online]. Available: http://www.imagefusion.org/.ftpquota.
  19. X. Li and S. Y. Qin, "Efficient fusion for infrared and visible images based on compressive sensing principle," IET Image Processing, vol. 5, no. 2, pp. 141-147, 2011. https://doi.org/10.1049/iet-ipr.2010.0084