KSII Transactions on Internet and Information Systems (TIIS)

Korean Society for Internet Information (한국인터넷정보학회)
권호 표지
DOI QR코드

DOI QR Code

Multi-focus Image Fusion using Fully Convolutional Two-stream Network for Visual Sensors

  • Received : 2017.05.04
  • Accepted : 2017.11.24
  • Published : 2018.05.31
Download PDF
⟨ Previous Next ⟩

Abstract

We propose a deep learning method for multi-focus image fusion. Unlike most existing pixel-level fusion methods, either in spatial domain or in transform domain, our method directly learns an end-to-end fully convolutional two-stream network. The framework maps a pair of different focus images to a clean version, with a chain of convolutional layers, fusion layer and deconvolutional layers. Our deep fusion model has advantages of efficiency and robustness, yet demonstrates state-of-art fusion quality. We explore different parameter settings to achieve trade-offs between performance and speed. Moreover, the experiment results on our training dataset show that our network can achieve good performance with subjective visual perception and objective assessment metrics.

Keywords

References

  1. Abdipour, M. and M. Nooshyar, "Multi-focus image fusion using sharpness criteria for visual sensor networks in wavelet domain," Computers & Electrical Engineering, vol. 51, p. 74-88, 2016. https://doi.org/10.1016/j.compeleceng.2016.03.011
  2. Petrovic, V.S. and C.S. Xydeas, "Gradient-Based Multiresolution Image Fusion," IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society, vol. 13, no. 2, p. 228-237, 2004. https://doi.org/10.1109/TIP.2004.823821
  3. Piella, G., "A general framework for multiresolution image fusion: from pixels to regions," Information Fusion, vol. 4, no. 4, p. 259-280, 2003. https://doi.org/10.1016/S1566-2535(03)00046-0
  4. Goshtasby, A.A. and S. Nikolov, "Image fusion: Advances in the state of the art," Information Fusion. Vol. 8, no. 2, p. 114-118, 2007. https://doi.org/10.1016/j.inffus.2006.04.001
  5. Zhang, Z. and R.S. Blum, "A categorization of multiscale-decomposition-based image fusion schemes with a performance study for a digital camera application," Proceedings of the IEEE , vol. 87, no. 8, p. 1315-1326, 1999. https://doi.org/10.1109/5.775414
  6. Forster, B., et al., "Complex wavelets for extended depth-of-field: A new method for the fusion of multichannel microscopy images," Microscopy Research & Technique, vol. 65, no.v1-2, p. 33-42, 2004. https://doi.org/10.1002/jemt.20092
  7. Mount, D.M., N.S. Netanyahu, and J.L. Moigne, "Efficient algorithms for robust feature matching, Pattern Recognition, vol. 32, no. 1, p. 17-38, 1999. https://doi.org/10.1016/S0031-3203(98)00086-7
  8. Stockman, G., S. Kopstein, and S. Benett, "Matching Images to Models for Registration and Object Detection via Clustering," IEEE Transactions on Pattern Analysis & Machine Intelligence, vol. 4, no. 3, p. 229-41, 1982.
  9. Benediktsson, J.A. and I. Kanellopoulos, "Classification of multisource and hyperspectral data based on decision fusion," IEEE Transactions on Geoscience & Remote Sensing, 37, no. 3, p. 1367-1377, 1999. https://doi.org/10.1109/36.763301
  10. Wang, Z., et al., "A comparative analysis of image fusion methods," IEEE Transactions on Geoscience & Remote Sensing, vol. 43, no. 6, p. 1391-1402, 2005. https://doi.org/10.1109/TGRS.2005.846874
  11. Cvejic, N., et al., "Region-Based Multimodal Image Fusion using ICA Bases," in Proc. of International Conference on Image Processing, Atlanta, Georgia, USA. p. 1801-1804, 2006.
  12. Li, S., X. Kang, and J. Hu, "Image fusion with guided filtering," IEEE Transactions on Image Processing A Publication of the IEEE Signal Processing Society, vol. 22, no. 7, p. 2864-2875, 2013. https://doi.org/10.1109/TIP.2013.2244222
  13. Li, S., J.T. Kwok, and Y. Wang, "Multifocus image fusion using artificial neural networks," Pattern Recognition Letters, vol. 23, no. 8, p. 985-997, 2002. https://doi.org/10.1016/S0167-8655(02)00029-6
  14. Li, S., et al., "Fusing images with different focuses using support vector machines," Neural Networks IEEE Transactions on, vol. 15, no. 6, p. 1555-61, 2004. https://doi.org/10.1109/TNN.2004.837780
  15. Mamatha, S.G., S.A. Rahim, and C.P. Raj, "Feature-level multi-focus image fusion using neural network and image enhancement, Global," Global Journal of Computer Science & Technology, vol. 12, no. 10-F, 2012.
  16. Rani, C.M.S., P.S.V.S. Rao, and V. Vijayakumar, "Improved Block Based Feature Level Image Fusion Technique Using Contourlet with Neural Network," International Journal of Soft Computing & Engineering, vol. 3, no. 4, 2012.
  17. Jain, V. and H.S. Seung," Natural Image Denoising with Convolutional Networks," in Proc. of Conference on Neural Information Processing Systems, Vancouver, British Columbia, Canada. p. 769-776, 2008.
  18. Dong, C., et al., "Image Super-Resolution Using Deep Convolutional Networks," Pattern Analysis & Machine Intelligence IEEE Transactions on, vol. 38, no. 2, p. 295-307, 2016. https://doi.org/10.1109/TPAMI.2015.2439281
  19. Long, J., E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA. p. 3431-3440, 2015.
  20. Zhong, J., et al, "Image Fusion and Super-Resolution with Convolutional Neural Network," in Proc. of Chinese Conference on Pattern Recognition, Chengdu, China: Springer Singapore, 2016.
  21. Liu, Y., et al., "Multi-focus image fusion with a deep convolutional neural network," Information Fusion, vol. 36, p. 191-207, 2017. https://doi.org/10.1016/j.inffus.2016.12.001
  22. Feichtenhofer, C., A. Pinz, and A. Zisserman, "Convolutional Two-Stream Network Fusion for Video Action Recognition," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA. p. 1933-1941, 2016.
  23. Nair, V. and G.E. Hinton, "Rectified Linear Units Improve Restricted Boltzmann Machines," in Proc. of Icml, p. 807-814, 2015.
  24. Noh, H., S. Hong, and B. Han, "Learning Deconvolution Network for Semantic Segmentation," in Proc. of IEEE International Conference on Computer Vision Santiago, Chile, p. 1520-1528, 2015.
  25. Jia, et al., "Caffe: Convolutional Architecture for Fast Feature Embedding," in Proc. of Eprint Arxiv, p. 675-678, 2014.
  26. Lecun, Y., et al., "Gradient-based learning applied to document recognition," in Proc. of Proceedings of the IEEE, vol. 86, no. 11, p. 2278-2324, 1998. https://doi.org/10.1109/5.726791
  27. Rumelhart, D.E., G.E. Hinton, and R.J. Williams, "Learning representations by back-propagating errors," Nature, vol. 323, no. 6088, p. 533-536, 1986. https://doi.org/10.1038/323533a0
  28. Bottou, L., "Large-Scale Machine Learning with Stochastic Gradient Descent," in Proc. of International Conference on Computational Statistics,Paris France, p. 177-186, 2010.
  29. Kingma, D. and J. Ba, "Adam: A Method for Stochastic Optimization," in Proc. of International Conference for Learning Representations, San Diego, USA, 2015.
  30. Mao, X.J., C. Shen, and Y.B. Yang, "Image Restoration Using Convolutional Auto-encoders with Symmetric Skip Connections," in Proc. of arXiv preprint arXiv, 1606.08921, 2016.
  31. He, K. and J. Sun, Convolutional Neural Networks at Constrained Time Cost, in IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA. p. 5353-5360.
  32. Zeiler, M.D. and R. Fergus, "Visualizing and Understanding Convolutional Networks," in Proc. of European Conference on Computer Vision, Zurich, Switzerland, p. 818-833, 2014.
  33. Glasner, D., S. Bagon, and M. Irani, "Super-resolution from a single image," in Proc. of International Conference on Computer Vision, Kyoto, Japan, p. 349-356, 2009.
  34. Li, H., B.S. Manjunath, and S.K. Mitra, "Multi-sensor image fusion using the wavelet transform," in Proc. of Image Processing, Proceedings. ICIP-94., IEEE International Conference, 1994.
  35. Burt, P.J. and E.H. Adelson, "The Laplacian Pyramid as a Compact Image Code," Readings in Computer Vision, vol. 31, no. 4, p. 671-679, 1987.
  36. Liu, Z., et al., "Image fusion by using steerable pyramid," Pattern Recognition Letters, vol. 22, no. 9, p. 929-939, 2001. https://doi.org/10.1016/S0167-8655(01)00047-2
  37. Toet, A., "Image fusion by a ratio of low-pass pyramid," Pattern Recognition Letters, vol. 9, no. 4, p. 245-253, 1989. https://doi.org/10.1016/0167-8655(89)90003-2
  38. Aslantas V, Toprak A N, "A pixel based multi-focus image fusion method," Optics Communications, vol. 332, no. 4, pp. 350-358, 2014. https://doi.org/10.1016/j.optcom.2014.07.044
  39. Toet, A., J.M. Valeton, and L.J. Van Ruyven, "Merging thermal and visual images by a contrast pyramid," Optical Engineering, vol. 28, no. 7, p. 789-792, 1989.
  40. Hossny, M., S. Nahavandi, and D. Creighton, "Comments on 'Information measure for performance of image fusion," Electronics Letters, vol. 44, no. 18, p. 1066-1067, 2008. https://doi.org/10.1049/el:20081754
  41. Xydeas, C.S. and V. Petrovic, "Objective image fusion performance measure," Military Technical Courier, vol. 36, no. 4, p. 308-309, 2000.
  42. Piella, G. and H. Heijmans, "A new quality metric for image fusion," in Proc. of International Conference on Image Processing, Barcelona, Catalonia, Spain, p. III-173-176, 2003.
  43. Shannon C E., "A Mathematical Theory of Communication[J]," Bell Labs Technical Journal, vol. 27, no. 4, pp. 379-423, 1948. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
  44. J. Deng, A. Berg, S. Satheesh, H. Su, A. Khosla, and L. Fei-Fei, ILSVRC-2012, 2012.
  45. Perronnin F, AVA, "A large-scale database for aesthetic visual analysis," in Proc. of IEEE Conference on Computer Vision and Pattern Recognition, IEEE Computer Society, pp. 2408-2415, 2012.
  46. Li H, Chai Y, Li Z., "A new fusion scheme for multifocus images based on focused pixels detection," Machine Vision & Applications, vol. 24, no. 6, 1167-1181, 2013. https://doi.org/10.1007/s00138-013-0502-4
  47. Li H, Chai Y, Li Z., "Multi-focus image fusion based on nonsubsampled contourlet transform and focused regions detection," Optik - International Journal for Light and Electron Optics, vol. 124, no. 1, 40-51, 2013. https://doi.org/10.1016/j.ijleo.2011.11.088
  48. Gao G, Xu L, Feng D.,"Multi-focus image fusion based on non-subsampled shearlet transform," Iet Image Processing, vol. 7, no. 6, pp. 633-639, 2013. https://doi.org/10.1049/iet-ipr.2012.0558
  49. Haghighat M B A, Aghagolzadeh A, Seyedarabi H., "Multi-focus image fusion for visual sensor networks in DCT domain," Computers & Electrical Engineering, vol. 37, no. 5, pp. 789-797, 2011. https://doi.org/10.1016/j.compeleceng.2011.04.016

Cited by

  1. Multifocus Image Fusion Using Wavelet-Domain-Based Deep CNN vol.2019, pp.None, 2019, https://doi.org/10.1155/2019/4179397
  2. Ensemble of CNN for multi-focus image fusion vol.51, pp.None, 2019, https://doi.org/10.1016/j.inffus.2019.02.003
  3. CNNs hard voting for multi-focus image fusion vol.11, pp.4, 2020, https://doi.org/10.1007/s12652-019-01199-0
  4. DCNN Optimization Using Multi-Resolution Image Fusion vol.14, pp.11, 2020, https://doi.org/10.3837/tiis.2020.11.003