Journal of Information Processing Systems

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

Convolutional Neural Network Based Multi-feature Fusion for Non-rigid 3D Model Retrieval

  • Zeng, Hui (Beijing Engineering Research Center of Industrial Spectrum Imaging, School of Automation and Electrical Engineering, University of Science and Technology Beijing) ;
  • Liu, Yanrong (Beijing Engineering Research Center of Industrial Spectrum Imaging, School of Automation and Electrical Engineering, University of Science and Technology Beijing) ;
  • Li, Siqi (Beijing Engineering Research Center of Industrial Spectrum Imaging, School of Automation and Electrical Engineering, University of Science and Technology Beijing) ;
  • Che, JianYong (The Tiantan Park Management Office) ;
  • Wang, Xiuqing (Vocational & Technical Institute, Hebei Normal University)
  • Received : 2017.05.03
  • Accepted : 2017.07.27
  • Published : 2018.02.28
Download PDF
⟨ Previous Next ⟩

Abstract

This paper presents a novel convolutional neural network based multi-feature fusion learning method for non-rigid 3D model retrieval, which can investigate the useful discriminative information of the heat kernel signature (HKS) descriptor and the wave kernel signature (WKS) descriptor. At first, we compute the 2D shape distributions of the two kinds of descriptors to represent the 3D model and use them as the input to the networks. Then we construct two convolutional neural networks for the HKS distribution and the WKS distribution separately, and use the multi-feature fusion layer to connect them. The fusion layer not only can exploit more discriminative characteristics of the two descriptors, but also can complement the correlated information between the two kinds of descriptors. Furthermore, to further improve the performance of the description ability, the cross-connected layer is built to combine the low-level features with high-level features. Extensive experiments have validated the effectiveness of the designed multi-feature fusion learning method.

Keywords

E1JBB0_2018_v14n1_176_f0001.png 이미지

Fig. 1. The HKS multi-scale shape distributions of human models (a) and ant models (b).

E1JBB0_2018_v14n1_176_f0002.png 이미지

Fig. 2. The WKS multi-scale shape distributions of human models (a) and ant models (b).

E1JBB0_2018_v14n1_176_f0003.png 이미지

Fig. 3. Our multi-feature fusion network, where “conv”, ”pool”, “cc”, ”fc1” and “fc2” represent “convolutional”,“mean-pooling”, “cross-connected layers”, “first fully-connected layer of the HKS/WKS feature” and“second fully-connected layer of the HKS/WKS feature” respectively.

E1JBB0_2018_v14n1_176_f0004.png 이미지

Fig. 4. Example 3D models of the McGill 3D shape benchmark.

Table 1. Description of each conventional neural network

E1JBB0_2018_v14n1_176_t0001.png 이미지

Table 2. Retrieval results compared with single-feature based methods

E1JBB0_2018_v14n1_176_t0002.png 이미지

Table 3. Retrieval results compared with other methods

E1JBB0_2018_v14n1_176_t0003.png 이미지

References

  1. Y. Matsuda, N. Miura, A. Nagasaka, H. Kiyomizu, and T. Miyatake, "Finger-vein authentication based on deformation-tolerant feature-point matching," Machine Vision and Applications, vol. 27, no. 2, pp. 237-250, 2016. https://doi.org/10.1007/s00138-015-0745-3
  2. A. Frome, D. Huber, R. Kolluri, T. Bulow, and J. Malik, "Recognizing objects in range data using regional point descriptors," in Proceedings of the European Conference on Computer Vision, Prague, Czech Republic, 2004, pp. 224-237.
  3. H. Chen and B. Bhanu, "3D free-form object recognition in range images using local surface patches," Pattern Recognition Letters, vol. 28, no. 10, pp. 1252-1262, 2007. https://doi.org/10.1016/j.patrec.2007.02.009
  4. A. Flint, A. Dick, and A. Van den Hengel, "Local 3D structure recognition in range images," IET Computer Vision, vol. 2, no. 4, pp. 208-217, 2008 https://doi.org/10.1049/iet-cvi:20080037
  5. A. E. Johnson and M. Hebert, "Surface matching for object recognition in complex three-dimensional scenes," Image and Vision Computing, vol. 16, no. 9-10, pp. 635-651, 1998. https://doi.org/10.1016/S0262-8856(98)00074-2
  6. A. E. Johnson and M. Hebert, "Using spin images for efficient object recognition in cluttered 3D scenes," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 21, no. 5, pp. 443-449, 1999.
  7. G. Hetzel, B. Leibe, P. Levi, and B. Schiele, "3D object recognition from range images using local feature histograms," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Kauai, HI, 2001, pp. 394-399.
  8. Y. Guo, F. Sohel, M. Bennamoun, M. Lu, and J. Wan, "Rotational projection statistics for 3D local surface description and object recognition," International Journal of Computer Vision, vol. 105, no. 1, pp. 63-86, 2013. https://doi.org/10.1007/s11263-013-0627-y
  9. R. B. Rusu, N. Blodow, and M. Beetz, "Fast point feature histograms (FPFH) for 3D registration," in Proceedings of the IEEE International Conference on Robotics and Automation, Kobe, Japan, 2009, pp. 3212- 3217.
  10. F. Tombari, S. Salti, and L. Di Stefano, "Unique signatures of histograms for local surface description," in Proceedings of the European Conference on Computer Vision, Crete, Greece, 2010, pp. 356-369.
  11. S. Salti, F. Tombari, and L. Di Stefano, "SHOT: unique signatures of histograms for surface and texture description," Computer Vision & Image Understanding, vol. 125, no. 8, pp. 251-264, 2014. https://doi.org/10.1016/j.cviu.2014.04.011
  12. J. Sun, M. Ovsjanikov, and L. Guibas, "A concise and provably informative multi‐scale signature based on heat diffusion," Computer Graphics Forum, vol. 28, no. 5, pp. 1383-1392, 2009. https://doi.org/10.1111/j.1467-8659.2009.01515.x
  13. A. M. Bronstein, M. M. Bronstein, R. Kimmel, M. Mohmoudi, and G. Sapiro, "A Gromov-Hausdorff framework with diffusion geometry for topologically-robust non-rigid shape matching," International Journal of Computer Vision, vol. 89, no. 2-3, pp. 266-286, 2010. https://doi.org/10.1007/s11263-009-0301-6
  14. Z. Lian, A. Godil, B. Bustos, M. Daoudi, J. Hermans, S. Kawamura, et al., "A comparison of methods for non-rigid 3D shape retrieval," Pattern Recognition, vol. 46, no. 1, pp. 449-461, 2013. https://doi.org/10.1016/j.patcog.2012.07.014
  15. A. M. Bronstein, M. M. Bronstein, L. J. Guibas, and M. Ovsjanikow, "Shape Google: geometric words and expressions for invariant shape retrieval," ACM Transactions on Graphics, vol. 30, no. 1, pp. 623-636, 2011.
  16. M. Aubry, U. Schlickewei, and D. Cremers, "The wave kernel signature: a quantum mechanical approach to shape analysis," in Proceedings of the IEEE International Conference on Computer Vision Workshops, Barcelona, Spain, 2011, pp. 1626-1633.
  17. Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, "Gradient-based learning applied to document recognition," Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
  18. J. Xie, G. Dai, F. Zhu, E. K. Wong, and Y. Fang, "DeepShape: deep-learned shape descriptor for 3D shape retrieval," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, no. 7, pp. 1335-1345, 2017. https://doi.org/10.1109/TPAMI.2016.2596722
  19. R. Osada, T. Funkhouser, B. Chazelle, and D. Dokin, "Shape distributions," ACM Transactions on Graphics, vol. 21, no. 4, pp. 807-832, 2002. https://doi.org/10.1145/571647.571648
  20. A. Wang, J. Lu, J. Cai, T. J. Cham, and G. Wang "Large-margin multi-modal deep learning for RGB-D object recognition," IEEE Transactions on Multimedia, vol. 17, no. 11, pp. 1887-1898, 2015. https://doi.org/10.1109/TMM.2015.2476655
  21. H. Tabia, H. Laga, D. Picard, and P. H. Gosselin, "Covariance descriptors for 3D shape matching and retrieval," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Columbus, OH, 2014, pp. 4185-4192.
  22. A. Agathos, I. Pratikakis, P. Papadakis, S. J. Perantonis, P. N. Azariadis, and N. S. Sapidis, "Retrieval of 3D articulated objects using a graph-based representation," in Proceedings of the Eurographics Workshop on 3D Object Retrieval, Munich, Germany, 2009, pp. 29-36.
  23. H. Tabia, D. Picard, H. Laga, and P. H. Gosselin, "Compact vectors of locally aggregated tensors for 3D shape retrieval," in Proceedings of the Eurographics Workshop on 3D Object Retrieval, Girona, Spain, 2013, pp. 17-24.
  24. P. Papadakis, I. Pratikakis, T. Theoharis, G. Passalis, and S. Perantonis, "3D object retrieval using an efficient and compact hybrid shape descriptor," in Proceedings of the Eurographics Workshop on 3D Object Retrieval, Crete, Greece, 2008, pp. 9-16.
  25. G. Lavoue, "Combination of bag-of-words descriptors for robust partial shape retrieval," The Visual Computer, vol. 28, no. 9, pp. 931-942, 2012. https://doi.org/10.1007/s00371-012-0724-x
  26. Z. Lian, A. Godil, T. Fabry, T. Furuya, J. Hermans, R. Ohbuchi, et al., "SHREC'10 Track: non-rigid 3D shape retrieval," Proceedings of the Eurographics Workshop on 3D Object Retrieval, Zurich, Switzerland, 2015, pp. 107-120.