Journal of Information Processing Systems

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

Face Sketch Synthesis Based on Local and Nonlocal Similarity Regularization

  • Tang, Songze (Dept. of Criminal Science and Technology, Nanjing Forest Police College) ;
  • Zhou, Xuhuan (Dept. of Criminal Science and Technology, Nanjing Forest Police College) ;
  • Zhou, Nan (Dept. of Criminal Science and Technology, Nanjing Forest Police College) ;
  • Sun, Le (School of Computer and Software, Nanjing University of Information Science and Technology) ;
  • Wang, Jin (School of Computer & Communication Engineering, Changsha University of Science & Technology)
  • Received : 2018.05.21
  • Accepted : 2019.10.14
  • Published : 2019.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Face sketch synthesis plays an important role in public security and digital entertainment. In this paper, we present a novel face sketch synthesis method via local similarity and nonlocal similarity regularization terms. The local similarity can overcome the technological bottlenecks of the patch representation scheme in traditional learning-based methods. It improves the quality of synthesized sketches by penalizing the dissimilar training patches (thus have very small weights or are discarded). In addition, taking the redundancy of image patches into account, a global nonlocal similarity regularization is employed to restrain the generation of the noise and maintain primitive facial features during the synthesized process. More robust synthesized results can be obtained. Extensive experiments on the public databases validate the generality, effectiveness, and robustness of the proposed algorithm.

Keywords

Acknowledgement

This paper is supported in part by the National Natural Science Foundation of China (No. 61702269, 61971233, and 61671339), in part by the Natural Science Foundation of Jiangsu Province (No. BK20171074), and the Fundamental Research Fundsfor the Central Universities at Nanjing Forest Police College (No. LGZD201702).

References

  1. R. G. Uhl and N. da Vitoria Lobo, "A framework for recognizing a facial image from a police sketch," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, CA, 1996, pp. 586-593.
  2. N. Wang, X. Gao, L. Sun, and J. Li, "Anchored neighborhood index for face sketch synthesis," IEEE Transactions on Circuits and Systems for Video Technology, vol. 28, no. 9, pp. 2154-2163, 2017. https://doi.org/10.1109/tcsvt.2017.2709465
  3. J. Wang, H. Bao, W. Zhou, Q. Peng, and Y. Xu, "Automatic image-based pencil sketch rendering," Journal of Computer Science and Technology, vol. 17, no. 3, pp. 347-355, 2002. https://doi.org/10.1007/BF02947313
  4. X. Li and X. Cao, "A simple framework for face photo-sketch synthesis," Mathematical Problems in Engineering, vol. 2012, article no. 910719, 2016.
  5. Y. LeCun, Y. Bengio, and G. Hinton, "Deep learning," Nature, vol. 521, no. 7553, pp. 436-444, 2015. https://doi.org/10.1038/nature14539
  6. P. Isola, J. Y. Zhu, T. Zhou, and A. A. Efros, "Image-to-image translation with conditional adversarial networks," in Proceedings of the IEEE conference on Computer Vision And Pattern Recognition, Honolulu, HI, 2017, pp. 1125-1134.
  7. C. Ledig, L. Theis, F. Huszar, J. Caballero, A. Cunningham, A. Acosta, et al., "Photo-realistic single image super-resolution using a generative adversarial network," in Proceedings of the IEEE Conference on Computer Vision And Pattern Recognition, Honolulu, HI, 2017, pp. 4681-4690.
  8. C. Dong, C. C. Loy, K. He, and X. Tang, "Image super-resolution using deep convolutional networks," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38, no. 2, pp. 295-307, 2015. https://doi.org/10.1109/TPAMI.2015.2439281
  9. L. Zhang, L. Lin, X. Wu, S. Ding, and L. Zhang, "End-to-end photo-sketch generation via fully convolutional representation learning," in Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, Shanghai, China, 2015, pp. 627-634.
  10. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, "Generative adversarial nets," Advances in Neural Information Processing Systems, vol. 27, pp. 2672-2680, 2014.
  11. A. Dosovitskiy and T. Brox, "Generating images with perceptual similarity metrics based on deep networks," Advances in Neural Information Processing Systems, vol. 29, pp. 658-666, 2016.
  12. N. Wang, W. Zha, J. Li, and X. Gao, "Back projection: an effective postprocessing method for GAN-based face sketch synthesis," Pattern Recognition Letters, vol. 107, pp. 59-65, 2018. https://doi.org/10.1016/j.patrec.2017.06.012
  13. X. Tang and X. Wang, "Face photo recognition using sketch," in Proceedings of IEEE International Conference on Image Processing, Rochester, NY, 2002.
  14. Q. Liu, X. Tang, H. Jin, H. Lu, and S. Ma, "A nonlinear approach for face sketch synthesis and recognition," in Proceedings of 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, CA, 2005, pp. 1005-1010.
  15. S. T. Roweis and L. K. Saul, "Nonlinear dimensionality reduction by locally linear embedding," Science, vol. 290, no. 5500, pp. 2323-2326, 2000. https://doi.org/10.1126/science.290.5500.2323
  16. Y. Song, L. Bao, Q. Yang, and M. H. Yang, "Real-time exemplar-based face sketch synthesis," in Computer Vision-ECCV 2014. Cham: Springer, 2014, pp. 800-813.
  17. N. Wang, X. Gao, and J. Li, "Random sampling for fast face sketch synthesis," Pattern Recognition, vol. 76, pp. 215-227, 2018. https://doi.org/10.1016/j.patcog.2017.11.008
  18. S. Tang, L. Xiao, P. Liu, L. Huang, N. Zhou, and Y. Xu, "Pansharpening via sparse regression," Optical Engineering, vol. 56, no. 9, article no. 093105, 2017.
  19. J. Wright, Y. Ma, J. Mairal, G. Sapiro, T. S. Huang, and S. Yan, "Sparse representation for computer vision and pattern recognition," Proceedings of the IEEE, vol. 98, no. 6, pp. 1031-1044, 2010. https://doi.org/10.1109/JPROC.2010.2044470
  20. R. Qi, Y. Zhang, and H. Li, "Block sparse signals recovery via block backtracking-based matching pursuit method," Journal of Information Processing Systems, vol. 13, no. 2, pp. 360-369, 2017. https://doi.org/10.3745/JIPS.04.0030
  21. L. Chang, M. Zhou, Y. Han, and X. Deng, "Face sketch synthesis via sparse representation," in Proceedings of the 20th International Conference on Pattern Recognition, Istanbul, Turkey, 2010, pp. 2146-2149.
  22. X. Gao, N. Wang, D. Tao, and X. Li, "Face sketch-photo synthesis and retrieval using sparse representation," IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 8, pp. 1213-1226, 2012. https://doi.org/10.1109/TCSVT.2012.2198090
  23. S. Wang, L. Zhang, Y. Liang, and Q. Pan, "Semi-coupled dictionary learning with applications to image super-resolution and photo-sketch synthesis," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, 2012, pp. 2216-2223.
  24. X. Gao, J. Zhong, J. Li, and C. Tian, "Face sketch synthesis algorithm based on E-HMM and selective ensemble," IEEE Transactions on Circuits and Systems for Video Technology, vol. 18, no. 4, pp. 487-496, 2008. https://doi.org/10.1109/TCSVT.2008.918770
  25. X. Wang and X. Tang, "Face photo-sketch synthesis and recognition," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 11, pp. 1955-1967, 2008. https://doi.org/10.1109/TPAMI.2008.222
  26. H. Zhou, Z. Kuang, and K. Y. K. Wong, "Markov weight fields for face sketch synthesis," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, 2012, pp. 1091-1097.
  27. W. Zhang, X. Wang, and X. Tang, "Lighting and pose robust face sketch synthesis," in Computer Vision- ECCV2010. Heidelberg: Springer, 2010, pp. 420-433.
  28. C. Peng, X. Gao, N. Wang, D. Tao, X. Li, and J. Li, "Multiple representations-based face sketch-photo synthesis," IEEE Transactions on Neural Networks and Learning Systems, vol. 27, no. 11, pp. 2201-2215, 2016. https://doi.org/10.1109/TNNLS.2015.2464681
  29. C. Li, S. Zhao, K. Xiao, and Y. Wang, "Face recognition based on the combination of enhanced local texture feature and DBN under complex illumination conditions," Journal of Information Processing Systems, vol. 14, no. 1, pp. 191-204, 2018. https://doi.org/10.3745/JIPS.04.0060
  30. A. Buades, B. Coll, and J. M. Morel, "A non-local algorithm for image denoising," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, San Diego, CA, 2005, pp. 60-65.
  31. Z. Zha, X. Zhang, Q. Wang, Y. Bai, Y. Chen, L. Tang, and X. Liu, "Group sparsity residual constraint for image denoising with external nonlocal self-similarity prior," Neurocomputing, vol. 275, pp. 2294-2306, 2018. https://doi.org/10.1016/j.neucom.2017.11.004
  32. D. Liu, B. Wen, Y. Fan, C. C. Loy, and T. S. Huang, "Non-local recurrent network for image restoration," Advances in Neural Information Processing Systems, vol. 31, pp. 1673-1682, 2018.
  33. W. Zhang, X. Wang, and X. Tang, "Coupled information-theoretic encoding for face photo-sketch recognition," in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, 2011, pp. 513-520.
  34. A. Martinez and R. Benavente, "The AR face database," 1998; http://www.cat.uab.cat/Public/Publications/1998/MaB1998/CVCReport24.pdf.
  35. K. Messer, J. Matas, J. Kittler, J. Luettin, and G. Maitre, "XM2VTSDB: the extended M2VTS database," in Proceedings of the 2nd International Conference on Audio and Video-Based Biometric Person Authentication, Washington, DC, 1999.
  36. P. J. Phillips, H. Moon, S. A. Rizvi, and P. J. Rauss, "The FERET evaluation methodology for face-recognition algorithms," IEEE Transactions on Pattern Analysis and Machine intelligence, vol. 22, no. 10, pp. 1090-1104, 2000. https://doi.org/10.1109/34.879790
  37. L. Zhang, L. Zhang, X. Mou, and D. Zhang, "FSIM: a feature similarity index for image quality assessment," IEEE Transactions on Image Processing, vol. 20, no. 8, pp. 2378-2386, 2011. https://doi.org/10.1109/TIP.2011.2109730