Journal of the Korea Computer Graphics Society (한국컴퓨터그래픽스학회논문지)

Korea Computer Graphics Society (한국컴퓨터그래픽스학회)
권호 표지
DOI QR코드

DOI QR Code

Deep Learning based Color Restoration of Corrupted Black and White Facial Photos

딥러닝 기반 손상된 흑백 얼굴 사진 컬러 복원

  • Woo, Shin Jae (Dept. of Convergence Software, Hallym University) ;
  • Kim, Jong-Hyun (School of Software Application, Kangnam University) ;
  • Lee, Jung (Dept. of Convergence Software, Hallym University) ;
  • Song, Chang-Germ (Dept. of Convergence Software, Hallym University) ;
  • Kim, Sun-Jeong (Dept. of Convergence Software, Hallym University)
  • 신재우 (한림대학교 융합소프트웨어학과) ;
  • 김종현 (강남대학교 소프트웨어응용학부) ;
  • 이정 (한림대학교 융합소프트웨어학과) ;
  • 송창근 (한림대학교 융합소프트웨어학과) ;
  • 김선정 (한림대학교 융합소프트웨어학과)
  • Received : 2018.01.31
  • Accepted : 2018.05.28
  • Published : 2018.06.01
Download PDF
⟨ Previous Next ⟩

Abstract

In this paper, we propose a method to restore corrupted black and white facial images to color. Previous studies have shown that when coloring damaged black and white photographs, such as old ID photographs, the area around the damaged area is often incorrectly colored. To solve this problem, this paper proposes a method of restoring the damaged area of input photo first and then performing colorization based on the result. The proposed method consists of two steps: BEGAN (Boundary Equivalent Generative Adversarial Networks) model based restoration and CNN (Convolutional Neural Network) based coloring. Our method uses the BEGAN model, which enables a clearer and higher resolution image restoration than the existing methods using the DCGAN (Deep Convolutional Generative Adversarial Networks) model for image restoration, and performs colorization based on the restored black and white image. Finally, we confirmed that the experimental results of various types of facial images and masks can show realistic color restoration results in many cases compared with the previous studies.

본 논문에서는 손상된 흑백 얼굴 이미지를 컬러로 복원하는 방법을 제안한다. 기존 연구에서는 오래된 증명사진처럼 손상된 흑백 사진에 컬러화 작업을 하면 손상된 영역 주변이 잘못 색칠되는 경우가 있었다. 이와 같은 문제를 해결하기 위해 본 논문에서는 입력받은 사진의 손상된 영역을 먼저 복원한 후 그 결과를 바탕으로 컬러화를 수행하는 방법을 제안한다. 본 논문의 제안 방법은 BEGAN(Boundary Equivalent Generative Adversarial Networks) 모델 기반 복원과 CNN(Convolutional Neural Network) 기반 컬러화의 두 단계로 구성된다. 제안하는 방법은 이미지 복원을 위해 DCGAN(Deep Convolutional Generative Adversarial Networks) 모델을 사용한 기존 방법들과 달리 좀 더 선명하고 고해상도의 이미지 복원이 가능한 BEGAN 모델을 사용하고, 그 복원된 흑백 이미지를 바탕으로 컬러화 작업을 수행한다. 최종적으로 다양한 유형의 얼굴 이미지와 마스크에 대한 실험 결과를 통해 기존 연구에 비해 많은 경우에 사실적인 컬러 복원 결과를 보여줄 수 있음을 확인하였다.

Keywords

Acknowledgement

Supported by : 한국연구재단

References

  1. Radford, A., Metz, L., & Chintala, S., "Unsupervised representation learning with deep convolutional generative adversarial networks," arXiv preprint, arXiv:1511.06434, 2015.
  2. Yeh, R. A., Chen, C., Lim, T. Y., Schwing, A. G., Hasegawa-Johnson, M., & Do, M. N., "Semantic image inpainting with deep generative models," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5485-5493, 2017.
  3. Berthelot, D., Schumm, T., & Metz, L., "Began: Boundary equilibrium generative adversarial networks," arXiv preprint, arXiv: 1703.10717, 2017.
  4. LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W., & Jackel, L. D., "Backpropagation applied to handwritten zip code recognition," Neural computation, Vol. 1, No. 4, pp. 541-551, 1989. https://doi.org/10.1162/neco.1989.1.4.541
  5. Bertalmio, M., Sapiro, G., Caselles, V., & Ballester, G., "Image inpainting," Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pp. 417-424, 2000.
  6. Efros, A. A., and Leung T. K. ''Texture synthesis by non-parametric sampling," Proceedings of the Seventh IEEE International Conference on Computer Vision, Vol. 2, pp. 1033-1038, 1999.
  7. Hays, J., & Efros, A. A., "Scene completion using millions of photographs," ACM Transactions on Graphics, Vol. 26, No. 3, p. 4, 2007. https://doi.org/10.1145/1276377.1276382
  8. Pathak, D., krahenbuhl, P., Donahue, J. Darrell, T., & Efros, A. A., "Context encoders: Feature learning by inpainting," Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536-2544, 2016.
  9. Levin, A., Lischinski, D., & Weiss, Y., "Colorization using optimization," ACM transactions on graphics, Vol. 23, No. 3, pp. 689-694, 2004. https://doi.org/10.1145/1015706.1015780
  10. Welsh, T., Ashikhmin, M., & Mueller, K., "Transferring color to greyscale image," ACM Transactions on Graphics, Vol. 21, No. 3, pp. 277-280, 2002.
  11. Iizuka, S., Simo-Serra, E., & Ishikawa, H., "Let there be color!: joint end-to-end learning of global and local image priors for automatic linage colorization with simultaneous classification," ACM Transactions on Graphics, Vol. 35, No. 4, p. 110, 2016.
  12. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., & Bengio, Y., "Generative adversarial nets," Advances in neural information processing systems, pp. 2672-2680, 2014.
  13. Bengio, Y., "Learning deep architectures for AI," Foundations and trends(R) in Machine Learning, Vol. 2, No. 1, pp. 1-127, 2009. https://doi.org/10.1561/2200000006
  14. Arjovsky, M., Chintala, S., & Bottou, L., "Wasserstein gan," arXiv preprint, arXiv: 1701.07875, 2017.
  15. Perez, P., Gangnet, M., & Blake, A., "Poisson linage editing," ACM Transactions on graphics, Vol. 22, No. 3, pp. 313-318, 2003.
  16. Noh, H., Hong, S., & Han, B., "Learning deconvolution network for semantic segmentation," Proceedings of the IEEE International Conference on Computer Vision, pp. 1520-1528, 2015.
  17. Sun, T., Chen, Y., Wang, X., & Tang, X., "Deep learning face representation by joint identification-verification," Advances in neural information proceesing systems, pp. 1988-1996, 2014.
  18. Viola, P ., & Jones, M., "Rapid object detection using a boosted cascade of simple features," Proceedings of the 200I IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Vol. 1, pp. 1-1, 2001.
  19. Huang, G. B., Ramesh, M., Berg, T., & Learned-Miller, E., "Labeled faces in the wild: A database for studying face recognition in unconstrained environments," Technical Report 07-49, University of Massachusetts, Amherst, Vol. 1, No. 2, p. 3, 2007.