Journal of the Korean Society for Industrial and Applied Mathematics

Korean Society for Industrial and Applied Mathematics (한국산업응용수학회)
권호 표지
DOI QR코드

DOI QR Code

GRAYSCALE IMAGE COLORIZATION USING A CONVOLUTIONAL NEURAL NETWORK

  • JWA, MINJE (DEPARTMENT OF COMPUTATIONAL SCIENCE AND TECHNOLOGY, SEOUL NATIONAL UNIVERSITY) ;
  • KANG, MYUNGJOO (DEPARTMENT OF MATHEMATICAL SCIENCES, SEOUL NATIONAL UNIVERSITY)
  • Received : 2021.05.31
  • Accepted : 2021.06.24
  • Published : 2021.06.25
Download PDF
⟨ Previous Next ⟩

Abstract

Image coloration refers to adding plausible colors to a grayscale image or video. Image coloration has been used in many modern fields, including restoring old photographs, as well as reducing the time spent painting cartoons. In this paper, a method is proposed for colorizing grayscale images using a convolutional neural network. We propose an encoder-decoder model, adapting FusionNet to our purpose. A proper loss function is defined instead of the MSE loss function to suit the purpose of coloring. The proposed model was verified using the ImageNet dataset. We quantitatively compared several colorization models with ours, using the peak signal-to-noise ratio (PSNR) metric. In addition, to qualitatively evaluate the results, our model was applied to images in the test dataset and compared to images applied to various other models. Finally, we applied our model to a selection of old black and white photographs.

Keywords

Acknowledgement

Myungjoo Kang was supported by the NRF grant [2015R1A5A1009350][2021R1A2C3010887] and the ICT R&D program of MSIT/IITP[1711117093].

References

  1. H. Bahng, S. Yoo, W. Cho, D. Keetae Park, Z. Wu, X. Ma, and J. Choo. Coloring with words: Guiding image colorization through text-based palette generation. In Proceedings of the european conference on computer vision (eccv), pages 431-447, 2018.
  2. F. Baldassarre, D. G. MorIn, and L. Rodes-Guirao. Deep koalarization: Image colorization using cnns and ' inception-resnet-v2. arXiv preprint arXiv:1712.03400, 2017.
  3. A. Bugeau, V.-T. Ta, and N. Papadakis. Variational exemplar-based image colorization. IEEE Transactions on Image Processing, 23(1):298-307, 2013. https://doi.org/10.1109/TIP.2013.2288929
  4. Z. Cheng, Q. Yang, and B. Sheng. Deep colorization. In Proceedings of the IEEE International Conference on Computer Vision, pages 415-423, 2015.
  5. A. Y.-S. Chia, S. Zhuo, R. K. Gupta, Y.-W. Tai, S.-Y. Cho, P. Tan, and S. Lin. Semantic colorization with internet images. ACM Transactions on Graphics (TOG), 30(6):1-8, 2011.
  6. A. Deshpande, J. Lu, M.-C. Yeh, M. Jin Chong, and D. Forsyth. Learning diverse image colorization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6837-6845, 2017.
  7. I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
  8. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
  9. Y.-C. Huang, Y.-S. Tung, J.-C. Chen, S.-W. Wang, and J.-L. Wu. An adaptive edge detection based colorization algorithm and its applications. In Proceedings of the 13th annual ACM international conference on Multimedia, pages 351-354, 2005.
  10. S. Iizuka, E. Simo-Serra, and H. Ishikawa. Let there be color! joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Transactions on Graphics (ToG), 35(4):1-11, 2016.
  11. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
  12. P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125-1134, 2017.
  13. S. Kang, J. Choo, and J. Chang. Consistent comic colorization with pixel-wise background classification. In NIPS'17 Workshop on Machine Learning for Creativity and Design, 2017.
  14. D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  15. A. Levin, D. Lischinski, and Y. Weiss. Colorization using optimization. In ACM SIGGRAPH 2004 Papers, pages 689-694. 2004.
  16. Q. Luan, F. Wen, D. Cohen-Or, L. Liang, Y.-Q. Xu, and H.-Y. Shum. Natural image colorization. In Proceedings of the 18th Eurographics conference on Rendering Techniques, pages 309-320, 2007.
  17. K. Nazeri, E. Ng, and M. Ebrahimi. Image colorization using generative adversarial networks. In International conference on articulated motion and deformable objects, pages 85-94. Springer, 2018.
  18. T. M. Quan, D. G. Hildebrand, and W.-K. Jeong. Fusionnet: A deep fully residual convolutional neural network for image segmentation in connectomics. arXiv preprint arXiv:1612.05360, 2016.
  19. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252, 2015. https://doi.org/10.1007/s11263-015-0816-y
  20. Y.-W. Tai, J. Jia, and C.-K. Tang. Local color transfer via probabilistic segmentation by expectation-maximization. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 1, pages 747-754. IEEE, 2005.
  21. P. Vitoria, L. Raad, and C. Ballester. Chromagan: Adversarial picture colorization with semantic class distribution. In The IEEE Winter Conference on Applications of Computer Vision, pages 2445-2454, 2020.
  22. T. Welsh, M. Ashikhmin, and K. Mueller. Transferring color to greyscale images. In Proceedings of the 29th annual conference on Computer graphics and interactive techniques, pages 277-280, 2002.
  23. Y. Xiao, P. Zhou, and Y. Zheng. Interactive deep colorization with simultaneous global and local inputs. arXiv preprint arXiv:1801.09083, 2018.
  24. L. Yatziv and G. Sapiro. Fast image and video colorization using chrominance blending. IEEE transactions on image processing, 15(5):1120-1129, 2006. https://doi.org/10.1109/TIP.2005.864231
  25. S. Yoo, H. Bahng, S. Chung, J. Lee, J. Chang, and J. Choo. Coloring with limited data: Few-shot colorization via memory augmented networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 11283-11292, 2019.
  26. R. Zhang, P. Isola, and A. A. Efros. Colorful image colorization. In European conference on computer vision, pages 649-666. Springer, 2016.
  27. R. Zhang, J.-Y. Zhu, P. Isola, X. Geng, A. S. Lin, T. Yu, and A. A. Efros. Real-time user-guided image colorization with learned deep priors. arXiv preprint arXiv:1705.02999, 2017.
  28. J. Zhao, J. Han, L. Shao, and C. G. Snoek. Pixelated semantic colorization. International Journal of Computer Vision, pages 1-17, 2019.
  29. C. Zou, H. Mo, R. Du, X. Wu, C. Gao, and H. Fu. Lucss: Language-based user-customized colourization of scene sketches. arXiv preprint arXiv:1808.10544, 2018.
  30. C. Zou, H. Mo, C. Gao, R. Du, and H. Fu. Language-based colorization of scene sketches. ACM Transactions on Graphics (TOG), 38(6):1-16, 2019.
  31. C. Zou, Q. Yu, R. Du, H. Mo, Y.-Z. Song, T. Xiang, C. Gao, B. Chen, and H. Zhang. Sketchyscene: Richly-annotated scene sketches. In Proceedings of the European Conference on Computer Vision (ECCV), pages 421-436, 2018.