Journal of the Korea Computer Graphics Society (한국컴퓨터그래픽스학회논문지)

Korea Computer Graphics Society (한국컴퓨터그래픽스학회)
권호 표지
DOI QR코드

DOI QR Code

Optimized Normalization for Unsupervised Learning-based Image Denoising

비지도 학습 기반 영상 노이즈 제거 기술을 위한 정규화 기법의 최적화

  • Received : 2021.11.10
  • Accepted : 2021.11.25
  • Published : 2021.12.01
Download PDF
⟨ Previous Next ⟩

Abstract

Recently, deep learning-based denoising approaches have been actively studied. In particular, with the advances of blind denoising techniques, it become possible to train a deep learning-based denoising model only with noisy images in an image domain where it is impossible to obtain a clean image. We no longer require pairs of a clean image and a noisy image to obtain a restored clean image from the observation. However, it is difficult to recover the target using a deep learning-based denoising model trained by only noisy images if the distribution of the noisy image is far from the distribution of the clean image. To address this limitation, unpaired image denoising approaches have recently been studied that can learn the denoising model from unpaired data of the noisy image and the clean image. ISCL showed comparable performance close to that of supervised learning-based models based on pairs of clean and noisy images. In this study, we propose suitable normalization techniques for each purpose of architectures (e.g., generator, discriminator, and extractor) of ISCL. We demonstrate that the proposed method outperforms state-of-the-art unpaired image denoising approaches including ISCL.

최근 노이즈 제거를 위한 심층 학습 모델에 대한 연구가 활발하게 진행되고 있다. 특히 블라인드 노이즈 제거 (blind denoising) 기술이 발전하면서 깨끗한 영상을 얻기가 불가능한 영상의 영역에서 노이즈 영상만으로 심층 학습 기반 노이즈 제거 모델의 학습이 가능해졌다. 우리는 관찰된 노이즈 영상으로부터 깨끗한 영상을 얻기 위해 더는 깨끗한 영상과 노이즈 영상의 짝을 이루는 데이터를 필요하지 않는다. 하지만 노이즈 영상과 깨끗한 영상 간의 차이가 큰 데이터라면 노이즈 영상만으로 학습된 노이즈 제거 모델은 우리가 원하는 품질의 깨끗한 영상을 복원하기 어려울 것이다. 이 문제를 해결하기 위해서 짝지어지지 않는 깨끗한 영상과 노이즈 영상으로 학습한 모델 기반 노이즈 제거 기술은 최근 연구되고 있다. 가장 최신 기술인 ISCL은 깨끗한 영상과 노이즈 영상의 쌍을 기반으로 한 지도학습 기반 모델의 성능과 거의 근접한 성능을 보여 주었다. 우리는 제안된 방법이 ISCL을 포함한 다른 최신 짝을 이루지 않는 영상 기반 노이즈 제거 기술보다 성능이 우수함을 보여준다.

Keywords

Acknowledgement

본 연구는 과학기술정보통신부 재원의 정보통신기획평가원의 ICT명품인재양성 사업 (IITP-2021-2020-0-01819), 교육부 재원의 한국연구재단의 기초연구사업 (No. NRF-2021R1A6A1A13044830), 그리고 과학기술정보통신부 재원의 한국연구재단의 초융합AI원천기술개발사업 (NRF-2019M3E5D2A01063819)의 지원을 받아 수행되었음.

References

  1. K. Lee and W.-K. Jeong, "ISCL: Interdependent self-cooperative learning for unpaired image denoising," IEEE Trans. Med. Imag., vol. 40, no. 11, pp. 3238-3248, 2021. [Online]. Available: https://doi.org/10.1109/TMI.2021.3096142
  2. D. G. C. Hildebrand, B. J. Graham, and W.-C. A. Lee, "Grid-tape for fast nanoscale imaging," 2017.
  3. B. J. Graham, D. G. C. Hildebrand, and W.-C. A. Lee, "Grid-tape imaging stage," 2018.
  4. B. J. Graham, D. G. C. Hildebrand, A. T. Kuan, J. T. Maniates-Selvin, L. A. Thomas, B. L. Shanny, and W.-C. A. Lee, "High-throughput transmission electron microscopy with automated serial sectioning," bioRxiv, 2019. [Online]. Available: https://www.biorxiv.org/content/early/2019/06/02/657346
  5. T. Minh Quan, D. Grant Colburn Hildebrand, K. Lee, L. A. Thomas, A. T. Kuan, W.-C. Allen Lee, and W.-K. Jeong, "Removing imaging artifacts in electron microscopy using an asymmetrically cyclic adversarial network without paired training data," in Proc. IEEE Int. Conf. Comput. Vis. Workshop, Oct. 2019.
  6. K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, "Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising," IEEE Trans. Image Process., vol. 26, no. 7, pp. 3142-3155, Jul. 2017. https://doi.org/10.1109/TIP.2017.2662206
  7. Z. Huang, Z. Chen, Q. Zhang, G. Quan, M. Ji, C. Zhang, Y. Yang, X. Liu, D. Liang, H. Zheng, et al., "Cagan: a cycle-consistent generative adversarial network with attention for low-dose ct imaging," IEEE Trans. Comput. Imag., vol. 6, pp. 1203-1218, 2020. https://doi.org/10.1109/tci.2020.3012928
  8. D. Bashkirova, B. Usman, and K. Saenko, "Adversarial self-defense for cycle-consistent gans," in Proc. Adv. Neural Inf. Process. Syst, vol. 32, 2019.
  9. J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired image-to-image translation using cycle-consistent adversarial networks," in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 2223-2232.
  10. K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, "Image denoising by sparse 3-d transform-domain collaborative filtering," IEEE Trans. Image Process., vol. 16, no. 8, pp. 2080-2095, Aug. 2007. https://doi.org/10.1109/TIP.2007.901238
  11. S. Ioffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," ser. Proc. Int. Conf. Mach. Learn., F. Bach and D. Blei, Eds., vol. 37, Lille, France, Jul. 2015, pp. 448-456.
  12. J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala, and T. Aila, "Noise2Noise: Learning image restoration without clean data," in Proc. Int. Conf. Mach. Learn., J. Dy and A. Krause, Eds., vol. 80, Jul. 2018, pp. 2965-2974.
  13. K. Lee and W.-K. Jeong, "Noise2kernel: Adaptive self-supervised blind denoising using a dilated convolutional kernel architecture," 2020. [Online]. Available: https://arxiv.org/abs/2012.03623
  14. A. Krull, T.-O. Buchholz, and F. Jug, "Noise2void-learning denoising from single noisy images," in Proc. IEEE conf. Comput. Vis. Pattern Recognit., Jun. 2019, pp. 2129-2137.
  15. J. Batson and L. Royer, "Noise2Self: Blind denoising by self-supervision," in Proc. Int. Conf. Mach. Learn., Mar. 2019, pp. 524-533.
  16. Y. Quan, M. Chen, T. Pang, and H. Ji, "Self2Self with dropout: Learning self-supervised denoising from single image," in Proc. IEEE conf. Comput. Vis. Pattern Recognit., Jun. 2020, pp. 1890-1898.
  17. H. Liao, W.-A. Lin, S. K. Zhou, and J. Luo, "Adn: Artifact disentanglement network for unsupervised metal artifact reduction," IEEE Trans. Med. Imag., vol. 39, no. 3, pp. 634-643, 2019. https://doi.org/10.1109/tmi.2019.2933425
  18. Y. Huang, W. Xia, Z. Lu, Y. Liu, H. Chen, J. Zhou, L. Fang, and Y. Zhang, "Noise-powered disentangled representation for unsupervised speckle reduction of optical coherence tomography images," IEEE Trans. Med. Imag., 2020.
  19. H. Nam and H.-E. Kim, "Batch-instance normalization for adaptively style-invariant neural networks," in Proc. Adv. Neural Inf. Process. Syst, vol. 31, 2018, pp. 2558-2567.
  20. D. Ulyanov, A. Vedaldi, and V. Lempitsky, "Instance normalization: The missing ingredient for fast stylization," 2016. [Online]. Available: https://arxiv.org/abs/1607.08022
  21. P. Luo, R. Zhang, J. Ren, Z. Peng, and J. Li, "Switchable normalization for learning-to-normalize deep representation," IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 2, pp. 712-728, 2019.
  22. J. L. Ba, J. R. Kiros, and G. E. Hinton, "Layer normalization," 2016. [Online]. Available: https://arxiv.org/abs/1607.06450
  23. T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida, "Spectral normalization for generative adversarial networks," in Proc. Int. Conf. Learn. Represent, 2018.
  24. A. Brock, J. Donahue, and K. Simonyan, "Large scale gan training for high fidelity natural image synthesis," in Proc. Int. Conf. Learn. Represent, 2018.
  25. M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein generative adversarial networks," in Proc. Int. Conf. Mach. Learn., vol. 70, Aug. 2017, pp. 214-223.
  26. I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, "Improved training of wasserstein gans," in Proc. Adv. Neural Inf. Process. Syst, 2017, pp. 5767-5777.
  27. H. Chen, Y. Zhang, M. K. Kalra, F. Lin, Y. Chen, P. Liao, J. Zhou, and G. Wang, "Low-dose ct with a residual encoder-decoder convolutional neural network," IEEE Trans. Med. Imag., vol. 36, no. 12, pp. 2524-2535, 2017. https://doi.org/10.1109/TMI.2017.2715284
  28. Z. Hong, X. Fan, T. Jiang, and J. Feng, "End-to-end unpaired image denoising with conditional adversarial networks," in Proc. Assoc. Adv. Artific. Intell., Apr. 2020, pp. 4140-4149.
  29. Y. Wu and K. He, "Group normalization," in Eur. Conf. Comput. Vis., 2018, pp. 3-19.