Applied Microscopy

  • Volume 50
  • /
  • Pages.23.1-23.9
  • /
  • 2020
  • /
  • 2287-5123(pISSN)
  • /
  • 2287-4445(eISSN)
Korean Society of Microscopy (한국현미경학회)
권호 표지
DOI QR코드

DOI QR Code

Noise2Atom: unsupervised denoising for scanning transmission electron microscopy images

  • Feng Wang (Electron Microscopy Center, Empa, Swiss Federal Laboratories for Materials Science and Technology) ;
  • Trond R. Henninen (Electron Microscopy Center, Empa, Swiss Federal Laboratories for Materials Science and Technology) ;
  • Debora Keller (Electron Microscopy Center, Empa, Swiss Federal Laboratories for Materials Science and Technology) ;
  • Rolf Erni (Electron Microscopy Center, Empa, Swiss Federal Laboratories for Materials Science and Technology)
  • Received : 2020.08.06
  • Accepted : 2020.09.17
  • Published : 2020.12.31
⟨ Previous Next ⟩

Abstract

We propose an effective deep learning model to denoise scanning transmission electron microscopy (STEM) image series, named Noise2Atom, to map images from a source domain 𝓢 to a target domain 𝓒, where 𝓢 is for our noisy experimental dataset, and 𝓒 is for the desired clear atomic images. Noise2Atom uses two external networks to apply additional constraints from the domain knowledge. This model requires no signal prior, no noise model estimation, and no paired training images. The only assumption is that the inputs are acquired with identical experimental configurations. To evaluate the restoration performance of our model, as it is impossible to obtain ground truth for our experimental dataset, we propose consecutive structural similarity (CSS) for image quality assessment, based on the fact that the structures remain much the same as the previous frame(s) within small scan intervals. We demonstrate the superiority of our model by providing evaluation in terms of CSS and visual quality on different experimental datasets.

Keywords

Acknowledgement

We acknowledge founding from the European Research Council (ERC) under the European Union's Horizon 2020 program (grant agreement No. 681312).

References

  1. M. Arjovsky, S. Chintala, L. Bottou, Wasserstein GAN (2017). arXiv:1701.07875
  2. J. Batson, L. Royer, Noise2Self: Blind Denoising by Self-Supervision (2019). arXiv:1901.11365
  3. T.-O. Buchholz, M. Jordan, G. Pigino, F. Jug, in 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). Cryo-CARE: Content-Aware Image Restoration for Cryo-Transmission Electron Microscopy Data (IEEE, Venice, 2019a), pp. 502-506. ISBN 978-1-5386-3641-1. https://doi.org/10.1109/ISBI.2019.8759519
  4. T.-O. Buchholz, A. Krull, R. Shahidi, G. Pigino, G. Jekely, F. Jug, Content-aware image restoration for electron microscopy.Methods Cell Biol. 152, 277-289 (2019b). ISSN 0091-679X. https://doi.org/10.1016/bs.mcb.2019.05.001
  5. K. Cao, T. Zoberbier, J. Biskupek, A. Botos, R. L. McSweeney, A. Kurtoglu, C. T. Stoppiello, A. V. Markevich, E. Besley, T. W. Chamberlain, Comparison of atomic scale dynamics for the middle and late transition metal nanocatalysts. Nat. Commun. 9(1), 1-10 (2018). Publisher: Nature Publishing Group
  6. Y. Chang, L. Yan, H. Fang, S. Zhong, W. Liao, HSI-DeNet: Hyperspectral Image Restoration via Convolutional Neural Network. IEEE Trans. Geosci. Remote Sens. 57(2), 667-682 (2019). ISSN 1558-0644. https://doi.org/10.1109/TGRS.2018.2859203
  7. F. Chollet, in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Xception: Deep Learning with Depthwise Separable Convolutions (IEEE, Honolulu, 2017), pp. 1800-1807. ISBN, 978-1-5386-0457-1. https://doi.org/10.1109/CVPR.2017.195
  8. C. Dwyer, R. Erni, J. Etheridge, Measurement of effective source distribution and its importance for quantitative interpretation of STEM images. Ultramicroscopy. 110(8), 952-957 (2010). ISSN 0304-3991. https://doi.org/10.1016/j.ultramic.2010.01.007. http://www.sciencedirect.com/science/article/pii/S0304399110000100
  9. T. Furnival, R. K. Leary, P. A. Midgley, Denoising time-resolved microscopy image sequences with singular value thresholding. Ultramicroscopy. 178, 112-124 (2017). ISSN 03043991. https://doi.org/10.1016/j.ultramic.2016.05.005
  10. Z. Guo, Y. Sun, M. Jian, X. Zhang, Deep Residual Network with Sparse Feedback for Image Restoration. Appl. Sci. 8(12), 2417 (2018)
  11. K. He, X. Zhang, S. Ren, J. Sun, in Proceedings of the IEEE conference on computer vision and pattern recognition. Deep residual learning for image recognition, (2016), pp. 770-778. https://doi.org/10.1109/cvpr.2016.90
  12. T. R. Henninen, M. Bon, F. Wang, D. Passerone, R. Erni, in Angewandte Chemie International Edition. The Structure of Sub-nm Platinum Clusters at Elevated Temperatures, (2019). ISSN 14337851. https://doi.org/10.1002/anie.201911068
  13. T. R. Henninen, D. Keller, R. Erni, Structure matters - Direct in-situ observation of cluster nucleation at atomic scale in a liquid phase. In review (2020)
  14. V. S. Kadimesetty, S. Gutta, S. Ganapathy, P. K. Yalavarthy, Convolutional neural network-based robust denoising of low-dose computed tomography perfusion maps. IEEE Trans. Radiat. Plasma Med. Sci. 3(2), 137-152 (2018)
  15. D. Keller, T. R. Henninen, R. Erni, Formation of gold nanoparticles in a free-standing ionic liquid triggered by heat and electron irradiation. Micron. 117, 16-21 (2019). Publisher: Elsevier
  16. T. Kim, M. Cha, H. Kim, J. K. Lee, J. Kim, in Proceedings of the 34th International Conference on Machine Learning, vol. 70, ed. by D. Precup, Y.W. Teh. Learning to Discover Cross-Domain Relations with Generative Adversarial Networks (PMLR, International Convention Centre, Sydney, 2017), pp. 1857-1865. http://proceedings.mlr.press/v70/kim17a.html. Accessed 24 Sept 2020
  17. E. J. Kirkland, R. F. Loane, J. Silcox, Simulation of annular dark field STEM images using a modified multislice method. Ultramicroscopy. 23(1), 77-96 (1987). Publisher: Elsevier
  18. F. Kokkinos, S. Lefkimmiatis, Iterative Joint Image Demosaicking and Denoising Using a Residual Denoising Network. IEEE Trans. Image Process. 28(8), 4177-4188 (2019). ISSN 1941-0042. https://doi.org/10.1109/TIP.2019.2905991
  19. A. Krizhevsky, I. Sutskever, G. E. Hinton, in Advances in Neural Information Processing Systems, vol. 25, ed. by F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger. ImageNet Classification with Deep Convolutional Neural Networks (Curran Associates, Inc., 2012), pp. 1097-1105. http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neuralnetwor.Accessed 24 Sept 2020
  20. A. Krull, T.-O. Buchholz, F. Jug, in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Noise2void - learning denoising from single noisy images, (2019a). https://doi.org/10.1109/cvpr.2019.00223
  21. A. Krull, T. Vicar, F. Jug, Probabilistic Noise2Void: Unsupervised Content-Aware Denoising (2019b). arXiv:1906.00651
  22. J. Lehtinen, J. Munkberg, J. Hasselgren, S. Laine, T. Karras, M. Aittala, T. Aila, Noise2noise: Learning image restoration without clean data (2018). arXiv preprint arXiv:1803.04189
  23. K. Lin, T. H. Li, S. Liu, G. Li, in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. Real photographs denoising with noise domain adaptation and attentive generative adversarial network, (2019). https://doi.org/10.1109/cvprw.2019.00221
  24. P. Liu, H. Zhang, K. Zhang, L. Lin, W. Zuo, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. Multi-level wavelet-CNN for image restoration, (2018), pp. 773-782. https://doi.org/10.1109/cvprw.2018.00121
  25. A. Lucas, M. Iliadis, R. Molina, A. K. Katsaggelos, Using deep neural networks for inverse problems in imaging: Beyond analytical methods. IEEE Signal Process. Mag. 35, 20-36 (2018)
  26. B. Mildenhall, J. T. Barron, J. Chen, D. Sharlet, R. Ng, R. Carroll, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Burst denoising with kernel prediction networks, (2018), pp. 2502-2510. https://doi.org/10.1109/cvpr.2018.00265
  27. N. Moran, D. Schmidt, Y. Zhong, P. Coady, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Noisier2Noise: Learning to Denoise from Unpaired Noisy Data, (2020), pp. 12064-12072. https://doi.org/10.1109/cvpr42600.2020.01208
  28. M. Prakash, M. Lalit, P. Tomancak, A. Krul, F. Jug, in 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI). Fully Unsupervised Probabilistic Noise2Void, (2020), pp. 154-158. https://doi.org/10.1109/ISBI45749.2020.9098612
  29. M. Ran, J. Hu, Y. Chen, H. Chen, H. Sun, J. Zhou, Y. Zhang, Denoising of 3D magnetic resonance images using a residual encoder-decoder Wasserstein generative adversarial network. Med. Image Anal. 55, 165-180 (2019)
  30. O. Ronneberger, P. Fischer, T. Brox, in Medical Image Computing and Computer-Assisted Intervention - MICCAI 2015, Lecture Notes in Computer Science, ed. by N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi. U-Net: Convolutional Networks for Biomedical Image Segmentation (Cham. Springer International Publishing, 2015), pp. 234-241. ISBN 978-3-319-24574-4. https://doi.org/10.1007/978-3-319-24574-4_28
  31. Y. Song, Y. Zhu, X. Du, Dynamic Residual Dense Network for Image Denoising. Sensors. 19(17), 3809 (2019). https://doi.org/10.3390/s19173809. https://www.mdpi.com/1424-8220/19/17/3809
  32. Y. Su, Q. Lian, X. Zhang, B. Shi, X. Fan, Multi-scale cross-path concatenation residual network for Poisson denoising. IET Image Process. 13(8), 1295-1303 (2019)
  33. D. Ulyanov, A. Vedaldi, V. Lempitsky, Instance normalization: The missing ingredient for fast stylization, (2016). arXiv preprint arXiv:1607.08022
  34. F. Wang, A. Eljarrat, J. Muller, T. R. Henninen, R. Erni, C. T. Koch, Multi-resolution convolutional neural networks for inverse problems. Sci. Rep. 10(1), 1-11 (2020). ISSN 2045-2322. https://doi.org/10.1038/s41598-020-62484-z
  35. J. Wu, Z. Huang, J. Thoma, D. Acharya, L. Van Gool, in Proceedings of the European Conference on Computer Vision (ECCV). Wasserstein divergence for gans, (2018), pp. 653-668. https://doi.org/10.1007/978-3-030-01228-1_40
  36. W. Xie, Y. Li, X. Jia, Deep convolutional networks with residual learning for accurate spectral-spatial denoising. Neurocomputing. 312, 372-381 (2018)
  37. Z. Yi, H. Zhang, P. Tan, M. Gong, in Proceedings of the IEEE International Conference on Computer Vision. Dualgan: Unsupervised dual learning for image-to-image translation, (2017), pp. 2849-2857. https://doi.org/10.1109/iccv.2017.310
  38. K. Zhang, W. Zuo, L. Zhang, FFDNet: Toward a Fast and Flexible Solution for CNN based Image Denoising. IEEE Trans. Image Process. 27(9), 4608-4622 (2018). ISSN 1057-7149, 1941-0042. https://doi.org/10.1109/TIP.2018.2839891
  39. Y. Zhang, Y. Zhu, E. Nichols, Q. Wang, S. Zhang, C. Smith, S. Howard, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. A poisson-gaussian denoising dataset with real fluorescence microscopy images, (2019), pp. 11710-11718. https://doi.org/10.1109/cvpr.2019.01198
  40. J.-Y. Zhu, T. Park, P. Isola, A. A. Efros, in Proceedings of the IEEE International Conference on Computer Vision. Unpaired image-to-image translation using cycle-consistent adversarial networks, (2017), pp. 2223-2232. https://doi.org/10.1109/iccv.2017.244