Journal of Information Processing Systems

Korea Information Processing Society (한국정보처리학회)
권호 표지
DOI QR코드

DOI QR Code

Image Restoration and Object Removal Using Prioritized Adaptive Patch-Based Inpainting in a Wavelet Domain

  • Borole, Rajesh P. (Dept. of Electronics & Telecom, Shri Guru Gobind Singhji Institute of Engineering and Technology) ;
  • Bonde, Sanjiv V. (Dept. of Electronics & Telecom, Shri Guru Gobind Singhji Institute of Engineering and Technology)
  • Received : 2014.01.18
  • Accepted : 2015.04.15
  • Published : 2017.10.31
Download PDF
⟨ Previous Next ⟩

Abstract

Image restoration has been carried out by texture synthesis mostly for large regions and inpainting algorithms for small cracks in images. In this paper, we propose a new approach that allows for the simultaneous fill-in of different structures and textures by processing in a wavelet domain. A combination of structure inpainting and patch-based texture synthesis is carried out, which is known as patch-based inpainting, for filling and updating the target region. The wavelet transform is used for its very good multiresolution capabilities. The proposed algorithm uses the wavelet domain subbands to resolve the structure and texture components in smooth approximation and high frequency structural details. The subbands are processed separately by the prioritized patch-based inpainting with isophote energy driven texture synthesis at the core. The algorithm automatically estimates the wavelet coefficients of the target regions of various subbands using optimized patches from the surrounding DWT coefficients. The suggested performance improvement drastically improves execution speed over the existing algorithm. The proposed patch optimization strategy improves the quality of the fill. The fill-in is done with higher priority to structures and isophotes arriving at target boundaries. The effectiveness of the algorithm is demonstrated with natural and textured images with varying textural complexions.

Keywords

References

  1. M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, "Image inpainting," in Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'00), New Orleans, LA, 2000, pp. 417-424.
  2. C. Ballester, V. Caselles, J. Verdera, M. Bertalmio, and G. Sapiro, "A variational model for filling-in gray level and color images," in Proceedings of 8th IEEE International Conference on Computer Vision (ICCV2001), Vancouver, Canada, 2001, pp. 10-16.
  3. T. F. Chan and J. Shen, "Nontexture inpainting by curvature-driven diffusions," Journal of Visual Communication and Image Representation, vol. 12, no. 4, pp. 436-449, 2001. https://doi.org/10.1006/jvci.2001.0487
  4. G. Kanizsa, Organization in Vision: Essays on Gestalt Perception. New York, NY: Praeger, 1979.
  5. L. Pessoa, E. Thompson, and A. Noe, "Finding out about filling-in: a guide to perceptual completion for visual science and the philosophy of perception," Behavioral and Brain Sciences, vol. 24, no. 6, pp. 723-802, 1998.
  6. S. Sarkar and K. L. Boyer, "Integration, inference, and management of spatial information using Bayesian networks: perceptual organization," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 15, no. 3, pp. 256-274, 1993. https://doi.org/10.1109/34.204907
  7. D. Tschumperle and R. Deriche, "Vector valued image regularization with PDEs: a common framework for different applications," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 27, no. 4, pp. 506-517, 2005. https://doi.org/10.1109/TPAMI.2005.87
  8. S. Masnou and J. M. Morel, "Level lines based disocclusion," in Proceeding of International Conference on Image Processing (ICIP 98), Chicago, IL, 1998, pp. 259-263.
  9. A. C. Kokaram, R. D. Morris, W. J. Fitzgerald, and P. J. Rayner, "Interpolation of missing data in image sequences," IEEE Transactions on Image Processing, vol. 4, no. 11, pp. 1509-1519, 1995. https://doi.org/10.1109/83.469932
  10. M. Bertalmio, L. A. Vese, G. Sapiro, and S. Osher, "Simultaneous structure and texture image inpainting," in Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR2003), Madison WI, 2003, pp. 707-712.
  11. I. Drori, D. Cohen-Or, and H. Yeshurun, "Fragment based image completion," ACM Transactions on Graphics, vol. 22, no. 3, pp. 303-312, 2003. https://doi.org/10.1145/882262.882267
  12. P. Harrison, "A non-hierarchical procedure for re-synthesis of complex texture," in Proceeding of the 9th International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision (WSCG2001), Plzen, Czech Republic, 2001, pp. 190-197.
  13. R. Bornard, E. Lecan, L. Laborelli, and J. H. Chenot, "Missing data correction in still images and image sequences," in Proceedings of the 10th ACM International Conference on Multimedia, Juan les Pins, France, 2002, pp. 355-361.
  14. J. S. De Bonet, "Multiresolution sampling procedure for analysis and synthesis of texture images," in Proceedings of 24th annual ACM Conference Computer Graphics and Interactive Techniques (SIGGRAPH), Los Angeles, CA, 1997, pp. 361-368.
  15. D. Garber, "Computational models for texture analysis and texture synthesis," Ph.D. dissertation, University of Southern California, Los Angeles, CA, 1981.
  16. D. J. Heeger and J. R. Bergen, "Pyramid based texture analysis/synthesis," in Proceedings of the 22nd Annual Conference on Computer Graphics and Interactive Techniques, Los Angeles, CA, 1995, pp. 229-238.
  17. A. A. Efros and W. T. Freeman, "Image quilting for texture synthesis and transfer," in Proceeding of the 28th ACM Annual Conference on Computer Graphics, Los Angeles, CA, 2001, pp. 341-346.
  18. L. Liang, C. Liu, Y. Q. Xu, B. Guo, and H. Y. Shum, "Real-time texture synthesis by patch based sampling," ACM Transactions on Graphics, vol. 20, no. 3, pp. 127-150, 2001. https://doi.org/10.1145/501786.501787
  19. M. Ashikhmin, "Synthesizing natural textures," in Proceedings of the 2001 Symposium on Interactive 3D Graphics, Research Triangle Park, NC, 2001, pp. 217-226.
  20. J. Wu and Q. Ruan, "Object removal by cross isophotes exemplar based inpainting," in Proceeding of 18th International Conference on Pattern Recognition (ICPR2006), Hong Kong, 2006, pp. 810-813.
  21. A. Criminisi, P. Perez, and K. Toyama, "Region filling and object removal by exemplar based image inpainting," IEEE Transactions on Image Processing, vol. 13, no. 9, pp. 1200-1212, 2004. https://doi.org/10.1109/TIP.2004.833105
  22. Y. Chen, Q. Luan, H. Li, and O. Au, "Sketch guided texture based image inpainting," in Proceeding of IEEE International Conference on Image Processing (ICIP), Atlanta, GA, 2006, pp. 1997-2000.
  23. J. Sun, L. Yuan, J. Jia, and H. Y. Shum, "Image compilation with structure propagation," ACM Transactions on Graphics, vol. 24, no. 3, pp. 861-868, 2005. https://doi.org/10.1145/1073204.1073274
  24. T. F. Chan, J. Shen, and H. M. Zhou, "Total variation wavelet inpainting," Journal of Mathematical Imaging and Vision, vol. 25, no. 1, pp. 107-125, 2006. https://doi.org/10.1007/s10851-006-5257-3
  25. S. D. Rane, J. Remus, and G. Sapiro, "Wavelet domain reconstruction of lost blocks in wireless image transmission and packet switched networks," in Proceedings of IEEE International Conference on Image Processing (ICIP), Rochester, NY, 2002, pp. 309-312.
  26. K. Patwardhan and G. Sapiro, "Projection based image and video inpainting using wavelets," in Proceedings of IEEE International Conference on Image Processing (ICIP), Barcelona, Spain, 2003, pp. 857-860.
  27. D. Cho and T. D. Bui, "Image inpainting using wavelet based inter- and intra-scale dependency," in Proceedings of 19th International Conference on Pattern Recognition (ICPR), Tampa, FL, 2008, pp. 1-4.
  28. G. Hua and M. Orchard, "Image inpainting based on geometrical modeling of complexwavelet coefficients," in Proceeding of IEEE International Conference on Image Processing (ICIP), San Antonio, TX, 2007, pp. 553-556.
  29. F. Wang, D. Liang, N. Wang, Z. Cheng, and J. Tang, "An new method for image inpainting using wavelets," in Proceeding of International Conference on Multimedia Technology (ICMT), Hangzhou, China, 2011, pp. 201-204.
  30. Y. W. Wen, R. H. Chan, and A. M. Yip, "A primal-dual method for total-variation-based wavelet domain inpainting," IEEE Transactions on Image Processing, vol. 21, no. 1, pp. 106-114, 2012. https://doi.org/10.1109/TIP.2011.2159983
  31. L. Chen, X. Chen, and P. Guo, "Blind image restoration using divisional regularization and wavelet technique," in Proceeding of 4th International Conference on Natural Computation (ICNC'08), Jinan, China, 2008, pp. 476-480.
  32. R. H. Chan, Y. W. Wen, and A. M. Yip, "A fast optimization transfer algorithm for image inpainting in wavelet domains," IEEE Transactions on Image Processing, vol. 18, no. 7, pp. 1467-1476, 2009. https://doi.org/10.1109/TIP.2009.2019806
  33. J. C. Hung, C. H. Hwang, Y. C. Liao, N. C. Tang, and T. J. Chen, "Exemplar-based image inpainting base on structure construction," Journal of Software, vol. 3, no. 8, pp. 57-64, 2008.
  34. R. P. Borole and S. V. Bonde, "Image analysis & restoration by exemplar inpainting," in Proceedings of National Conference on Communication, Computing and Networking (NCCCNT'13), Nanded, India, 2013.
  35. R. P. Borole and S. V. Bonde, "Patch-based inpainting for object removal and region filling in images," Journal of Intelligent Systems, vol. 22, no. 3, pp. 335-350, 2013.
  36. R. C. Gonzalez and R. E. Wood, Digital Image Processing, 2nd ed. Upper Saddle River, NJ: Prentice Hall, 2002.
  37. S. G. Mallat, "A theory for multiresolution signal decomposition: the wavelet representation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 11, no. 7, pp. 674-693, 1989. https://doi.org/10.1109/34.192463
  38. A. Efros and T. K. Leung, "Texture synthesis by non-parametric sampling," in Proceeding of the 7th IEEE International Conference on Computer Vision, Kerkyra, Greece, 1999, pp. 1033-1038.
  39. D. Sun, L. Yuan, Y. Zhang, J. Zhang, and G. Pan, "Structure-aware image completion with texture propagation," in Proceeding of 6th International Conference on Image and Graphics (ICIG), Hefei, China, 2011, pp. 199-204.