Asia-pacific Journal of Multimedia Services Convergent with Art, Humanities, and Sociology (예술인문사회 융합 멀티미디어 논문지)

The Convergent Research Society Among Humanities, Sociology, Science and Technology (인문사회과학기술융합학회)
권호 표지
DOI QR코드

DOI QR Code

Image retrieval based on a combination of deep learning and behavior ontology for reducing semantic gap

시맨틱 갭을 줄이기 위한 딥러닝과 행위 온톨로지의 결합 기반 이미지 검색

  • Lee, Seung (College of Information and Communication Engineering, Sungkyunkwan Univ.) ;
  • Jung, Hye-Wuk (College of Liberal Arts and Interdisciplinary Studies, Kyonggi Univ.)
  • Received : 2019.09.25
  • Accepted : 2019.11.14
  • Published : 2019.11.30
⟨ Previous Next ⟩

Abstract

Recently, the amount of image on the Internet has rapidly increased, due to the advancement of smart devices and various approaches to effective image retrieval have been researched under these situation. Existing image retrieval methods simply detect the objects in a image and carry out image retrieval based on the label of each object. Therefore, the semantic gap occurs between the image desired by a user and the image obtained from the retrieval result. To reduce the semantic gap in image retrievals, we connect the module for multiple objects classification based on deep learning with the module for human behavior classification. And we combine the connected modules with a behavior ontology. That is to say, we propose an image retrieval system considering the relationship between objects by using the combination of deep learning and behavior ontology. We analyzed the experiment results using walking and running data to take into account dynamic behaviors in images. The proposed method can be extended to the study of automatic annotation generation of images that can improve the accuracy of image retrieval results.

최근 스마트 기기의 발전으로 인터넷상에 존재하는 이미지 데이터의 양이 급속하게 증가하는 상황에서 효과적인 이미지 검색을 위한 다양한 방법들이 연구되고 있다. 기존의 이미지 검색 방법들은 이미지에 존재하는 물체들을 단순하게 검출하여 각 물체들의 라벨 정보에 근거한 검색을 수행하기 때문에 사용자가 원하는 이미지와 검색 결과로 얻은 이미지 간에 의미적 차이인 시맨틱 갭(Semantic Gap)이 발생된다. 이미지 검색에서 발생하는 시맨틱 갭을 줄이기 위해, 본 논문에서는 딥러닝 기반의 다중 객체 분류 모듈과 사람의 행위를 분류하는 모듈을 연결하고, 이 모듈들에 행위 온톨로지를 결합하였다. 즉, 딥러닝과 행위 온톨로지의 결합을 기반으로 객체들 간의 연관성을 고려한 이미지 검색 시스템을 제안한다. 이미지에 포함된 동적인 행위를 고려하기 위해 Walking과 Running 데이터를 이용하여 실험한 결과를 분석하였다. 제안한 방법은 향후 이미지 검색 결과의 정확도를 높일 수 있는 영상의 자동 주석 생성 연구에 확장하여 적용할 수 있다.

Keywords

References

  1. R. Datta, D. Joshi, J. Li and J. Wang, Image Retrieval: Ideas, Influences, and Trends of the New Age. Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval. (2005), November 10-11, Hilton, Singapore.
  2. Y. Rui, T.S. Huang, and S.-F. Chang, Image retrieval: Current techniques, promising directions, and open issues. Journal of visual communication and image representation. (1999), Vol. 10, pp.39-62. https://doi.org/10.1006/jvci.1999.0413
  3. M. Singha and K. Hemachandran, Content based image retrieval using color and texture. Signal & Image Processing: An International Journal (SIPIJ). (2012), Vol. 3, pp.39-57.
  4. F. Long, H. Zhang and D. D. Feng, Fundamentals of content based image retrieval, Technological Fundamentals and Applications Springer-Verlag (2003)
  5. B. Dinakaran, J. Annapurna, and C. A. Kumar, Interactive image retrieval using text and image content. Cybernetics and Information Technologies. (2010), Vol. 10, pp. 20-30.
  6. H. H. Wang, D. Mohamad, and N. Ismail, Image Retrieval: Techniques, Challenge, and Trend. International conference on Machine Vision, Image processing and Pattern Analysis. (2009), Bangkok, Citeseer.
  7. N. Shanmugapriya and R. Nallusamy, A new content based image retrieval system using GMM and relevance feedback. Journal of Computer Science. (2013), Vol. 10, No. 2, pp. 330-340. https://doi.org/10.3844/jcssp.2014.330.340
  8. Z. Mehmood, F. Abbas, T. Mahmood, M. A. Javid, A. Rehman, and T. Nawaz, Content-based image retrieval based on visual words fusion versus features fusion of local and global features, Arabian Journal for Science and Engineering, (2018), pp. 1-20.
  9. Ansari, Mohd & Dixit, Manish & Kurchaniya, Diksha & Johari, Punit. An Effective Approach to an Image Retrieval using SVM Classifier. International Journal of Computer Sciences and Engineering. (2017), Vol. 5, pp. 62-72. https://doi.org/10.26438/ijcse/v5i9.6267
  10. Ansa Saju, I. Thusnavis Bella Mary, A. Vasuki, P. S. Lakshmi, Reduction of semantic gap using relevance feedback technique in image retrieval system. ICADIWT (2014), pp. 148-153.
  11. Bai, C., Huang, L., Pan, X,; Zheng, J., Chen, S., Optimization of deep convolutional neural network for large scale image retrieval. Neurocomputing, (2018), Vol. 303, pp. 60-67 https://doi.org/10.1016/j.neucom.2018.04.034
  12. Ouhda, M., El Asnaoui, K., Aksasse, B., Ouanan, M. Content-Based Image Retrieval Using Convolutional Neural Networks, Lecture Notes in Real-Time Intelligent Systems. Advances in Intelligent System and Computing, (2019), pp. 463-473.
  13. S. Ren, K. He, R. B. Girshick, and J. Sun, Faster R-CNN: towards real-time object detection with region proposal networks. In NIPS. (2015), pp. 91-99.
  14. C. Thurau and V. Hlava, Pose primitive based human action recognition in videos or still images. Proceedings of the in 2008 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). (2008).
  15. https://ko.wikipedia.org/wiki/Ontology, Jun 12(2019).
  16. C. Schuldt, I. Laptev and B. Caputo, Recognizing Human Actions: A Local SVM Approach. presented at Proceedings of the International Conference on Pattern Recognition (ICPR). (2004), pp. 32-36.
  17. J.-B. Lamy, Owlready: Ontology-oriented programming in Python with automatic classification and high level constructs for biomedical ontologies. Artificial Intelligence in Medicine. (2017), Vol. 80, pp. 11-28. https://doi.org/10.1016/j.artmed.2017.07.002