응용통계연구 (The Korean Journal of Applied Statistics)

  • 제34권2호
  • /
  • Pages.191-204
  • /
  • 2021
  • /
  • 1225-066X(pISSN)
  • /
  • 2383-5818(eISSN)
한국통계학회 (The Korean Statistical Society)
권호 표지
DOI QR코드

DOI QR Code

유사도 기반 이미지 캡션을 이용한 시각질의응답 연구

Using similarity based image caption to aid visual question answering

  • 강준서 (중앙대학교 응용통통계학과) ;
  • 임창원 (중앙대학교 응용통통계학과)
  • Kang, Joonseo (Department of Applied Statistics, Chung-Ang University) ;
  • Lim, Changwon (Department of Applied Statistics, Chung-Ang University)
  • 투고 : 2020.12.22
  • 심사 : 2021.01.27
  • 발행 : 2021.04.30
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

시각질의응답과 이미지 캡셔닝은 이미지의 특징과 문장의 언어적인 특징을 이해하는 것을 요구하는 작업이다. 따라서 두 가지 작업 모두 이미지와 텍스트를 연결해 줄 수 있는 공동 어텐션이 핵심이라고 할 수 있다. 본 논문에서는 MSCOCO 데이터 셋에 대하여 사전 훈련된 transformer 모델을 이용 하여 캡션을 생성한 후 이를 활용해 시각질의응답의 성능을 높이는 모델을 제안하고자 한다. 이때 질 문과 관계없는 캡션은 오히려 시각질의응답에서 답을 맞히는데 방해가 될 수 있기 때문에 질문과의 유사도를 기반으로 질문과 유사한 일부의 캡션을 활용하도록 하였다. 또한 캡션에서 불용어는 답을 맞히는데 영향을 주지 못하거나 방해가 될 수 있기 때문에 제거한 후에 실험을 진행하였다. 기존 시 각질의응답에서 이미지와 텍스트간의 공동 어텐션을 활용하여 좋은 성능을 보였던 deep modular co-attention network (MCAN)과 유사도 기반의 선별된 캡션을 사용하여 VQA-v2 데이터에 대하여 실험을 진행하였다. 그 결과 기존의 MCAN모델과 비교하여 유사도 기반으로 선별된 캡션을 활용했을 때 성능 향상을 확인하였다.

Visual Question Answering (VQA) and image captioning are tasks that require understanding of the features of images and linguistic features of text. Therefore, co-attention may be the key to both tasks, which can connect image and text. In this paper, we propose a model to achieve high performance for VQA by image caption generated using a pretrained standard transformer model based on MSCOCO dataset. Captions unrelated to the question can rather interfere with answering, so some captions similar to the question were selected to use based on a similarity to the question. In addition, stopwords in the caption could not affect or interfere with answering, so the experiment was conducted after removing stopwords. Experiments were conducted on VQA-v2 data to compare the proposed model with the deep modular co-attention network (MCAN) model, which showed good performance by using co-attention between images and text. As a result, the proposed model outperformed the MCAN model.

키워드

과제정보

이 논문은 2019년도 중앙대학교 연구장학기금 지원에 의한 것임.

참고문헌

  1. Anderson P, He X, Buehler C, Teney D, Johnson M, Gould S, and Zhang L (2018). Bottom-up and top-down attention for image captioning and visual question answering, IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6077-6086.
  2. Antol S, Agrawal A, Lu J, Mitchell M, Batra D, Zitnick L, and Parikh D (2015). VQA: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, 2425-2433.
  3. Ba JL, Kiros JR, and Hinton GE (2016). Layer Normalization, arXiv preprint arXiv:1607.06450.
  4. Chorowski JK, Bahdanau D, Serdyuk D, Cho K, and Bengio Y (2015). Attention-based models for speech recognition. In Advances in Neural Information Processing Systems (NIPS), 577-585.
  5. Herdade S, Kappeler A, Boakye K, and Soares J (2019). Image Captioning: Transforming Objects into Words. In Advances in Neural Information Processing Systems, Mit Press, Cambridge, MA, USA, 11137-11147.
  6. Kim JH, Jun J, and Zhang BT (2018). Bilinear attention networks. In Advances in Neural Information Processing Systems, 31, 1564-1574.
  7. Krishna R, Zhu Y, Groth O, Johnson J, Hata K, Kravitz J, Chen S, Kalantidis Y, Li LJ, Shamma DA, Bernstein MS, and Li FF (2016). Visual Genome: connecting language and vision using crowdsourced dense image annotations, arXiv preprint arXiv:1602.07332.
  8. Li Q, Tao Q, Joty S, Cai J, and Luo J (2018). VQA-E: Explaining, elaborating, and enhancing your answers for visual questions, arXiv preprint arXiv:1803.07464.
  9. Loper E and Bird S (2002). NLTK: The natural language toolkit, ETMTNLP '02: Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics, 1, 63--70.
  10. Lu J, Yang J, Batra D, and Parikh D (2017). Hierarchical question-image co-attention for visual question answering, arXiv preprint arXiv:1606.00061.
  11. Mnih V, Heess N, Graves A, and Kavukcuoglu K (2014). Recurrent models of visual attention. In Advances in neural information processing systems (NIPS), 2204-2212.
  12. Pennington J, Socher R, and Manning CD (2014). GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), 1532-1543.
  13. Teney D, Anderson P, He X, and Hengel A (2017). Tips and tricks for visual question answering: Learnings from the 2017 challenge, arXiv preprint arXiv:1708.02711.
  14. Vaswani A, Shazeer M, Parmar N, Uszkoreit J, Jones L, Gomez AN, Kaiser L, Polosukhin I (2017). Attention is all you need. In Advances in Neural Information Processing Systems, 6000-6010.
  15. Wu J, Hu Z, and Mooney R (2019). Generating question relevant captions to aid visual question answering, arXiv preprint arXiv:1906.00513.
  16. Xu K, Ba J, Kiros R, Cho K, Courville A, Salakhutdinov R, Zemel R, Bengio Y (2015). Show, attend and tell: Neural image caption generation with visual attention, arXiv preprint arXiv:1502.03044.
  17. Yu Z, Yu J, Cui Y, Tao D, and Tian Q (2019). Deep modular co-attention networks for visual question answering, arXiv preprint arXiv:1906.10770.
  18. Yu Z, Yu J, Xiang C, Fan J, and Tao D (2017). Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering. In Proceedings of the IEEE, 26, 2275-2290.