Abstract
Video information retrieval has become a very important issue because of the explosive increase in video data from Web content development. Meanwhile, content-based video analysis using visual features has been the main source for video information retrieval and browsing. Content in video can be represented with content-based analysis techniques, which can extract various features from audio-visual data such as frames, shots, colors, texture, or shape. Moreover, similarity between videos can be measured through content-based analysis. However, a movie that is one of typical types of video data is organized by story as well as audio-visual data. This causes a semantic gap between significant information recognized by people and information resulting from content-based analysis, when content-based video analysis using only audio-visual data of low level is applied to information retrieval of movie. The reason for this semantic gap is that the story line for a movie is high level information, with relationships in the content that changes as the movie progresses. Information retrieval related to the story line of a movie cannot be executed by only content-based analysis techniques. A formal model is needed, which can determine relationships among movie contents, or track meaning changes, in order to accurately retrieve the story information. Recently, story-based video analysis techniques have emerged using a social network concept for story information retrieval. These approaches represent a story by using the relationships between characters in a movie, but these approaches have problems. First, they do not express dynamic changes in relationships between characters according to story development. Second, they miss profound information, such as emotions indicating the identities and psychological states of the characters. Emotion is essential to understanding a character's motivation, conflict, and resolution. Third, they do not take account of events and background that contribute to the story. As a result, this paper reviews the importance and weaknesses of previous video analysis methods ranging from content-based approaches to story analysis based on social network. Also, we suggest necessary elements, such as character, background, and events, based on narrative structures introduced in the literature. We extract characters' emotional words from the script of the movie Pretty Woman by using the hierarchical attribute of WordNet, which is an extensive English thesaurus. WordNet offers relationships between words (e.g., synonyms, hypernyms, hyponyms, antonyms). We present a method to visualize the emotional pattern of a character over time. Second, a character's inner nature must be predetermined in order to model a character arc that can depict the character's growth and development. To this end, we analyze the amount of the character's dialogue in the script and track the character's inner nature using social network concepts, such as in-degree (incoming links) and out-degree (outgoing links). Additionally, we propose a method that can track a character's inner nature by tracing indices such as degree, in-degree, and out-degree of the character network in a movie through its progression. Finally, the spatial background where characters meet and where events take place is an important element in the story. We take advantage of the movie script to extracting significant spatial background and suggest a scene map describing spatial arrangements and distances in the movie. Important places where main characters first meet or where they stay during long periods of time can be extracted through this scene map. In view of the aforementioned three elements (character, event, background), we extract a variety of information related to the story and evaluate the performance of the proposed method. We can track story information extracted over time and detect a change in the character's emotion or inner nature, spatial movement, and conflicts and resolutions in the story.
웹의 발전과 콘텐츠 산업의 팽창으로 비디오 데이터가 폭발적으로 증가함에 따라 데이터의 정보 검색은 매우 중요한 문제가 되었다. 그동안 비디오 데이터의 정보 검색과 브라우징을 위해 비디오의 프레임(frame)이나 숏(shot)으로부터 색채(color)와 질감(texture), 모양(shape)과 같은 시각적 특징(features)들을 추출하여 비디오의 내용을 표현하고 유사도를 측정하는 내용 기반(content-based)방식의 비디오 분석이 주를 이루었다. 영화는 하위 레벨의 시청각적 정보와 상위 레벨의 스토리 정보를 포함하고 있다. 저차원의 시각적 특징을 통해 내용을 표현하는 내용 기반 분석을 영화에 적용할 경우 내용 기반 분석과 인간이 인지하는 영화의 내용 사이에는 의미적 격차(semantic gap)가 발생한다. 왜냐하면 영화의 스토리는 시간의 진행에 따라 그 내용이 변하고, 관점에 따라 주관적 해석이 가능한 고차원의 의미정보이기 때문이다. 따라서 스토리 차원의 정보 검색을 위해서는 스토리를 모델링하는 정형화된 모형이 필요하다. 최근 들어 소셜 네트워크 개념을 활용한 스토리 기반의 비디오 분석 방법들이 등장하고 있다. 그러나 영화 속 등장인물들의 소셜 네트워크를 통해 스토리를 표현하는 이 방법들은 몇 가지 문제점들을 드러내고 있다. 첫째, 등장인물들의 관계에만 초점이 맞추어져 있으며, 스토리 진행에 따른 등장인물들의 관계 변화를 역동적으로 표현하지 못한다. 둘째, 등장인물의 정체성과 심리상태를 보여주는 감정(emotion)과 같은 심층적 정보를 간과하고 있다. 셋째, 등장인물 이외에 스토리를 구성하는 사건과 배경에 대한 정보들을 반영하지 못하고 있다. 따라서 본 연구는 기존의 스토리 기반의 비디오 분석 방법들의 한계를 살펴보고, 문제 해결을 위해 문학 이론에서 제시하고 있는 서사 구조에 근거하여 스토리 모델링에 필요한 요소들을 인물, 배경, 사건의 세 가지 측면에서 제시하고자 한다.
