• Korean Journal of Child Studies
• /
• v.34 no.4
• /
• pp.163-177
• /
• 2013

Digital technology has advanced rapidly and it is anticipated that multimodal ways of meaning-making will become increasingly important. Consequently, teaching multimodal literacies is becoming a major issue in education. This study focuses on the use of picture diaries as a means of teaching multimodal literacies. Picture diaries are one of the basic and unique multimodal texts used in lower elementary level classes in Korea. A further advantage is that it is a promising text model which can be taught in unplugged ways. In order to explore the educational implications of using such picture diaries, this study sought to analyze the ways in which twenty four $1_{st}$ graders in an elementary school constructed meaning with written language and pictures in composing picture diaries. 251 picture diaries composed during several months of their $1_{st}$ grade period were analyzed based on the constant comparative method. The results indicated that the students utilized both written language and pictures in diverse and creative ways to provide their audience with more comprehensive meaning. These results indicate that teachers need to consider their students as active multimodal meaning-makers and provide their students with more opportunities to practice multimodal meaning-making and share their experiences.

• Journal of English Language & Literature
• /
• v.56 no.4
• /
• pp.575-596
• /
• 2010

The modes of narratives and communication have expanded due to social and cultural changes and technological development. Thus texts have become multimodal and media hybridities and media crossover have been increasing as well. Multimodality requires new literacy to understand and interpret those multimodal texts other than existing traditional literacy approaches. The New London Group (2000) argues that multiliteracies are needed to serve today's changing multimodal texts. Kress (2003) also argues, visual texts have been prevailing, being mingled with other modes of texts such as linguistic, audio, gestural, and spatial modes. Literary texts are not exception in this trend of multimodality. The recent renaissance of comics, in particular, the new light on graphic novels can be interpreted in this historical vein. In comparison to comics, no consensus has been made in defining graphic novels, however, many studies have been recently conducted in order to look into the potential of graphic novels in building multiliteracies. In this paper, the graphic novel as a literary genre are explored from a histocial perspective and the definition of graphic novels was attempted to be made. In the light of multiliteracies, this paper presented cases that show how graphic novels can be utilized to build multiliteracies. Lastly, the use of graphic novels for English as a foreign language was introduced as well. The author hopes that at the age of multimodality, the potential graphic novels have in language and literacy education can be taken into account by language teachers and students in expanding their territory of literacy.

• English Language & Literature Teaching
• /
• v.17 no.4
• /
• pp.229-250
• /
• 2011

Despite the social turn in literacy studies, few empirical studies have investigated the practical applications and learning experiences of multimodal composition pedagogy. Using a qualitative research approach, this study examines undergraduates' experiences in producing multimodal texts. Findings report that students' experiences in a multimodal composition class epitomize enjoyable learning. Students enjoyed their learning process because (a) the multimodal literacy curriculum filled the pedagogical gap between the conventional school-sponsored alphabetic literacy pedagogy and widespread out-of-school multimodal literacy practices and (b) the usefulness of the curriculum helped students enhance their intrinsic motivation to learn and compose. By questioning fundamental assumptions about what counts as knowledge in the current ecology of literacies, the authors argue for a dynamic view of literacy into practice.

• 한국HCI학회:학술대회논문집
• /
• 2009.02a
• /
• pp.278-284
• /
• 2009

Multimodal data to have several modalities such as videos, images, sounds and texts for one contents is increasing. Since this type of data has ill-defined format, it is not easy to represent the crossmodal information for them explicitly. So, we proposed new method to extract and analyze vision-language crossmodal association information using the documentaries video data about the nature. We collected pairs of images and captions from 3 genres of documentaries such as jungle, ocean and universe, and extracted a set of visual words and that of text words from them. We found out that two modal data have semantic association on crossmodal association information from this analysis.

• Journal of the Korea Convergence Society
• /
• v.9 no.1
• /
• pp.419-431
• /
• 2018

In this paper, we propose a novel method for monitoring mood trend of Twitter users by analyzing their daily tweets for a long period. Then, to more accurately understand their tweets, we analyze all types of content in tweets, i.e., texts and emoticons, and images, thus develop a multimodal sentiment analysis method. In the proposed method, two single-modal analyses first are performed to extract the users' moods hidden in texts and images: a lexicon-based and learning-based text classifier and a learning-based image classifier. Thereafter, the extracted moods from the respective analyses are combined into a tweet mood and aggregated a daily mood. As a result, the proposed method generates a user daily mood flow graph, which allows us for monitoring the mood trend of users more intuitively. For evaluation, we perform two sets of experiment. First, we collect the data sets of 40,447 data. We evaluate our method via comparing the state-of-the-art techniques. In our experiments, we demonstrate that the proposed multimodal analysis method outperforms other baselines and our own methods using text-based tweets or images only. Furthermore, to evaluate the potential of the proposed method in monitoring users' mood trend, we tested the proposed method with 40 depressive users and 40 normal users. It proves that the proposed method can be effectively used in finding depressed users.

• The Journal of the Convergence on Culture Technology
• /
• v.5 no.3
• /
• pp.9-23
• /
• 2019

The artificial intelligence speaker market is in a new age of mounting displays. This study aimed to analyze the difference of experience using artificial intelligent speakers in terms of usage context, according to the presence or absence of displays. This was achieved by using semantic network analysis to determine how the online review texts of Amazon Echo Show and Echo Plus consisted of different UX issues with structural differences. Based on the physical context and the social context of the user experience, the ego network was constructed to draw out major issues. Results of the analysis show that users' expectation gap is generated according to the display presence, which can lead to negative experiences. Also, it was confirmed that the Multimodal interface is more utilized in the kitchen than in the bedroom, and can contribute to the activation of communication among family members. Based on these findings, we propose a user experience strategy to be considered in display type speakers to be launched in Korea in the future.

• Journal of Korean Elementary Science Education
• /
• v.42 no.1
• /
• pp.161-177
• /
• 2023

Scientific explanation is composed of various representations such as texts, diagrams, and graphs, and each representation contributes to expanding scientific meaning by connecting similar but slightly different meanings as a 'mode'. Based on this perspective of social semiotics, we aimed to identify the characteristics of meaning formation demonstrated in students' science writing (verbal mode) and science drawing (visual mode) and to discuss the pedagogical use of multimodal representations. To that end, 18 science drawings and 18 scientific writings constructed by science-gifted elementary students on air pressure were collected. The characteristics of the drawn and written explanations were then analyzed from the affordance perspective in social semiotics. In science drawing, students showed a tendency to use the affordance of the visual mode to infer concrete changes from the particle view, such as the movement of air particles, the number of air particles, and the collision of particles. In science writing, students used the affordance of the verbal mode mainly to infer the causal relationship between the concept of air pressure and other related factors at an abstract level. Based on those results, we discuss the educational implications and provide concrete examples of how to use the unique affordances of each form to complement one another.

• Korean Journal of Cognitive Science
• /
• v.21 no.1
• /
• pp.157-175
• /
• 2010

Understanding dialogue participant's emotion is important as well as decoding the explicit message in human communication. It is well known that non-verbal elements are more suitable for conveying speaker's emotions than verbal elements. Written texts, however, contain a variety of linguistic units that express emotions. This study aims at analyzing components for constructing an emotion ontology, that provides us with numerous applications in Human Language Technology. A majority of the previous work in text-based emotion processing focused on the classification of emotions, the construction of a dictionary describing emotion, and the retrieval of those lexica in texts through keyword spotting and/or syntactic parsing techniques. The retrieved or computed emotions based on that process did not show good results in terms of accuracy. Thus, more sophisticate components analysis is proposed and the linguistic factors are introduced in this study. (1) 5 linguistic types of emotion expressions are differentiated in terms of target (verbal/non-verbal) and the method (expressive/descriptive/iconic). The correlations among them as well as their correlation with the non-verbal expressive type are also determined. This characteristic is expected to guarantees more adaptability to our ontology in multi-modal environments. (2) As emotion-related components, this study proposes 24 emotion types, the 5-scale intensity (-2~+2), and the 3-scale polarity (positive/negative/neutral) which can describe a variety of emotions in more detail and in standardized way. (3) We introduce verbal expression-related components, such as 'experiencer', 'description target', 'description method' and 'linguistic features', which can classify and tag appropriately verbal expressions of emotions. (4) Adopting the linguistic tag sets proposed by ISO and TEI and providing the mapping table between our classification of emotions and Plutchik's, our ontology can be easily employed for multilingual processing.

• International Journal of Computer Science & Network Security
• /
• v.21 no.4
• /
• pp.103-114
• /
• 2021

A conceptual model can be used to manage complexity in both the design and implementation phases of the system development life cycle. Such a model requires a firm grasp of the abstract principles on which a system is based, as well as an understanding of the high-level nature of the representation of entities and processes. In this context, models can have distinct architectural characteristics. This paper discusses model multiplicity (e.g., unified modeling language [UML]), model singularity (e.g., object-process methodology [OPM], thinging machine [TM]), and a heterogeneous model that involves multiplicity and singularity. The basic idea of model multiplicity is that it is not possible to present all views in a single representation, so a number of models are used, with each model representing a different view. The model singularity approach uses only a single unified model that assimilates its subsystems into one system. This paper is concerned with current approaches, especially in software engineering texts, where multimodal UML is introduced as the general-purpose modeling language (i.e., UML is modeling). In such a situation, we suggest raising the issue of multiplicity versus singularity in modeling. This would foster a basic appreciation of the UML advantages and difficulties that may be faced during modeling, especially in the educational setting. Furthermore, we advocate the claim that a multiplicity of views does not necessitate a multiplicity of models. The model singularity approach can represent multiple views (static, behavior) without resorting to a collection of multiple models with various notations. We present an example of such a model where the static representation is developed first. Then, the dynamic view and behavioral representations are built by incorporating a decomposition strategy interleaved with the notion of time.