• Journal of Multimedia Information System
• /
• v.7 no.2
• /
• pp.107-114
• /
• 2020

This paper proposes a wearable audio book application using two wearable devices, a smartwatch and a hearables. We review requirements of what could be a killer wearable application and design our application based on these elicited requirements. To distinguish our application, we present 7 scenarios and introduce several wearable interaction modalities. To show feasibility of our approach, we design and implement our proof-of-concept prototype on Android emulator as well as on a commercial smartwatch. We thoroughly address how different interaction modalities are designed and implemented in the Android platform. Lastly, we show latency of the multi-modal and alternative interaction modalities that can be gracefully handled in wearable audio application use cases.

• Sleep Medicine and Psychophysiology
• /
• v.11 no.1
• /
• pp.5-9
• /
• 2004

Obstructive sleep apnea (OSA) is a common disorder characterized by recurrent cessation of breathing due to complete or partial upper airway occlusion during sleep. The incompetent tone of palatal, pharngeal, and glossal muscles which fail to maintain airway patency during sleep causes narrowing of the airway dimension and increased resistance of breathing. The identification of the sites of upper airway obstruction in patients with OSA is important in understanding the pathogenesis and deciding the treatment modality of snoring and/or OSA. Various upper airway imaging modalities have been used to assess upper airway size and precise localization of the sites of upper airway obstruction during sleep. Dynamic imaging modalities enabled assessment of dimensional changes in the upper airway during respiration and sleep. This article focused on reviews of various upper airway imaging modalities, especially dynamic upper airway imaging studies providing important information on the pathogenesis of OSA.

• Journal of the Ergonomics Society of Korea
• /
• v.16 no.1
• /
• pp.85-96
• /
• 1997

This article reviews the effects of VDT tasks on multiple resources for processing and storage in short-term working memory. MD and PD method were introduced toevaluate the modalities (auditory-visual) in the multiple resources model. The subjects conducted 2 sessions of 50 minites VDT tasks. Before, between and after VDT tasks, MD, PD task performance scores and CFF(critical flicker frequency0 values were measured. The review suggested that the modalities of human information processing in working memory were affected by VDT tasks with different task contents.

• Proceedings of the IEEK Conference
• /
• 2003.07d
• /
• pp.1475-1478
• /
• 2003

When multimedia contents are adapted to different terminals in the ubiquitous computing environment, the contents' modalities can be converted variously. In this paper, we present an efficient user preference that enable user to specify his choices on the modalities of the adapted contents. We also propose the methods to systematically integrate the user preference into the content adaptation process. This modality conversion preference has been developed as a description tool for MPEG-21 Digital Item Adaptation (DIA).

• Journal of Broadcast Engineering
• /
• v.8 no.4
• /
• pp.339-350
• /
• 2003

The Part 7 of MPEG-21, called Digital Item Adaptation, aims at an interoperable transparent access of multimedia contents in heterogeneous environments. This standard facilitates the development of Universal Multimedia Access (UMA) systems, which adapt the rich multimedia contents to provide user the best possible presentation under the constraints of various terminals and network connections. Content adaptation has two major aspects: one is modality conversion that converts content from one modality (e.g. video) to different modalities (e.g. image) the other is content scaling that changes the titrates (or qualities) of the contents without converting their modalities. At the output of adaptation process, the highly-subjective qualities of adapted contents nay vary widely with respect to point-of-views of different providers and different users. So, user should have some control on the adaptation process. In this paper, we describe two description tools of user characteristics, the presentation priority preference and the modality conversion preference, which allow user to have flexible choices on the qualities and modalities of output contents. We also present a systematic approach to integrate these user preferences into the adaptation process. These description tools are developed in the process of MPEG-21 standardization.

• Journal of information and communication convergence engineering
• /
• v.20 no.3
• /
• pp.174-180
• /
• 2022

Emotion recognition is an essential component of complete interaction between human and machine. The issues related to emotion recognition are a result of the different types of emotions expressed in several forms such as visual, sound, and physiological signal. Recent advancements in the field show that combined modalities, such as visual, voice and electroencephalography signals, lead to better result compared to the use of single modalities separately. Previous studies have explored the use of multiple modalities for accurate predictions of emotion; however the number of studies regarding real-time implementation is limited because of the difficulty in simultaneously implementing multiple modalities of emotion recognition. In this study, we proposed an emotion recognition system for real-time emotion recognition implementation. Our model was built with a multithreading block that enables the implementation of each modality using separate threads for continuous synchronization. First, we separately achieved emotion recognition for each modality before enabling the use of the multithreaded system. To verify the correctness of the results, we compared the performance accuracy of unimodal and multimodal emotion recognitions in real-time. The experimental results showed real-time user emotion recognition of the proposed model. In addition, the effectiveness of the multimodalities for emotion recognition was observed. Our multimodal model was able to obtain an accuracy of 80.1% as compared to the unimodality, which obtained accuracies of 70.9, 54.3, and 63.1%.

• Journal of Information Processing Systems
• /
• v.13 no.1
• /
• pp.8-25
• /
• 2017

This survey paper explores the application of multimodal feedback in automated systems for motor learning. In this paper, we review the findings shown in recent studies in this field using rehabilitation and various motor training scenarios as context. We discuss popular feedback delivery and sensing mechanisms for motion capture and processing in terms of requirements, benefits, and limitations. The selection of modalities is presented via our having reviewed the best-practice approaches for each modality relative to motor task complexity with example implementations in recent work. We summarize the advantages and disadvantages of several approaches for integrating modalities in terms of fusion and frequency of feedback during motor tasks. Finally, we review the limitations of perceptual bandwidth and provide an evaluation of the information transfer for each modality.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.8 no.2
• /
• pp.483-503
• /
• 2014

In this paper, human action recognition using pyramid histograms of oriented gradients and collaborative multi-task learning is proposed. First, we accumulate global activities and construct motion history image (MHI) for both RGB and depth channels respectively to encode the dynamics of one action in different modalities, and then different action descriptors are extracted from depth and RGB MHI to represent global textual and structural characteristics of these actions. Specially, average value in hierarchical block, GIST and pyramid histograms of oriented gradients descriptors are employed to represent human motion. To demonstrate the superiority of the proposed method, we evaluate them by KNN, SVM with linear and RBF kernels, SRC and CRC models on DHA dataset, the well-known dataset for human action recognition. Large scale experimental results show our descriptors are robust, stable and efficient, and outperform the state-of-the-art methods. In addition, we investigate the performance of our descriptors further by combining these descriptors on DHA dataset, and observe that the performances of combined descriptors are much better than just using only sole descriptor. With multimodal features, we also propose a collaborative multi-task learning method for model learning and inference based on transfer learning theory. The main contributions lie in four aspects: 1) the proposed encoding the scheme can filter the stationary part of human body and reduce noise interference; 2) different kind of features and models are assessed, and the neighbor gradients information and pyramid layers are very helpful for representing these actions; 3) The proposed model can fuse the features from different modalities regardless of the sensor types, the ranges of the value, and the dimensions of different features; 4) The latent common knowledge among different modalities can be discovered by transfer learning to boost the performance.

• Proceedings of the Korean Institute of Information and Commucation Sciences Conference
• /
• 2022.05a
• /
• pp.416-418
• /
• 2022

Glioma is a type of brain tumor that occurs in glial cells and is classified into two types: high hrade hlioma with a poor prognosis and low grade glioma. Magnetic resonance imaging (MRI) as a non-invasive method is widely used in glioma diagnosis research. Studies to obtain complementary information by combining multiple modalities to overcome the incomplete information limitation of single modality are being conducted. In this study, we developed a 3D CNN-based model that applied input-level fusion to MRI of four modalities (T1, T1Gd, T2, T2-FLAIR). The trained model showed classification performance of 0.8926 accuracy, 0.9688 sensitivity, 0.6400 specificity, and 0.9467 AUC on the validation data. Through this, it was confirmed that the grade of glioma was effectively classified by learning the internal relationship between various modalities.

• Journal of the Optical Society of Korea
• /
• v.18 no.2
• /
• pp.151-155
• /
• 2014

Although various ophthalmic imaging methods, including fundus photography and optical coherence tomography, have been applied for effective diagnosis of ocular diseases with high spatial resolution, most of them are limited by shallow imaging penetration depth and a narrow field of view. Also, many of those imaging modalities are optimized to provide microscopic anatomical information, while functional or cellular information is lacking. Compared to other ocular imaging modalities, photoacoustic imaging can achieve relatively deep penetration depth and provide more detailed functional and cellular data based on photoacoustic signal generation from endogenous contrast agents such as hemoglobin and melanin. In this paper, array-based ultrasound and photoacoustic imaging was demonstrated to visualize pigmentation in the eye as well as overall ocular structure. Fresh porcine eyes were visualized using a real-time ultrasound micro-imaging system and an imaging probe supporting laser pulse delivery. In addition, limited photoacoustic imaging field of view was improved by an imaging probe tilting method, enabling visualization of most regions of the retina covered in the ultrasound imaging.