Journal of the Korea Society of Computer and Information (한국컴퓨터정보학회논문지)

Korean Society of Computer Information (한국컴퓨터정보학회)
권호 표지
DOI QR코드

DOI QR Code

Intelligent User Pattern Recognition based on Vision, Audio and Activity for Abnormal Event Detections of Single Households

  • Jung, Ju-Ho (Computer Information Technology, Korea National University of Transportation) ;
  • Ahn, Jun-Ho (Computer Information Technology, Korea National University of Transportation)
  • Received : 2019.03.08
  • Accepted : 2019.04.07
  • Published : 2019.05.31
Download PDF
⟨ Previous Next ⟩

Abstract

According to the KT telecommunication statistics, people stayed inside their houses on an average of 11.9 hours a day. As well as, according to NSC statistics in the united states, people regardless of age are injured for a variety of reasons in their houses. For purposes of this research, we have investigated an abnormal event detection algorithm to classify infrequently occurring behaviors as accidents, health emergencies, etc. in their daily lives. We propose a fusion method that combines three classification algorithms with vision pattern, audio pattern, and activity pattern to detect unusual user events. The vision pattern algorithm identifies people and objects based on video data collected through home CCTV. The audio and activity pattern algorithms classify user audio and activity behaviors using the data collected from built-in sensors on their smartphones in their houses. We evaluated the proposed individual pattern algorithm and fusion method based on multiple scenarios.

Keywords

CPTSCQ_2019_v24n5_59_f0001.png 이미지

Fig. 1. State Chart Diagram for Vision Pattern Algorithm

CPTSCQ_2019_v24n5_59_f0002.png 이미지

Fig. 2. State Chart Diagram for Audio Pattern Algorithm

CPTSCQ_2019_v24n5_59_f0003.png 이미지

Fig. 3. State Chart Diagram for Activity Pattern Algorithm

CPTSCQ_2019_v24n5_59_f0004.png 이미지

Fig. 4. State Chart Diagram for Fusion Method

CPTSCQ_2019_v24n5_59_f0005.png 이미지

Fig. 5. Experiment example using Vision Pattern Algorithm

CPTSCQ_2019_v24n5_59_f0006.png 이미지

Fig. 6. Tensorflow Object Detection Faster R-cnn inception v2 algorithm apply to videos

CPTSCQ_2019_v24n5_59_f0007.png 이미지

Fig. 7. Patterns Occurring in Daily life

CPTSCQ_2019_v24n5_59_f0008.png 이미지

Fig. 8. Patterns that occur where noise is present

CPTSCQ_2019_v24n5_59_f0009.png 이미지

Fig. 9. Pattern Occurring in quiet places

CPTSCQ_2019_v24n5_59_f0010.png 이미지

Fig. 10. Application developed to measure acceleration sensor

CPTSCQ_2019_v24n5_59_f0011.png 이미지

Fig. 11. Activity patterns while carrying smartphones

CPTSCQ_2019_v24n5_59_f0012.png 이미지

Fig. 12. Abnormal event patterns while carrying smartphones

Table 1. Vision Pattern Algorithm of Recall, Precision, Accuracy

CPTSCQ_2019_v24n5_59_t0001.png 이미지

Table 2. Audio Pattern Algorithm of Recall, Precision, Accuracy

CPTSCQ_2019_v24n5_59_t0002.png 이미지

Table 3. Activity Pattern Algorithm of Recall, Precision, Accuracy

CPTSCQ_2019_v24n5_59_t0003.png 이미지

Table 4. Comparison Scenarios of Fusion with Each Algorithm

CPTSCQ_2019_v24n5_59_t0004.png 이미지

References

  1. "Launching Big Data LifeProg: Analyzing and Utilizing Travel and Staying Patterns", https://m.post.naver.com/viewer/postView.nhn?volumeNo=8897860&memberNo=30305360
  2. "Home Accident Statistics: Is Your Home as Safe as You Think?", https://www.asecurelife.com/home-accidentstatistics/
  3. Glen Debard, Marc Mertens, Toon Goedeme, Tinne Tuytelaars and Bart Vanrumst, "Three Ways to Improve the Performance of Real-Life Camera-Based Fall Detection Systems", Journal of Sensors (2017)
  4. Miao Yu, Liyun Gong, Stefanos Kollias, "Computer vision based fall detection by a convolutional neural network", ACM (2017)
  5. Koldo de Miguel, Alberto Brunete, Miguel Hernando and Ernesto Gambao, "Home Camera-Based Fall Detection System for the Elderly", Multidisciplinary Digital Publishing Institute (MDPI), Sensors, 21(2017)
  6. Fouzi Harroua, Nabil Zerroukib, Ying Suna, Amrane Houacineb, "Vision-based fall detection system for improving safety of elderly people", IEEE Instrumentation and Measurement Society, 21, (2017)
  7. Huang J, Rathod V, Sun C, Zhu M, Korattikara A, Fathi A, Fischer I, Wojna Z, Song Y, Guadarrama S, Murphy K., Speed/accuracy trade-offs for modern convolutional object detectors. CVPR 2017, https://github.com/tenso rflow/models/tree/master/research/object_detection
  8. L. Vuegen B. Van Den Broeck P. Karsmakers J. F. Gemmeke B. Vanrumste H. Van hamme, "an mfcc-gmm approach for event detection and classification", IEEE AASP Challenge on Detection and Classification of Acoustic Scenes and Events, pp.1-3 2013
  9. Minkyu Lim, Donghyun Lee, Hosung Park, Yoseb Kang, Junseok Oh, Jeong-Sik Park, Gil-Jin Jang and Ji-Hwan Kim, "Convolutional Neural Network based Audio Event Classification," KSII Transactions on Internet and Information Systems, vol. 12, no. 6, pp. 2748-2760, 2018. DOI: 10.3837/tiis.2018.06.01.
  10. Subhasmita Sahoo, Aurobinda Routray, "Detecting Aggression in Voice Using Inverse Filtered Speech Features", IEEE Transactions on Affective Computing ( Volume: 9, Issue: 2, April-June 1 2018 ), pp.217 - 226, DOI: 10.1109/TAFFC.2016.2615607
  11. Monisha Mohan, Arun P.S, ACCELEROMETER-BASED HUMAN FALL DETECTION AND RESPONSE USING SMARTPHONES, International Journal of Computer Engineering In Research Trends,5, 2017
  12. Zishan Zahidul Islam, Syed Mahir Tazwar, Md. Zahidul Islam, Seiichi Serikawa and Md. Atiqur Rahman Ahad, Automatic Fall Detection System of Unsupervised Elderly People Using Smartphone, IIAE International Conference on Intelligent Systems and Image Processing, 7, 2017
  13. Jose Antonio Santoyo-Ramón, Eduardo Casilari OrcID and Jose Manuel Cano-Garcia, "Analysis of a Smartphone-Based Architecture with Multiple Mobility Sensors for Fall Detection with Supervised Learning", Sensors 2018, 18(4), 1155; doi:10.3390/s18041155
  14. Junho Ahn, Hwijune Park, Juho Jung, Gwang Lee, "Unusual Event Detection Algorithm via Personalized Daily Activity and Vision Patterns for Single Households", International Journal of Engineering &Technology; Vol 8, No 1.4 (2019): Special Issue 4, doi:http://dx.doi.org/10.14419/ijet.v8i1.4.25465
  15. Juho Jung, Junho Ahn, "Intelligent Abnormal Event Detection Algorithm for Single Households at Home via Daily Audio and Vision Patterns", Journal of Internet Computing and Services, doi:http://dx.doi.org/10.7472/jksii.2019.20.1.77

Cited by

  1. 영상, 음성, 활동, 먼지 센서를 융합한 딥러닝 기반 사용자 이상 징후 탐지 알고리즘 vol.21, pp.5, 2020, https://doi.org/10.7472/jksii.2020.21.5.109
  2. 스마트 홈 사용자를 위한 라이다, 영상, 오디오 센서를 이용한 인공지능 이상징후 탐지 알고리즘 vol.22, pp.3, 2019, https://doi.org/10.7472/jksii.2021.22.3.17