ETRI Journal

Electronics and Telecommunications Research Institute (한국전자통신연구원)
권호 표지
DOI QR코드

DOI QR Code

Real-world multimodal lifelog dataset for human behavior study

  • Chung, Seungeun (Artificial Intelligence Laboratory, Electronics and Telecommunications Research Institute) ;
  • Jeong, Chi Yoon (Artificial Intelligence Laboratory, Electronics and Telecommunications Research Institute) ;
  • Lim, Jeong Mook (Artificial Intelligence Laboratory, Electronics and Telecommunications Research Institute) ;
  • Lim, Jiyoun (Artificial Intelligence Laboratory, Electronics and Telecommunications Research Institute) ;
  • Noh, Kyoung Ju (Artificial Intelligence Laboratory, Electronics and Telecommunications Research Institute) ;
  • Kim, Gague (Artificial Intelligence Laboratory, Electronics and Telecommunications Research Institute) ;
  • Jeong, Hyuntae (Artificial Intelligence Laboratory, Electronics and Telecommunications Research Institute)
  • Received : 2020.11.26
  • Accepted : 2021.07.01
  • Published : 2022.06.10
Download PDF
⟨ Previous Next ⟩

Abstract

To understand the multilateral characteristics of human behavior and physiological markers related to physical, emotional, and environmental states, extensive lifelog data collection in a real-world environment is essential. Here, we propose a data collection method using multimodal mobile sensing and present a long-term dataset from 22 subjects and 616 days of experimental sessions. The dataset contains over 10 000 hours of data, including physiological, data such as photoplethysmography, electrodermal activity, and skin temperature in addition to the multivariate behavioral data. Furthermore, it consists of 10 372 user labels with emotional states and 590 days of sleep quality data. To demonstrate feasibility, human activity recognition was applied on the sensor data using a convolutional neural network-based deep learning model with 92.78% recognition accuracy. From the activity recognition result, we extracted the daily behavior pattern and discovered five representative models by applying spectral clustering. This demonstrates that the dataset contributed toward understanding human behavior using multimodal data accumulated throughout daily lives under natural conditions.

Keywords

Acknowledgement

This work was supported by Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government. [21ZS1100, Core Technology Research for Self-Improving Integrated Artificial Intelligence System].

References

  1. D. Anguita et al., A public domain dataset for human activity recognition using smartphones, Esann 3 (2013), 3.
  2. O. Banos et al., Design, implementation and validation of a novel open framework for agile development of mobile health applications, Biomed. Eng. Online 14 (2015), no. S2, S6.
  3. D. Micucci, M. Mobilio, and P. Napoletano, Unimib shar: A dataset for human activity recognition using acceleration data from smartphones, Appl. Sci. 7 (2017), no. 10, 1101. https://doi.org/10.3390/app7101101
  4. A. Reiss and D. Stricker, Introducing a new benchmarked dataset for activity monitoring, in Proc. Int. Symp. Wearable Comput., (Newcastle, UK), June 2012, pp. 108-109.
  5. D. Roggen et al., Collecting complex activity datasets in highly rich networked sensor environments, in Proc. Int. Conf. Netw. Sens. Syst. (INSS), (Kassel, Germany), June 2010, pp. 233-240.
  6. S. Chung et al., Sensor data acquisition and multimodal sensor fusion for human activity recognition using deep learning, Sensors 19 (2019), no. 7, 1716. https://doi.org/10.3390/s19071716
  7. C. Y. Jeong and M. Kim, An energy-efficient method for human activity recognition with segment-level change detection and deep learning, Sensors 19 (2019), no. 17, article no. 3688.
  8. O. F. Ince et al., Human activity recognition with analysis of angles between skeletal joints using a rgb-depth sensor, ETRI J. 42 (2020), no. 1, 78-89. https://doi.org/10.4218/etrij.2018-0577
  9. M. Kim and C. Y. Jeong, Label-preserving data augmentation for mobile sensor data, Multidimens. Syst. Signal Process. 32 (2021), no. 1, 115-129. https://doi.org/10.1007/s11045-020-00731-2
  10. C. Gurrin, A. F. Smeaton, and A. R. Doherty, Lifelogging: Personal big data, Found. Trends Inf. Ret. 8 (2014), no. 1, 1-125, doi:10.1561/1500000033.
  11. A. J. Sellen and S. Whittaker, Beyond total capture: A constructive critique of lifelogging, Commun. ACM 53 (2010), no. 5, 70-77, doi:10.1145/1735223.1735243.
  12. SIGMM, Lsc '18: Proceedings of the 2018 ACM Workshop on the Lifelog Search Challenge, ACM, New York, NY, USA, 2018.
  13. C. Gurrin et al., LTA 2016: The first workshop on lifelogging tools and applications, in Proc. ACM Int. Conf. Multimed. (New York, NY, USA), Oct. 2016, pp. 1487-1488.
  14. D. T. Dang Nguyen et al., Overview of imageCLEFlifelog 2019: Solve my life puzzle and lifelog moment retrieval, in Proc. Conf. Labs Eval. Forum. (Lugano, Switzerland), Sept. 2019.
  15. C. Gurrin et al., Overview of the NTCIR-14 lifelog-3 task, in Proc. NTCIR Conf. (Tokyo, Japan), June 2019, pp. 14-26.
  16. Y. Vaizman, K. Ellis, and G. Lanckriet, Recognizing detailed human context in the wild from smartphones and smartwatches, IEEE Pervasive Comput. 16 (2017), no. 4, 62-74. https://doi.org/10.1109/MPRV.2017.3971131
  17. Y. Vaizman et al., Extrasensory app: Data collection in-the-wild with rich user interface to self-report behavior, in Proc. CHI Conf. Hum. Factors Comput. Syst. (New York, NY, USA), Apr. 2018, pp. 1-12.
  18. Y. Vaizman, N. Weibel, and G. Lanckriet, Context recognition in-the-wild: Unified model for multi-modal sensors and multi-label classification, Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1 (2018), no. 4, 1-22.
  19. G. M. Harari et al., Smartphone sensing methods for studying behavior in everyday life, Curr. Opin. Behav. Sci. 18 (2017), 83-90. https://doi.org/10.1016/j.cobeha.2017.07.018
  20. L. A. Clark and D. Watson, Mood and the mundane: Relations between daily life events and self-reported mood, J. Pers. Soc. Psychol. 54 (1988), no. 2, 296-308. https://doi.org/10.1037/0022-3514.54.2.296
  21. P. R. Giacobbi, H. A. Hausenblas, and N. Frye, A naturalistic assessment of the relationship between personality, daily life events, leisure-time exercise, and mood, Psychol. Sport Exerc. 6 (2005), no. 1, 67-81. https://doi.org/10.1016/j.psychsport.2003.10.009
  22. S. N. Rogers et al., The addition of mood and anxiety domains to the university of washington quality of life scale, Head & Neck: J. Sci. Specialties Head Neck 24 (2002), no. 6, 521-529. https://doi.org/10.1002/hed.10106
  23. K. J. Stewart et al., Are fitness, activity, and fatness associated with health-related quality of life and mood in older persons?, J. Cardiopulm. Rehabil. Prev. 23 (2003), no. 2, 115-121. https://doi.org/10.1097/00008483-200303000-00009
  24. K. H. Kim, S. W. Bang, and S. R. Kim, Emotion recognition system using short-term monitoring of physiological signals, Med. Biol. Eng. Comput. 42 (2004), no. 3, 419-427. https://doi.org/10.1007/BF02344719
  25. L. Shu et al., A review of emotion recognition using physiological signals, Sensors 18 (2018), no. 7, 2074. https://doi.org/10.3390/s18072074
  26. J. Kim and E. Andre, Emotion recognition based on physiological changes in music listening, IEEE Trans. Pattern Anal. Mach. Intell. 30 (2008), no. 12, 2067-2083. https://doi.org/10.1109/TPAMI.2008.26
  27. S. Koelstra et al., Deap: A database for emotion analysis; using physiological signals, IEEE Trans. Affect. Comput. 3 (2011), no. 1, 18-31. https://doi.org/10.1109/T-AFFC.2011.15
  28. T. Song et al., Mped: A multi-modal physiological emotion database for discrete emotion recognition, IEEE Access 7 (2019), 12177-12191. https://doi.org/10.1109/access.2019.2891579
  29. M.-F. Shao et al., Sleep quality and quality of life in female shift-working nurses, J. Adv. Nurs. 66 (2010), no. 7, 1565-1572. https://doi.org/10.1111/j.1365-2648.2010.05300.x
  30. J. Zeitlhofer et al., Sleep and quality of life in the austrian population, Acta Neurol. Scand. 102 (2000), no. 4, 249-257. https://doi.org/10.1034/j.1600-0404.2000.102004249.x
  31. B. Bower et al., Poor reported sleep quality predicts low positive affect in daily life among healthy and mood-disordered persons, J. Sleep Res. 19 (2010), no. 2, 323-332. https://doi.org/10.1111/j.1365-2869.2009.00816.x
  32. D. K. Thomsen et al., Rumination-relationship with negative mood and sleep quality, Pers. Individ. Differ. 34 (2003), no. 7, 1293-1301. https://doi.org/10.1016/S0191-8869(02)00120-4
  33. K. J. Reid et al., Aerobic exercise improves self-reported sleep and quality of life in older adults with insomnia, Sleep Med. 11 (2010), no. 9, 934-940. https://doi.org/10.1016/j.sleep.2010.04.014
  34. S. Ancoli-Israel et al., The role of actigraphy in the study of sleep and circadian rhythms, Sleep 26 (2003), no. 3, 342-392. https://doi.org/10.1093/sleep/26.3.342
  35. A. D. Krystal and J. D. Edinger, Measuring sleep quality, Sleep Med. 9 (2008), no. 1, S10-S17. https://doi.org/10.1016/S1389-9457(08)70011-X
  36. D. J. Buysse et al., The pittsburgh sleep quality index: A new instrument for psychiatric practice and research, Psychiatry Res. 28 (1989), no. 2, 193-213. https://doi.org/10.1016/0165-1781(89)90047-4
  37. G. Landry, J. Best, and T. Liu-Ambrose, Measuring sleep quality in older adults: A comparison using subjective and objective methods, Front. Aging Neurosci. 7 (2015), article no. 166.
  38. S. Shiffman, A. A. Stone, and M. R. Hufford, Ecological momentary assessment, Annu. Rev. Clin. Psychol. 4 (2008), 1-32. https://doi.org/10.1146/annurev.clinpsy.3.022806.091415
  39. D.-T. Dang-Nguyen et al., in Proc. Int. Workshop Content-based Multimed. Indexing (New York, NY, USA), June 2017, pp. 1-6.
  40. C. Gurrin et al., Ntcir lifelog: The first test collection for lifelog research, in Proc. Int. ACM SIGIR Conf. Res. Dev. Inf. Retr. (New York, NY, USA), July 2016, pp. 705-708.
  41. D.-T. Dang-Nguyen et al., Overview of imageCLEFlifelog 2017: Lifelog retrieval and summarization, in Proc. ImageCLEF 2017 (Dublin, Ireland), Sept. 2017.
  42. D.-T. Dang-Nguyen et al., Overview of imageCLEFlifelog 2018: Daily living understanding and lifelog moment retrieval, in Proc. Conf. Labs Eval. Forum. (Avignon, France), Sept. 2018.
  43. J. A. Russell, A circumplex model of affect, J. Pers. Soc. Psychol. 39 (1980), no. 6, article no. 1161.
  44. Empatica, E4 wristband: Real-time physiological signals, Available from https://empatica.com/research/e4 [last accessed March 2021].
  45. Withings, Sleep tracking mat, Available from https://www.withings.com/kr/en/sleep [last accessed March 2021].
  46. M. Barzegar, A. Sadeghi-Niaraki, and M. Shakeri, Spatial experience based route finding using ontologies, ETRI J. 42 (2020), no. 2, 247-257. https://doi.org/10.4218/etrij.2017-0246
  47. F. Bergadano, Keyed learning: An adversarial learning framework-formalization, challenges, and anomaly detection applications, ETRI J. 41 (2019), no. 5, 608-618. https://doi.org/10.4218/etrij.2019-0140
  48. J. K. Bii, R. Rimiru, and R. W. Mwangi, Adaptive boosting in ensembles for outlier detection: Base learner selection and fusion via local domain competence, ETRI J. 42 (2020), no. 6, 886-898. https://doi.org/10.4218/etrij.2019-0205