Development of a Machine-Learning based Human Activity Recognition System including Eastern-Asian Specific Activities

Abstract

The purpose of this study is to develop a human activity recognition (HAR) system, which distinguishes 13 activities, including five activities commonly dealt with in conventional HAR researches and eight activities from the Eastern-Asian culture. The eight special activities include floor-sitting/standing, chair-sitting/standing, floor-lying/up, and bed-lying/up. We used a 3-axis accelerometer sensor on the wrist for data collection and designed a machine learning model for the activity classification. Data clustering through preprocessing and feature extraction/reduction is performed. We then tested six machine learning algorithms for recognition accuracy comparison. As a result, we have achieved an average accuracy of 99.7% for the 13 activities. This result is far better than the average accuracy of current HAR researches based on a smartwatch (89.4%). The superiority of the HAR system developed in this study is proven because we have achieved 98.7% accuracy with publically available 'pamap2' dataset of 12 activities, whose conventionally met the best accuracy is 96.6%.

1. Introduction

Human activity recognition (HAR), a technique for automatically recognizing a person's behavior, has been an important research topic due to its applicability to various pervasive computing fields such as healthcare, gaming and sports, and general-purpose monitoring system [1]. HAR methods use sensors (human-body embedded sensors and environmental sensors) or more sophisticated techniques such as image processing [2]. However, the use of environmental sensors or image processing may not be adequate because it may require a high cost for system implementation. In particular, the image processing method may have personal privacy problems, so the controversy continues [3]. Therefore, this study aims to develop a method of recognizing human activities using embedded sensors mounted on a person, which mitigates cost and privacy issues.

We can summarize the process of the HAR system as (1) acquiring sensor data from activities (Data Acquisition), (2) classifying patterns from the data (Feature Selection), (3) deciding the activity of the newly acquired data (Activity Recognition). Consequently, HAR system development involves (1) deciding the sensors and devices to collect data, (2) collecting and processing data, (3) developing a learning system that can classify activities based on the collected sensor data [4]. [Figure 1] shows the overall process and three main components of the HAR system.

(Figure 1) HAR process and components

However, even though attaching various sensors to the body, including smart-clothes, can improve the HAR rate, it also causes inconvenience to everyday life [5]. Smartphones have been used as an alternative to reduce this discomfort. Still, they are not always carried on by a person, and also the determination of the appropriate parts of the body to place the device has been a long issue [6]. Consequently, there have been various attempts to classify human activities using sensors built into the smartwatch, minimizing disruption to daily activities [7].

Today's smartwatches usually come with several sensors, at least with accelerometer, gyroscope, and magnetometer. In the field of HAR, these sensors are used individually or in combination. Although there are slight differences in methods, they are not very large in performance. Considering the computational overhead, using a single sensor can be useful for HAR [4]. Therefore, in this study, an artificial intelligent classification system is proposed, which acquires data using a single 3-axis acceleration sensor mounted on the wrist.

One of the originalities of the classification system developed in this study is that it deals with Eastern-Asian culture's distinctive activities in addition to the five typical activities handled by other studies [4]. The eight special activities include floor-sitting/ standing, chair-sitting/standing, floor-lying/up, and bed-lying/up, which are commonly exercised in the culture. [Table 1] lists the target activities of this study.

(Table 1) 13 activities of the study

We expect that the system provides distinctive activity recognition for the Korean and Japanese. Besides, to minimize discomfort in activities and reduce computational complexity, we used a wrist-mounted device and its accelerometer for the system development. In other words, this study provides an artificial intelligence classification system that distinguishes 13 daily activities popular in the Eastern-Asian culture using only an accelerometer worn on the wrist.

We used the supervised conventional machine learning (CML) method as a method of artificial intelligence classification because we are classifying known intermediate-sized activities (See Section 2.3 for the rationale). We tested six representative CML methods [4, 8], such as k-Nearest Neighbours (kNN), Decision Tree (DT), Random Forest (RF), Support Vector Machine (SVM), Multilayer Perceptron (MLP), and Extreme Gradient Boosting (XGB), and compared their recognition rates to identify the most suitable classification algorithm.

Data sensing, preprocessing (filtering and segmentation), feature extraction and reduction, and machine learning and classification are performed to increase the HAR efficiency. We have tested various algorithms for each step in the process and compared their performances. We found the selection of the dimensionality reduction algorithm is the most critical factor for the classification performance.

For the 13 target activities, this study achieved an average recognition rate of 99.7% through acquisition, filtering, feature extraction, dimensionality reduction, and classification of the 3-axis accelerometer data. This recognition rate significantly exceeds the performance of 150 machine learning studies surveyed in paper [4]. In the survey, the average recognition rates of the standalone device, smartphone, and smartwatch are 93.2%, 91.8%, and 89.4%, respectively. The system also achieved a recognition rate of 98.7% with a publically available data set whose currently reported the highest recognition rate is 96.6%.

The composition of the paper is as follows.

Section 1 describes the purpose and scope of this study. In Section 2, we look at the framework for the HAR system development and examine the related researches and their HAR performances. Section 3 describes the data acquisition method, data preprocessing, feature extraction and dimensionality reduction, and CML classifiers used. Section 4 compares and analyzes factors and performance that affect the accuracy of the classification. In particular, we examine the recognition rate of various dimensionality reduction and CML methods to derive the most suitable model for our HAR system. Section 5 presents the conclusion of the study.

2. Background and Related Work

There have been various attempts to recognize human activities automatically. As mentioned in Section 1, we can categorize the HAR system into a type using sensors and a type using images. The former includes sensors attached to the body and the use of environmental sensors. However, for computational efficiency and privacy issues, many studies are carried out with portable electronic devices (especially smartphones and smartwatches) with embedded sensors.

2.1 Device and Sensors

Wearable sensors carried on or attached to the body are frequently used in the HAR research. Among them, smartphones or smartwatches embedded accelerometer, gyroscope, and magnetometer are typical. In the related studies, sensors generated signals from activities, such as acceleration, angular velocity, magnetic fields, tilt, shock, vibration, rotation, etc. are used. These data become the primary sources to infer human activities [2, 9]. In most studies, an accelerometer becomes a fundamental unit, and they may also utilize a gyroscope or magnetometer. A survey [4] that analyzed more than 250 related studies published since 2015 reports an average of 92.5% HAR accuracy with the accelerometer's sole use. It also says 93.0% accuracy with gyroscope and 92.8% accuracy with the magnetometer. Even the type of activities handled by each study differs, this figure tells that the accelerometer alone can produce activity recognition rates close to those of using multiple sensors. Based on this analysis, this study implements a HAR system using only a single accelerometer.

2.2 Data Manipulation

2.2.1 Preprocessing

As shown in [Figure 1], the signal acquired from the sensor goes through filtering and window segmentation preprocessing. The filtering removes outside elements such as noise or other artifacts to form accurate sensor data.

The main concern in the window segmentation is how often the signal is measured and how many measurements are needed (window-size) to identify activities. According to [2], signals are usually measured 20 to 100 times per second, and the window-size varies from 0.08 to 30 seconds. However, the prerequisite of the window-size determination is that it should cover at least one full action. Also, this should be in the form of 2ⁿ because only with the format an accurate conversion from time-domain signals to frequent-domain signals is possible, which are needed to define the activities' features.

2.2.2 Features

It is possible to classify activities using machine learning only when the characteristics (features) of activities acquired from the sensors are distinguishable. In general, we extract features from time-domain and frequency-domain signals obtained from sensors and may use them independently or in combination. The time-domain feature refers to the behavior characteristics based on various statistical concepts. Commonly used time-domain features include signal magnitude area, standard deviation, median, variance, skewness, zero-crossing rate, autoregressive coefficient, peak-to-peak values [2]. Frequency-domain features, such as spectral energy, entropy, and dominant frequency, indicate the components extracted from the frequency spectrum constituting the signal. One derives the signal from the time-domain signals through a transformation algorithm such as Fast Fourier Transformation (FFT) [2].

After defining the features, appropriate features to use for machine learning and classification needs to be selected because there may be overlapping components among them. Also, too many features may impose a computational burden on the classification system. Therefore, methods to find a subset of features (feature selection) that provide competitive results are used [2]. However, time-consuming trial-and-error performance testing is unavoidable for the feature selection. Accordingly, researchers have proposed various methods for automatic feature extraction and reduction. These methods, called deep feature extraction in general, automatically extract appropriate features using a deep neural network [10].

An autoencoder is one of the deep feature extraction methods that creates a new set of features in a smaller dimension from the original features. It has a structure in which the hidden layer of the neural network narrows toward the center. We encode input features using this structure and create new features. We can restore near original features from the decoded features. In other words, the neural network learns to encode crucial features that represent the initial input with fewer features. The representative autoencoder includes regularized autoencoders and variational autoencoders [11].

2.3 Human Activity Classification

CML and deep learning (DL) techniques are commonly used machine learning classification methods for HAR researches. CML establishes a stochastic model that learns values associated with the input. kNN, DT, RF, SVM, and MLP methods are often used [12], and XGB is used much in recent researches [8]. DL method focuses on data representation and automatically finds optimal features from input data [13]. Typical examples of DL include Convolutional Neural Network (CNN), Recurrent Neural Networks (RNN), Long Short-Term Memory Network (LSTM), and Extreme Learning Machine (ELM).

A survey [4] that deals with HAR studies after the year 2015 reports that the number of HAR studies using CML is more than that of DL (95:54). When the amount of data is not large, the accuracy of the CML methods is not significantly different from that of DL (92.3%:92.8%). Therefore, it may be more useful to use CML methods when the amount of data is not large, and the computational overhead is a concern. Thus, in this study, we focused on the models that classify activities using CML methods with about 7.2K counts of 13 human activities.

3. HAR System

In this section, we describe the various components of the HAR system we have developed in this study. Focuses are on the data preprocessing, including noise filtering and segmentation, feature extraction, dimensionality reduction, and the CML classifiers.

3.1 Data Acquisition

We used a wearable sensor mounted on the wrist with a 3-axis acceleration (EBIMU - 9DOFV2) sensor and a bluetooth module (FB - 744AS) for activity signal acquisition. The embedded module on the wrist continuously sends the data to the server for data accumulation and processing. As we have discussed in Section 2.1, we only used the 3-axis acceleration data from the 9-axis inertial sensor.

As given in [Table 1], there are 13 human activities to recognize: one stationary activity and 12 transitional activities. The stationary activity includes all static postures such as standing, sitting, and lying. The common sensor sampling rate used for the HAR researches ranges from 20 Hz to 100 Hz [14]. We used a 40 Hz sampling frequency (40 times per second).

3.2 Preprocessing

Preprocessing includes filtering and segmentation [2]. In this step, the reliability of the data increases by removing noise and extreme outliers of the signal. The most commonly used filters are Butterworth filter, Kalman filter, and moving average filter [15]. Among them, we used the moving average filter for this study. [Figure 2] shows the raw data and the data after filtering. Some studies argued that the filtering may or may not affect the classification performance[16], but we found that noise filtering affects the classification performance.

(Figure 2) 3-axis accelerometer signals before and after filtering. (Unfiltered X, Y, Z signals and filtered MA-X, MA-Y, MA-Z signals)

The segmentation window is set to 3.2 seconds, beginning from when one action starts. This interval is large enough to contain one whole transitional movement. 'Large enough' means the transitional and stationary activities that we try to recognize completely occurs in this window, or similar patterns repeat. Other researches also reported that this window size is sufficient for HAR [17]. The window size is determined to be 3.2 seconds so that the count of data is in the form of power of 2. We mentioned the rationale for this in section 2.2.1.

3.3 HAR Process

3.3.1 Time and Frequency Domain Features

To construct a HAR classifier, we should be able to distinguish the characteristics of activities. These characteristics are collectively called features (see Section 2.2.2). There are two categories for features time-domain and frequency-domain features. [Table 2] lists the features used in this study.

(Table 2) Features used

* for x, y, z axis, respectively

3.3.2 Feature Extraction and Reduction

There may be features with overlapping characteristics or little effect on classification. Therefore, removing or compressing overlapped or meaningless entries from these features can reduce the feature dimension for better classification and minimize the computational overhead for machine learning. In this study, we used the Adversarial Autoencoder (AAE) [18] for feature reduction. It uses the adversarial network [19] as a regularization method to solve the fracture problem. AAE showed excellent performance for our HAR study when compared to other dimensionality reduction methods. We provide performance comparisons of AAE and other reduction methods in Section 4.

The clustering performance of the AAE varies depending on how to set the learning-prior-distribution (LPD) of the feature space. To determine the LPD suitable for clustering the 13 activity data, we compared the performance of 'swiss roll' distribution and 'gaussian mixture' distribution. As a performance measure, we used the Adjusted Rand Index (ARI) score [20], which indicates the degree of cluster separation. The comparison result shows that 'swiss roll' is better suited for our HAR system. We also compared the clustering performance of well-known HAR dimensionality reduction methods, such as Principal Component Analysis (PCA), kPCA, Linear Discriminant Analysis (LDA), and Variational AutoEncoder (VAE) [2], to see if the AAE method outperforms them.

For a fair comparison, we reduced the 258 features from 13 activities to the same dimension (3 dimensions*), and the result is in [Figure 3]. We can see an outstanding performance of the AAE (swiss roll) with 0.998 ARI points. However, since the clustering performance is not the sole factor that affects the classification performance, we conducted a performance evaluation of CML classifiers using the features derived from different dimensionality reduction methods.

(Figure 3) Clustering performance (ARI score) of five dimensionality reduction methods on training data

3.3.3 Classification Algorithm

In this paper, we developed the HAR system using six CML models (kNN, Decision Tree, Random Forest, SVM, MLP, and XGB) and compared their performance. We used the 'scikit-learn' library (ver.0.22.1) [21] and 'Tensorflow' library (tensorflow-gpu, ver.1.15.0) [22] in 'python' programming language for the implementation of each model.

(Table 3) Classification performance comparisons based on 13 activities dataset built for this study

* dimension of 3

For the signal data collection, we subjected six people (two males and four females) of the age group 20 to 25. We measured the natural behaviors of the subjects as they do in their ordinary living. A total of over 7.2K data were obtained, with about 600 data for each of the 13 activities.

We then used the features from the signals that have undergone dimensionality reduction as the input to the classification methods.

4. Experimental Results and Analysis

4.1 Experimental Setting

We used the accelerometer signal from the wrist-worn device to classify 13 activities in [Table 1]. We tested and compared the performances of the models with the moving-average filter, 258 features, five methods for feature extraction and reduction, and six types of machine learning algorithms. In all cases, we used 7.2K activity data setting the ratio of training and test data set 8:2.

4.2 Experimental Results

[Table 3] shows the results of the measurements evaluated with the classification accuracy. We tested six CML classification models with the input of raw data and extracted features. Also, we used five dimensionality reduction methods mentioned in section 3.3.2 to produce features reduced to 3 dimensions. For a fair comparison, we averaged the accuracy of 10 performance tests using the k-fold cross-validation technique, with the 'k' value set to 10 as is in [23].

From the result, we could see the most dominant factor affecting the HAR performance is the choice of dimensionality reduction method. With the LDA reduction, we acquired 89.7% accuracy (average of 88.5%) and achieved 99.7% (average of 99.6%) accuracy with the AAE reduction. As we can see from the table, the choice of CML methods is not a significant factor with a proper dimensionality reduction method. One can use any of kNN, RF, DNN, and XGB algorithms for the activity classification.

To demonstrate the adequacy of our classifier for the Eastern-Asian specific activities, we provide individual accuracy for the 13 activities of this study in [Table 4]. The result shows near-perfect recognition rates for the Eastern-Asian specific activities.

(Table 4) Classification accuracies for each activity

4.3 Robustness Verification

From the experiment, we have achieved classification accuracy of 99.7% through moving-average filtering for raw data, 258 extracted features, AAE feature reduction to three dimensions, and CML using the XGB model and three others. The result dramatically exceeds the average recognition rate of HAR research based on acceleration sensors (the average recognition rates of standalone/smartphone/smartwatch are 93.2%, 91.8%, and 89.4%, respectively) [4].

However, we may not urge our model's superiority because activities handled by other studies are not the same as ours. Therefore, we used the public data 'pamap2' [24] to verify our model's classification performance. 'Pamap2' includes 12 basic activities from nine subjects with around 6.3K data counts (see Table 5). The public data consists of signals from the 3-axis accelerometers worn on the wrist, chest, and ankle. It also includes signals from the gyroscope, magnetometer, and electrocardiogram. Since the purpose of this discussion is to prove our model's robustness, 12 activities based on the wrist acceleration sensor data were extracted and used. We evaluated the performance of our HAR model against the latest study performed on 'pamap2.' [Table 5] provides average recognition accuracies for all activities and F1 scores (called F-measure) [2] for each activity.

From the result, we could confirm that the model developed in this study (using moving average filter, AAE, and XBG) is useful to classify our 13 activities as well as other activities. With the 'pamap2' data, we could achieve an average of 98.7%. This result exceeds the highest recognition rate of 96.6% of the latest study[25], which uses 'pamap2' data for the activity classification. From this analysis, we could confirm that the systems we have developed in this study work well not only for the 13 activities defined in this study but also for other human activities.

(Table 5) Classification performance comparisons using ‘pamap2’ public activity dataset

5. Conclusion

In this study, we developed an artificial intelligent classification system that distinguishes 13 human activities using a wrist-mounted accelerometer. Activities normal in the Eastern-Asian culture (Korea and Japan), such as floor-sitting/ standing, chair-sitting/standing, floor-lying/up, bed-lying/up, are included.

To develop a human activity classification system, we followed a typical HAR process of sensing, preprocessing, feature extraction/reduction, and machine learning/classification. We identified appropriate methods for each step through performance comparisons of five dimensionality reduction methods and six CML models using over 7.2k data for 13 activities. With the AAE autoencoder with 'swiss roll' distribution created features as input to the four CML models (kNN, RF, DNN, and XGB), we achieved the classification accuracy of 99.7%. This result is far superior to the accuracy of current HAR research based on a smartwatch (89.4%). The superiority of the HAR system developed in this study is also demonstrated through an experiment using a publically available 'pamap2' dataset of 12 activities, where 98.7% accuracy of our model outperformed the conventionally achieved highest accuracy of 96.6%.

In this research, we have concentrated on the CML algorithms for activity classification. As future research, however, it will be very interesting to see if DL algorithms can also provide near-perfect HAR performance as in the CML. Finding out the essential DL factors that affect HAR performance will be useful to make the machine learning system more versatile in general.