Robust Sentiment Classification of Metaverse Services Using a Pre-trained Language Model with Soft Voting

Abstract

Metaverse services generate text data, data of ubiquitous computing, in real-time to analyze user emotions. Analysis of user emotions is an important task in metaverse services. This study aims to classify user sentiments using deep learning and pre-trained language models based on the transformer structure. Previous studies collected data from a single platform, whereas the current study incorporated the review data as "Metaverse" keyword from the YouTube and Google Play Store platforms for general utilization. As a result, the Bidirectional Encoder Representations from Transformers (BERT) and Robustly optimized BERT approach (RoBERTa) models using the soft voting mechanism achieved a highest accuracy of 88.57%. In addition, the area under the curve (AUC) score of the ensemble model comprising RoBERTa, BERT, and A Lite BERT (ALBERT) was 0.9458. The results demonstrate that the ensemble combined with the RoBERTa model exhibits good performance. Therefore, the RoBERTa model can be applied on platforms that provide metaverse services. The findings contribute to the advancement of natural language processing techniques in metaverse services, which are increasingly important in digital platforms and virtual environments. Overall, this study provides empirical evidence that sentiment analysis using deep learning and pre-trained language models is a promising approach to improving user experiences in metaverse services.

1. Introduction

Ubiquitous computing is frequently utilized to document events from a social, physical, and economic point of view. In particular, data of ubiquitous computing mainly consists of the information generated by geographical factors and time stamps of samples sequentially observed from mobile and human sensing devices or simulations in various fields [1]. With advancements in big data, many researchers and industries have focused on the technology required to process it. Web pages and reviews are examples of unstructured data that comprise most digital information. Millions of such text data are continually being generated as streaming data. Hence, natural language processing (NLP) tasks such as event detection in documents or calculating similarities must be promptly processed to obtain correct information [2]. Therefore, tasks related to NLP face considerable challenges in terms of scalability in text-data processing.

One area where ubiquitous computing has had a significant impact is in mobile application usage, which has become an integral part of daily life for many people worldwide [3]. People worldwide spend times watching videos via different video applications such as YouTube, Twitch, and TikTok [4]. These platforms allow them to share their opinions about a video through the comments function. Several researchers have attempted to conduct sentiment analysis based on the data gathered with these comments. However, in previous studies, despite the large amount of comment data generated by YouTube, only a limited amount could be utilized using the YouTube API. Additionally, because the analysis was performed using data from a single platform, there was difficulty in examining large amounts of data from users across multiple platforms [5–7].

Another common activity among mobile users is socialization through applications. As the metaverse expands its horizons, different technologies are being adopted by an increasing number of platforms and applications in their services [8]. Moreover, many users experience diverse services such as learning, travel, and lectures through metaverse applications. However, few studies have been undertaken on mobile applications that support metaverse services. As the number of users and companies that use metaverse services in mobile applications increases, it is crucial to analyze user experiences based on big data collected from users to provide improved services [9].

This study presents an improved classification model that can analyze both the user reviews and comments of mobile applications generated in real-time from YouTube videos. The domain for the metaverse service was specified, and the number of data points was 450,890. YouTube video data were collected using "Metaverse" as a keyword. Additionally, four mobile applications that provide metaverse services were selected from the Google Play Store: Roblox, Minecraft, Animal Crossing, and Zepeto. These application services have recorded a maximum of 150 thousand monthly users since they began and are highly successful [10–12]. The authors implemented the Bidirectional Encoder Representations from Transformers (BERT) model, A Lite BERT (ALBERT), and Robustly optimized BERT approach (RoBERTa) model with an ensemble model, enabling them to build a sentiment classification model. This study is noteworthy because it combines natural language data from diverse platforms in the metaverse domain and suggests a novel deep learning approach.

2. Related Works

2.1 Applications of Machine Learning in Various Domains

With the development of machine-learning technology, research on classification and prediction has been conducted in various domains. Table 1 shows a summary of related research conducted in diverse domains. Lee et al. [13] used algorithms to predict customer satisfaction for mobile health services and demonstrated that extreme gradient boosting (XGBoost) can be adopted to forecast users’ sentiments. To predict sentiment using restaurant feedback on quality and price, Zahoor et al. [14] confirmed that Random Forest algorithm has higher accuracy than other algorithms. Elnagar et al. [15] achieved improved performance in the categorization of Arabic text using deep learning methodologies (i.e., convolutional-Gated Recurrent Unit (GRU), Attention-GRU). Additionally, Jung et al. [16] predicted Bitcoin price trends using data from Reddit and LexisNexis over a period of more than four years. The study found that the use of extreme gradient boosting (XGBoost) and a combination of sentiment and technical indicators can produce significant results with an accuracy of 90.57% and an area under the receiver operating characteristic curve value (AUC) of 97.48%. Munisami et al. [17] presented a recognition system that identifies plants based on images of their leaves, which can be useful for various professionals. The system utilized pre-processing, feature extraction techniques, and a pattern matcher, achieving an accuracy of 83.5% which improved to 87.3% with the use of color histogram information. Yang et al. [18] designed a fault location method based on deep neural evolution network (FL-DNEN) to extract hidden fault characteristics from alarm information collected from cloud data center, in order to address the vulnerabilities that cloud computing faces due to network failures. By adopting this approach, they achieved a fault location accuracy of up to 92%. Lee et al. [19] leveraged extensive data from news articles and academic papers to analyze Environmental, Social and Governance (ESG) discourse. Employing the Bidirectional Encoder Representations from Transformers (BERTopic), the researchers identified significant keywords, and tracked the evolution of topic patters over time. Yang et al. [20] conducted a comprehensive assessment of various core-level influencing factors to evaluate the quality of link transmission. Additionally, the authors introduced an unsupervised learning-based modeling methodology, a self-organizing feature mapping (SOFM)-based routing, and spectrum allocation algorithm. Furthermore, the application of machine learning techniques is being widely utilized in metaverse services and environments. Moztarzadeh et al. [21] confirmed the applicability of disease digitalization using machine learning within the metaverse environment. Specifically, the gradient boosting algorithm was recognized as a potent and efficient approach for digitizing cancer. Tunca et al. [22] aimed to facilitate the understanding of the metaverse. The authors sought to grasp the positive and negative features of emotions. In addition, the correlations with the metaverse concept and related concepts was elucidated. For this purpose, they employed unsupervised machine learning-based Leximancer and Valence Aware Dictionary and sEntiment Reasoner (VADER) for natural language data analysis. Abou El-Magd et al. [23] designed a classification model using machine learning and real-world data to enhance the interaction between the metaverse and physical reality. They achieved a performance of 99.9% using SVM's three-dimension kernel model. The performance varies depending on the domain. Therefore, many studies are being conducted to confirm the model’s capability in each field.

Table 1. Summary of Machine Learning Research in Diverse Domains

2.2 Sentiment Analysis Utilizing Social Network Services Data

Sentiment analysis, also referred to opinion mining, analyzes a person's thoughts and feelings using natural language processing techniques [24]. Recently, many researchers have demonstrated an understanding in classifying emotions. In particular, different social network services, including blogs, reviews, and articles, have been employed to obtain the general public’s thoughts on a specific topic or circumstance. Lee et al. [25] indicated that an algorithm combining VADER and machine learning is an effective methodology for sentiment analysis using metaverse mobile application service reviews. La et al. [26] found that logistic regression is an appropriate model for analyzing the online comments of tourists. Luo and Xu[27] used online restaurant reviews to discover the model that could predict positive or negative scores. Novendri et al. [28] adjusted YouTube comments to build sentiment classifiers and achieved successful results using Naïve Bayes. Hence, collecting and leveraging user reviews is the most advantageous approach for researchers to understand user opinions and feelings.

3. Method

In this study, the authors considered a robust deep learning model that could be used on various platforms. Previous studies normally trained a deep learning model using the data collected from a single platform to classify sentiments. However, this approach has its limitations with regard to generalization. To alleviate this shortcoming, review data from YouTube and Google Play Stores were used to train the deep learning model. The flow of architecture of the proposed system is illustrated in Fig. 1.

Fig. 1. Architecture of the total process.

3.1 Data Collection

The authors collected 49,306 review data from 18 videos posted on YouTube using "metaverse" as a keyword. More specifically, YouTube videos were selected from platforms publishing news, documentaries, and technical information, as well as from individual YouTubers who introduced technology and issues. Moreover, 401,584 review data were collected from four applications supporting metaverse services: Roblox, Minecraft, Animal Crossing, and Zepeto in the Google Play Store. the number of data points was 450,890 (Table 2) and examples of the data used can be found in Table 3.

Table 2. Information on the collected data

Table 3. Examples of collected data

3.2 Data Preprocessing

The authors labeled the data to classify the positive, negative, and neutral classes in the YouTube review data. Subsequently, the data from the neutral class were excluded from this study. Existing studies indicate that reviews with four- or five-star ratings out of five were considered positive, whereas those with less than three stars were considered negative [29]. Additionally, the authors removed unnecessary duplication of data.

Most datasets have differences in the number of data points that belong to each class. This disproportion can degrade the model performance. To reduce the dissymmetry, down sampling was utilized to balance the number of classes [30,31]. To train a deep learning model, a training dataset and a test dataset were randomly separated from the entire dataset in a ratio of 8:2. Subsequently, to avoid overfitting, 20% of the training dataset was used as the validation dataset.

3.3 Deep Learning Model and Pre-trained Language Model

Various deep learning models have been used to solve NLP tasks. Convolutional Neural Network (CNN) and Long Short-Term Memory (LSTM) are representative deep learning models. In this study, a CNN and Bidirectional LSTM (BiLSTM) were used. In addition, after the advent of the transformer model, there have been many cases in which natural language processing tasks have been solved by fine-tuning them with pre-trained language models using the transformer. The authors applied representative pre-trained models, BERT, RoBERTa, and ALBERT, in HuggingFace with appropriate tokenizers.

3.3.1 Bidirectional Encoder Representations from Transformers (BERT)

The BERT model was created using the encoder part of the transformer. With large-capacity text data, a pre-trained bidirectional language model (LM) through various tasks, such as masked LM and next sentence prediction (NSP) [32]. As a result, BERT shows strength in natural language understanding tasks. Our study utilized classification among tasks that could be performed using BERT.

3.3.2 Robustly Optimized BERT Approach (RoBERTa)

RoBERTa is an enhanced model of BERT that generally performs better than it. The training dataset used in BERT was 16GB of data from BookCorpus and WIKIPEDIA. However, RoBERTa employed 160GB of the training dataset in five domains with different capacities. In addition, RoBERTa does not apply NSP, which is one of the BERT operations used for training. While BERT used a static mask pattern for every epoch, RoBERTa implemented the various mask patterns using the dynamic mask technique [33].

3.3.3 A Lite BERT (ALBERT)

ALBERT, the lite version of BERT, reduces the number of parameters while minimizing the performance degradation of BERT using two methods. First, a linear layer that reduces the dimension of the embedding layer through factorized embedding parameterization and restores it to an existing scale of a hidden size is added. Second, the parameters of the attention and feed-forward network (FFN) layers were shared through cross-layer parameter sharing. Additionally, because the NSP was not helpful in improving the performance, Sentence Order Prediction (SOP), a more difficult task, was adopted [34].

3.4 Ensemble Classifier

After training the single models, the authors configured the voting classifiers. There are two types of voting classifiers: hard voting classifiers that predict using a majority vote and soft voting classifiers that average the predicted probabilities obtained from each model and select the best class [35,36]. This study constructed ensemble models employing soft voting. Because the soft voting classifier combines predictions from different models, it usually achieves better outcomes than the other base models [37]. Therefore, the proposed study designed several ensemble models for all possible combinations using deep-learning models: BERT, RoBERTa, and ALBERT.

4. Performance Evaluation

The experimental setup and classification assessment are presented in this section. The authors experimentally verified the use of ensemble deep-learning models.

4.1 Evaluation Methodology

Ten deep learning classifiers were used for the verification. The evaluation was conducted by adjusting the confusion matrix that dealt with the accuracy (1), recall (2), precision (3), and F1-score (4) [38]. Additionally, the authors measured the area under the curve (AUC), which correlates to the section under the receiver operating characteristic (ROC) curve. AUC is a metric indicating how well a classifier distinguishes classes; the closer the score is to 1, the better the performance [39]. Table 4 defines the confusion matrix.

Accuracy = (TP+TN) / (TP+TN+FP+FN)       (1)

Recall = TP / (TP+FN)       (2)

Precision = TP / (TP+FP)       (3)

F1-score = 2 × (Recall×Precision) / (Recall+Precision)       (4)

Table 4. Confusion matrix

4.2 Results

The results show two conspicuous ensemble models, as depicted in Table 5 and Figs. 2-4. As can be seen, the first (BERT and RoBERTa) and second ensemble models (BERT, RoBERTa, and ALBERT) both demonstrate high accuracies (88.57% and 88.47%, respectively). In addition, the AUC score of the ensemble model combining the above three models was 0.9458, which is close to 1. Therefore, the ensemble model employing RoBERTa performed better than the other combinations.

Table 5. Results of the deep learning models

Fig. 2. Results of accuracy and precision Note. (ALB: ALBERT, RoB: RoBERTa, Ensemble: BERT + ALBERT + RoBERTa).

Fig. 3. Results of recall and F1-score Note. (ALB: ALBERT, RoB: RoBERTa, Ensemble: BERT + ALBERT + RoBERTa).

Fig. 4. Results of AUC Note. (ALB: ALBERT, RoB: RoBERTa, Ensemble: BERT + ALBERT + RoBERTa).

5. Conclusion

The growth of mobile technology and the internet has led to the creation of big data in real-time across many industries. Many researchers have focused on technology that processes big data [1]. The majority of big data consists of text data. Therefore, text mining in data analytics is emerging as a powerful technique for extracting new information and classifying latent meanings from unstructured text data [40]. In addition, the increasing availability of online text data has created opportunities for an advanced understanding of customer preferences based on user-generated content [41]. In this context, sentiment analysis research that examines user experience has significantly improved user satisfaction.

In previous studies that attempted sentiment analysis using YouTube comment data, the analysis was implemented with a small amount of data [5–7]. Furthermore, when using review data from mobile applications, single-platform data were employed [9,25]. One limitation is that this model was not robust because of the lack of diversity. In this study, the authors provide deep learning models that can process big data collected from multiple platforms (i.e., YouTube and four mobile applications) for a specific domain. In addition, the sentiment classifier was evaluated by applying ten different models. As a result, the ensemble of BERT and RoBERTa showed a high accuracy of 88.57%.

Following the practical implications, two types of ensemble models performed well: the first ensemble model (BERT and RoBERTa) and the second ensemble model (BERT, RoBERTa, and ALBERT). These findings could be applied generally in domains relevant to metaverse services, since they were obtained using large amounts of data taken from many platforms. Additionally, in industries with limited computing resources, it is recommended to perform user analysis based on the RoBERTa model because the overall performance is good when combined with the RoBERTa model. In summary, the authors developed a classifier that performed well in the metaverse domain. Thus, the results of this study are useful for investigating how consumers perceive metaverse-related services.

From a theoretical perspective, the authors confirmed that models utilizing user-generated content such as online reviews can predict user satisfaction. Online reviews are natural language data generated by various sectors, suggesting that they can be used to improve services. Therefore, this study reinforces a novel methodological addition to existing literature on metaverse services.

The following are this study's limitations: First, only English data were used. Therefore, further studies that consider other languages and cultures should be undertaken. Second, because the data were collected using metaverse keywords, our discovery can only be used for metaverse services. Therefore, other fields utilizing them require additional validation.