Methodology for Identifying Issues of User Reviews from the Perspective of Evaluation Criteria: Focus on a Hotel Information Site

사용자 리뷰의 평가기준 별 이슈 식별 방법론: 호텔 리뷰 사이트를 중심으로

  • Byun, Sungho (Graduate School of Business IT, Kookmin University) ;
  • Lee, Donghoon (Graduate School of Business IT, Kookmin University) ;
  • Kim, Namgyu (School of Management Information Systems, Kookmin University)
  • 변성호 (국민대학교 비즈니스IT전문대학원) ;
  • 이동훈 (국민대학교 비즈니스IT전문대학원) ;
  • 김남규 (국민대학교 비즈니스IT전문대학원)
  • Received : 2016.08.17
  • Accepted : 2016.09.24
  • Published : 2016.09.30


As a result of the growth of Internet data and the rapid development of Internet technology, "big data" analysis has gained prominence as a major approach for evaluating and mining enormous data for various purposes. Especially, in recent years, people tend to share their experiences related to their leisure activities while also reviewing others' inputs concerning their activities. Therefore, by referring to others' leisure activity-related experiences, they are able to gather information that might guarantee them better leisure activities in the future. This phenomenon has appeared throughout many aspects of leisure activities such as movies, traveling, accommodation, and dining. Apart from blogs and social networking sites, many other websites provide a wealth of information related to leisure activities. Most of these websites provide information of each product in various formats depending on different purposes and perspectives. Generally, most of the websites provide the average ratings and detailed reviews of users who actually used products/services, and these ratings and reviews can actually support the decision of potential customers in purchasing the same products/services. However, the existing websites offering information on leisure activities only provide the rating and review based on one stage of a set of evaluation criteria. Therefore, to identify the main issue for each evaluation criterion as well as the characteristics of specific elements comprising each criterion, users have to read a large number of reviews. In particular, as most of the users search for the characteristics of the detailed elements for one or more specific evaluation criteria based on their priorities, they must spend a great deal of time and effort to obtain the desired information by reading more reviews and understanding the contents of such reviews. Although some websites break down the evaluation criteria and direct the user to input their reviews according to different levels of criteria, there exist excessive amounts of input sections that make the whole process inconvenient for the users. Further, problems may arise if a user does not follow the instructions for the input sections or fill in the wrong input sections. Finally, treating the evaluation criteria breakdown as a realistic alternative is difficult, because identifying all the detailed criteria for each evaluation criterion is a challenging task. For example, if a review about a certain hotel has been written, people tend to only write one-stage reviews for various components such as accessibility, rooms, services, or food. These might be the reviews for most frequently asked questions, such as distance between the nearest subway station or condition of the bathroom, but they still lack detailed information for these questions. In addition, in case a breakdown of the evaluation criteria was provided along with various input sections, the user might only fill in the evaluation criterion for accessibility or fill in the wrong information such as information regarding rooms in the evaluation criteria for accessibility. Thus, the reliability of the segmented review will be greatly reduced. In this study, we propose an approach to overcome the limitations of the existing leisure activity information websites, namely, (1) the reliability of reviews for each evaluation criteria and (2) the difficulty of identifying the detailed contents that make up the evaluation criteria. In our proposed methodology, we first identify the review content and construct the lexicon for each evaluation criterion by using the terms that are frequently used for each criterion. Next, the sentences in the review documents containing the terms in the constructed lexicon are decomposed into review units, which are then reconstructed by using the evaluation criteria. Finally, the issues of the constructed review units by evaluation criteria are derived and the summary results are provided. Apart from the derived issues, the review units are also provided. Therefore, this approach aims to help users save on time and effort, because they will only be reading the relevant information they need for each evaluation criterion rather than go through the entire text of review. Our proposed methodology is based on the topic modeling, which is being actively used in text analysis. The review is decomposed into sentence units rather than considering the whole review as a document unit. After being decomposed into individual review units, the review units are reorganized according to each evaluation criterion and then used in the subsequent analysis. This work largely differs from the existing topic modeling-based studies. In this paper, we collected 423 reviews from hotel information websites and decomposed these reviews into 4,860 review units. We then reorganized the review units according to six different evaluation criteria. By applying these review units in our methodology, the analysis results can be introduced, and the utility of proposed methodology can be demonstrated.


  1. Archak, N., A. Ghose and P. G. Ipeirotis, "Deriving the Pricing Power of Product Features by Mining Consumer Reviews," Management Science, Vol.57, No.8(2011), 1485-1509.
  2. Bjorkelund, E., T. H. Burnett and K. Norvag, "A Study of Opinion Mining and Visualization of Hotel Reviews," In Proceedings of the 14th International Conference on Information Integration and Web-based Applications & Services (IIWAS '12), 2012.
  3. Buneman, P., "Semistructured data," In Proceedings of the Sixteenth ACM SIGACTSIGMOD-SIGART Symposium on Principles of Database Systems (PODS '97), 1997.
  4. Chae, S. H., J. I. Lim and J. Y. Kang, "A Comparative Analysis of Social Commerce and Open Market Using User Reviews in Korean Mobile Commerce," Journal of Intelligence and Information Systems, Vol.21, No.4(2015), 53-77.
  5. Choi, J. U., H. J. Ryu, D. B. Yu, N. R. Kim and Y. H. Kim, "System Design for Analysis and Evaluation of E-commerce Products Using Review Sentiment Word Analysis," KIISE Transactions on Computing Practices, Vol.22, No.5(2016), 209-217.
  6. Gamon, M., A. Aue, S. Corston-Oliver and E. Ringger, "Pulse: Mining Customer Opinions from Free Text," In Proceedings of the 6th International Conference on Advances in Intelligent Data Analysis (IDA '05), 2005.
  7. Jeon, B. K. and H. C. Ahn, "A Collaborative Filtering System Combined with Users' Review Mining : Application to the Recommendation of Smartphone Apps," Journal of Intelligence and Information Systems, Vol.21, No.2(2015), 1-18.
  8. Kim, J. Y. and D. S. Kim, "A Study on the Method for Extracting the Purpose-Specific Customized Information from Online Product Reviews based on Text Mining," The Journal of Society for e-Business Studies, Vol.21, No.2(2016), 151-161.
  9. Kim, K. H., "Design and Implementation Online Customer Reviews Analysis System based on Dependency Network Model," The Journal of the Korea Contents Association, Vol.10, No.11(2010), 30-37.
  10. Lee, S. H., J. Cui and J. W. Kim, "Sentiment analysis on movie review through building modified sentiment dictionary by movie genre," Journal of Intelligence and Information Systems, Vol.22, No.2(2016), 97-113.
  11. Lee, Y. J., J. H. Ji, G. Woo and H. G. Cho, "TRIB: A Clustering and Visualization System for Responding Comments on Blogs," The KIPS Transactions : Part D, Vol.16, No.5(2009), 817-824.
  12. Lee, Y. J., I. J. Jung and G. Woo, "Extracting and Visualizing Dispute comments and Relations on Internet Forum Site," The Journal of the Korea Contents Association, Vol.12, No.2 (2012), 40-51.
  13. Liu, B., M. Hu and J. Cheng, "Opinion Observer: Analyzing and Comparing Opinions on the Web," In Proceedings of the 14th International Conference on World Wide Web (WWW '05), 2005, 342-351.
  14. Liu, C. and N. Kim, "Methodology for Improving the Reliability of the Rating System for Leisure Activity Information Sites : Focusing on a Movie Information Site," Journal of Tourism and Leisure Research, Vol.27, No.7(2015), 187-200.
  15. Mun, S. M., G. N. Kim, G. C. Choi and K. W. Lee, "Movie Recommended System base on Analysis for the User Review utilizing Ontology Visualization," Design Convergence Study, Vol.15, No.2(2016), 347-368.
  16. Scaffidi, C., K. Bierhoff, E. Chang, M. Felker, H. Ng and C. Jin, "Red Opal: Product-Feature Scoring from Reviews," In Proceedings of the 8th ACM Conference on Electronic Commerce (EC '07), 2007.
  17. Yang, J. Y., J. S. Myung and S. G. Lee, "A Product Review Summarization System Using a Scoring of Features," The Journal of Society for e-Business Studies Symposium and other publications, Society for e-Business Studies, 2008.
  18. Yang, J. Y., J. S. Myung and S. G. Lee, "A Sentiment Classification Method Using Context. Information in Product Review Summarization," Journal of KISS: Databases, Vol.36, No.4(2009), 254-262.
  19. Yeon, J. H., D. J. Lee, J. H. Shim and S. G. Lee, "Product Review Data and Sentiment Analytical Processing Modeling," The Journal of Society for e-Business Studies, Vol.16, No.4(2011), 125-137.
  20. Yeon. H. B., S. J. Yoo, H. S. Jang, D. I. Han and Y. Jang, "Design and Implementation of a Web Crawling System for the Reviews of Korean Restaurants in the U.S.," Korea Computer Congress Symposium, Vol.2013, No.6(2013), 283-285.