Design and Implementation of MongoDB-based Unstructured Log Processing System over Cloud Computing Environment (클라우드 환경에서 MongoDB 기반의 비정형 로그 처리 시스템 설계 및 구현)
-
- Journal of Internet Computing and Services
- /
- v.14 no.6
- /
- pp.71-84
- /
- 2013
Log data, which record the multitude of information created when operating computer systems, are utilized in many processes, from carrying out computer system inspection and process optimization to providing customized user optimization. In this paper, we propose a MongoDB-based unstructured log processing system in a cloud environment for processing the massive amount of log data of banks. Most of the log data generated during banking operations come from handling a client's business. Therefore, in order to gather, store, categorize, and analyze the log data generated while processing the client's business, a separate log data processing system needs to be established. However, the realization of flexible storage expansion functions for processing a massive amount of unstructured log data and executing a considerable number of functions to categorize and analyze the stored unstructured log data is difficult in existing computer environments. Thus, in this study, we use cloud computing technology to realize a cloud-based log data processing system for processing unstructured log data that are difficult to process using the existing computing infrastructure's analysis tools and management system. The proposed system uses the IaaS (Infrastructure as a Service) cloud environment to provide a flexible expansion of computing resources and includes the ability to flexibly expand resources such as storage space and memory under conditions such as extended storage or rapid increase in log data. Moreover, to overcome the processing limits of the existing analysis tool when a real-time analysis of the aggregated unstructured log data is required, the proposed system includes a Hadoop-based analysis module for quick and reliable parallel-distributed processing of the massive amount of log data. Furthermore, because the HDFS (Hadoop Distributed File System) stores data by generating copies of the block units of the aggregated log data, the proposed system offers automatic restore functions for the system to continually operate after it recovers from a malfunction. Finally, by establishing a distributed database using the NoSQL-based Mongo DB, the proposed system provides methods of effectively processing unstructured log data. Relational databases such as the MySQL databases have complex schemas that are inappropriate for processing unstructured log data. Further, strict schemas like those of relational databases cannot expand nodes in the case wherein the stored data are distributed to various nodes when the amount of data rapidly increases. NoSQL does not provide the complex computations that relational databases may provide but can easily expand the database through node dispersion when the amount of data increases rapidly; it is a non-relational database with an appropriate structure for processing unstructured data. The data models of the NoSQL are usually classified as Key-Value, column-oriented, and document-oriented types. Of these, the representative document-oriented data model, MongoDB, which has a free schema structure, is used in the proposed system. MongoDB is introduced to the proposed system because it makes it easy to process unstructured log data through a flexible schema structure, facilitates flexible node expansion when the amount of data is rapidly increasing, and provides an Auto-Sharding function that automatically expands storage. The proposed system is composed of a log collector module, a log graph generator module, a MongoDB module, a Hadoop-based analysis module, and a MySQL module. When the log data generated over the entire client business process of each bank are sent to the cloud server, the log collector module collects and classifies data according to the type of log data and distributes it to the MongoDB module and the MySQL module. The log graph generator module generates the results of the log analysis of the MongoDB module, Hadoop-based analysis module, and the MySQL module per analysis time and type of the aggregated log data, and provides them to the user through a web interface. Log data that require a real-time log data analysis are stored in the MySQL module and provided real-time by the log graph generator module. The aggregated log data per unit time are stored in the MongoDB module and plotted in a graph according to the user's various analysis conditions. The aggregated log data in the MongoDB module are parallel-distributed and processed by the Hadoop-based analysis module. A comparative evaluation is carried out against a log data processing system that uses only MySQL for inserting log data and estimating query performance; this evaluation proves the proposed system's superiority. Moreover, an optimal chunk size is confirmed through the log data insert performance evaluation of MongoDB for various chunk sizes.
Conserving the recored cultural inheritance is actually the duty of all of us. Above all, the management and conservation of archives and documents is up to archivists who have technical knowledge about archival science. Archivists have to not only conserve archives and documents but also carry out classifying and appraising them in order to define them as current historic ones. The fundamental education about archival science is made up of history and law. Because Archive is the organisation which manage archives and documents produced by legal and administrative actions. Although there are still arguments about technical knowledge and degree archivists have to acquire, most of them prefer the studies related with history and emphasize legal studies to be the general boundary of archivits' ideology and trust. The training course about conservation of archives is conducted in about 9 National Archives of Torino, Milano, Venezia, Genova, Bologna, Parma, Roma, Napoli, Palermo. The training course in 19th was mostly based on the lectures of Phaleography, Diplomatics. There were not the education about archival science yet. Toward the end of 19th and 20th, people stressed the most basic subject in the training course of National Archive was not Phaleography and Diplomatics but archival science. The goal of archival science is to study the institution and organisation transferring archives and documents to Archive. And also it help archivists not wander about with ignorance of organisational and original procedures and divisions but know exactly theirs works. Like this, the studies on institution and organisation have got in the saddle as a branch of archival science since a few ten years. While archival science didn't evoke sympathy among people and experienced the tedious and difficult path in italy and other countries, Archive was managed by experts of other branches. As a result, there were a lot of faults in Archival Science. Specializing training course for Italian archivists came into being under the backdrop of Social Science Institute of Roma National University in 1925. The archival course of universities accomplished by the studies of history, law and economy. And such as Eugenio Casanova and Giorgio Cencetti were devoted archival science was abled to settle down in national archive. The training course for experts of 'archival science, 'Phaleography and Diplomatics' in National Archive of Bologna(Archivio di Stato di Bologna) is one of courses conducted in 17 National Archives in italy. This course is gratuitous and made up of 8 subjects(Archivistica, Paleografia, Diplomatica, Storia dell' Archivio, Notariato e documenti privati, istituzione medievale, istituzione moderna, istituzione contemporanea) students have to complete for two years. Students can receive the degree through passing twice written exam and once oral test. After department of Culture and education finally puts the marks of students, the chief Nationa Archive of Bologna confer the degree of 'archival science Phaleography and Diplomatics' on students passing the exams. This degree authenticates trainees' qualification which enables him to work at the archive in province, district and administrative capital city and archive of comunity and so on. Italian training course naturally leads archivists to keep in contact with valuable cultural inheritance through training in Archive. And it shows the intention to strengthen the affinity with each documents in the spot of archival management before training archivists. Also this is appraised as one of positive policies to conserve the local cultual inheritante in connection with the original qualitity of national archive with testify the history of each region. Traning course for archivist in Italy shows us the way how we have to prepare and proceed it. First, from producing documents to conserving than forever there has introduced 'original order that is to say a general rule to respect the first order given at the time producing documents'. Management of administrative documents is related consistently with one of historical documents. Second, the traning course for archivist is managing around 17 national archives. because italian national archive lay stress not or rducation of theory bus on train for archivest working in the first time of archival science. Third, diplomatics and phaleography for studies about historical document support archives. Forth, the studies on history id proceeding by cooperation between archivist and historian around archive. How our duties is non continuinf disputer who has to conserve and manage document and archives, but traing experts who having ability, vision and flexible thought, responsibility about archivals.
We investigated what factors motivate gamers to participate in a guild and why they continue to be engaged as members of the guild. We find that, based on the result of focus group interviews with MMORPG gamers, social support and self-esteem factors play important roles. Considering both prior research and the focus group interviews we have conducted, we define social support and character control as independent variables. Character identity, guild identity, and self-esteem are proposed as mediating variables while guild flow and game loyalty as dependent variables. Accordingly, we develop the research model and hypotheses, and verify them empirically. Based on our experiences of playing the WoW game, we proposed a research model and conducted focus-group interviews (FGIs). FGIs involve formulating a hypothesis and then collecting some relevant data. FGIs were conducted face-to-face with students of C University in Korea. We formulated structured interview schedules, and the questions were based on our research variables and personal experiences. The questions for the interviews encompassed the following areas: (a) the demographic characteristics of the focus group; (b) the number of years for which respondents had played online games; (c) the motive for starting a game; (d) the number of game-characters assumed by each gamer; (e) the type of game played; and (f) other issues such as the reasons for involvement in the play, the willingness to reuse the game in case new versions were released, etc. On average, it took two hours to interview each of three groups. A primary set of FGIs was conducted with three groups on the premise that there would be some differences caused by character race (Horde vs. Alliance) or by playable server (Normal vs. Combat). With respect to the manner of playing, we found that guild members shared information, felt a sense of belonging, and played computer games for quite a long time through the guild; however, they did not undergo these experiences when playing alone. Gamers who belonged to a specific guild helped other players without expecting compensation for that, freely shared information about the game, gave away items for free, and more generous with other members who made mistakes. The guild members were aware of the existence other members and experienced a sense of belonging through interactions with, and evaluations from, other players. It was clear that social support was shown within the guild and that it played an important role as a major research variable. Based on the results of the first FGIs, a second set of in-depth FGIs was carried out with a focus on the psychology of the individual within the guild and the social community of the guild. The second set of FGIs also focused on the guild's offline meetings. Gamers, over all, recognize the necessity of joining a community, not only off-line but also online world of the guild. They admit that the guild is important for them to easily and conveniently enjoy playing online computer games. The active behavior and positive attitudes of existing guild members can motivate new members of the guild to adapt themselves to the guild environment. They then adopt the same behaviors and attitudes of established guild members. In this manner, the new members of the guild strengthen the bonds with other gamers while feeling a sense of belonging, and developing social identity, thereby. It was discovered that the interaction among guild members and the social support encouraged new gamers to quickly develop a sense of social identity and increase their self-esteem. The guild seemed to play the role of socializing gamers. Sometimes, even in the real world, the guild members helped one another; therefore, the features of the guild also spilled over to the offline environment. We intend to use self-esteem, which was found through the second set of FGIs, as an important research variable. To collect data, an online survey was designed with a questionnaire to be completed by WoW gamers, who belong to a guild. The survey was registered on the best three domestic game-sites: 'WoW playforum,' 'WoW gamemeca,' and 'Wow invent.' The selected items to be measured in the questionnaire were decided based on prior research and data from FGIs. To verify the content of the questionnaire, we carried out a pilot test with the same participants to point out ambiguous questions as a way to ensure maximum accuracy of the survey result. A total of 244 responses were analyzed from the 250 completed questionnaires. The SEM analysis was used to test goodness-of-fit of the model. As a result, we found important results as follows: First, according to the statistics, social support had statistically significant impacts on character control, character identity, guild identity and self-esteem. Second, character control had significant effects on character identity, guild identity and self-esteem. Third, character identity shows its clear impact on self-esteem and game loyalty. Fourth, guild identity affected self-esteem, guild flow and game loyalty. Fifth, self-esteem had a positive influence on the guild flow. These days, the number of virtual community is rising along with its significance largely because of the nature of the online games. Accordingly, this study is designed to clarify the psychological relationship between gamers within the guild that has been generally established by gamers to play online games together. This study focuses on the relationships in which social support influences guild flow or game loyalty through character control, character identity, guild identity, and self-esteem, which are present within a guild in the MMORPG game environment. The study results are as follows. First, the effects of social support on character control, character identity, guild identity and self-esteem are proven to be statistically significant. It was found that character control improves character identity, guild identity and self-esteem. Among the seven variables, social support, which is derived from FGIs, plays an important role in this study. With the active support of other guild members, gamers can improve their ability to develop good characters and to control them. Second, character identity has a positive effect on self-esteem and game loyalty, while guild identity has a significant effect on self-esteem, guild flow and game loyalty. Self-esteem affects guild flow. It was found that the higher the character and guild identities become, the greater the self-esteem is established. Contrary to the findings of prior research, our study results indicate that the relationship between character identity and guild flow is not significant. Rather, it was found that character identity directly affects game players' loyalty. Even though the character identity had no direct effect on increasing guild flow, it has indirectly affected guild flow through self-esteem. The significant relationship between self-esteem and guild flow indicates that gamers achieve flow, i.e., a feeling of pleasure and excitement through social support. Several important implications of this study should be noted. First, both qualitative and quantitative methods were used to conduct this study. Through FGIs, it was observed that both social support and self-esteem are important variables. Second, because guilds had been rarely studied, this research is expected to play an important role in the online community. Third, according to the result, six hypotheses (H1, H5, H6, H7, H8, and H11) setup based on FGIs, were statistically significant; thus, we can suggest the corresponding relationships among the variables as a guideline for follow-up research. Our research is significant as it has following implications: first, the social support of the guild members is important when establishing character control, character identity, guildidentity and self-esteem. It is also a major variable that affects guild flow and game loyalty. Second, character control when improved by social support shows notable influence on the development of character identity, guild identity and self-esteem. Third, character identity and guild identity are major factors to help establish gamers' own self-esteem. Fourth, character identity affects guild flow through self-esteem and game loyalty. The gamers usually express themselves through characters; the higher character identity is, the more loyalty a gamer has. Fifth, guild identity, established within the guild, has clear effects on self-esteem, guild flow and game loyalty. Sixth, qualitative and quantitative methods are employed to conduct this study. Based on the results of focus group interviews and SEM analysis, we find that the social support by guild members and psychological factors are significant in strengthening the flow of guild and loyalty to the game. As such, game developers should provide some extra functions for guild community, through which gamers can play online games in collaboration with one another. Also, we suggest that positive self-esteem which is built up through social support can help gamers achieve higher level of flow and satisfaction, which will consequently contribute to minimizing the possibility for the players to develop negative attitude toward the guild they belong to.