Design and Implementation of MongoDB-based Unstructured Log Processing System over Cloud Computing Environment (클라우드 환경에서 MongoDB 기반의 비정형 로그 처리 시스템 설계 및 구현)
-
- Journal of Internet Computing and Services
- /
- v.14 no.6
- /
- pp.71-84
- /
- 2013
Log data, which record the multitude of information created when operating computer systems, are utilized in many processes, from carrying out computer system inspection and process optimization to providing customized user optimization. In this paper, we propose a MongoDB-based unstructured log processing system in a cloud environment for processing the massive amount of log data of banks. Most of the log data generated during banking operations come from handling a client's business. Therefore, in order to gather, store, categorize, and analyze the log data generated while processing the client's business, a separate log data processing system needs to be established. However, the realization of flexible storage expansion functions for processing a massive amount of unstructured log data and executing a considerable number of functions to categorize and analyze the stored unstructured log data is difficult in existing computer environments. Thus, in this study, we use cloud computing technology to realize a cloud-based log data processing system for processing unstructured log data that are difficult to process using the existing computing infrastructure's analysis tools and management system. The proposed system uses the IaaS (Infrastructure as a Service) cloud environment to provide a flexible expansion of computing resources and includes the ability to flexibly expand resources such as storage space and memory under conditions such as extended storage or rapid increase in log data. Moreover, to overcome the processing limits of the existing analysis tool when a real-time analysis of the aggregated unstructured log data is required, the proposed system includes a Hadoop-based analysis module for quick and reliable parallel-distributed processing of the massive amount of log data. Furthermore, because the HDFS (Hadoop Distributed File System) stores data by generating copies of the block units of the aggregated log data, the proposed system offers automatic restore functions for the system to continually operate after it recovers from a malfunction. Finally, by establishing a distributed database using the NoSQL-based Mongo DB, the proposed system provides methods of effectively processing unstructured log data. Relational databases such as the MySQL databases have complex schemas that are inappropriate for processing unstructured log data. Further, strict schemas like those of relational databases cannot expand nodes in the case wherein the stored data are distributed to various nodes when the amount of data rapidly increases. NoSQL does not provide the complex computations that relational databases may provide but can easily expand the database through node dispersion when the amount of data increases rapidly; it is a non-relational database with an appropriate structure for processing unstructured data. The data models of the NoSQL are usually classified as Key-Value, column-oriented, and document-oriented types. Of these, the representative document-oriented data model, MongoDB, which has a free schema structure, is used in the proposed system. MongoDB is introduced to the proposed system because it makes it easy to process unstructured log data through a flexible schema structure, facilitates flexible node expansion when the amount of data is rapidly increasing, and provides an Auto-Sharding function that automatically expands storage. The proposed system is composed of a log collector module, a log graph generator module, a MongoDB module, a Hadoop-based analysis module, and a MySQL module. When the log data generated over the entire client business process of each bank are sent to the cloud server, the log collector module collects and classifies data according to the type of log data and distributes it to the MongoDB module and the MySQL module. The log graph generator module generates the results of the log analysis of the MongoDB module, Hadoop-based analysis module, and the MySQL module per analysis time and type of the aggregated log data, and provides them to the user through a web interface. Log data that require a real-time log data analysis are stored in the MySQL module and provided real-time by the log graph generator module. The aggregated log data per unit time are stored in the MongoDB module and plotted in a graph according to the user's various analysis conditions. The aggregated log data in the MongoDB module are parallel-distributed and processed by the Hadoop-based analysis module. A comparative evaluation is carried out against a log data processing system that uses only MySQL for inserting log data and estimating query performance; this evaluation proves the proposed system's superiority. Moreover, an optimal chunk size is confirmed through the log data insert performance evaluation of MongoDB for various chunk sizes.
In Smart Education, the roles of digital textbook is very important as face-to-face media to learners. The standardization of digital textbook will promote the industrialization of digital textbook for contents providers and distributers as well as learner and instructors. In this study, the following three objectives-oriented digital textbooks are looking for ways to standardize. (1) digital textbooks should undertake the role of the media for blended learning which supports on-off classes, should be operating on common EPUB viewer without special dedicated viewer, should utilize the existing framework of the e-learning learning contents and learning management. The reason to consider the EPUB as the standard for digital textbooks is that digital textbooks don't need to specify antoher standard for the form of books, and can take advantage od industrial base with EPUB standards-rich content and distribution structure (2) digital textbooks should provide a low-cost open market service that are currently available as the standard open software (3) To provide appropriate learning feedback information to students, digital textbooks should provide a foundation which accumulates and manages all the learning activity information according to standard infrastructure for educational Big Data processing. In this study, the digital textbook in a smart education environment was referred to open digital textbook. The components of open digital textbooks service framework are (1) digital textbook terminals such as smart pad, smart TVs, smart phones, PC, etc., (2) digital textbooks platform to show and perform digital contents on digital textbook terminals, (3) learning contents repository, which exist on the cloud, maintains accredited learning, (4) App Store providing and distributing secondary learning contents and learning tools by learning contents developing companies, and (5) LMS as a learning support/management tool which on-site class teacher use for creating classroom instruction materials. In addition, locating all of the hardware and software implement a smart education service within the cloud must have take advantage of the cloud computing for efficient management and reducing expense. The open digital textbooks of smart education is consdered as providing e-book style interface of LMS to learners. In open digital textbooks, the representation of text, image, audio, video, equations, etc. is basic function. But painting, writing, problem solving, etc are beyond the capabilities of a simple e-book. The Communication of teacher-to-student, learner-to-learnert, tems-to-team is required by using the open digital textbook. To represent student demographics, portfolio information, and class information, the standard used in e-learning is desirable. To process learner tracking information about the activities of the learner for LMS(Learning Management System), open digital textbook must have the recording function and the commnincating function with LMS. DRM is a function for protecting various copyright. Currently DRMs of e-boook are controlled by the corresponding book viewer. If open digital textbook admitt DRM that is used in a variety of different DRM standards of various e-book viewer, the implementation of redundant features can be avoided. Security/privacy functions are required to protect information about the study or instruction from a third party UDL (Universal Design for Learning) is learning support function for those with disabilities have difficulty in learning courses. The open digital textbook, which is based on E-book standard EPUB 3.0, must (1) record the learning activity log information, and (2) communicate with the server to support the learning activity. While the recording function and the communication function, which is not determined on current standards, is implemented as a JavaScript and is utilized in the current EPUB 3.0 viewer, ths strategy of proposing such recording and communication functions as the next generation of e-book standard, or special standard (EPUB 3.0 for education) is needed. Future research in this study will implement open source program with the proposed open digital textbook standard and present a new educational services including Big Data analysis.
The conflict between major corporations and small and medium-sized businesses is being aggravated, the trickle down effect is not working properly, and, as the controversy surrounding the effectiveness of the business limiting system continues to swirl, the plan proposed to protect the business domain of small and medium-sized businesses, resolve polarization between these businesses and large corporations, and protect small family run stores is the suitable business type designation system for small and medium-sized businesses. The current status of carrying out this system of selecting suitable business types among small and medium-sized businesses involves receiving applications for 234 items among the suitable business types and items from small and medium-sized businesses in manufacturing, and then selecting the items of the consultative group by analyzing and investigating the actual conditions. Suitable business type designation in the service industry will involve designation with priority on business types that are experiencing social conflict. Three major classifications of the service industry, related to the livelihood of small and medium-sized businesses, will be first designated, and subsequently this will be expanded sequentially. However, there is the concern that when designated as a suitable business type or item, this will hinder the growth motive for small to medium-sized businesses, and designation all cause decrease in consumer welfare. Also it is highly likely that it will operate as a prior regulation, cause side-effects by limiting competition systematically, and also be in violation against the main regulations of the FTA system. Moreover, it is pointed out that the system does not sufficiently reflect reverse discrimination factor against large corporations. Because conflict between small to medium sized businesses and large corporations results from the expansion of corporations to the service industry, which is unrelated to their key industry, it is necessary to introduce an advanced contract method like a master franchise or local franchise system and to develop local small to medium sized businesses through a franchise system to protect these businesses and dealers. However, this method may have an effect that contributes to stronger competitiveness of small to medium sized franchise businesses by advancing their competitiveness and operational methods a step further, but also has many negative aspects. First, as revealed by the Ministry of Knowledge Economy, the franchise industry is contributing to the strengthening of competitiveness through the economy of scale by organizing existing individual proprietors and increasing the success rate of new businesses. It is also revealed to be a response measure by the government to stabilize the economy of ordinary people and is emphasized as a 'useful way' to revitalize the service industry and improve the competitiveness of individual proprietors, and has been involved in contributions to creating jobs and expanding the domestic market by providing various services to consumers. From this viewpoint, franchises fit the purpose of the suitable business type system and is not something that is against it. Second, designation as a suitable business type may decrease investment for overseas expansion, R&D, and food safety, as well negatively affect the expansion of overseas corporations that have entered the domestic market, due to the contraction and low morale of large domestic franchise corporations that have competitiveness internationally. Also because domestic franchise businesses are hard pressed to secure competitiveness with multinational overseas franchise corporations that are operating in Korea, the system may cause difficulty for domestic franchise businesses in securing international competitiveness and also may result in reverse discrimination against these overseas franchise corporations. Third, the designation of suitable business type and item can limit the opportunity of selection for consumers who have up to now used those products and can cause a negative effect that reduces consumer welfare. Also, because there is the possibility that the range of consumer selection may be reduced when a few small to medium size businesses monopolize the market, by causing reverse discrimination between these businesses, the role of determining the utility of products must be left ot the consumer not the government. Lastly, it is desirable that this is carried out with the supplementation of deficient parts in the future, because fair trade is already secured with the enforcement of the franchise trade law and the best trade standard of the Fair Trade Commission. Overlapping regulations by the suitable business type designation is an excessive restriction in the franchise industry. Now, it is necessary to establish in the domestic franchise industry an environment where a global franchise corporation, which spreads Korean culture around the world, is capable of growing, and the active support by the government is needed. Therefore, systems that do not consider the process or background of the growth of franchise businesses and harm these businesses for the sole reason of them being large corporations must be removed. The inhibition of growth to franchise enterprises may decrease the sales of franchise stores, in some cases even bankrupt them, as well as cause other problems. Therefore the suitable business type system should not hinder large corporations, and as both small dealers and small to medium size businesses both aim at improving competitiveness and combined growth, large corporations, small dealers and small to medium sized businesses, based on their mutual cooperation, should not include franchise corporations that continue business relations with them in this system.