• Title/Summary/Keyword: Web-based DB system

Search Result 195, Processing Time 0.029 seconds

Design and Implementation of MongoDB-based Unstructured Log Processing System over Cloud Computing Environment (클라우드 환경에서 MongoDB 기반의 비정형 로그 처리 시스템 설계 및 구현)

  • Kim, Myoungjin;Han, Seungho;Cui, Yun;Lee, Hanku
    • Journal of Internet Computing and Services
    • /
    • v.14 no.6
    • /
    • pp.71-84
    • /
    • 2013
  • Log data, which record the multitude of information created when operating computer systems, are utilized in many processes, from carrying out computer system inspection and process optimization to providing customized user optimization. In this paper, we propose a MongoDB-based unstructured log processing system in a cloud environment for processing the massive amount of log data of banks. Most of the log data generated during banking operations come from handling a client's business. Therefore, in order to gather, store, categorize, and analyze the log data generated while processing the client's business, a separate log data processing system needs to be established. However, the realization of flexible storage expansion functions for processing a massive amount of unstructured log data and executing a considerable number of functions to categorize and analyze the stored unstructured log data is difficult in existing computer environments. Thus, in this study, we use cloud computing technology to realize a cloud-based log data processing system for processing unstructured log data that are difficult to process using the existing computing infrastructure's analysis tools and management system. The proposed system uses the IaaS (Infrastructure as a Service) cloud environment to provide a flexible expansion of computing resources and includes the ability to flexibly expand resources such as storage space and memory under conditions such as extended storage or rapid increase in log data. Moreover, to overcome the processing limits of the existing analysis tool when a real-time analysis of the aggregated unstructured log data is required, the proposed system includes a Hadoop-based analysis module for quick and reliable parallel-distributed processing of the massive amount of log data. Furthermore, because the HDFS (Hadoop Distributed File System) stores data by generating copies of the block units of the aggregated log data, the proposed system offers automatic restore functions for the system to continually operate after it recovers from a malfunction. Finally, by establishing a distributed database using the NoSQL-based Mongo DB, the proposed system provides methods of effectively processing unstructured log data. Relational databases such as the MySQL databases have complex schemas that are inappropriate for processing unstructured log data. Further, strict schemas like those of relational databases cannot expand nodes in the case wherein the stored data are distributed to various nodes when the amount of data rapidly increases. NoSQL does not provide the complex computations that relational databases may provide but can easily expand the database through node dispersion when the amount of data increases rapidly; it is a non-relational database with an appropriate structure for processing unstructured data. The data models of the NoSQL are usually classified as Key-Value, column-oriented, and document-oriented types. Of these, the representative document-oriented data model, MongoDB, which has a free schema structure, is used in the proposed system. MongoDB is introduced to the proposed system because it makes it easy to process unstructured log data through a flexible schema structure, facilitates flexible node expansion when the amount of data is rapidly increasing, and provides an Auto-Sharding function that automatically expands storage. The proposed system is composed of a log collector module, a log graph generator module, a MongoDB module, a Hadoop-based analysis module, and a MySQL module. When the log data generated over the entire client business process of each bank are sent to the cloud server, the log collector module collects and classifies data according to the type of log data and distributes it to the MongoDB module and the MySQL module. The log graph generator module generates the results of the log analysis of the MongoDB module, Hadoop-based analysis module, and the MySQL module per analysis time and type of the aggregated log data, and provides them to the user through a web interface. Log data that require a real-time log data analysis are stored in the MySQL module and provided real-time by the log graph generator module. The aggregated log data per unit time are stored in the MongoDB module and plotted in a graph according to the user's various analysis conditions. The aggregated log data in the MongoDB module are parallel-distributed and processed by the Hadoop-based analysis module. A comparative evaluation is carried out against a log data processing system that uses only MySQL for inserting log data and estimating query performance; this evaluation proves the proposed system's superiority. Moreover, an optimal chunk size is confirmed through the log data insert performance evaluation of MongoDB for various chunk sizes.

Constructing a Metadata Database to Enhance Internet Retrieval of Educational Materials

  • Oh Sam-Gyun
    • Journal of the Korean Society for Library and Information Science
    • /
    • v.32 no.3
    • /
    • pp.143-156
    • /
    • 1998
  • This paper reports the GEM (Gateway to Educational Materials) project whose goal is to develop an operational framework to provide the K-12 teachers in the world with 'one-stop/any-stop' access to thousands of lesson plans, curriculum units and other Internet-based educational resources. To the IS-element Dublin Core base package, the GEM project added an 8-element, domain-specific GEM package. The GEM project employed the conceptual data modeling approach to designing the GEM database, used the Sybase relational database management system (RDBMS) to construct the backend database for storing the metadata of educational resources, and also employed the active server page (ASP) technology to provide Web interfaces to that database. The consortium members catalog lesson plans and other Internet-based educational resources using a cataloging module program that produces HTML meta tags. A harvest program collects these meta tags across the Internet and outputs an ASCII file that conforms to the standard agreed by the consortium members. A parser program processes this file to enter meta tags automatically into appropriate relational tables in the Sybase database. The conceptual/logical schemas of Dublin Core and GEM profile are presented. The advantages of conceptual modeling approach to manage metadata are discussed. A prototype system that provides access to the GEM metadata is available at http://lis.skku.ac.kr/gem/.

  • PDF

Smart Synthetic Path Search System for Prevention of Hazardous Chemical Accidents and Analysis of Reaction Risk (반응 위험성분석 및 사고방지를 위한 스마트 합성경로 탐색시스템)

  • Jeong, Joonsoo;Kim, Chang Won;Kwak, Dongho;Shin, Dongil
    • Korean Chemical Engineering Research
    • /
    • v.57 no.6
    • /
    • pp.781-789
    • /
    • 2019
  • There are frequent accidents by chemicals during laboratory experiments and pilot plant and reactor operations. It is necessary to find and comprehend relevant information to prevent accidents before starting synthesis experiments. In the process design stage, reaction information is also necessary to prevent runaway reactions. Although there are various sources available for synthesis information, including the Internet, it takes long time to search and is difficult to choose the right path because the substances used in each synthesis method are different. In order to solve these problems, we propose an intelligent synthetic path search system to help researchers shorten the search time for synthetic paths and identify hazardous intermediates that may exist on paths. The system proposed in this study automatically updates the database by collecting information existing on the Internet through Web scraping and crawling using Selenium, a Python package. Based on the depth-first search, the path search performs searches based on the target substance, distinguishes hazardous chemical grades and yields, etc., and suggests all synthetic paths within a defined limit of path steps. For the benefit of each research institution, researchers can register their private data and expand the database according to the format type. The system is being released as open source for free use. The system is expected to find a safer way and help prevent accidents by supporting researchers referring to the suggested paths.

A Study on the Data Organization of Specification Information for reference of Design Information (설계정보 참조를 위한 시방정보의 자료구조화에 관한 연구)

  • Kim Jae-hyun;Song Younk-Kyou;Kim Uk
    • Korean Journal of Construction Engineering and Management
    • /
    • v.2 no.3 s.7
    • /
    • pp.92-100
    • /
    • 2001
  • The architectural drawing, construction project specification, etc. are included in the contract of a documents. However, construction project specification, for being documentation, is not utilized to such an extent. The reason is that specification information is difficult in collecting information in relation to the architectural drawing, material finishing list and other architectural information. Therefore, an integrated model, which can be associated with other architectural information, is needed, and a DB based on this integrated model must be established in order for it to be utilized in design, construction, and management. The DB, which is established through this process, must be updated according to modification in design, and construction. Furthermore the specification must be in document on the web for reference. Consequently in this research, the structure of integrated model has been introduced, and it has made the search and preparation of the integrated model on the Internet, using the specification information DB established for the mutual reference of DB, possible. The improvements of construction project specification standards are expected by this system. Also, it will bring about Improvements upon claim prevention, and design, construction, management qualities. Furthermore, it will make the use of information more convenient in practical business such as order agency, design service and building site.

  • PDF

A Web Based Archery Game Support System (웹기반 궁도경기지원시스템 개발)

  • Kwak, Chang-Uk;Byun, Jeongyong
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2012.11a
    • /
    • pp.1754-1757
    • /
    • 2012
  • 전국의 궁도인구는 만 여명에 이른다. 매년 1~2주를 주기로 개최되는 이 대회는 참가접수부터 운영까지 오로지 오프라인으로만 진행되어 경기외적인 비용과 시간이 상당히 낭비된다. 이에 본 연구에서는 이용자의 접근성이 향상되고, 플랫폼에 독립적인 웹기반의 경기지원시스템을 구현하고자 한다. 시스템에 접근하는 이용자는 접수와 결과를 확인하는 대회 전/후에 집중되므로 이 때의 효율적인 DB Access 기술이 필요하다. 따라서 본 논문에서는 DBCP 기술을 이용하여 DB Connection부분에서 보다 효율적인 시스템으로 개발하고자 한다.

Design and Implementation of Virtual Tutoring Estimation System on the Internet Environment (인터넷 환경에서 가상 학습평가 시스템 설계 및 구현)

  • 이석호;김창수;황현숙
    • Journal of Korea Multimedia Society
    • /
    • v.1 no.2
    • /
    • pp.204-214
    • /
    • 1998
  • The existing tutoring estimation systems using the authoring tools or HTML forms on the Web environment don't have been improved inconvenient factors of uni-directional interactive systems. In this paper, we design and implement the interactive tutoring estimation system which is conveniently considered to user on the internet environment. The proposed system is divided into teacher and student modules, which have the functions of the new problem insertion and deletion in database, regular examination marking for the former and lecture/test selection, scoring look, etc for the letter. The environment of server system in this thesis is implemented using the IDC file and SQL database for internet access based on Windows NT, client systems are composed to Windows 95 environment based on Explorer or Netscape to access for interactive tutoring systems.

  • PDF

Study of Speech Recognition System Using the Java (자바를 이용한 음성인식 시스템에 관한 연구)

  • Choi, Kwang-Kook;Kim, Cheol;Choi, Seung-Ho;Kim, Jin-Young
    • The Journal of the Acoustical Society of Korea
    • /
    • v.19 no.6
    • /
    • pp.41-46
    • /
    • 2000
  • In this paper, we implement the speech recognition system based on the continuous distribution HMM and Browser-embedded model using the Java. That is developed for the speech analysis, processing and recognition on the Web. Client sends server through the socket to the speech informations that extracting of end-point detection, MFCC, energy and delta coefficients using the Java Applet. The sewer consists of the HMM recognizer and trained DB which recognizes the speech and display the recognized text back to the client. Because of speech recognition system using the java is high error rate, the platform is independent of system on the network. But the meaning of implemented system is merged into multi-media parts and shows new information and communication service possibility in the future.

  • PDF

WSN Data Visualization using Augmented Reality (증강현실을 통한 WSN 데이터 가시화)

  • Park, Jin-Gwan;Jung, Min-A;Kim, Kyoung-Ho;Lee, Seong-Ro
    • Journal of the Institute of Electronics and Information Engineers
    • /
    • v.50 no.12
    • /
    • pp.107-116
    • /
    • 2013
  • We proposed the WSN monitoring system applied the augmented reality to visualize effectively an indoor WSN. To implement system, we used wireless sensor network, indoor location determination, location-based augmented reality technology. First, we composed the wireless sensor networks indoors and implement web server and then get data from server DB using Android phones. Then, we obtained the (x, y) coordinates using the triangulation method from RSSI of three point of the strongest signal strength of the AP's. Also, we adjusted coordinates using the Kalman filter. Finally, we inserted the adjusted coordinates to the latitude and the longitude of the Mixare that use the GPS signal, and we got location of user and wireless sensor in the server DB. After that, we implemented augmented reality system using the android phone and wireless sensor location and data and real life image.

Design and Implementation of Indoor Air Hazardous Substance Detection Mobile System based on IoT Platform (IoT platform 기반 실내 대기 위험 물질 감지 모바일 시스템 설계 및 구현)

  • Yang, Oh-Seok;Kim, Yeong-Uk;Lee, Hong-Lo
    • Journal of Korea Society of Industrial Information Systems
    • /
    • v.24 no.6
    • /
    • pp.43-53
    • /
    • 2019
  • In recent years, there have been many cases of damage to indoor air hazardous materials, and major damage due to the lack of quick action. In this regard, the system is intended to establish for sending push messages to the user's mobile when the concentration of hazardous substances is exceeded. This system extracts data with IoT system such as Arduino and Raspberry Pi and then constructs database through MongoDB and MySQL in cloud computing system. The database is imported through the application server using NodeJS and sent to the application for visualization. Also, when receiving signals about a dangerous situation in IoT system, push message is sent using Google FCM library. Mobile application is developed using Android Web view, and page to enter Web view is developed using HTML5 (HTML, Javascript CSS). The application of this system enables real-time monitoring of indoor air-dangerous substances. In addition, real-time information on indoor/outdoor detection location and concentration can be sent to the user's mobile in case of a risk situation, which can be expected to help the user respond quickly.

Total Solution System for Applications using DPMS (DPMS 통합 원서 지원 시스템 개발)

  • Park, Chang-Ho;Hwang, Hu-Mor
    • Proceedings of the KIEE Conference
    • /
    • 2001.07d
    • /
    • pp.2678-2680
    • /
    • 2001
  • We develop a total solution system for applications, which is called a data processing module system(DPMS), based on ASP, PDF, JAVA, and MSSQL. The DPMS system provides a realtime telematic guidance in the fields of application, recruiting, and management. The user documents the PDF file on the Web browser and store the file in DB for correction. The DPMS discovered the difficulties in data input due to 2 byte problem of Korean language. This result leads to the usage of Korean language as well as others. Test results through free-charge access confirm that the DPMS outperforms the conventional systems based on ASP with HTML in view of simplicity and confidentiality.

  • PDF