• Title/Summary/Keyword: 정형로그

Search Result 20, Processing Time 0.038 seconds

Development of integrated management solution through log analysis based on Big Data (빅데이터기반의 로그분석을 통한 통합 관리 솔루션 개발)

  • Kang, Sun-Kyoung;Lee, Hyun-Chang;Shin, Seong-Yoon
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2017.10a
    • /
    • pp.541-542
    • /
    • 2017
  • In this paper, we intend to develop an integrated management solution that can be easily operated by integrating complex and various cloud environments. This has the advantage that users and administrators can conveniently solve problems by collecting and analyzing fixed log data and unstructured log data based on big data and realizing integrated monitoring in real time. Hypervisor log pattern analysis technology will be able to manage existing complex and various cloud environment more efficiently.

  • PDF

Design and Implementation of MongoDB-based Unstructured Log Processing System over Cloud Computing Environment (클라우드 환경에서 MongoDB 기반의 비정형 로그 처리 시스템 설계 및 구현)

  • Kim, Myoungjin;Han, Seungho;Cui, Yun;Lee, Hanku
    • Journal of Internet Computing and Services
    • /
    • v.14 no.6
    • /
    • pp.71-84
    • /
    • 2013
  • Log data, which record the multitude of information created when operating computer systems, are utilized in many processes, from carrying out computer system inspection and process optimization to providing customized user optimization. In this paper, we propose a MongoDB-based unstructured log processing system in a cloud environment for processing the massive amount of log data of banks. Most of the log data generated during banking operations come from handling a client's business. Therefore, in order to gather, store, categorize, and analyze the log data generated while processing the client's business, a separate log data processing system needs to be established. However, the realization of flexible storage expansion functions for processing a massive amount of unstructured log data and executing a considerable number of functions to categorize and analyze the stored unstructured log data is difficult in existing computer environments. Thus, in this study, we use cloud computing technology to realize a cloud-based log data processing system for processing unstructured log data that are difficult to process using the existing computing infrastructure's analysis tools and management system. The proposed system uses the IaaS (Infrastructure as a Service) cloud environment to provide a flexible expansion of computing resources and includes the ability to flexibly expand resources such as storage space and memory under conditions such as extended storage or rapid increase in log data. Moreover, to overcome the processing limits of the existing analysis tool when a real-time analysis of the aggregated unstructured log data is required, the proposed system includes a Hadoop-based analysis module for quick and reliable parallel-distributed processing of the massive amount of log data. Furthermore, because the HDFS (Hadoop Distributed File System) stores data by generating copies of the block units of the aggregated log data, the proposed system offers automatic restore functions for the system to continually operate after it recovers from a malfunction. Finally, by establishing a distributed database using the NoSQL-based Mongo DB, the proposed system provides methods of effectively processing unstructured log data. Relational databases such as the MySQL databases have complex schemas that are inappropriate for processing unstructured log data. Further, strict schemas like those of relational databases cannot expand nodes in the case wherein the stored data are distributed to various nodes when the amount of data rapidly increases. NoSQL does not provide the complex computations that relational databases may provide but can easily expand the database through node dispersion when the amount of data increases rapidly; it is a non-relational database with an appropriate structure for processing unstructured data. The data models of the NoSQL are usually classified as Key-Value, column-oriented, and document-oriented types. Of these, the representative document-oriented data model, MongoDB, which has a free schema structure, is used in the proposed system. MongoDB is introduced to the proposed system because it makes it easy to process unstructured log data through a flexible schema structure, facilitates flexible node expansion when the amount of data is rapidly increasing, and provides an Auto-Sharding function that automatically expands storage. The proposed system is composed of a log collector module, a log graph generator module, a MongoDB module, a Hadoop-based analysis module, and a MySQL module. When the log data generated over the entire client business process of each bank are sent to the cloud server, the log collector module collects and classifies data according to the type of log data and distributes it to the MongoDB module and the MySQL module. The log graph generator module generates the results of the log analysis of the MongoDB module, Hadoop-based analysis module, and the MySQL module per analysis time and type of the aggregated log data, and provides them to the user through a web interface. Log data that require a real-time log data analysis are stored in the MySQL module and provided real-time by the log graph generator module. The aggregated log data per unit time are stored in the MongoDB module and plotted in a graph according to the user's various analysis conditions. The aggregated log data in the MongoDB module are parallel-distributed and processed by the Hadoop-based analysis module. A comparative evaluation is carried out against a log data processing system that uses only MySQL for inserting log data and estimating query performance; this evaluation proves the proposed system's superiority. Moreover, an optimal chunk size is confirmed through the log data insert performance evaluation of MongoDB for various chunk sizes.

A Study on Web-log Analysis for CRM based on Internet Business (인터넷 비즈니스 기반의 고객관계관리(CRM)을 위한 웹 로그 분석에 관한 연구)

  • Kim, Jae-Hyung;Noh, Hyo-Won;Kim, Nam-Ho;Chong, Jong-Wha
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2000.04a
    • /
    • pp.10-15
    • /
    • 2000
  • 개별화 웹 마케팅은 본질적으로 고객지향의 패러다임이다. 즉, 개별 고객의 특수한 니즈를 개별적으로 파악해서 각각의 고객에게 차별화된 서비스를 제공하는 것이 그 핵심이다. 웹 서버의 로그파일에 데이터마이닝의 연관규칙 기술을 이용하게 되면 고객행동 패턴의 파악 및 예측을 위한 기법으로 활용할 수 있다. 본 연구에서는 웹 사용자의 교차 판매를 위한 원투원 마케팅에 필요한 접근패턴을 분석하고자 하며, 이는 웹서버 로그파일 분석을 통하여 이루어진다. 분석하고자 하는 웹서버 로그파일은 기존의 데이터웨어하우스의 원천 데이터들과는 다르게 비정형적인 데이터 구조를 가지고있다. 이들 비정형 데이터 처리와 교차판매 지원을 위한 데이터마이닝 모델링, 이를 통한 원투원 마케팅 모델 제시, 그리고 이의 활용이 고객관계관리(CRM)에 미치는 효과를 제시한다.

  • PDF

Spark-based Network Log Analysis Aystem for Detecting Network Attack Pattern Using Snort (Snort를 이용한 비정형 네트워크 공격패턴 탐지를 수행하는 Spark 기반 네트워크 로그 분석 시스템)

  • Baek, Na-Eun;Shin, Jae-Hwan;Chang, Jin-Su;Chang, Jae-Woo
    • The Journal of the Korea Contents Association
    • /
    • v.18 no.4
    • /
    • pp.48-59
    • /
    • 2018
  • Recently, network technology has been used in various fields due to development of network technology. However, there has been an increase in the number of attacks targeting public institutions and companies by exploiting the evolving network technology. Meanwhile, the existing network intrusion detection system takes much time to process logs as the amount of network log increases. Therefore, in this paper, we propose a Spark-based network log analysis system that detects unstructured network attack pattern. by using Snort. The proposed system extracts and analyzes the elements required for network attack pattern detection from large amount of network log data. For the analysis, we propose a rule to detect network attack patterns for Port Scanning, Host Scanning, DDoS, and worm activity, and can detect real attack pattern well by applying it to real log data. Finally, we show from our performance evaluation that the proposed Spark-based log analysis system is more than two times better on log data processing performance than the Hadoop-based system.

Design of Streaming based Unstructured-Data Collecting Framework in IoT Environment (IoT 환경에서 스트리밍 기반의 비정형 데이터 수집 프레임워크 설계)

  • Lee, Hoo-Young;Park, Koo-Rack;Kim, Dong-Hyun
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2017.01a
    • /
    • pp.57-58
    • /
    • 2017
  • 사물인터넷 환경의 다양한 기기에서는 매초마다 시스템 로그 데이터, 온도, 습도, 조도 및 위치 정보 등과 같은 데이터를 지속적으로 생성한다. 이렇게 생성된 데이터는 기기 안에서 대부분 소멸되거나 수집된다 하더라도 시스템 개선의 일부 목적으로 활용하는데 그칠 뿐이다. 본 논문에서는 각각의 사물인터넷 기기에서 발생하는 비정형 데이터를 스트리밍 방식을 통해 수집 서버로 전송하고 이를 유연한 스키마 구조를 가지는 NoSQL 데이터베이스에 적재하는 프레임워크 설계를 제안한다. 이렇게 수많은 장비로부터 수집된 로그 및 센싱 데이터는 빅데이터 분석을 통해 산업의 현장에서 생산성 향상을 위해 사용할 수 있으며 공공의 목적으로 도심지의 교통문제 해소와 재난 및 재해 예측에 활용될 수 있다.

  • PDF

NFV Log Analysis using Machine Learning (머신러닝을 활용한 NFV 시스템 로그 분석)

  • Oh, SeongKeun;Yu, HeonChang
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2017.11a
    • /
    • pp.118-120
    • /
    • 2017
  • 모바일 이동통신망의 Core 노드들은 2G CDMA, 3G WCDMA, 4G LTE 교환기를 비롯하여 IMS 및 다양한 부가장비들로 이루어져 있다. 최근 5G로 진화하는 과정에는 NFV(Network Function Virtualization)가 그 중심에 서 있다. NFV 환경에서는 기존 통신 노드와 다르게 범용서버 및 범용 운영체제가 주축이 되어, 일반 IT 툴로도 통신망 내부 노드의 로그분석이 용이해 졌다. 또한 다양하고 복잡한 Core 네트워크에서 빅데이터로 발생하는 로그 또한 머신러닝으로 분석이 가능하며, 운용에 활용할 수 있다. 따라서 본 연구에서는 vDPI, vMMSGW OS 로그를 대상으로 분석하였으며, 잠재되어 있는 문제점들을 확인할 수 있었다. 또한 어플리케이션의 비정형화 된 로그에서도 비정상적인 패턴들을 발견하여 대용량 트래픽이 발생하며 SLA가 유난히 높은 통신환경에서도 비지도 머신러닝 분석이 유용함을 확인하였다.

NoSQL-based User Behavior Detection System in Cloud Computing Environment (NoSQL 기반 클라우드 사용자 행동 탐지 시스템 설계)

  • Ahn, Kwang-Min;Lee, Bong-Hwan
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2012.10a
    • /
    • pp.804-807
    • /
    • 2012
  • Cloud service provider has to protect client's information securely since all the resources are offered by the service provider, and a large number of users share the resources. In this paper, a NoSQL-based anomaly detection system is proposed in order to enhance the security of mobile cloud services. The existing integrated security management system that uses a relational database can not be used for real-time processing of data since security log from a variety of security equipment and data from cloud node have different data format with unstructured features. The proposed system can resolve the emerging security problem because it provides real time processing and scalability in distributed processing environment.

  • PDF

Security Operation Implementation through Big Data Analysis by Using Open Source ELK Stack (오픈소스 ELK Stack 활용 정보보호 빅데이터 분석을 통한 보안관제 구현)

  • Hyun, Jeong-Hoon;Kim, Hyoung-Joong
    • Journal of Digital Contents Society
    • /
    • v.19 no.1
    • /
    • pp.181-191
    • /
    • 2018
  • With the development of IT, hacking crimes are becoming intelligent and refined. In Emergency response, Big data analysis in information security is to derive problems such as abnormal behavior through collecting, storing, analyzing and visualizing whole log including normal log generated from various information protection system. By using the full log data, including data we have been overlooked, we seek to detect and respond to the abnormal signs of the cyber attack from the early stage of the cyber attack. We used open-source ELK Stack technology to analyze big data like unstructured data that occur in information protection system, terminal and server. By using this technology, we can make it possible to build an information security control system that is optimized for the business environment with its own staff and technology. It is not necessary to rely on high-cost data analysis solution, and it is possible to accumulate technologies to defend from cyber attacks by implementing protection control system directly with its own manpower.

A Study of improving reliability on prediction model by analyzing method Big data (빅데이터 분석방법을 이용한 예측모형의 신뢰도 향상에 관한 연구)

  • Song, Min-Gu;Kim, Sun-Bae
    • Journal of Digital Convergence
    • /
    • v.11 no.6
    • /
    • pp.103-112
    • /
    • 2013
  • Traditional method of establishing prediction model is usually using formal data stored in Data Base. However, nowadays advent of "smart" era brought by ground-breaking development of communication system makes informal data to dominate overall data, such 80% in total. Therefore, conventional method using formal data as establishing predicting model would be untrustworthy means in present. In other words, it is indispensible to make prediction model credible including informal data(SNS, image, video) and semi-formal data(log data). In this study, we increase credibility of predicting model adapting Bigdata method and comparing reliability of conventional measurement to real-data.

A Study on implementation model for security log analysis system using Big Data platform (빅데이터 플랫폼을 이용한 보안로그 분석 시스템 구현 모델 연구)

  • Han, Ki-Hyoung;Jeong, Hyung-Jong;Lee, Doog-Sik;Chae, Myung-Hui;Yoon, Cheol-Hee;Noh, Kyoo-Sung
    • Journal of Digital Convergence
    • /
    • v.12 no.8
    • /
    • pp.351-359
    • /
    • 2014
  • The log data generated by security equipment have been synthetically analyzed on the ESM(Enterprise Security Management) base so far, but due to its limitations of the capacity and processing performance, it is not suited for big data processing. Therefore the another way of technology on the big data platform is necessary. Big Data platform can achieve a large amount of data collection, storage, processing, retrieval, analysis, and visualization by using Hadoop Ecosystem. Currently ESM technology has developed in the way of SIEM (Security Information & Event Management) technology, and to implement security technology in SIEM way, Big Data platform technology is essential that can handle large log data which occurs in the current security devices. In this paper, we have a big data platform Hadoop Ecosystem technology for analyzing the security log for sure how to implement the system model is studied.