• Proceedings of the Korea Information Processing Society Conference
• /
• 2014.04a
• /
• pp.461-463
• /
• 2014

네트워크 공격이 다양해지고 빈번하게 발생함에 따라 이에 따라 해킹 공격의 유형을 파악하기 위해 다양한 보안 솔루션이 생겨났다. 그 중 하나인 통합보안관리시스템은 다양한 로그 관리와 분석을 통해 보안 정책을 세워 차후에 있을 공격에 대비할 수 있지만 기존 통합보안관리시스템은 대부분 관계형 데이터베이스의 사용으로 급격히 증가하는 데이터를 감당하지 못한다. 많은 정보를 가지는 로그데이터의 유실 방지 및 시스템 저하를 막기 위해 대용량의 로그 데이터를 처리하는 방식이 필요해짐에 따라 분산처리에 특화되어 있는 하둡 에코시스템을 이용하여 늘어나는 데이터에 따라 유연하게 대처할 수 있고 기존 NoSQL 로그 저장방식에서 나아가 로그 저장단계에서 정규화를 사용하여 처리, 저장 능력을 향상시켜 실시간 처리 및 저장, 확장성이 뛰어난 하둡 기반의 로그 수집 시스템을 제안하고자 한다.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2015.10a
• /
• pp.194-196
• /
• 2015

시스템에 이상 징후가 발생하거나 해킹을 당했을 때, 전문가들은 가장 먼저 로그 파일을 확인한다. 이처럼 로그파일을 관리하고 분석하는 것은 시스템을 관리 하는 것에 있어서 필수불가결하다. 하지만 보안을 담당하는 장비에서 발생하는 로그들은 저장 공간의 한계 때문에 일부만 저장되었다가 사라지거나 HDD가 없는 보안장비들은 로그를 남길 수 없다. 따라서 이러한 단점을 해결하기 위해 본 논문에서는 보안 로그 수집과 분석에 하둡 에코 시스템을 접목시켜 방대한 로그를 저장하고, 이를 R프로그래밍으로 분석 할 수 있는 시스템 모델을 제안한다. 제안한 시스템 모델을 구현하기 위한 아키텍처에 대해서도 상세한 결과를 서술하였다.

• Journal of Advanced Navigation Technology
• /
• v.18 no.5
• /
• pp.468-475
• /
• 2014

The recommend system is getting more difficult real time recommend by lager preference data set, computing power and recommend algorithm. For this reason, recommend system is proceeding actively one's studies toward distribute processing method of large preference data set. This paper studied distribute processing method of large preference data set using hadoop distribute processing platform and mahout machine learning library. The recommend algorithm is used Co-occurrence Matrix similar to item Collaborative Filtering. The Co-occurrence Matrix can do distribute processing by many node of hadoop cluster, and it needs many computation scale but can reduce computation scale by distribute processing. This paper has simplified distribute processing of co-occurrence matrix by changes over from four stage to three stage. As a result, this paper can reduce mapreduce job and can generate recommend file. And it has a fast processing speed, and reduce map output data.

• Journal of the Korea Convergence Society
• /
• v.9 no.10
• /
• pp.13-19
• /
• 2018

MapReduce, the Hadoop's essential core technology, is most commonly used to process big data based on the Hadoop distributed file system. However, the existing MapReduce-based big data processing techniques have a feature of dividing and storing files in blocks predefined in the Hadoop distributed file system, thus wasting huge infrastructure resources. Therefore, in this paper, we propose an efficient MapReduce-based big data processing scheme. The proposed method enhances the storage efficiency of a big data infrastructure environment by converting and compressing the data to be processed into a data format in advance suitable for processing by MapReduce. In addition, the proposed method solves the problem of the data processing time delay arising from when implementing with focus on the storage efficiency.

• Journal of KIISE
• /
• v.45 no.2
• /
• pp.99-105
• /
• 2018

One of the spatial statistical analysis, hotspot analysis is one of easy method of see spatial patterns. It is based on the concept that "Adjacent ones are more relevant than those that are far away". However, in hotspot analysis is spatial adjacency must be considered, Therefore, distributed processing is not easy. In this paper, we proposed a distributed algorithm design for hotspot spatial analysis. Its performance was compared to standalone system and Hadoop, Spark based processing. As a result, it is compare to standalone system, Performance improvement rate of Hadoop at 625.89% and Spark at 870.14%. Furthermore, performance improvement rate is high at Spark processing than Hadoop at as more large data set.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2015.10a
• /
• pp.184-186
• /
• 2015

본 논문에서는 PTAM을 위한 새로운 분산 스트림 프로세싱 시스템을 제안한다. PTAM은 하나의 시스템에서 동작하도록 설계되었다. 이는 PTAM이 가지고 있는 한계점을 말해주는 부분인데, PTAM은 Bundle Adjustment의 계산 부하가 커지는 경우에 map을 구축하는데 있어 많은 시간과 리소스가 필요하다. 이에 하둡을 통해 계산 부하를 분산하고, PE(Processing Element)를 Xeon phi 시스템을 통해 동작되는 시스템을 제안한다.

• Journal of the Korea Academia-Industrial cooperation Society
• /
• v.16 no.1
• /
• pp.726-734
• /
• 2015

Relational databases used by structuralizing data are the most widely used in data management at present. However, in relational databases, service becomes slower as the amount of data increases because of constraints in the reading and writing operations to save or query data. Furthermore, when a new task is added, the database grows and, consequently, requires additional infrastructure, such as parallel configuration of hardware, CPU, memory, and network, to support smooth operation. In this paper, in order to improve the web information services that are slowing down due to increase of data in the relational databases, we implemented a model to extract a large amount of data quickly and safely for users by processing Hadoop Distributed File System (HDFS) files after sending data to HDFSs and unifying and reconstructing the data. We implemented our model in a Web-based civil affairs system that stores image files, which is irregular data processing. Our proposed system's data processing was found to be 0.4 sec faster than that of a relational database system. Thus, we found that it is possible to support Web information services with a Hadoop-based big data processing technique in order to process a large amount of data, as in conventional relational databases. Furthermore, since Hadoop is open source, our model has the advantage of reducing software costs. The proposed system is expected to be used as a model for Web services that provide fast information processing for organizations that require efficient processing of big data because of the increase in the size of conventional relational databases.

• Journal of the Korea Institute of Information and Communication Engineering
• /
• v.22 no.4
• /
• pp.575-580
• /
• 2018

This paper discusses system for real-time monitoring of patient waiting time in hospitals based on open-source platform. It is necessary to make use of open-source projects to develop a high-performance stream processing system, which analyzes and processes stream data in real time, with less cost. The Hadoop ecosystem is a well-known big data processing platform consisting of numerous open-source subprojects. This paper first defines several requirements for the monitoring system, and selects a few projects from the Hadoop ecosystem that are suited to meet the requirements. Then, the paper proposes system architecture and a detailed module design using Apache Spark, Apache Kafka, and so on. The proposed system can reduce development costs by using open-source projects and by acquiring data from legacy hospital information system. High-performance and fault-tolerance of the system can also be achieved through distributed processing.

• Proceedings of the Korea Information Processing Society Conference
• /
• 2013.05a
• /
• pp.56-58
• /
• 2013

최근 페이스북이나 트위터와 같은 소셜네트워크 서비스를 포함하여 대용량의 빅데이터에 대한 처리와 분석이 중요한 이슈로 다뤄지고 있으며, 사용자들이 끊임없이 쏟아내는 데이터로 인해서 이러한 데이터들을 어떻게 다룰 것인지, 혹은 어떻게 분석하여 의미 있고, 가치 있는 것으로 가공할 것인지가 중요한 사안으로 여겨지고 있다. 이러한 빅데이터 관리 도구로써 하둡은 빅데이터의 처리와 분석에 있어서 가장 해결에 근접한 도구로 평가받고 있다. 이 논문은 하둡의 주요 구성요소인 HDFS(Hadoop Distributed File System)와 JAVA에 기반하여 제작되는 온라인 대용량 저장소 시스템의 가장 기본적인 요소인 온라인 데이터 저장소를 직접 설계하고 제작하고, 구현하여 봄으로써 대용량 저장소의 구현 방식에 대한 이슈를 다뤄보도록 한다.

• The Journal of the Korea institute of electronic communication sciences
• /
• v.9 no.7
• /
• pp.791-797
• /
• 2014

In order to create an environment for Apache Hadoop for parallel distributed processing system of Bigdata, by connecting a plurality of computers, or to configure the node, using the configuration of the virtual nodes on a single computer it is necessary to build a cloud fading environment. However, be constructed in practice for education in these systems, there are many constraints in terms of cost and complex system configuration. Therefore, it is possible to be used as training for educational institutions and beginners in the field of Bigdata processing, development of learning systems and inexpensive practical is urgent. Based on the Raspberry Pi board, training and analysis of Big data processing, such as Hadoop and NoSQL is now the design and implementation of a learning system of parallel distributed processing of possible Bigdata in this study. It is expected that Bigdata parallel distributed processing system that has been implemented, and be a useful system for beginners who want to start a Bigdata and education.