• International Journal of Contents
• /
• v.14 no.1
• /
• pp.28-33
• /
• 2018

In this paper, we design and implement a distributed, moving objects management system for processing locations and sensor data from smart black boxes. The proposed system is designed and implemented based on Apache Kafka, Apache Spark & Spark Streaming, Hbase, HDFS. Apache Kafka is used to collect the data from smart black boxes and queries from users. Received location data from smart black boxes and queries from users becomes input of Apache Spark Streaming. Apache Spark Streaming preprocesses the input data for indexing. Recent location data and indexes are stored in-memory managed by Apache Spark. Old data and indexes are flushed into HBase later. We perform experiments to show the throughput of the index manager. Finally, we describe the implementation detail in Scala function level.

• Journal of Korea Society of Digital Industry and Information Management
• /
• v.13 no.2
• /
• pp.77-85
• /
• 2017

Due to the expansion of accommodation space caused by increase of population along with lifestyle changes, most of people spend their time indoor except for the travel time. Because of this, environmental change of indoor is very important, and it affects people's health and economy in resources. But, most of people don't acknowledge the importance of indoor environment. Thus, monitoring system for sustaining and managing indoor environment systematically is needed, and big data clusters should be used in order to save and manage numerous sensor data collected from many spaces. In this paper, we design a big data cluster for the indoor environment monitoring in order to store the sensor data and monitor unit of the huge building Implementation design big data cluster-based system for the analysis, and a distributed file system and building a Hadoop, HBase for big data processing. Also, various sensor data is saved for collection, and effective indoor environment management and health enhancement through monitoring is expected.

• Journal of the Korea Society of Computer and Information
• /
• v.20 no.8
• /
• pp.77-83
• /
• 2015

The storage and retrieval of multimedia data is becoming increasingly important in many application areas including record management, video(CCTV) management and Internet of Things (IoT). In these applications, the files containing multimedia that need to be stored and managed is tremendous and constantly scaling. In this paper, we propose a technique to retrieve a very large number of files, in multimedia format, using the Hadoop Framework. Our strategy is based on the management of metadata that describes the characteristic of files that are stored in Hadoop Distributed File System (HDFS). The metadata schema is represented in Hbase and looked up using SQL On Hadoop (Hive, Tajo). Both the Hbase, Hive and Tajo are part of the Hadoop Ecosystem. Preliminary experiment on multimedia data files stored in HDFS shows the viability of the proposed strategy.

• Journal of KIISE
• /
• v.41 no.9
• /
• pp.686-698
• /
• 2014

Recently, large-scale streaming sensor data have emerged due to explosive supply of smart phones, diffusion of IoT and Cloud computing technology, and generalization of IoT devices. Also, researches on combination of semantic web technology are being actively pushed forward by increasing of requirements for creating new value of data through data sharing and mash-up in large-scale environments. However, we are faced with big issues due to large-scale and streaming data in the inference field for creating a new knowledge. For this reason, we propose the RDFS rule based parallel reasoning scheme to service by processing large-scale streaming sensor data with the semantic web technology. In the proposed scheme, we run in parallel each job of Rete network algorithm, the existing rule inference algorithm and sharing data using the HBase, a hadoop database, as a public storage. To achieve this, we implement our system and evaluate performance through the AWS data of the weather center as large-scale streaming sensor data.

• The Journal of Korea Institute of Information, Electronics, and Communication Technology
• /
• v.9 no.3
• /
• pp.298-305
• /
• 2016

Various NoSQL databases are more excellent to process a large amount of big data than existing relational databases such as MySQL, PostgreSQL and Oracle. Among widely used NoSQL databases, performance of HBase, Cassandra, MongoDB and Redis was comparatively assessed. For distributed processing of a large amount of data, 12 servers were connected through switching hub and Ubuntu was installed as operating system. As for benchmark tool, YCSB was applied. Read and update ratios changed from 50% and 50%, 95% and 5% and finally, 100% and 0% and each of them was assessed as 200,000 commands developed into 1,200,000 commands for each case. Cassandra was most excellent with transaction processing per second while MongoDB was most excellent with the number of processes carried out per unit time.

• KEPCO Journal on Electric Power and Energy
• /
• v.4 no.2
• /
• pp.67-73
• /
• 2018

There are many types of power facilities such as transformers, switches, and energy storage devices in the micro grid environment. However, with the development of IoT technology, opportunities to acquire sensor information such as temperature, pressure, and humidity are provided. In the existing micro grid environment, the communication protocols such as MMS transport protocol in IEC 61850 standard is applied in accordance with the integrated operation between the power facilities and the platform. Therefore, to accommodate IoT data, a gateway technology that can link IoT data to a data collection device (FEP) based on IEC 61850 is required. In this paper, we propose IEC 61850 based IoT gateway platform prototype for microgrid operating system linkage. The gateway platform consists of an IoT protocol interface module (MQTT, CoAP, AMQP) and database, IEC 61850 server. For databases, We used open source based NoSQL databases, Hbase and MongoDB, to store JSON data. We verified the interoperability between the IoT protocol and the IEC 61850 protocol using Sisco's MMS EASY Lite.

• The Journal of the Korea institute of electronic communication sciences
• /
• v.7 no.3
• /
• pp.591-596
• /
• 2012

Recently, the efficient management system for photovoltaic power plants has been required due to the continuously increasing construction of photovoltaic power plants. In this paper, we propose a cloud-based intelligent management system for many photovoltaic power plants. The proposed system stores the measured data of power plants using Hadoop HBase which is a column-oriented database, and processes the calculations of performance, efficiency, and prediction the amount of power generation by parallel processing based on Map-Reduce model. And, Web-based data visualization module allows the administrator to provide information in various forms.

• Proceedings of the Korean Institute of Information and Commucation Sciences Conference
• /
• 2013.10a
• /
• pp.237-239
• /
• 2013

Data so as to meet key values are preserved at Column-Family Stores such as Cassandra, HBase, Hypertable, and Amazon Simple DB in the Big Data environment. In this paper, with referring to Cassandra, we define column-family data stores and its structure. And then, we check out their characteristics such as consistency, transaction, availability, retrieval function (basic queries and advance queries) with CQL (Cassandra Query Language), and expandability. Also, we appropriate or inappropriate subjects for application of column-family stores.

• Journal of Internet Computing and Services
• /
• v.20 no.5
• /
• pp.49-55
• /
• 2019

Organizations design and operate business process models to achieve their goals efficiently and systematically. With the advancement of IT technology, the number of items that computer systems can participate in and the process becomes huge and complicated. This phenomenon created a more complex and subdivide flow of business process.The process instances that contain workcase and events are larger and have more data. This is an essential resource for process mining and is used directly in model discovery, analysis, and improvement of processes. This event log is getting bigger and broader, which leads to problems such as capacity management and I / O load in management of existing row level program or management through a relational database. In this paper, as the event log becomes big data, we have found the problem of management limit based on the existing original file or relational database. Design and apply schemes to archive and analyze large event logs through Hadoop, an open source distributed file system, and HBase, a NoSQL database system.

• Journal of the Korea Institute of Information and Communication Engineering
• /
• v.22 no.2
• /
• pp.382-389
• /
• 2018

Recently cyber attacks can cause serious damage on various information systems. Log data analysis would be able to resolve this problem. Security log analysis system allows to cope with security risk properly by collecting, storing, and analyzing log data information. In this paper, a security log analysis system is designed and implemented in order to analyze security log data using the Logstash in the Elasticsearch, a distributed search engine which enables to collect and process various types of log data. The Kibana, an open source data visualization plugin for Elasticsearch, is used to generate log statistics and search report, and visualize the results. The performance of Elasticsearch-based security log analysis system is compared to the existing log analysis system which uses the Flume log collector, Flume HDFS sink and HBase. The experimental results show that the proposed system tremendously reduces both database query processing time and log data analysis time compared to the existing Hadoop-based log analysis system.