• Journal of Digital Contents Society
• /
• v.18 no.1
• /
• pp.149-158
• /
• 2017

Big data have been presenting us with exciting opportunities and challenges in economic development. For instance, in the agriculture sector, mixing up of various agricultural data (e.g., weather data, soil data, etc.), and subsequently analyzing these data deliver valuable and helpful information to farmers and agribusinesses. However, massive data in agriculture are generated in every minute through multiple kinds of devices and services such as sensors and agricultural web markets. It leads to the challenges of big data problem including data collection, data storage, and data analysis. Although some systems have been proposed to address this problem, they are still restricted either in the type of data, the type of storage, or the size of data they can handle. In this paper, we propose a novel design of a platform for collecting and analyzing agricultural big data. The proposed platform supports (1) multiple methods of collecting data from various data sources using Flume and MapReduce; (2) multiple choices of data storage including HDFS, HBase, and Hive; and (3) big data analysis modules with Spark and Hadoop.

• Journal of the Korea Institute of Information and Communication Engineering
• /
• v.19 no.8
• /
• pp.1797-1804
• /
• 2015

Integrated log management system can help to predict the risk of security and contributes to improve the security level of the organization, and leads to prepare an appropriate security policy. In this paper, we have designed and implemented a Hadoop-based log analysis system by using distributed database model which can store large amount of data and reduce analysis time by automating log collecting procedure. In the proposed system, we use the HBase in order to store a large amount of data efficiently in the scale-out fashion and propose an easy data storing scheme for analysing data using a Hadoop-based normal expression, which results in improving data processing speed compared to the existing system.

• Journal of Convergence for Information Technology
• /
• v.7 no.5
• /
• pp.177-182
• /
• 2017

As major ICT technologies such as IoT, cloud computing, and Big Data are being applied to manufacturing, smart factories are beginning to be built. The key of smart factory implementation is the ability to acquire and analyze data of the factory. Therefore, the need for a big data analysis platform is increasing. The purpose of this study is to construct a platform for big data analysis of manufacturing process and propose integrated method for analysis. The proposed platform is a RHadoop-based structure that integrates analysis tool R and Hadoop to distribute a large amount of datasets. It can store and analyze big data collected in the unit process and factory in the automation system directly in HBase, and it has overcome the limitations of RDB - based analysis. Such a platform should be developed in consideration of the unit process suitability for smart factories, and it is expected to be a guide to building IoT platforms for SMEs that intend to introduce smart factories into the manufacturing process.

• Journal of the Korea Convergence Society
• /
• v.12 no.7
• /
• pp.45-51
• /
• 2021

In accordance with the rapid non-face-to-face environment and mobile first strategy, the explosive increase and creation of many structured/unstructured data every year demands new decision making and services using big data in all fields. However, there have been few reference cases of using the Hadoop Ecosystem, which uses the rapidly increasing big data every year to collect and load big data into a standard platform that can be applied in a practical environment, and then store and process well-established big data in a relational database. Therefore, in this study, after collecting unstructured data searched by keywords from social network services based on Hadoop 2.0 through three virtual machine servers in the Spring Framework environment, the collected unstructured data is loaded into Hadoop Distributed File System and HBase based on the loaded unstructured data, it was designed and implemented to store standardized big data in a relational database using a morpheme analyzer. In the future, research on clustering and classification and analysis using machine learning using Hive or Mahout for deep data analysis should be continued.

• Smart Media Journal
• /
• v.8 no.3
• /
• pp.88-94
• /
• 2019

Livestock disease is a very important issue in the livestock industry because if livestock disease is not responded quickly enough, its damage can be devastating. To solve the issues involving the occurrence of livestock disease, it is necessary to diagnose in advance the status of livestock disease and develop systematic and scientific livestock feeding technologies. However, there is a lack of domestic studies on such technologies in Korea. This paper, therefore, proposes Livestock Disease Forecasting and Livestock Farm Integrated Control System using Cloud Computing to quickly manage livestock disease. The proposed system collects a variety of livestock data from wireless sensor networks and application. Moreover, it saves and manages the data with the use of the column-oriented database Hadoop HBase, a column-oriented database management system. This provides livestock disease forecasting and livestock farm integrated controlling service through MapReduce Model-based parallel data processing. Lastly, it also provides REST-based web service so that users can receive the service on various platforms, such as PCs or mobile devices.

• Ocean Systems Engineering
• /
• v.9 no.1
• /
• pp.79-95
• /
• 2019

The material analysis of an offshore structure is generally conducted in the contract design phase for the price quotation of a new offshore project. This analysis is conducted manually by an engineer, which is time-consuming and can lead to inaccurate results, because the data size from previous projects is too large, and there are so many materials to consider. In this study, the piping materials in an offshore structure are analyzed for contract design using a big data framework. The big data technologies used include HDFS (Hadoop Distributed File System) for data saving, Hive and HBase for the database to handle the saved data, Spark and Kylin for data processing, and Zeppelin for user interface and visualization. The analyzed results show that the proposed big data framework can reduce the efforts put toward contract design in the estimation of the piping material cost.

• Proceedings of the Korean Information Science Society Conference
• /
• 2011.06b
• /
• pp.377-380
• /
• 2011

음악은 인류의 대표적인 예술로서 오랜 세월동안 사랑을 받아왔다. 그 오래된 세월만큼이나 인류가 만들어온 음악의 수는 방대하다. 방대한 음악이 IT기술의 발달과 인터넷의 확산을 통하여 온라인 음악시장을 형성하였고 음악은 디지털 음원으로 관리되게 되었다. 이러한 디지털 음원을 효과적으로 검색하기 위한 방법은 많이 연구되었다. 그리고 검색을 도와줄 대량의 디지털 음원 자료들을 저장하고 관리하는 기법에 관한 연구가 필요하다. 본 논문에서는 대용량 자료를 처리하는 기술로 관심 받고 있는 하둡을 통하여 이 문제를 연구하였다. 하둡의 맵리듀스, HDFS 그리고 HBase를 이용하여 음악 내용기반검색을 설계하였다. 본 시스템은 음악 검색 시스템을 관리하고 유지하는데 있어서 컴퓨팅자원을 절약함으로써 비용을 절감 효과를 얻을 수 있다.

• KSII Transactions on Internet and Information Systems (TIIS)
• /
• v.9 no.8
• /
• pp.3182-3202
• /
• 2015

Flexibly expanding the storage capacity required to process a large amount of rapidly increasing unstructured log data is difficult in a conventional computing environment. In addition, implementing a log processing system providing features that categorize and analyze unstructured log data is extremely difficult. To overcome such limitations, we propose and design a MongoDB-based unstructured log processing system (MdbULPS) for collecting, categorizing, and analyzing log data generated from banks. The proposed system includes a Hadoop-based analysis module for reliable parallel-distributed processing of massive log data. Furthermore, because the Hadoop distributed file system (HDFS) stores data by generating replicas of collected log data in block units, the proposed system offers automatic system recovery against system failures and data loss. Finally, by establishing a distributed database using the NoSQL-based MongoDB, the proposed system provides methods of effectively processing unstructured log data. To evaluate the proposed system, we conducted three different performance tests on a local test bed including twelve nodes: comparing our system with a MySQL-based approach, comparing it with an Hbase-based approach, and changing the chunk size option. From the experiments, we found that our system showed better performance in processing unstructured log data.

• Proceedings of the Korean Institute of Information and Commucation Sciences Conference
• /
• 2014.05a
• /
• pp.351-354
• /
• 2014

The emergence of big data has brought many changes to the database management environment. the each amount of big data will increase, but each data size is smaller and simpler. This feature was required to a new data processing techniques. Accordingly, A variety database technology was provided to Specializing in big data processing. It is defined as NoSQL. NoSQL is how to use each different, according to the data characteristics. It is difficult to define one. In this paper, Classified according to the characteristics of each type of NoSQL Appropriate NoSQL is proposed.

• The Journal of Bigdata
• /
• v.4 no.2
• /
• pp.207-220
• /
• 2019

Recent developments in information and communication technology has enabled the deployment of sensor based data to provide real-time services. In Korea, The Korea Transportation Safety Authority is collecting driving information of all commercial vehicles through a fitted digital tachograph (DTG). This information gathered using DTG can be utilized in various ways in the field of transportation. Notably in autonomous driving, the real-time analysis of this information can be used to prevent or respond to dangerous driving behavior. However, there is a limit to processing a large amount of data at a level suitable for real-time services using a traditional database system. In particular, due to a such technical problem, the processing of large quantity of traffic big data for real-time commercial vehicle operation information analysis has never been attempted in Korea. In order to solve this problem, this study optimized the new database server system and confirmed that a real-time service is possible. It is expected that the constructed database system will be used to secure base data needed to establish digital twin and autonomous driving environments.