• Title/Summary/Keyword: Big data Processing

Search Result 1,063, Processing Time 0.033 seconds

Big Data Conceptualization and Policy Design on Data Sovereignty (빅데이터의 개념적 논의와 데이터 주권에 대한 정책설계)

  • Moon, Hyejung
    • Annual Conference of KIPS
    • /
    • 2013.05a
    • /
    • pp.911-914
    • /
    • 2013
  • 빅데이터가 이전의 대용량정보와 비교하여 어떠한 개념적인 의미를 지니는지 정책설계과정에 따라 이론적으로 논의하고, 이 시대 이슈가 되는 데이터 주권에 대하여 저작권과 CCL을 사례로 ICT정책의 설계방안을 제시한다. 사례분석의 결과 빅데이터 시대 데이터 주권에 대한 정책은 법, 시장, 기술, 규범 측면에서 균형 있게 설계되어야 하며 기술구조를 기초로 사회문제에 대한 규제구조를 설계하고 정책을 집행해야 한다.

Big Data Processing and Monitoring System based on Vehicle Data (차량 데이터 기반 빅데이터 처리 및 모니터링 시스템)

  • Shin, Dong-Yun;Kim, Ju-Ho;Lee, Seung-Hae;Shin, Dong-Jin;Oh, Jae-Kon;Kim, Jeong-Joon
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.19 no.3
    • /
    • pp.105-114
    • /
    • 2019
  • As the Industrial Revolution progressed, Big Data technologies were used to develop a system that instantly identified the consequences of older vehicles using mobile devices. First, data from the vehicle was collected using the OBD2 sensor, and the data collected was stored in the raspberry pie, giving it the same situation that the raspberry pie was driving. In the event that vehicle data is generated, the data is collected in real time, stored in multiple nodes, and visualized and printed based on the processed, refined, processed and processed data. We can use Big Data in this process and quickly process vehicle data to identify it effectively through mobile devices.

A Review on the Management of Water Resources Information based on Big Data and Cloud Computing (빅 데이터와 클라우드 컴퓨팅 기반의 수자원 정보 관리 방안에 관한 검토)

  • Kim, Yonsoo;Kang, Narae;Jung, Jaewon;Kim, Hung Soo
    • Journal of Wetlands Research
    • /
    • v.18 no.1
    • /
    • pp.100-112
    • /
    • 2016
  • In recent, the direction of water resources policy is changing from the typical plan for water use and flood control to the sustainable water resources management to improve the quality of life. This change makes the information related to water resources such as data collection, management, and supply is becoming an important concern for decision making of water resources policy. We had analyzed the structured data according to the purpose of providing information on water resources. However, the recent trend is big data and cloud computing which can create new values by linking unstructured data with structured data. Therefore, the trend for the management of water resources information is also changing. According to the paradigm change of information management, this study tried to suggest an application of big data and cloud computing in water resources field for efficient management and use of water. We examined the current state and direction of policy related to water resources information in Korea and an other country. Then we connected volume, velocity and variety which are the three basic components of big data with veracity and value which are additionally mentioned recently. And we discussed the rapid and flexible countermeasures about changes of consumer and increasing big data related to water resources via cloud computing. In the future, the management of water resources information should go to the direction which can enhance the value(Value) of water resources information by big data and cloud computing based on the amount of data(Volume), the speed of data processing(Velocity), the number of types of data(Variety). Also it should enhance the value(Value) of water resources information by the fusion of water and other areas and by the production of accurate information(Veracity) required for water management and prevention of disaster and for protection of life and property.

Addressing Big Data solution enabled Connected Vehicle services using Hadoop (Hadoop을 이용한 스마트 자동차 서비스용 빅 데이터 솔루션 개발)

  • Nkenyereye, Lionel;Jang, Jong-Wook
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.19 no.3
    • /
    • pp.607-612
    • /
    • 2015
  • As the amount of vehicle's diagnostics data increases, the actors in automotive ecosystem will encounter difficulties to perform a real time analysis in order to simulate or to design new services according to the data gathered from the connected cars. In this paper, we have conducted a study of a Big Data solution that expresses the essential deep analytics to process and analyze vast quantities of vehicles on board diagnostics data generated by cars. Hadoop and its ecosystems have been deployed to process a large data and delivered useful outcomes that may be used by actors in automotive ecosystem to deliver new services to car owners. As the Intelligent transport system is involved to guarantee safety, reduce rate of crash and injured in the accident due to speed, addressing big data solution based on vehicle diagnostics data is upcoming to monitor real time outcome from it and making collection of data from several connected cars, facilitating reliable processing and easier storage of data collected.

Real Time Distributed Parallel Processing to Visualize Noise Map with Big Sensor Data and GIS Data for Smart Cities (스마트시티의 빅 센서 데이터와 빅 GIS 데이터를 융합하여 실시간 온라인 소음지도로 시각화하기 위한 분산병렬처리 방법론)

  • Park, Jong-Won;Sim, Ye-Chan;Jung, Hae-Sun;Lee, Yong-Woo
    • Journal of Internet Computing and Services
    • /
    • v.19 no.4
    • /
    • pp.1-6
    • /
    • 2018
  • In smart cities, data from various kinds of sensors are collected and processed to provide smart services to the citizens. Noise information services with noise maps using the collected sensor data from various kinds of ubiquitous sensor networks is one of them. This paper presents a research result which generates three dimensional (3D) noise maps in real-time for smart cities. To make a noise map, we have to converge many informal data which include big image data of geographical Information and massive sensor data. Making such a 3D noise map in real-time requires the processing of the stream data from the ubiquitous sensor networks in real-time and the convergence operation in real-time. They are very challenging works. We developed our own methodology for real-time distributed and parallel processing for it and present it in this paper. Further, we developed our own real-time 3D noise map generation system, with the methodology. The system uses open source softwares for it. Here in this paper, we do introduce one of our systems which uses Apache Storm. We did performance evaluation using the developed system. Cloud computing was used for the performance evaluation experiments. It was confirmed that our system was working properly with good performance and the system can produce the 3D noise maps in real-time. The performance evaluation results are given in this paper, as well.

A Design and Development of Big Data Indexing and Search System using Lucene (루씬을 이용한 빅데이터 인덱싱 및 검색시스템의 설계 및 구현)

  • Kim, DongMin;Choi, JinWoo;Woo, ChongWoo
    • Journal of Internet Computing and Services
    • /
    • v.15 no.6
    • /
    • pp.107-115
    • /
    • 2014
  • Recently, increased use of the internet resulted in generation of large and diverse types of data due to increased use of social media, expansion of a convergence of among industries, use of the various smart device. We are facing difficulties to manage and analyze the data using previous data processing techniques since the volume of the data is huge, form of the data varies and evolves rapidly. In other words, we need to study a new approach to solve such problems. Many approaches are being studied on this issue, and we are describing an effective design and development to build indexing engine of big data platform. Our goal is to build a system that could effectively manage for huge data set which exceeds previous data processing range, and that could reduce data analysis time. We used large SNMP log data for an experiment, and tried to reduce data analysis time through the fast indexing and searching approach. Also, we expect our approach could help analyzing the user data through visualization of the analyzed data expression.

Dynamic Cluster Management of Hadoop Distributed Filesystem (하둡 분산 파일시스템의 동적 클러스터 관리 기법)

  • Ryu, Wooseok
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2016.10a
    • /
    • pp.435-437
    • /
    • 2016
  • Hadoop Distributed File System(HDFS) is a file system for distributed processing of big data by replicating data to distributed data nodes. HDFS cluster shows a great scalability up to thousands of nodes, but it assumes a exclusive node cluster with numerous nodes for the big data processing. Various operational-purpose worker systems used by office are hardly considered as a part of cluster. This paper discusses this problem and proposes a dynamic cluster management technique to increase storage capability and analytic performance of hadoop cluster. The propsed technique can add legacy systems to the cluster and can remove them from the cluster dynamically depending on their availability.

  • PDF

Building Modeling for Unstructured Data Analysis Using Big Data Processing Technology (빅데이터 처리 기술을 활용한 비정형데이터 분석 모델링 구축)

  • Kim, Jung-Hoon;Kim, Sung-Jin;Kwon, Gi-Yeol;Ju, Da-Hye;Oh, Jae-Yong;Lee, Jun-Dong
    • Proceedings of the Korean Society of Computer Information Conference
    • /
    • 2020.07a
    • /
    • pp.253-255
    • /
    • 2020
  • 기업 및 기관 데이터는 워드프로세서, 프레젠테이션, 이메일, open api, 엑셀, XML, JSON 등과 같은 텍스트 기반의 비정형 데이터로 구성되어 있습니다. 텍스트 마이닝(Textmining)을 통해서 자연어 처리 및 기계학습 등의 기술을 이용하여 정보의 추출부터 요약·분류·군집·연관도 분석 등의 과정을 수행울 진행한다. 다양한 시각화 데이터를 보여줄 수 있는 다양한 모델 구축을 진행한 후 민원 신청 내용을 분석 및 변환 작업을 진행한다. 본 논문은 AI 기술과 빅데이터를 활용하여 민원을 분석을 하여 알맞은 부서에 민원을 자동으로 할당해 주는 기술을 다룬다.

  • PDF

Big Data Analytics of Construction Safety Incidents Using Text Mining (텍스트 마이닝을 활용한 건설안전사고 빅데이터 분석)

  • Jeong Uk Seo;Chie Hoon Song
    • Journal of the Korean Society of Industry Convergence
    • /
    • v.27 no.3
    • /
    • pp.581-590
    • /
    • 2024
  • This study aims to extract key topics through text mining of incident records (incident history, post-incident measures, preventive measures) from construction safety accident case data available on the public data portal. It also seeks to provide fundamental insights contributing to the establishment of manuals for disaster prevention by identifying correlations between these topics. After pre-processing the input data, we used the LDA-based topic modeling technique to derive the main topics. Consequently, we obtained five topics related to incident history, and four topics each related to post-incident measures and preventive measures. Although no dominant patterns emerged from the topic pattern analysis, the study holds significance as it provides quantitative information on the follow-up actions related to the incident history, thereby suggesting practical implications for the establishment of a preventive decision-making system through the linkage between accident history and subsequent measures for reccurrence prevention.

Mobile-based Big Data Processing and Monitoring Technology in IoT Environment (IoT 환경에서 모바일 기반 빅데이터 처리 및 모니터링 기술)

  • Lee, Seung-Hae;Kim, Ju-Ho;Shin, Dong-Youn;Shin, Dong-Jin;Park, Jeong-Min;Kim, Jeong-Joon
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.18 no.6
    • /
    • pp.1-9
    • /
    • 2018
  • In the fourth industrial revolution, which has become an issue now, we have been able to receive instant analysis results faster than the existing slow speed through various Big Data technologies, and to conduct real-time monitoring on mobile and web. First, various irregular sensor Data is generated using IoT device, Raspberry Pi. Sensor Data is collected in real time, and the collected data is distributed and stored using several nodes. Then, the stored Sensor Data is processed and refined. Visualize and output the analysis result after analysis. By using these methods, we can train the human resources required for Big Data and mobile related fields using IoT, and process data efficiently and quickly. We also provide information that can confirm the reliability of research results through real time monitoring.