• Title/Summary/Keyword: Big Data Processing Technology

Search Result 388, Processing Time 0.043 seconds

Monitoring of Virtual Machines in the Eucalyptus Cloud

  • Nandimandalam, Mohan Krishna Varma;Choi, Eunmi
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2013.11a
    • /
    • pp.169-171
    • /
    • 2013
  • Cloud computing provides access to big volumes of data and computational resources through various services. Cloud computing also supports to process these volumes of data using set of computers. Cloud computing can satisfy resource requirements through virtualization technology. Eucalyptus is an open source cloud computing environment helps the users to setup their own private cloud based on virtualization. In this paper, monitoring of virtual machines is explained with the eucalyptus cloud setup.

Big Data Conceptualization and Policy Design on Data Sovereignty (빅데이터의 개념적 논의와 데이터 주권에 대한 정책설계)

  • Moon, Hyejung
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2013.05a
    • /
    • pp.911-914
    • /
    • 2013
  • 빅데이터가 이전의 대용량정보와 비교하여 어떠한 개념적인 의미를 지니는지 정책설계과정에 따라 이론적으로 논의하고, 이 시대 이슈가 되는 데이터 주권에 대하여 저작권과 CCL을 사례로 ICT정책의 설계방안을 제시한다. 사례분석의 결과 빅데이터 시대 데이터 주권에 대한 정책은 법, 시장, 기술, 규범 측면에서 균형 있게 설계되어야 하며 기술구조를 기초로 사회문제에 대한 규제구조를 설계하고 정책을 집행해야 한다.

A Study on the Prediction Model for Student Dropout (학생 중도탈락 예측 모델에 관한 연구)

  • Lee, JongHyuk;Kim, DaeHak;Gil, JoonMin
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2018.05a
    • /
    • pp.37-40
    • /
    • 2018
  • 빅데이터 산업 부상과 함께 교육 데이터 분석 분야가 새롭게 주목받고 있다. 교육 현장에서 학습 데이터의 양과 종류는 꾸준히 증가하고 있고 이를 분석하기 위한 정보기술도 계속 발전하고 있다. 한편, 학교 교육은 사회적 성취와 밀접한 관련이 있어 사회이동의 중요한 수단이 되는 만큼 학교 교육으로부터 이탈할 위험이 있는 학생들을 조기에 발견하여 이탈을 방지하는 것은 매우 중요하다. 본 논문은 대학생의 중도탈락을 예방하기 위해 로지스틱 회귀분석과 다층 퍼셉트론 기법을 이용해 학습 데이터를 분석하여 예측 모델을 생성하고 해당 모델을 평가한다. 평가 결과, 다층 퍼셉트론 모델이 로지스틱 회귀분석 모델에 비해 정확도와 재현율은 우수하였지만 정밀도는 약간 저조하였다.

A Method on Associated Document Recommendation with Word Correlation Weights (단어 연관성 가중치를 적용한 연관 문서 추천 방법)

  • Kim, Seonmi;Na, InSeop;Shin, Juhyun
    • Journal of Korea Multimedia Society
    • /
    • v.22 no.2
    • /
    • pp.250-259
    • /
    • 2019
  • Big data processing technology and artificial intelligence (AI) are increasingly attracting attention. Natural language processing is an important research area of artificial intelligence. In this paper, we use Korean news articles to extract topic distributions in documents and word distribution vectors in topics through LDA-based Topic Modeling. Then, we use Word2vec to vector words, and generate a weight matrix to derive the relevance SCORE considering the semantic relationship between the words. We propose a way to recommend documents in order of high score.

A Case Study on Product Production Process Optimization using Big Data Analysis: Focusing on the Quality Management of LCD Production (빅데이터 분석 적용을 통한 공정 최적화 사례연구: LCD 공정 품질분석을 중심으로)

  • Park, Jong Tae;Lee, Sang Kon
    • Journal of Information Technology Services
    • /
    • v.21 no.2
    • /
    • pp.97-107
    • /
    • 2022
  • Recently, interest in smart factories is increasing. Investments to improve intelligence/automation are also being made continuously in manufacturing plants. Facility automation based on sensor data collection is now essential. In addition, we are operating our factories based on data generated in all areas of production, including production management, facility operation, and quality management, and an integrated standard information system. When producing LCD polarizer products, it is most important to link trace information between data generated by individual production processes. All systems involved in production must ensure that there is no data loss and data integrity is ensured. The large-capacity data collected from individual systems is composed of key values linked to each other. A real-time quality analysis processing system based on connected integrated system data is required. In this study, large-capacity data collection, storage, integration and loss prevention methods were presented for optimization of LCD polarizer production. The identification Risk model of inspection products can be added, and the applicable product model is designed to be continuously expanded. A quality inspection and analysis system that maximizes the yield rate was designed by using the final inspection image of the product using big data technology. In the case of products that are predefined as analysable products, it is designed to be verified with the big data knn analysis model, and individual analysis results are continuously applied to the actual production site to operate in a virtuous cycle structure. Production Optimization was performed by applying it to the currently produced LCD polarizer production line.

Design and Implementation of Efficient Storage and Retrieval Technology of Traffic Big Data (교통 빅데이터의 효율적 저장 및 검색 기술의 설계와 구현)

  • Kim, Ki-su;Yi, Jae-Jin;Kim, Hong-Hoi;Jang, Yo-lim;Hahm, Yu-Kun
    • The Journal of Bigdata
    • /
    • v.4 no.2
    • /
    • pp.207-220
    • /
    • 2019
  • Recent developments in information and communication technology has enabled the deployment of sensor based data to provide real-time services. In Korea, The Korea Transportation Safety Authority is collecting driving information of all commercial vehicles through a fitted digital tachograph (DTG). This information gathered using DTG can be utilized in various ways in the field of transportation. Notably in autonomous driving, the real-time analysis of this information can be used to prevent or respond to dangerous driving behavior. However, there is a limit to processing a large amount of data at a level suitable for real-time services using a traditional database system. In particular, due to a such technical problem, the processing of large quantity of traffic big data for real-time commercial vehicle operation information analysis has never been attempted in Korea. In order to solve this problem, this study optimized the new database server system and confirmed that a real-time service is possible. It is expected that the constructed database system will be used to secure base data needed to establish digital twin and autonomous driving environments.

  • PDF

Study on Proactive Data Process Orchestration in Distributed Cloud

  • Jong-Sub Lee;Seok-Jae Moon
    • International journal of advanced smart convergence
    • /
    • v.13 no.3
    • /
    • pp.135-142
    • /
    • 2024
  • Recently, along with digital transformation, technologies such as cloud computing, big data, and artificial intelligence have been actively introduced. In a situation where these technological changes are progressing rapidly, it is often difficult to manage processes efficiently using existing simple workflow management methods. Companies providing current cloud services are adopting virtualization technologies, including virtual machines (VMs) and containers, in their distributed system infrastructure for automated application deployment. Accordingly, this paper proposes a process-based orchestration system for integrated execution of corporate process-oriented workloads by integrating the potential of big data and machine learning technologies. This system consists of four layers as components for performing workload processes. Additionally, a common information model is applied to the data to efficiently integrate and manage the various formats and uses of data generated during the process creation stage. Moreover, a standard metadata protocol is introduced to ensure smooth exchange between data. This proposed system utilizes various types of data storage to store process data, metadata, and analysis models. This enables flexible management and efficient processing of data.

A Study on Distributed Processing of Big Data and User Authentication for Human-friendly Robot Service on Smartphone (인간 친화적 로봇 서비스를 위한 대용량 분산 처리 기술 및 사용자 인증에 관한 연구)

  • Choi, Okkyung;Jung, Wooyeol;Lee, Bong Gyou;Moon, Seungbin
    • Journal of Internet Computing and Services
    • /
    • v.15 no.1
    • /
    • pp.55-61
    • /
    • 2014
  • Various human-friendly robot services have been developed and mobile cloud computing is a real time computing service that allows users to rent IT resources what they want over the internet and has become the new-generation computing paradigm of information society. The enterprises and nations are actively underway of the business process using mobile cloud computing and they are aware of need for implementing mobile cloud computing to their business practice, but it has some week points such as authentication services and distributed processing technologies of big data. Sometimes it is difficult to clarify the objective of cloud computing service. In this study, the vulnerability of authentication services on mobile cloud computing is analyzed and mobile cloud computing model is constructed for efficient and safe business process. We will also be able to study how to process and analyze unstructured data in parallel to this model, so that in the future, providing customized information for individuals may be possible using unstructured data.

Design and Implementation of Big Data Streaming Query Processing System for Realtime Power Plant Sensor data (실시간 발전소 시설 장비 센서 데이터에 대한 빅데이터 스트리밍 질의 처리 시스템 설계 및 구현)

  • Um, Jung-Ho;Yu, Chan Hee;Sarda, Komal;Park, Kyongseok
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2020.11a
    • /
    • pp.88-91
    • /
    • 2020
  • 발전 시설은 연간 무중단으로 운영되어야 하고, 고장이 발생하면 손해가 막대하기 때문에 발전 시설 장비에는 수십만 개의 센서 데이터가 설치되어 있다. 본 논문에서는 효율적인 센서 데이터의 수집과 시설 모니터링 및 고장 예측 등을 위한 빅데이터 스트리밍 질의 처리 시스템을 설계 및 구현하였다. 또한 실시간 데이터 수집의 효율적인 관리를 위해 인코딩 방식을 설계하였으며, 데이터 전송 성능을 측정하여 문자열로 데이터를 전송하는 것보다 평균 12%, 최대 32% 데이터 처리 성능이 향상됨을 보였다. 또한, 스트리밍 데이터에 대한 윈도우 질의 처리 성능을 측정하여 약 0.97초의 평균 집계 질의 처리 시간이 소요됨을 확인하였다. 향후에는 고장 감지를 위한 인공지능 추론 모델을 제안하는 빅데이터 스트리밍 질의 처리 시스템에 적용할 예정이다.

The Efficient Method of Parallel Genetic Algorithm using MapReduce of Big Data (빅 데이터의 MapReduce를 이용한 효율적인 병렬 유전자 알고리즘 기법)

  • Hong, Sung-Sam;Han, Myung-Mook
    • Journal of the Korean Institute of Intelligent Systems
    • /
    • v.23 no.5
    • /
    • pp.385-391
    • /
    • 2013
  • Big Data is data of big size which is not processed, collected, stored, searched, analyzed by the existing database management system. The parallel genetic algorithm using the Hadoop for BigData technology is easily realized by implementing GA(Genetic Algorithm) using MapReduce in the Hadoop Distribution System. The previous study that the genetic algorithm using MapReduce is proposed suitable transforming for the GA by MapReduce. However, they did not show good performance because of frequently occurring data input and output. In this paper, we proposed the MRPGA(MapReduce Parallel Genetic Algorithm) using improvement Map and Reduce process and the parallel processing characteristic of MapReduce. The optimal solution can be found by using the topology, migration of parallel genetic algorithm and local search algorithm. The convergence speed of the proposal method is 1.5 times faster than that of the existing MapReduce SGA, and is the optimal solution can be found quickly by the number of sub-generation iteration. In addition, the MRPGA is able to improve the processing and analysis performance of Big Data technology.