• Title/Summary/Keyword: Bigdata Center Processing System

Search Result 8, Processing Time 0.026 seconds

A Study on Data Processing Technology based on a open source R to improve utilization of the Geostationary Ocean Color Imager(GOCI) Products (천리안해양관측위성 산출물 활용성 향상을 위한 오픈소스 R 기반 데이터 처리기술 연구)

  • OH, Jung-Hee;CHOI, Hyun-Woo;LEE, Chol-Young;YANG, Hyun;HAN, Hee-Jeong
    • Journal of the Korean Association of Geographic Information Studies
    • /
    • v.22 no.4
    • /
    • pp.215-228
    • /
    • 2019
  • HDF5 data format is used to effectively store and distribute large volume of Geostationary Ocean Color Imager(GOCI) satellite data. The Korea Ocean Satellite Center has developed and provided a GOCI Data Processing System(GDPS) for general users who are not familiar with HDF5 format. Nevertheless, it is not easy to merge and process Hierarchical Data Format version5(HDF5) data that requires an understanding of satellite data characteristics, needs to learn how to use GDPS, and stores location and attribute information separately. Therefore, the open source R and rhdf5, data.table, and matrixStats packages were used to develop algorithm that could easily utilize satellite data in HDF5 format without the need for the process of using GDPS.

Performance Optimization of Big Data Center Processing System - Big Data Analysis Algorithm Based on Location Awareness

  • Zhao, Wen-Xuan;Min, Byung-Won
    • International Journal of Contents
    • /
    • v.17 no.3
    • /
    • pp.74-83
    • /
    • 2021
  • A location-aware algorithm is proposed in this study to optimize the system performance of distributed systems for processing big data with low data reliability and application performance. Compared with previous algorithms, the location-aware data block placement algorithm uses data block placement and node data recovery strategies to improve data application performance and reliability. Simulation and actual cluster tests showed that the location-aware placement algorithm proposed in this study could greatly improve data reliability and shorten the application processing time of I/O interfaces in real-time.

Image analysis method and system for multi-center Medical bigdata research (다기관 의료 빅데이터 연구를 위한 영상 분석 방법 및 시스템)

  • Kim, Seung-Jin;Jeong, Chang-Won;Kim, Tae-Hoon;Jun, Hong Yong;No, Si-Hyeong;Kim, Ji-Eon;Lee, Yun Oh;Yoon, Kwon-Ha
    • Annual Conference of KIPS
    • /
    • 2018.10a
    • /
    • pp.428-429
    • /
    • 2018
  • 본 논문에서는 다기관 의료영상 분석 방법 및 시스템을 제안한다. 다기관 연구에 참여하는 기관에게 분석 가이드 및 분석 프로그램을 제공하여 표준화된 영상분석 연구를 지원하고자 한다. 이를 위해 동일한 프로토콜로 표준화된 영상을 획득 및 분석하고 결과를 공유하는 분산형 연구방법을 제시한다. 제안하는 시스템은 개인정보보호법 및 보안문제가 강조되고 있는 의료현장에 적합한 시스템으로 다양한 다기관 의료 빅데이터 분석 연구에 활용될 것으로 기대된다.

Infrastructure Anomaly Analysis for Data-center Failure Prevention: Based on RRCF and Prophet Ensemble Analysis (데이터센터 장애 예방을 위한 인프라 이상징후 분석: RRCF와 Prophet Ensemble 분석 기반)

  • Hyun-Jong Kim;Sung-Keun Kim;Byoung-Whan Chun;Kyong-Bog, Jin;Seung-Jeong Yang
    • The Journal of Bigdata
    • /
    • v.7 no.1
    • /
    • pp.113-124
    • /
    • 2022
  • Various methods using machine learning and big data have been applied to prevent failures in Data Centers. However, there are many limitations to referencing individual equipment-based performance indicators or to being practically utilized as an approach that does not consider the infrastructure operating environment. In this study, the performance indicators of individual infrastructure equipment are integrated monitoring and the performance indicators of various equipment are segmented and graded to make a single numerical value. Data pre-processing based on experience in infrastructure operation. And an ensemble of RRCF (Robust Random Cut Forest) analysis and Prophet analysis model led to reliable analysis results in detecting anomalies. A failure analysis system was implemented to facilitate the use of Data Center operators. It can provide a preemptive response to Data Center failures and an appropriate tuning time.

Establishing a Sustainable Future Smart Education System (지속가능한 미래형 스마트교육 시스템 구축 방안)

  • Park, Ji-Hyeon;Choi, Jae-Myeong;Park, Byoung-Lyoul;Kang, Heau-Jo
    • Journal of Advanced Navigation Technology
    • /
    • v.16 no.3
    • /
    • pp.495-503
    • /
    • 2012
  • As modern society rapidly changes, the field of education has also developed speedily. Since Edunet system developed in 1996, many different systems are developing continuously such as Center for Teaching and Learning, cyber home learning systems, diagnosis prescribing systems, video systems, teaching and counseling, and study management systems. However, the aforementioned systems have had not great response from the educational consumers due to a lack of interconnection. There are several reasons for it. One of the reasons is that program administrators did not carefully consider the continuity of each programs but established a brand new system whenever they need rather than predict or consider the future needs. The suitable system for smart education should be one big integrated system based on many different data analysis and processing. The system should also supply educational consumers various and useful information by adopting the idea of bigdata rather than a single sign on system connecting each independent system. The cloud computing system should be established as a system that can be managed not as simple compiled files and application programs but as various contents and DATA.

A Study on the Real-time Recommendation Box Recommendation of Fulfillment Center Using Machine Learning (기계학습을 이용한 풀필먼트센터의 실시간 박스 추천에 관한 연구)

  • Dae-Wook Cha;Hui-Yeon Jo;Ji-Soo Han;Kwang-Sup Shin;Yun-Hong Min
    • The Journal of Bigdata
    • /
    • v.8 no.2
    • /
    • pp.149-163
    • /
    • 2023
  • Due to the continuous growth of the E-commerce market, the volume of orders that fulfillment centers have to process has increased, and various customer requirements have increased the complexity of order processing. Along with this trend, the operational efficiency of fulfillment centers due to increased labor costs is becoming more important from a corporate management perspective. Using historical performance data as training data, this study focused on real-time box recommendations applicable to packaging areas during fulfillment center shipping. Four types of data, such as product information, order information, packaging information, and delivery information, were applied to the machine learning model through pre-processing and feature-engineering processes. As an input vector, three characteristics were used as product specification information: width, length, and height, the characteristics of the input vector were extracted through a feature engineering process that converts product information from real numbers to an integer system for each section. As a result of comparing the performance of each model, it was confirmed that when the Gradient Boosting model was applied, the prediction was performed with the highest accuracy at 95.2% when the product specification information was converted into integers in 21 sections. This study proposes a machine learning model as a way to reduce the increase in costs and inefficiency of box packaging time caused by incorrect box selection in the fulfillment center, and also proposes a feature engineering method to effectively extract the characteristics of product specification information.

System Architecture of the Integrated Data Safety Zone for the Secured Application of Transportation-specific Mobility Data (교통 분야 모빌리티 데이터의 안전한 활용을 위한 통합데이터안심구역 시스템 아키텍처 개발)

  • Hyoungkun Lee;Keedong Yoo
    • The Journal of The Korea Institute of Intelligent Transport Systems
    • /
    • v.22 no.3
    • /
    • pp.88-103
    • /
    • 2023
  • With the recent advancement of 4th Industrial Revolution technology, transportation systems are generating large amounts of mobility data related to the individual movement trajectories of vehicles and people. There are many constraints on utilizing mobility data containing personal information. Thus, in South Korea, the processing and generation of pseudonymized information and the analysis and utilization of this information have been managed in a dual manner by applying separate agencies and technologies through the revision of the Data 3 Act and the enactment of the Data Basic Act. However, this dual approach fails to securely support the entire data lifecycle and suffers from inefficiencies in terms of processing time and cost. Therefore, to compensate for the problems of the existing Expert Data Combination System and Data Safety Zone, this study proposes an Integrated Data Safety Zone Framework that integrates and unifies the process of generating, processing, analyzing, and utilizing mobility data. The integrated process for data processing was redesigned, and common requirements and core technologies were derived. The result is an architecture for a next-generation Integrated Data Safety Zone system that can manage and utilize the entire life cycle of mobility data at one stop.

Construction of Artificial Intelligence Training Platform for Multi-Center Clinical Research (다기관 임상연구를 위한 인공지능 학습 플랫폼 구축)

  • Lee, Chung-Sub;Kim, Ji-Eon;No, Si-Hyeong;Kim, Tae-Hoon;Yoon, Kwon-Ha;Jeong, Chang-Won
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.9 no.10
    • /
    • pp.239-246
    • /
    • 2020
  • In the medical field where artificial intelligence technology is introduced, research related to clinical decision support system(CDSS) in relation to diagnosis and prediction is actively being conducted. In particular, medical imaging-based disease diagnosis area applied AI technologies at various products. However, medical imaging data consists of inconsistent data, and it is a reality that it takes considerable time to prepare and use it for research. This paper describes a one-stop AI learning platform for converting to medical image standard R_CDM(Radiology Common Data Model) and supporting AI algorithm development research based on the dataset. To this, the focus is on linking with the existing CDM(common data model) and model the system, including the schema of the medical imaging standard model and report information for multi-center research based on DICOM(Digital Imaging and Communications in Medicine) tag information. And also, we show the execution results based on generated datasets through the AI learning platform. As a proposed platform, it is expected to be used for various image-based artificial intelligence researches.