• Title/Summary/Keyword: a distributed storage

Search Result 611, Processing Time 0.029 seconds

Implement of MapReduce-based Big Data Processing Scheme for Reducing Big Data Processing Delay Time and Store Data (빅데이터 처리시간 감소와 저장 효율성이 향상을 위한 맵리듀스 기반 빅데이터 처리 기법 구현)

  • Lee, Hyeopgeon;Kim, Young-Woon;Kim, Ki-Young
    • Journal of the Korea Convergence Society
    • /
    • v.9 no.10
    • /
    • pp.13-19
    • /
    • 2018
  • MapReduce, the Hadoop's essential core technology, is most commonly used to process big data based on the Hadoop distributed file system. However, the existing MapReduce-based big data processing techniques have a feature of dividing and storing files in blocks predefined in the Hadoop distributed file system, thus wasting huge infrastructure resources. Therefore, in this paper, we propose an efficient MapReduce-based big data processing scheme. The proposed method enhances the storage efficiency of a big data infrastructure environment by converting and compressing the data to be processed into a data format in advance suitable for processing by MapReduce. In addition, the proposed method solves the problem of the data processing time delay arising from when implementing with focus on the storage efficiency.

Improving Fault Tolerance for High-capacity Shared Distributed File Systems using the Rotational Lease Under Network Partitioning (대용량 공유 분산 화일 시스템에서 망 분할 시 순환 리스를 사용한 고장 감내성 향상)

  • Tak, Byung-Chul;Chung, Yon-Dohn;Kim, Myoung-Ho
    • Journal of KIISE:Databases
    • /
    • v.32 no.6
    • /
    • pp.616-627
    • /
    • 2005
  • In the shared storage file system, systems can directly access the shared storage device through specialized data-only subnetwork unlike in the network attached file server system. In this shared-storage architecture, data consistency is maintained by some designated set of lock servers which use control network to send and receive the lock information. Furthermore, lease mechanism is introduced to cope with the control network failure. But when the control network is partitioned, participating systems can no longer make progress after the lease term expires until the network recovers. This paper addresses this limitation and proposes a method that allows partitioned systems to make progress under the partition of control network. The proposed method works in a manner that each participating system is rotationally given a predefined lease term periodically. It is also shown that the proposed mechanism always preserves data consistency.

Design and Implementation of Context-Aware Middleware for Distributed Ubiquitous Environments (분산 유비쿼터스 환경을 위한 상황 인식 미들웨어의 설계 및 구현)

  • Kim, En-Young;Oh, Dong-Yeol
    • Journal of the Korea Society of Computer and Information
    • /
    • v.11 no.5 s.43
    • /
    • pp.105-114
    • /
    • 2006
  • Context-Awareness is a important technology to support optimized services automatically by recognizing various context informations in ubiquitous environments. Previous middlewares which supports ubiquitous environments used a centralized storage and DBMS to store and manage context informations and service contents. Centralized management of context informations and service contents sometimes hinders the autonomy of moving node and interoperability between difference middlewares. In this paper, we design the systems which stores context informations in moving node by distributed form and shares service contents between middlewares in distributed ubiquitous environments. And it provides Context-Aware scripts to supports the definition of context informations reasoning and execution of services. It verifies the usefulness of the designed systems by applying the scenario of music playing service based on context awareness

  • PDF

Intelligent Coordination Method of Multiple Distributed Resources for Harmonic Current Compensation in a Microgrid

  • Kang, Hyun-Koo;Yoo, Choel-Hee;Chung, Il-Yop;Won, Dong-Jun;Moon, Seung-Il
    • Journal of Electrical Engineering and Technology
    • /
    • v.7 no.6
    • /
    • pp.834-844
    • /
    • 2012
  • Nonlinear electronic loads draw harmonic currents from the power grids that can cause energy loss, miss-operation of power equipment, and other serious problems in the power grids. This paper proposes a harmonic compensation method using multiple distributed resources (DRs) such as small distributed generators (DGs) and battery energy storage systems (BESSs) that are integrated to the power grids through power inverters. For harmonic compensation, DRs should inject additional apparent power to the grids so that certain DRs, especially operated in proximity to their rated power, may possibly reach their maximum current limits. Therefore, intelligent coordination methods of multiple DRs are required for efficient harmonic current compensation considering the power margins of DRs, energy cost, and the battery state-of-charge. The proposed method is based on fuzzy multi-objective optimization so that DRs can cooperate with other DRs to eliminate harmonic currents with optimizing mutually conflicting multi-objectives.

Object-oriented Prototype Framework For Tightly Coupled GIS-based Hydrologic Modeling (객체지향성 프로그래밍 방법을 통한 GIS 연계의 수문모델)

  • Kang, Kwang-Min;Rim, Chang-Soo;Yoon, Sei-Eui
    • Journal of Korea Water Resources Association
    • /
    • v.45 no.6
    • /
    • pp.597-606
    • /
    • 2012
  • With the availability of multi-scale hydrologic data in public domain depending on DEM size, there is a need for a modeling framework that is capable of using these data to simulate hydrologic processes at multiple scales for different topographic and climate conditions for distributed hydrologic model. To address this need, an object-oriented approach, called Geographic and Hydrologic Information System Modeling Objects (GHISMO), is developed. Main hydrologic approaches in GHISMO are storage-release for direct runoff and SCS curve number method for infiltration part. This paper presents conceptual and structural framework of storage-release concept including its application to two watersheds will be presented.

Design and Implementation of a Performance Monitoring and Configuration Management Tool for SANtopia (SANtopia를 위한 성능 감시 및 구성 관리 도구의 설계 및 구현)

  • Rim Kee-Wook;Na Yong-Hi;Min Byoung-Joon;Seo Dea-Wha;Shin Bum-Joo
    • Journal of Internet Computing and Services
    • /
    • v.4 no.1
    • /
    • pp.53-65
    • /
    • 2003
  • I/O processing speed of relatively sluggish storage devices incurs overall performance degradation in computer systems. As a solution to improve the situation, SAN(Storage Area Network) has been proposed. In order to utilize a SAN system more effectively, where storage devices are directly connected with a high speed network such as Fibre Channel, a proper management tool is needed. In this paper, we present a design and implementation of a performance monitoring and configuration management tool for the SANtopia system which supports global file sharing in the SAN environment The developed tool is to monitor the performance of the Linux hosts composing the SANtopia system and to manage the configuration of the hosts and storage devices. It also supports GUI(Graphic User Interface) environment using the JAVA programming language.

  • PDF

Development of Climate & Environment Data System for Big Data from Climate Model Simulations (대용량 기후모델자료를 위한 통합관리시스템 구축)

  • Lee, Jae-Hee;Sung, Hyun Min;Won, Sangho;Lee, Johan;Byu, Young-Hwa
    • Atmosphere
    • /
    • v.29 no.1
    • /
    • pp.75-86
    • /
    • 2019
  • In this paper, we introduce a novel Climate & Environment Database System (CEDS). The CEDS is developed by the National Institute of Meteorological Sciences (NIMS) to provide easy and efficient user interfaces and storage management of climate model data, so improves work efficiency. In uploading the data/files, the CEDS provides an option to automatically operate the international standard data conversion (CMORization) and the quality assurance (QA) processes for submission of CMIP6 variable data. This option increases the system performance, removes the user mistakes, and increases the level of reliability as it eliminates user operation for the CMORization and QA processes. The uploaded raw files are saved in a NAS storage and the Cassandra database stores the metadata that will be used for efficient data access and storage management. The Metadata is automatically generated when uploading a file, or by the user inputs. With the Metadata, the CEDS supports effective storage management by categorizing data/files. This effective storage management allows easy and fast data access with a higher level of data reliability when requesting with the simple search words by a novice. Moreover, the CEDS supports parallel and distributed computing for increasing overall system performance and balancing the load. This supports the high level of availability as multiple users can use it at the same time with fast system-response. Additionally, it deduplicates redundant data and reduces storage space.

A Study of the Active Resonance Damper for a DC Distributed Power System with Parallel Pulsed Power Loads (병렬펄스부하를 갖는 직류배전시스템을 위한 능동 공진 댐퍼에 대한 연구)

  • La, Jae-Du;Lee, Byung-Hun;Chang, Han-Sol;Woo, Hyun-Min;Kim, Young-Seok
    • The Transactions of The Korean Institute of Electrical Engineers
    • /
    • v.61 no.9
    • /
    • pp.1289-1295
    • /
    • 2012
  • An active resonance damper for a DC Distributed Power System(DPS) with parallel loads is presented. Each pulse power load in a DC DPS comprises both a resistive power load and a step-up converter. The step-up converter behave as constant power load(CPL) when tightly regulated and usually cause a negative impedance instability problem. Furthermore, when an input filter is connected to a large constant power load, the instability of DC bus voltage. In this paper, a bidirectional DC/DC converter with a reduced storage capacitor quantitatively are proposed as a active resonance damper, to mitigate the voltage instability on the bus. The validity of the proposed method was confirmed by simulation and experimental works.

Presentation Planning for Distributed VoD Systems (분산 VoD 시스템을 위한 프리젠테이션 플래닝)

  • Hwang, In-Jun;Byeon, Gwang-Jun
    • The Transactions of the Korea Information Processing Society
    • /
    • v.7 no.2S
    • /
    • pp.577-593
    • /
    • 2000
  • A distributed video-on-demand (VoD) system is one where collection of video data is located at dispersed sites across a computer network. In a single site environment, a local video server retrieves video data from its local storage device. However, in the setting of a distributed VoD system, when a customer requests a movie from the local server, the server may need to interact with other servers located across the network. In this paper, we present three types of presentation plans that a local server must construct in order to satisfy the customer request. Informally speaking, a presentation plan is a temporally synchronized detailed sequence of steps that the local server must perform for presenting the requested movie to the customer. This involves obtaining commitments from other video servers, obtaining commitments from the network service provider, as well as making commitments of local resources, within the limitations of available bandwidth, available buffer, and customer data consumption rates. Furthermore, for evaluating the goodness of a presentation plan, we introduce two measures of optimality for presentation plans: minimizing wait time for a customer, and minimizing access bandwidth is used. We develop algorithms to compute optimal presentation plans for all three types, and carry out extensive experiments to compare their performance. We have also mathematically proved certain results for the presentation plans that had previously been verified experimentally in the literature.

  • PDF

A Digital Convergence Platform based on the MPEG-21 Multimedia Framework (MPEG-21기반 디지털 컨버젼스 플랫폼 기술)

  • Oh, Hwa-Yong;Kim, Dong-Hwan;Lee, Eun-Seo;Chang, Tae-Gyu
    • The Transactions of The Korean Institute of Electrical Engineers
    • /
    • v.56 no.5
    • /
    • pp.987-989
    • /
    • 2007
  • This paper describes a digital convergence platform(DCP) which is implemented based on the MPEG-21 multimedia framework. DCP is a newly proposed solution in this research for the convergence service of future home multimedia environment. DCP is a common platform designed to have the feature of reconfigurability, by means of S/W. which supports to run diverse digital multimedia services. A distributed peer to peer service and transaction model is also a new feature realized in DCP using the MPEG-21 multimedia framework. A prototype DCP is implemented to verify its functions of multimedia service and transactions. The developed DCPs are networked with IP clustering storage systems for the distributed service of multimedia. Successful streaming services of the MPEG-2/4 video and audio are verified with the implemented test-bed system of DCP.