• Title/Summary/Keyword: distributed data storage

Search Result 296, Processing Time 0.029 seconds

Design and Implementation of parallel Media server in current system environment (기존 시스템 환경에서의 병렬 미디어 서버의 설계 및 구현)

  • 김경훈;류재상;김서균;남지승
    • Proceedings of the IEEK Conference
    • /
    • 2000.06c
    • /
    • pp.97-100
    • /
    • 2000
  • As network resources have become faster and demands for multimedia service through network have increased, the demand for Media server system has increased. These kinds of media server solve their bottle neck problem of internal storage device by using parallel system which takes advantage of fast network resource. Many vendors have suggested each of their media server system to solve these problem radically, but most of them require major modification of infra component and additional drawback has added. For example, storage mechanism for specific media requires new file system which is totally different from traditional one, and algorithm for enhancing performance may not suit for traditional operating system environment. In this paper, we designed a parallel media server based on web interface of traditional system and implemented a program for media server. Implemented server system performs parallel processing through web interface without any modification of traditional system, and controls which is related to merging load by distributed data is charged only to client and control server and consequently load of storage server can be minimized. And also, data transfer protocol for streaming media includes Retransfer algorithm and client Admission control policy relevant to performance of whole system.

  • PDF

Prospect of Information Technology and Its Application to Regional Agricultural Meteorology (지역농업기상지원을 위한 정보화기술 전망 및 활용)

  • Lee, Byong-Lyol
    • Proceedings of The Korean Society of Agricultural and Forest Meteorology Conference
    • /
    • 2003.09a
    • /
    • pp.189-201
    • /
    • 2003
  • Grid is a new Information Technology (IT) concept of "super Internet" for high-performance computing: worldwide collections of high-end resources - such as supercomputers, storage, advanced instruments and immerse environments. The Grid is expected to bring together geographically and organizationally dispersed computational resources, such as CPUs, storage systems, communication systems, real-time data sources and instruments, and human collaborators. The term "the Grid" was coined in the mid l990s to denote a proposed distributed computing infrastructure for advanced science and engineering. The term computational Grids refers to infrastructures aimed at allowing users to access and/or aggregate potentially large numbers of powerful and sophisticated resources. More formally, Grids are defined as infrastructure allowing flexible, secure, and coordinated resource sharing among dynamic collections of individuals, institutions and resources referred to as virtual Organizations. GRID is an emerging IT as a kind of next generation Internet technology which will fit very well with Agrometeorological services in the future. I believe that it would contribute to the resource sharing in AgroMeteorology by providing super computing power, virtual storage, and efficient data exchanges, especially for developing countries that are suffering from the lack of resources for their agmet services at national level. Thus, the establishment of CAgM-GRID based on existing RAMINSII is proposed as a part of FWIS of WMO.part of FWIS of WMO.

  • PDF

A Four-Layer Robust Storage in Cloud using Privacy Preserving Technique with Reliable Computational Intelligence in Fog-Edge

  • Nirmala, E.;Muthurajkumar, S.
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.14 no.9
    • /
    • pp.3870-3884
    • /
    • 2020
  • The proposed framework of Four Layer Robust Storage in Cloud (FLRSC) architecture involves host server, local host and edge devices in addition to Virtual Machine Monitoring (VMM). The goal is to protect the privacy of stored data at edge devices. The computational intelligence (CI) part of our algorithm distributes blocks of data to three different layers by partially encoded and forwarded for decoding to the next layer using hash and greed Solomon algorithms. VMM monitoring uses snapshot algorithm to detect intrusion. The proposed system is compared with Tiang Wang method to validate efficiency of data transfer with security. Hence, security is proven against the indexed efficiency. It is an important study to integrate communication between local host software and nearer edge devices through different channels by verifying snapshot using lamport mechanism to ensure integrity and security at software level thereby reducing the latency. It also provides thorough knowledge and understanding about data communication at software level with VMM. The performance evaluation and feasibility study of security in FLRSC against three-layered approach is proven over 232 blocks of data with 98% accuracy. Practical implications and contributions to the growing knowledge base are highlighted along with directions for further research.

A Study of File Replacement Policy in Data Grid Environments (데이터 그리드 환경에서 파일 교체 정책 연구)

  • Park, Hong-Jin
    • The KIPS Transactions:PartA
    • /
    • v.13A no.6 s.103
    • /
    • pp.511-516
    • /
    • 2006
  • The data grid computing provides geographically distributed storage resources to solve computational problems with large-scale data. Unlike cache replacement policies in virtual memory or web-caching replacement, an optimal file replacement policy for data grids is the one of the important problems by the fact that file size is very large. The traditional file replacement policies such as LRU(Least Recently Used) LCB-K(Least Cost Beneficial based on K), EBR(Economic-based cache replacement), LVCT(Least Value-based on Caching Time) have the problem that they have to predict requests or need additional resources to file replacement. To solve theses problems, this paper propose SBR-k(Sized-based replacement-k) that replaces files based on file size. The results of the simulation show that the proposed policy performs better than traditional policies.

An Iterative Algorithm for the Bottom Up Computation of the Data Cube using MapReduce (맵리듀스를 이용한 데이터 큐브의 상향식 계산을 위한 반복적 알고리즘)

  • Lee, Suan;Jo, Sunhwa;Kim, Jinho
    • Journal of Information Technology and Architecture
    • /
    • v.9 no.4
    • /
    • pp.455-464
    • /
    • 2012
  • Due to the recent data explosion, methods which can meet the requirement of large data analysis has been studying. This paper proposes MRIterativeBUC algorithm which enables efficient computation of large data cube by distributed parallel processing with MapReduce framework. MRIterativeBUC algorithm is developed for efficient iterative operation of the BUC method with MapReduce, and overcomes the limitations about the storage size and processing ability caused by large data cube computation. It employs the idea from the iceberg cube which computes only the interesting aspect of analysts and the distributed parallel process of cube computation by partitioning and sorting. Thus, it reduces data emission so that it can reduce network overload, processing amount on each node, and eventually the cube computation cost. The bottom-up cube computation and iterative algorithm using MapReduce, proposed in this paper, can be expanded in various way, and will make full use of many applications.

Design and Implementation of Distributed File Transfer Server (파일 공유를 위한 분산 파일 서버 설계 및 구현)

  • 박주영;고석주;강신각
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2002.11a
    • /
    • pp.614-618
    • /
    • 2002
  • Although there are a lot of file sharing mechanisms in the Internet such as NFS and P2P, FTP is one of the most common mechanisms used until now. In this paper, we propose a more enhanced file sharing mechanism, the Distributed File Server (DFS) method. The most remarkable feature of DFS is that it can be simply adopted in the current Internet FTP services by locating a DFS node between existing FTP servers and clients without changing any existing file transfer environment. To verify the proposed mechanism we implemented a DFS prototype on RedHat 7.2 system and we finally show that it can reduce the file storage problem caused by big-sized multimedia data.

  • PDF

Distributed In-Memory Caching Method for ML Workload in Kubernetes (쿠버네티스에서 ML 워크로드를 위한 분산 인-메모리 캐싱 방법)

  • Dong-Hyeon Youn;Seokil Song
    • Journal of Platform Technology
    • /
    • v.11 no.4
    • /
    • pp.71-79
    • /
    • 2023
  • In this paper, we analyze the characteristics of machine learning workloads and, based on them, propose a distributed in-memory caching technique to improve the performance of machine learning workloads. The core of machine learning workload is model training, and model training is a computationally intensive task. Performing machine learning workloads in a Kubernetes-based cloud environment in which the computing framework and storage are separated can effectively allocate resources, but delays can occur because IO must be performed through network communication. In this paper, we propose a distributed in-memory caching technique to improve the performance of machine learning workloads performed in such an environment. In particular, we propose a new method of precaching data required for machine learning workloads into the distributed in-memory cache by considering Kubflow pipelines, a Kubernetes-based machine learning pipeline management tool.

  • PDF

Hydrological Drought Assessment of Agricultural Reservoirs based on SWSI in Geum River Basin (SWSI에 기반한 금강권역 농업용 저수지의 수문학적 가뭄평가)

  • Ahn, So-Ra;Park, Jong-Yoon;Jung, In-Kyun;Na, Sang-Jin;Kim, Seong-Joon
    • Journal of The Korean Society of Agricultural Engineers
    • /
    • v.51 no.5
    • /
    • pp.35-49
    • /
    • 2009
  • This study proposes a method to evaluate agricultural reservoirs drought by modifying SWSI (Surface Water Supply Index). The method was applied to Geum river basin and the results were represented as spatially distributed information. The SWSI evaluates hydrological drought of watershed unit by selectively applying one or all of the components of snowpack, precipitation, streamflow and reservoir storage. South Korea has 22 % of agricultural area, and rice paddy covers 64 % among them. Usually paddy fields scattered along stream are irrigated by so many small agricultural reservoirs. It is difficult to evaluate agriculture drought by the little information and large number of agricultural reservoirs. In this study, seven agricultural reservoirs over 10 million ton storage capacity were selected in Geum river basin, and the SWSI was evaluated for both upstream and downstream of the reservoirs using 16 years data (1991-2006). Using the results, multiple regression analyses with precipitation and reservoir storage as variables were conducted and the equations were applied to other watersheds. The spatial results by applying regression equations showed that the severe and moderate drought conditions of July and September in 1994, June in 1995, and May in 2001 were well expressed by the watershed unit.

User-Centric Disaster Recovery System Based on Proxy Re-Encryption Using Blockchain and Distributed Storage (블록체인과 분산 스토리지를 활용한 프록시 재암호화 기반의 사용자 중심 재해 복구 시스템)

  • Park, Junhoo;Kim, Geunyoung;Kim, Junseok;Ryou, Jaecheol
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.31 no.6
    • /
    • pp.1157-1169
    • /
    • 2021
  • The disaster recovery refers to policies and procedures to ensure continuity of services and minimize loss of resources and finances in case of emergency situations such as natural disasters. In particular, the disaster recovery method by the cloud service provider has advantages such as management flexibility, high availability, and cost effectiveness. However, this method has a dependency on a service provider and has a structural limitation in which a user cannot be involved in personal data. In this paper, we propose a protocol using proxy re-encryption for data confidentiality by removing dependency on service providers by backing up user data using blockchain and distributed storage. The proposed method is implemented in Ethereum and IPFS environments, and presents the performance and cost required for backup and recovery operations.

Query Optimization on Large Scale Nested Data with Service Tree and Frequent Trajectory

  • Wang, Li;Wang, Guodong
    • Journal of Information Processing Systems
    • /
    • v.17 no.1
    • /
    • pp.37-50
    • /
    • 2021
  • Query applications based on nested data, the most commonly used form of data representation on the web, especially precise query, is becoming more extensively used. MapReduce, a distributed architecture with parallel computing power, provides a good solution for big data processing. However, in practical application, query requests are usually concurrent, which causes bottlenecks in server processing. To solve this problem, this paper first combines a column storage structure and an inverted index to build index for nested data on MapReduce. On this basis, this paper puts forward an optimization strategy which combines query execution service tree and frequent sub-query trajectory to reduce the response time of frequent queries and further improve the efficiency of multi-user concurrent queries on large scale nested data. Experiments show that this method greatly improves the efficiency of nested data query.