• Title/Summary/Keyword: distributed storage systems

Search Result 207, Processing Time 0.027 seconds

Recent Trends of Universal Data Compression (유니버샬 데이터 압축의 최근의 연구동향)

  • 박지환;진용옥
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.16 no.10
    • /
    • pp.901-913
    • /
    • 1991
  • Data compression has important application in the areas of file storage and distributed computer systems. The universal data compression achieves asymptotically optimum cimpression ratio for strings generated by any stationary ergodic source without a priori source probabilities.The paper describes the principle and the recent research trends on universal data compression. And its applications.

  • PDF

Query Optimization on Large Scale Nested Data with Service Tree and Frequent Trajectory

  • Wang, Li;Wang, Guodong
    • Journal of Information Processing Systems
    • /
    • v.17 no.1
    • /
    • pp.37-50
    • /
    • 2021
  • Query applications based on nested data, the most commonly used form of data representation on the web, especially precise query, is becoming more extensively used. MapReduce, a distributed architecture with parallel computing power, provides a good solution for big data processing. However, in practical application, query requests are usually concurrent, which causes bottlenecks in server processing. To solve this problem, this paper first combines a column storage structure and an inverted index to build index for nested data on MapReduce. On this basis, this paper puts forward an optimization strategy which combines query execution service tree and frequent sub-query trajectory to reduce the response time of frequent queries and further improve the efficiency of multi-user concurrent queries on large scale nested data. Experiments show that this method greatly improves the efficiency of nested data query.

Smart Sensor Management System Supporting Service Plug-In in MQTT-Based IIoT Applications

  • Lee, Young-Ran;Kim, Sung-Ki
    • Journal of Multimedia Information System
    • /
    • v.9 no.3
    • /
    • pp.209-218
    • /
    • 2022
  • Industrial IoT applications, including smart factories, require two problem-solving to build data monitoring systems required by services from distributed IoT sensors (smart sensors). One is to overcome proprietary protocols, data formats, and hardware differences and to uniquely identify and connect IoT sensors, and the other is to overcome the problem of changing the server-side data storage structure and sensor data transmission format according to the addition or change of service or IoT sensors. The IEEE 1451.4 standard-based or IPMI specification-based smart sensor technology supports the development of plug-and-play sensors that solve the first problem. However, there is a lack of research that requires a second problem-solving, which requires support for the plug-in of IoT sensors into remote services. To propose a solution for the integration of these two problem-solving, we present a IoT sensor platform, a service system architecture, and a service plugin protocol for the MQTT-based IIoT application environment.

Eliminating Redundant Data for Storage Efficiency on Distributed File Systems (저장 공간의 효율성을 위한 분산 파일 시스템의 중복 데이터 제거 기법)

  • Kim, Jung Hoon;Lim, ByoungHong;Eom, Young Ik
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2009.11a
    • /
    • pp.111-112
    • /
    • 2009
  • 최근 IT 분야의 키워드 중 하나인 클라우드 컴퓨팅에서, 분산 파일 시스템의 선택은 대용량의 데이터를 관리하기 위해 매우 중요하다. 오픈소스 분산 파일 시스템 중 하나인 HDFS는 데이터의 효율적인 저장과 검색의 장점을 통해 최근 널리 사용되고 있다. HDFS는 데이터를 3단계 중복저장을 통해 신뢰성을 보장한다. 그러나 이러한 중복저장은 데이터 저장의 효율성 저하의 문제점을 갖고 있다. 따라서 본 논문에서는 MD5 해쉬 기술을 적용한 중복 데이터 제거 기법을 제안한다. 본 기법은 시뮬레이션을 통해 저장 공간의 효율성을 향상의 결과를 확인하였다.

A Practical Method of a Distributed Information Resources Based on a Mediator for the u-Learning Environment (유비쿼터스 학습(u-Learning)을 위한 미디에이터 기반의 분산정보 활용방법)

  • Joo, Kil-Hong
    • Journal of The Korean Association of Information Education
    • /
    • v.9 no.1
    • /
    • pp.79-86
    • /
    • 2005
  • With the rapid advance of computer and communication technology, the amount of data transferred is also increasing more than ever. The recent trend of education systems is connecting related information semantically in different systems in order to improve the utilization of computerized information Therefore, Web-based teaching-learning is developing in the ubiquitous learning direction that learners select and organize the contents, time and order of learning by themselves. That is, it is evolving to provide teaching-learning environment adaptive to individual learners' characteristics (their level of knowledge, pattern of study, areas of interest). This paper proposes the efficient evaluation method of learning contents in a mediator for the integration of heterogeneous information resources. This means that the autonomy of a remote server can be preserved to the highest degree. In addition, this paper proposes the adaptive optimization of learning contents such that available storage in a mediator can be highly utilized at any time. In order to differentiate the recent usage of a learning content from the past, the accumulated usage frequency of a learning content decays as time goes by.

  • PDF

Design and Implementation of a Metadata Structure for Large-Scale Shared-Disk File System (대용량 공유디스크 파일 시스템에 적합한 메타 데이타 구조의 설계 및 구현)

  • 이용주;김경배;신범주
    • Journal of KIISE:Computer Systems and Theory
    • /
    • v.30 no.1
    • /
    • pp.33-49
    • /
    • 2003
  • Recently, there have been large storage demands for manipulating multimedia data. To solve the tremendous storage demands, one of the major researches is the SAN(Storage Area Network) that provides the local file requests directly from shared-disk storage and also eliminates the server bottlenecks to performance and availability. SAN also improve the network latency and bandwidth through new channel interface like FC(Fibre Channel). But to manipulate the efficient storage network like SAN, traditional local file system and distributed file system are not adaptable and also are lack of researches in terms of a metadata structure for large-scale inode object such as file and directory. In this paper, we describe the architecture and design issues of our shared-disk file system and provide the efficient bitmap for providing the well-formed block allocation in each host, extent-based semi flat structure for storing large-scale file data, and two-phase directory structure of using Extendible Hashing. Also we describe a detailed algorithm for implementing the file system's device driver in Linux Kernel and compare our file system with the general file system like EXT2 and shard disk file system like GFS in terms of file creation, directory creation and I/O rate.

Design and Implementation of an Expert Search System Using Academic Data in Big Data Processing Platforms (빅데이터 처리 플랫폼에서 학술 데이터를 사용한 전문가 검색 시스템 설계 및 구현)

  • Choi, Dojin;Kim, Minsoo;Kim, Daeyun;Lee, Seohee;Han, Jinsu;Seo, Indeok;Lim, Jongtae;Bok, Kyoungsoo;Yoo, Jaesoo
    • The Journal of the Korea Contents Association
    • /
    • v.17 no.3
    • /
    • pp.100-114
    • /
    • 2017
  • Most of the researchers establish research directions to conduct the study of new fields by getting advice from experts or through the papers of experts. The existing academic data search services provide paper information by field but do not provide experts by field. Therefore, users should decide experts by field using the searched papers by themselves. In this paper, we design and implement an expert search system by discipline through big data processing based on papers that have been published in the academic societies. The proposed system utilizes distributed big data storage systems to store and manage large papers. We also discriminate experts and analyze data related to the experts by using distributed big data processing technologies. The processed results are provided through web pages when a user searches for experts. The user can get a lot of helps for the research of a particular field since the proposed system recommends the experts of the corresponding research field.

Optimal Sizing Method of Distributed Energy Resources for a Stand-alone Microgrid by using Reliability-based Genetic Algorithm (신뢰도 기반의 유전자알고리즘을 활용한 독립형 마이크로그리드 내 분산형전원 최적용량 산정 방법)

  • Baek, Ja-Hyun;Han, Soo-Kyung;Kim, Dae-Sik;Han, Dong-Hwa;Lee, Hansang;Cho, Soo-Hwan
    • The Transactions of The Korean Institute of Electrical Engineers
    • /
    • v.66 no.5
    • /
    • pp.757-764
    • /
    • 2017
  • As the reduction of greenhouse gases(GHGs) emission has become a global issue, the microgrid markets are growing rapidly. With the sudden changes in the market, Korean government suggested a new business model called 'Self-Sufficient Energy Islands'. Its main concern is a stand-alone microgrid composed of Distributed Energy Resources(DERs) such as Renewable Energy Sources(RESs), Energy Storage System(ESS) and Fuel Cell, in order to minimize the emission of GHGs. According to these trend, this paper is written to propose an optimal sizing method of DERs in a stand-alone microgrid by using Genetic Algorithm(GA), one of the representative stochastic methods. It is to minimize the net present cost with the variables, size of RESs and ESS. In the process for optimization, the sunless days are considered as additional constraints. Through the case study analysis, the size of DERs installed in a microgrid system has been computed using the proposed method in MATLAB. And the result of MATLAB is compared with that of HOMER(Hybrid Optimization of Multiple Energy Resources), a well-known energy modeling software.

Service Architecture Models For Fog Computing: A Remedy for Latency Issues in Data Access from Clouds

  • Khalid, Adnan;Shahbaz, Muhammad
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.11 no.5
    • /
    • pp.2310-2345
    • /
    • 2017
  • With the emergence of the Internet of Things (IoT) the world is projecting towards a scenario where every object in the world (including humans) acts as a sender and receiver of data and if we were to see that concept mature we would soon be talking of billions more users of the cloud networks. The cloud technology is a very apt alternative to permanent storage when it comes to bulk storage and reporting. It has however shown weaknesses concerning real-time data accessibility and processing. The bandwidth availability of the cloud networks is limited and combined with the highly centralized storage structure and geographical vastness of the network in terms of distance from the end user the cloud just does not seem like a friendly environment for real-time IOT data. This paper aims at highlighting the importance of Flavio Bonomi's idea of Fog Computing which has been glamorized and marketed by Cisco but has not yet been given a proper service architecture that would explain how it would be used in terms of various service models i-e IaaS, PaaS and SaaS, of the Cloud. The main contribution of the paper would be models for IaaS, PaaS and SaaS for Fog environments. The paper would conclude by highlighting the importance of the presented models and giving a consolidated overview of how they would work. It would also calculate the respective latencies for fog and cloud to prove that our models would work. We have used CloudSim and iFogSim to show the effectiveness of the paradigm shift from traditional cloud architecture to our Fog architecture.

A Study on Traffic Analysis and Hierarchical Program Allocation for Distributed VOD Systems (분산 VOD 시스템의 트래픽 분석과 계층적 프로그램 저장에 관한 연구)

  • Lee, Tae-Hoon;Kim, Yong-Deak
    • The Transactions of the Korea Information Processing Society
    • /
    • v.4 no.8
    • /
    • pp.2080-2091
    • /
    • 1997
  • It is generally recognized that Video On Demand (VOD) service will become a promising interactive service in the emerging broadband integrated services digital networks. A centralized VOD system, all programs are stored in a single VOD server which is linked to each user via exchanges, is applicable when a small number of users enjoys the VOD service. However, in case of large service penetration, it is very important to solve the problems of bandwidth and load concentrating in the central video server(CVS) and program transmission network. In this paper, the architecture of the video distribution service network is studied, then a traffic characteristics and models for VOD system are established, and proposed program allocation method to video servers. For this purpose, we present an analysis of program storage amount in each LVS(Local Video Server), transmission traffic volume between LVSs, and link traffic volume between CVS and LVSs, according to changing the related factors such as demand, the number of LVS, vision probability, etc. A method for finding out storage capacity in LVSs is also presented on the basis of the tradeoffs among program storage cost, link traffic cost, and transmission cost.

  • PDF