• Title/Summary/Keyword: Distributed memory

Search Result 397, Processing Time 0.027 seconds

A Comparative Analysis of Recursive Query Algorithm Implementations based on High Performance Distributed In-Memory Big Data Processing Platforms (대용량 데이터 처리를 위한 고속 분산 인메모리 플랫폼 기반 재귀적 질의 알고리즘들의 구현 및 비교분석)

  • Kang, Minseo;Kim, Jaesung;Lee, Jaegil
    • Journal of KIISE
    • /
    • v.43 no.6
    • /
    • pp.621-626
    • /
    • 2016
  • Recursive query algorithm is used in many social network services, e.g., reachability queries in social networks. Recently, the size of social network data has increased as social network services evolve. As a result, it is almost impossible to use the recursive query algorithm on a single machine. In this paper, we implement recursive query on two popular in-memory distributed platforms, Spark and Twister, to solve this problem. We evaluate the performance of two implementations using 50 machines on Amazon EC2, and real-world data sets: LiveJournal and ClueWeb. The result shows that recursive query algorithm shows better performance on Spark for the Livejournal input data set with relatively high average degree, but smaller vertices. However, recursive query on Twister is superior to Spark for the ClueWeb input data set with relatively low average degree, but many vertices.

Research for Efficient Massive File I/O on Parallel Programs (병렬 프로그램에서의 효율적인 대용량 파일 입출력 방식의 비교 연구)

  • Hwang, Gyuhyeon;Kim, Youngtae
    • Journal of Internet Computing and Services
    • /
    • v.18 no.2
    • /
    • pp.53-60
    • /
    • 2017
  • Since processors are handling inputs and outputs independently on distributed memory computers, different file input/output methods are used. In this paper, we implemented and compared various file I/O methods to show their efficiency on distributed memory parallel computers. The implemented I/O systems are as following: (i) parallel I/O using NFS, (ii) sequential I/O on the host processor and domain decomposition, (iii) MPI-IO. For performance analysis, we used a separated file server and multiple processors on one or two computational servers. The results show the file I/O with NFS for inputs and sequential output with domain composition for outputs are best efficient respectively. The MPI-IO result shows unexpectedly the lowest performance.

GLOVE: Distributed Shared Memory Based Parallel Visualization Tool for Massive Scientific Dataset (GLOVE: 대용량 과학 데이터를 위한 분산공유메모리 기반 병렬 가시화 도구)

  • Lee, Joong-Youn;Kim, Min Ah;Lee, Sehoon;Hur, Young Ju
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.5 no.6
    • /
    • pp.273-282
    • /
    • 2016
  • Visualization tool can be divided by three components - data I/O, visual transformation and interactive rendering. In this paper, we present requirements of three major components on visualization tools for massive scientific dataset and propose strategies to develop the tool which satisfies those requirements. In particular, we present how to utilize open source softwares to efficiently realize our goal. Furthermore, we also study the way to combine several open source softwares which are separately made to produce a single visualization software and optimize it for realtime visualization of massiv espatio-temporal scientific dataset. Finally, we propose a distributed shared memory based scientific visualization tool which is called "GLOVE". We present a performance comparison among GLOVE and well known open source visualization tools such as ParaView and VisIt.

Scalable RDFS Reasoning Using the Graph Structure of In-Memory based Parallel Computing (인메모리 기반 병렬 컴퓨팅 그래프 구조를 이용한 대용량 RDFS 추론)

  • Jeon, MyungJoong;So, ChiSeoung;Jagvaral, Batselem;Kim, KangPil;Kim, Jin;Hong, JinYoung;Park, YoungTack
    • Journal of KIISE
    • /
    • v.42 no.8
    • /
    • pp.998-1009
    • /
    • 2015
  • In recent years, there has been a growing interest in RDFS Inference to build a rich knowledge base. However, it is difficult to improve the inference performance with large data by using a single machine. Therefore, researchers are investigating the development of a RDFS inference engine for a distributed computing environment. However, the existing inference engines cannot process data in real-time, are difficult to implement, and are vulnerable to repetitive tasks. In order to overcome these problems, we propose a method to construct an in-memory distributed inference engine that uses a parallel graph structure. In general, the ontology based on a triple structure possesses a graph structure. Thus, it is intuitive to design a graph structure-based inference engine. Moreover, the RDFS inference rule can be implemented by utilizing the operator of the graph structure, and we can thus design the inference engine according to the graph structure, and not the structure of the data table. In this study, we evaluate the proposed inference engine by using the LUBM1000 and LUBM3000 data to test the speed of the inference. The results of our experiment indicate that the proposed in-memory distributed inference engine achieved a performance of about 10 times faster than an in-storage inference engine.

Intelligent Control by Immune Network Algorithm Based Auto-Weight Function Tuning

  • Kim, Dong-Hwa
    • 제어로봇시스템학회:학술대회논문집
    • /
    • 2002.10a
    • /
    • pp.120.2-120
    • /
    • 2002
  • In this paper auto-tuning scheme of weight function in the neural networks has been suggested by immune algorithm for nonlinear process. A number of structures of the neural networks are considered as learning methods for control system. A general view is provided that they are the special cases of either the membership functions or the modification of network structure in the neural networks. On the other hand, since the immune network system possesses a self organizing and distributed memory, it is thus adaptive to its external environment and allows a PDP (parallel distributed processing) network to complete patterns against the environmental situation. Also. It can provi..

  • PDF

Web-Based Organizational Memory Acquisition by Using a Fuzzy Cognitive Map (퍼지인식도를 이용한 웹기반 조직지식획득에 관한 연구)

  • 이건창
    • Journal of Intelligence and Information Systems
    • /
    • v.5 no.2
    • /
    • pp.79-97
    • /
    • 1999
  • Knowledge management (KM) is emerging as a robust management mechanism with which an organization can remain highly intelligent and competitive in a turbulent market. Organization knowledge is at the heart of KM success. As a vehicle of acquiring organizational knowledge in a distributed decision-making environment, we applied a fuzzy cognitive map (FMM) technique and proved its effectiveness in a distributed knowledge management environment. Our approach was applied to the financial statement analysis problem, yielding a robust result.

  • PDF

Distributed File Placement and Coverage Expansion Techniques for Network Throughput Enhancement in Small-cell Network (소형셀 네트워크 전송용량 향상을 위한 분산 파일저장 및 커버리지 확장 기법)

  • Hong, Jun-Pyo
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.22 no.1
    • /
    • pp.183-189
    • /
    • 2018
  • This paper proposes distributed file placement and coverage expansion techniques for mitigating the traffic bottleneck in backhaul for small-cell networks. In order to minimize the backhaul load with limited memory space, the proposed scheme controls the coverage and file placement of base station according to file popularity distribution and memory space of base stations. In other words, since the cache hit ratio is low when there is small memory capacity or widespread file popularity distribution, the base stations expand its coverage and cache different set of files for the user located in overlapped area to exploit multiple cached file sets of base stations. Our simulation results show that the proposed scheme outperforms the conventional cache strategy in terms of network throughput when there is small memory capacity or widespread file popularity distribution.

An Adaptive Prefetching Technique for Software Distributed Shared Memory Systems (소프트웨어 분산공유메모리시스템을 위한 적응적 선인출 기법)

  • Lee, Sang-Kwon;Yun, Hee-Chul;Lee, Joon-Won;Maeng, Seung-Ryoul
    • Journal of KIISE:Computer Systems and Theory
    • /
    • v.28 no.9
    • /
    • pp.461-468
    • /
    • 2001
  • Though shared virtual memory (SVM) system promise low cost solutions for high performance computing they suffer from long memory latencies. These latencies are usually caused by repetitive invalidations on shared data. Since shared data are accessed through synchronization and the patterns by which threads synchronizes are repetitive, a prefetching scheme bases on such repetitiveness would reduce memory latencies. Based on this observation, we propose a prefetching technique which predicts future access behavior by analyzing access history per synchronization variable. Our technique was evaluated on an 8-node SVM system using the SPLASH-2 benchmark. The results show the our technique could achieve 34%~45% reduction in memory access latencies.

  • PDF

Distributed Simulator for General Control System in CEMTool

  • Lee, Tai-Ri;Lee, Young-Sam;Lee, Kwan-Ho;Kwon, Wook-Hyun
    • 제어로봇시스템학회:학술대회논문집
    • /
    • 2003.10a
    • /
    • pp.2230-2234
    • /
    • 2003
  • This paper proposes a distributed simulator for general control system in CEMTool. Systems can be described by SIMTool likes the simulink in Matlab. For distributed simulation, we can seperate any system into several parallel subsystems in SIMTool. The number of parallel subsystem can be determined by the system's property. After seperation, parallel simulator will do initialization, one-step-ahead simulation, block-distribution and ordering and so on. Finally, simulator will create independent C codes and executive files for each subsystem. The whole system is fulfilled by several PCs, and each PC executes one subsystem. There are communications among these subsystem using reflective memory or ethernet. We have made several experiments, and the 5-stand cold rolling mill control system is our main target. The result of parallel simulation has shown effective speedup in comparison with one pc simulation.

  • PDF