• Title/Summary/Keyword: Massive Data Processing

Search Result 231, Processing Time 0.029 seconds

GLOVE: Distributed Shared Memory Based Parallel Visualization Tool for Massive Scientific Dataset (GLOVE: 대용량 과학 데이터를 위한 분산공유메모리 기반 병렬 가시화 도구)

  • Lee, Joong-Youn;Kim, Min Ah;Lee, Sehoon;Hur, Young Ju
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.5 no.6
    • /
    • pp.273-282
    • /
    • 2016
  • Visualization tool can be divided by three components - data I/O, visual transformation and interactive rendering. In this paper, we present requirements of three major components on visualization tools for massive scientific dataset and propose strategies to develop the tool which satisfies those requirements. In particular, we present how to utilize open source softwares to efficiently realize our goal. Furthermore, we also study the way to combine several open source softwares which are separately made to produce a single visualization software and optimize it for realtime visualization of massiv espatio-temporal scientific dataset. Finally, we propose a distributed shared memory based scientific visualization tool which is called "GLOVE". We present a performance comparison among GLOVE and well known open source visualization tools such as ParaView and VisIt.

Design of the Node Decision Scheme for Processing Queries on Sensor Network Environments (센서 네트워크 환경에서 질의 처리를 위한 노드 선정 기법의 설계)

  • Kim, Dong Hyun
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.16 no.10
    • /
    • pp.2224-2229
    • /
    • 2012
  • Since sensor data are inserted into a data set continuously, continuous queries should be evaluated for searching data. To processing the continuous queries, it is required to build a query index on each sensor node and to transmit result data appropriate for query predicates. However, if query predicates are transferred to all sensor nodes, massive messages are required. In this paper, we propose the node decision scheme using the sensor node decision tree in order to diminish messages. The entry of a leaf node in the node decision tree represents a sensor node and defines the data region of the sensor node. When a user query is issued, sensor nodes are decided by intersecting between data regions of the tree with the query predicates of the user query, and then the query predicates are transmitted to the selected sensor nodes. We also implement the proposed sensor node decision tree and evaluate the experiments for the tree.

Performance Evaluation and Verification of MMX-type Instructions on an Embedded Parallel Processor (임베디드 병렬 프로세서 상에서 MMX타입 명령어의 성능평가 및 검증)

  • Jung, Yong-Bum;Kim, Yong-Min;Kim, Cheol-Hong;Kim, Jong-Myon
    • Journal of the Korea Society of Computer and Information
    • /
    • v.16 no.10
    • /
    • pp.11-21
    • /
    • 2011
  • This paper introduces an SIMD(Single Instruction Multiple Data) based parallel processor that efficiently processes massive data inherent in multimedia. In addition, this paper implements MMX(MultiMedia eXtension)-type instructions on the data parallel processor and evaluates and analyzes the performance of the MMX-type instructions. The reference data parallel processor consists of 16 processors each of which has a 32-bit datapath. Experimental results for a JPEG compression application with a 1280x1024 pixel image indicate that MMX-type instructions achieves a 50% performance improvement over the baseline instructions on the same data parallel architecture. In addition, MMX-type instructions achieves 100% and 51% improvements over the baseline instructions in energy efficiency and area efficiency, respectively. These results demonstrate that multimedia specific instructions including MMX-type have potentials for widely used many-core GPU(Graphics Processing Unit) and any types of parallel processors.

Cloud-Based Automation System to Process Data from Astronomy Observation (대용량 천문 관측 자료처리를 위한 클라우드 기반 자동화 시스템)

  • Yeom, Jae-Keun;Yu, Jung-Lok;Yim, Hong-Suh;Kim, Myung-Jin;Park, Jintae;Lee, Hee-Jae;Moon, Hong-Kyu;Choi, Young-Jun;Roh, Dong-Goo;Oh, Young-Seok;Bae, Young-Ho
    • KIISE Transactions on Computing Practices
    • /
    • v.23 no.1
    • /
    • pp.45-56
    • /
    • 2017
  • In astronomy, the amount of data generated using wide-field optical telescope has increased exponentially. However, the fixed-size small-scale computing environment and the complexity of data analysis tools, results in difficulties to process the massive observation data collected. To resolve this problem, we propose a cloud-based automation system for the efficient processing of the enormous data gathered. The proposed system consists of a Workflow Execution Manager which manages various workflow templates and controls the execution of workflows instantiated from theses templates, and an Elastic Resource Manager that dynamically adds/deletes computing resources, according to the amount of data analysis requests. To show the effectiveness of our proposed system, we exhaustively explored a board spectrum of experiments, like elastic resources allocation, system load, etc. Finally, we describe the best practice case of DEEP-SOUTH scheduling system as an example application.

Design of Indexing Agent for Semantic-based Video Retrieval (의미기반 비디오 검색을 위한 인덱싱 에이전트의 설계)

  • Lee, Jong-Hee;Oh, Hae-Seok
    • The KIPS Transactions:PartB
    • /
    • v.10B no.6
    • /
    • pp.687-694
    • /
    • 2003
  • According to the rapid increase of multimedia data quantity recently, various means of video data search has been desired. In order to process video data effectively, it is required that the content information of video data is loaded in database and semantic-based retrieval method can be available for various query of users. Currently existent contents-based video retrieval systems search by single method such as annotation-based or feature-based retrieval, and show low search efficiency and requires many efforts of system administrator or annotator form less perfect automatic processing. In this paper, we propose semantic-based video retrieval system which support semantic retrieval of various users by feature-based retrieval and annotation-based retrieval of massive video data. By user's fundamental query and selection of image for key frame that extracted from query, the agent gives the detail shape for annotation of extracted key frame. Also, key frame selected by user become query image and searches the most similar key frame through feature based retrieval method that propose. Therefore, we design the system that can heighten retrieval efficiency of video data through semantic-based retrieval.

A Partial Encryption Method for the Efficiency and the Security Enhancement of Massive Data Transmission in the Cloud Environment (클라우드 환경에서의 대용량 데이터 전송의 효율성과 보안성 강화를 위한 부분 암호화 방법)

  • Jo, Sung-Hwan;Han, Gi-Tae
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.6 no.9
    • /
    • pp.397-406
    • /
    • 2017
  • In case of using the existing encrypted algorithm for massive data encryption service under the cloud environment, the problem that requires much time in data encryption come to the fore. To make up for this weakness, a partial encryption method is used generally. However, the existing partial encryption method has a disadvantage that the encrypted data can be inferred due to the remaining area that is not encrypted. This study proposes a partial encryption method of increasing the encryption speed and complying with the security standard in order to solve this demerit. The proposed method consists of 3 processes such as header formation, partial encryption and block shuffle. In step 1 Header formation process, header data necessary for the algorithm are generated. In step 2 Partial encryption process, a part of data is encrypted, using LEA (Lightweight Encryption Algorithm), and all data are transformed with XOR of data in the unencrypted part and the block generated in the encryption process. In step 3 Block shuffle process, the blocks are mixed, using the shuffle data stored with the random arrangement form in the header to carry out encryption by transforming the data into an unrecognizable form. As a result of the implementation of the proposed method, applying it to a mobile device, all the encrypted data were transformed into an unrecognizable form, so the data could not be inferred, and the data could not be restored without the encryption key. It was confirmed that the proposed method could make prompt treatment possible in encrypting mass data since the encryption speed is improved by approximately 273% or so compared to LEA which is Lightweight Encryption Algorithm.

A Study on the Power System Restoration Simulator (전력계통 고장복구 교육 시스템에 관한 연구)

  • Lee H.J.;Park S.M.;Lee K.S.;Lee J.G.;Min S.W.;Han C.K.;Park J.K.;Moon Y.H.
    • The Transactions of the Korean Institute of Electrical Engineers A
    • /
    • v.54 no.7
    • /
    • pp.323-327
    • /
    • 2005
  • This paper presents an operator training simulator for power system restoration against massive black-out. The system is designed especially focused on the generality and convenient setting up for initial condition of simulation. The former is accomplished by using power flow calculation methodology, and PSS/E data is used to define the initial situation. The proposed simulator consists of three major components - the power flow(PF) module, data conversion(COW) module and GU subsystem. PF module calculates power flow, and then checks overvoltage of buses and overflow of lines. COW module composes an Y-Bus array and a data base at each restoration action. The initial Y-Bus array is constructed from PSS/E data. The user friendly GUI subsystem is developed including graphic editor and built-in operation manual. As a result, the maximum processing time for one step operation is 15 seconds, which is adequate for training purpose. Comparison with PSS/E simulation proves the accuracy and reliability of the training system.

A study on the power system restoration simulator (전력계통 고장복군 교육 시스템에 관한 연구)

  • Lee, H.J.;Kim, J.M.;Lee, K.S.;Park, S.M.;Song, I.J.;Lee, N.H.;Bae, J.C.;Hwang, B.H.
    • Proceedings of the KIEE Conference
    • /
    • 2003.07a
    • /
    • pp.181-183
    • /
    • 2003
  • This paper presents an operator training simulator for power system restoration against massive black-out. The system is designed especially focused on the generality and convenient setting up for initial condition of simulation. The former is accomplished by using on line calculation methodology, and PSS/E data is used to define the initial situation. The proposed simulator consists of three major components - the Power flow(PF) module, data conversion(CONV) module and CUI subsystem. PF module calculates power flow, and then checks overvoltage of buses and overflow of lines. CONV module composes an Y-Bus array and a data base at each restoration action. The initial Y-Bus array is constructed from PSS/E data. The user friendly GUI module is developed including graphic editor and built-in operation manual. As a result, the maximum processing time for one step operation is 15 seconds, which is adequate for training purpose. Comparison with PSS/E simulation proves the accuracy and reliability of the training system.

  • PDF

Real Time Distributed Parallel Processing to Visualize Noise Map with Big Sensor Data and GIS Data for Smart Cities (스마트시티의 빅 센서 데이터와 빅 GIS 데이터를 융합하여 실시간 온라인 소음지도로 시각화하기 위한 분산병렬처리 방법론)

  • Park, Jong-Won;Sim, Ye-Chan;Jung, Hae-Sun;Lee, Yong-Woo
    • Journal of Internet Computing and Services
    • /
    • v.19 no.4
    • /
    • pp.1-6
    • /
    • 2018
  • In smart cities, data from various kinds of sensors are collected and processed to provide smart services to the citizens. Noise information services with noise maps using the collected sensor data from various kinds of ubiquitous sensor networks is one of them. This paper presents a research result which generates three dimensional (3D) noise maps in real-time for smart cities. To make a noise map, we have to converge many informal data which include big image data of geographical Information and massive sensor data. Making such a 3D noise map in real-time requires the processing of the stream data from the ubiquitous sensor networks in real-time and the convergence operation in real-time. They are very challenging works. We developed our own methodology for real-time distributed and parallel processing for it and present it in this paper. Further, we developed our own real-time 3D noise map generation system, with the methodology. The system uses open source softwares for it. Here in this paper, we do introduce one of our systems which uses Apache Storm. We did performance evaluation using the developed system. Cloud computing was used for the performance evaluation experiments. It was confirmed that our system was working properly with good performance and the system can produce the 3D noise maps in real-time. The performance evaluation results are given in this paper, as well.

Multi-Target Tracking System based on Neural Network Data Association Algorithm (신경회로망 데이터 연관 알고리즘에 근거한 다중표적 추적 시스템)

  • 이진호;류충상;김은수
    • Journal of the Korean Institute of Telematics and Electronics A
    • /
    • v.29A no.11
    • /
    • pp.70-77
    • /
    • 1992
  • Generally, the conventional tracking algorithms are very limited in the practical applications because of that the computation load is exponentially increased as the number of targets being tracked is increase. Recently, to overcome this kind of limitation, some new tracking methods based on neural network algorithms which have learning and parallel processing capabilities are introduced. By application of neural networks to multi-target tracking problems, the tracking system can be made computationally independent of the number of objects being tracked, through their characteristics of massive parallelism and dense interconnectivity. In this paper, a new neural network tracking algorithm, which has capability of adaptive target tracking with little increase of the amount of calculation under the clutter and noisy environments, is suggested and the possibility of real-time multi-target tracking system based on neural networks is also demonstrated through some good computer simulation results.

  • PDF