• Title/Summary/Keyword: Parallel data processing

Search Result 751, Processing Time 0.133 seconds

Data Sampling-based Angular Space Partitioning for Parallel Skyline Query Processing (데이터 샘플링을 통한 각 기반 공간 분할 병렬 스카이라인 질의처리 기법)

  • Chung, Jaehwa
    • The Journal of Korean Association of Computer Education
    • /
    • v.18 no.5
    • /
    • pp.63-70
    • /
    • 2015
  • In the environment that the complex conditions need to be satisfied, skyline query have been applied to various field. To processing a skyline query in centralized scheme, several techniques have been suggested and recently map/reduce platform based approaches has been proposed which divides data space into multiple partitions for the vast volume of multidimensional data. However, the performances of these approaches are fluctuated due to the uneven data loading between servers and redundant tasks. Motivated by these issues, this paper suggests a novel technique called MR-DEAP which solves the uneven data loading using the random sampling. The experimental result gains the proposed MR-DEAP outperforms MR-Angular and MR-BNL scheme.

A Hadoop-based Multimedia Transcoding System for Processing Social Media in the PaaS Platform of SMCCSE

  • Kim, Myoungjin;Han, Seungho;Cui, Yun;Lee, Hanku;Jeong, Changsung
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.6 no.11
    • /
    • pp.2827-2848
    • /
    • 2012
  • Previously, we described a social media cloud computing service environment (SMCCSE). This SMCCSE supports the development of social networking services (SNSs) that include audio, image, and video formats. A social media cloud computing PaaS platform, a core component in a SMCCSE, processes large amounts of social media in a parallel and distributed manner for supporting a reliable SNS. Here, we propose a Hadoop-based multimedia system for image and video transcoding processing, necessary functions of our PaaS platform. Our system consists of two modules, including an image transcoding module and a video transcoding module. We also design and implement the system by using a MapReduce framework running on a Hadoop Distributed File System (HDFS) and the media processing libraries Xuggler and JAI. In this way, our system exponentially reduces the encoding time for transcoding large amounts of image and video files into specific formats depending on user-requested options (such as resolution, bit rate, and frame rate). In order to evaluate system performance, we measure the total image and video transcoding time for image and video data sets, respectively, under various experimental conditions. In addition, we compare the video transcoding performance of our cloud-based approach with that of the traditional frame-level parallel processing-based approach. Based on experiments performed on a 28-node cluster, the proposed Hadoop-based multimedia transcoding system delivers excellent speed and quality.

The Algorithm Design and Implemention for Operation using a Matrix Table in the WAVE system (WAVE 시스템에서 행렬 테이블로 연산하기 위한 알고리즘 설계 및 구현)

  • Lee, Dae-Sik;You, Young-Mo;Lee, Sang-Youn;Jang, Chung-Ryong
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.37 no.4A
    • /
    • pp.189-196
    • /
    • 2012
  • A WAVE(Wireless Access for Vehicular Environment) system is a vehicle communication technology. The system provides the services to prevent vehicle accidents that might occur during driving. Also, it is used to provide various services such as monitoring vehicle management and system failure. However, the scrambler bit operation of WAVE system becomes less efficient in the organizations of software and hardware design because the parallel processing is impossible. Although scrambler algorithm proposed in this paper has different processing speed depending on input data 8 bit, 16 bit, 32 bit, and 64 bit. it improves the processing speed of the operation because it can make parallel processing possible depending on the input unit.

Analysis on the GPU Performance according to Hierarchical Memory Organization (계층적 메모리 구성에 따른 GPU 성능 분석)

  • Choi, Hongjun;Kim, Jongmyon;Kim, Cheolhong
    • The Journal of the Korea Contents Association
    • /
    • v.14 no.3
    • /
    • pp.22-32
    • /
    • 2014
  • Recently, GPGPU has been widely used for general-purpose processing as well as graphics processing by providing optimized hardware for parallel processing. Memory system has big effects on the performance of parallel processing units such as GPU. In the GPU, hierarchical memory architecture is implemented for high memory bandwidth. Moreover, both memory address coalescing and memory request merging techniques are widely used. This paper analyzes the GPU performance according to various memory organizations. According to our simulation results, GPU performance improves by 15.5%, 21.5%, 25.5%, 30.9% as adding 8KB L1, 16KB L1, 32KB L1, 64KB L1 cache, respectively, compared to case without L1 cache. However, experimental results show that some benchmarks decrease performance since memory transaction increases due to data dependency. Moreover, average memory access latency is increased as the depth of hierarchical cache level increases when cache miss occurs significantly.

A Noise Robust Speech Recognition Method Using Model Compensation Based on Speech Enhancement (음성 개선 기반의 모델 보상 기법을 이용한 강인한 잡음 음성 인식)

  • Shen, Guang-Hu;Jung, Ho-Youl;Chung, Hyun-Yeol
    • The Journal of the Acoustical Society of Korea
    • /
    • v.27 no.4
    • /
    • pp.191-199
    • /
    • 2008
  • In this paper, we propose a MWF-PMC noise processing method which enhances the input speech by using Mel-warped Wiener Filtering (MWF) at pre-processing stage and compensates the recognition model by using PMC (Parallel Model Combination) at post-processing stage for speech recognition in noisy environments. The PMC uses the residual noise extracted from the silence region of enhanced speech at pre-processing stage to compensate the clean speech model and thus this method is considered to improve the performance of speech recognition in noisy environments. For recognition experiments we dew.-sampled KLE PBW (Phoneme Balanced Words) 452 word speech data to 8kHz and made 5 different SNR levels of noisy speech, i.e., 0dB. 5dB, 10dB, 15dB and 20dB, by adding Subway, Car and Exhibition noise to clean speech. From the recognition results, we could confirm the effectiveness of the proposed MWF-PMC method by obtaining the improved recognition performances over all compared with the existing combined methods.

The Application and Integration of an Improvement Technique for Layers of NETCONF (NETCONF 계층에 대한 개선 기법 적용 및 통합)

  • Lee, YangMin;Lee, JaeKee
    • Journal of KIISE
    • /
    • v.43 no.2
    • /
    • pp.256-268
    • /
    • 2016
  • Modern networks consisting of various heterogeneous equipment are often installed in a distributed manner. Thus the NETCONF standard was established to manage networks centrally and efficiently. In this paper, we present a method that integrates each NETCONF layer into a single system based on the results of previous studies. In the RPC Layer, an asynchronous communication channel and parallel processes are possible using multi-threading. In the Operation Layer, operational efficiency is increased by using a data group with dependencies between the equipment configuration data and by improving the data structure, enabling efficiently processing of XML queries even with multiple managers. The data modeling techniques and grouping methods in the Content Layer are presented in detail for interoperability between the Operation Layer and the Content Layer. Finally, the GUI program was implemented and its implementation is reported. We performed an experiment comparing the improved NETCONF with the standard NETCONF to measure factors, such as query processing ratio, query processing speed, and CPU utilization. The improved NETCONF demonstrated excellent query processing ratio and query processing speed, whereas the standard NETCONF had excellent CPU utilization.

A Study on implementation model for security log analysis system using Big Data platform (빅데이터 플랫폼을 이용한 보안로그 분석 시스템 구현 모델 연구)

  • Han, Ki-Hyoung;Jeong, Hyung-Jong;Lee, Doog-Sik;Chae, Myung-Hui;Yoon, Cheol-Hee;Noh, Kyoo-Sung
    • Journal of Digital Convergence
    • /
    • v.12 no.8
    • /
    • pp.351-359
    • /
    • 2014
  • The log data generated by security equipment have been synthetically analyzed on the ESM(Enterprise Security Management) base so far, but due to its limitations of the capacity and processing performance, it is not suited for big data processing. Therefore the another way of technology on the big data platform is necessary. Big Data platform can achieve a large amount of data collection, storage, processing, retrieval, analysis, and visualization by using Hadoop Ecosystem. Currently ESM technology has developed in the way of SIEM (Security Information & Event Management) technology, and to implement security technology in SIEM way, Big Data platform technology is essential that can handle large log data which occurs in the current security devices. In this paper, we have a big data platform Hadoop Ecosystem technology for analyzing the security log for sure how to implement the system model is studied.

Performance Analysis of Distributed Parallel Processing Schemes for Large Data in Cloud Computing (클라우드 컴퓨팅에서의 대규모 데이터를 위한 분산 병렬 처리 기법의 성능분석)

  • Hong, Seung-Tae;Chang, Jae-Woo
    • Proceedings of the Korean Association of Geographic Inforamtion Studies Conference
    • /
    • 2010.09a
    • /
    • pp.111-118
    • /
    • 2010
  • 최근 IT 분야에서 인터넷을 기반으로 IT 자원들을 서비스 형태로 제공하는 클라우드 컴퓨팅에 대한 연구가 활발히 진행되고 있다. 한편, 효율적인 클라우드 컴퓨팅을 제공하기 위해서는, 막대한 양의 데이터를 수많은 서버들에 분산 처장하고 관리하기 위한 분산 데이터 처장 기법 빛 분산 병렬 처리 기법에 대한 연구가 필수적이다. 이를 위해 본 논문에서는 대표적인 분산 병렬 처리 기법에 대해 살펴보고, 이를 비교 분석한다. 마지막으로 Hadoop 기반 클러스터를 구축하고 이를 통해서 대규모 데이터를 위한 분산 병렬 처리 기법에 대한 성능평가를 수행한다.

  • PDF

The automatic tire classfying vision system for real time processing (실시간 처리를 위한 타이어 자동 선별 비젼 시스템)

  • 박귀태;김진헌;정순원;송승철
    • 제어로봇시스템학회:학술대회논문집
    • /
    • 1992.10a
    • /
    • pp.358-363
    • /
    • 1992
  • The tire manufacturing process demands classification of tire types when the tires are transferred between the inner processes. Though most processes are being well automated, the classification relies greatly upon the visual inspection of humen. This has been an obstacle to the factory automation of tire manufacturing companies. This paper proposes an effective vision systems which can be usefully applied to the tire classification process in real time. The system adopts a parallel architecture using multiple transputers and contains the algorithms of preprocesssing for character recognition. The system can be easily expandable to manipulate the large data that can be processed seperately.

  • PDF

3D numerical simulation of temperature on Pilot tube

  • Ying Wang;Baogeng Ding
    • 한국전산유체공학회:학술대회논문집
    • /
    • 2006.05a
    • /
    • pp.248-251
    • /
    • 2006
  • Multi-physics problem is considered for the Pitot tube located in uniform freon gas flow with high Mach number and the 3D numerical results of temperature on Pitot tube is given. The model is created by using structural module of ANSYS, the grids are obtained by ICEM, and the problem is solved and the data post-processing is done by CFX.

  • PDF