• Title/Summary/Keyword: Parallel data processing

Search Result 751, Processing Time 0.032 seconds

Processing large-scale data with Apache Spark (Apache Spark를 활용한 대용량 데이터의 처리)

  • Ko, Seyoon;Won, Joong-Ho
    • The Korean Journal of Applied Statistics
    • /
    • v.29 no.6
    • /
    • pp.1077-1094
    • /
    • 2016
  • Apache Spark is a fast and general-purpose cluster computing package. It provides a new abstraction named resilient distributed dataset, which is capable of support for fault tolerance while keeping data in memory. This type of abstraction results in a significant speedup compared to legacy large-scale data framework, MapReduce. In particular, Spark framework is suitable for iterative machine learning applications such as logistic regression and K-means clustering, and interactive data querying. Spark also supports high level libraries for various applications such as machine learning, streaming data processing, database querying and graph data mining thanks to its versatility. In this work, we introduce the concept and programming model of Spark as well as show some implementations of simple statistical computing applications. We also review the machine learning package MLlib, and the R language interface SparkR.

Development of Network Based MT Data Processing System (네트워크에 기반한 MT자료의 처리기술 개발 연구)

  • Lee Heuisoon;Kwon Byung-Doo;Chung Hojoon;Oh Seokhoon
    • Geophysics and Geophysical Exploration
    • /
    • v.3 no.2
    • /
    • pp.53-60
    • /
    • 2000
  • The server/client systems using the web protocol and distribution computing environment by network was applied to the MT data processing based on the Java technology. Using this network based system, users can get consistent and stable results because the system has standard analysing methods and has been tested from many users through the internet. Users can check the MT data processing at any time and get results during exploration to reduce the exploration time and money. The pure/enterprised Java technology provides facilities to develop the network based MT data processing system. Web based socket communication and RMI technology are tested respectively to produce the effective and practical client application. Intrinsically, the interpretation of MT data performing the inversion and data process requires heavy computational ability. Therefore we adopt the MPI parallel processing technique to fit the desire of in situ users and expect the effectiveness for the control and upgrade of programing codes.

  • PDF

3D feature profile simulation for nanoscale semiconductor plasma processing

  • Im, Yeon Ho
    • Proceedings of the Korean Vacuum Society Conference
    • /
    • 2015.08a
    • /
    • pp.61.1-61.1
    • /
    • 2015
  • Nanoscale semiconductor plasma processing has become one of the most challenging issues due to the limits of physicochemical fabrication routes with its inherent complexity. The mission of future and emerging plasma processing for development of next generation semiconductor processing is to achieve the ideal nanostructures without abnormal profiles and damages, such as 3D NAND cell array with ultra-high aspect ratio, cylinder capacitors, shallow trench isolation, and 3D logic devices. In spite of significant contributions of research frontiers, these processes are still unveiled due to their inherent complexity of physicochemical behaviors, and gaps in academic research prevent their predictable simulation. To overcome these issues, a Korean plasma consortium began in 2009 with the principal aim to develop a realistic and ultrafast 3D topography simulator of semiconductor plasma processing coupled with zero-D bulk plasma models. In this work, aspects of this computational tool are introduced. The simulator was composed of a multiple 3D level-set based moving algorithm, zero-D bulk plasma module including pulsed plasma processing, a 3D ballistic transport module, and a surface reaction module. The main rate coefficients in bulk and surface reaction models were extracted by molecular simulations or fitting experimental data from several diagnostic tools in an inductively coupled fluorocarbon plasma system. Furthermore, it is well known that realistic ballistic transport is a simulation bottleneck due to the brute-force computation required. In this work, effective parallel computing using graphics processing units was applied to improve the computational performance drastically, so that computer-aided design of these processes is possible due to drastically reduced computational time. Finally, it is demonstrated that 3D feature profile simulations coupled with bulk plasma models can lead to better understanding of abnormal behaviors, such as necking, bowing, etch stops and twisting during high aspect ratio contact hole etch.

  • PDF

FPGA based HW/SW co-design for vision based real-time position measurement of an UAV

  • Kim, Young Sik;Kim, Jeong Ho;Han, Dong In;Lee, Mi Hyun;Park, Ji Hoon;Lee, Dae Woo
    • International Journal of Aeronautical and Space Sciences
    • /
    • v.17 no.2
    • /
    • pp.232-239
    • /
    • 2016
  • Recently, in order to increase the efficiency and mission success rate of UAVs (Unmanned Aerial Vehicles), the necessity for formation flights is increased. In general, GPS (Global Positioning System) is used to obtain the relative position of leader with respect to follower in formation flight. However, it can't be utilized in environment where GPS jamming may occur or communication is impossible. Therefore, in this study, monocular vision is used for measuring relative position. General PC-based vision processing systems has larger size than embedded systems and is hard to install on small vehicles. Thus FPGA-based processing board is used to make our system small and compact. The processing system is divided into two blocks, PL(Programmable Logic) and PS(Processing system). PL is consisted of many parallel logic arrays and it can handle large amount of data fast, and it is designed in hardware-wise. PS is consisted of conventional processing unit like ARM processor in hardware-wise and sequential processing algorithm is installed on it. Consequentially HW/SW co-designed FPGA system is used for processing input images and measuring a relative 3D position of the leader, and this system showed RMSE accuracy of 0.42 cm ~ 0.51 cm.

Numerical analysis of two and three dimensional buoyancy driven water-exit of a circular cylinder

  • Moshari, Shahab;Nikseresht, Amir Hossein;Mehryar, Reza
    • International Journal of Naval Architecture and Ocean Engineering
    • /
    • v.6 no.2
    • /
    • pp.219-235
    • /
    • 2014
  • With the development of the technology of underwater moving bodies, the need for developing the knowledge of surface effect interaction of free surface and underwater moving bodies is increased. Hence, the two-phase flow is a subject which is interesting for many researchers all around the world. In this paper, the non-linear free surface deformations which occur during the water-exit of a circular cylinder due to its buoyancy are solved using finite volume discretization based code, and using Volume of Fluid (VOF) scheme for solving two phase flow. Dynamic mesh model is used to simulate dynamic motion of the cylinder. In addition, the effect of cylinder mass in presence of an external force is studied. Moreover, the oblique exit and entry of a circular cylinder with two exit angles is simulated. At last, water-exit of a circular cylinder in six degrees of freedom is simulated in 3D using parallel processing. The simulation errors of present work (using VOF method) for maximum velocity and height of a circular cylinder are less than the corresponding errors of level set method reported by previous researchers. Oblique exit shows interesting results; formation of waves caused by exit of the cylinder, wave motion in horizontal direction and the air trapped between the waves are observable. In 3D simulation the visualization of water motion on the top surface of the cylinder and the free surface breaking on the front and back faces of the 3D cylinder at the exit phase are observed which cannot be seen in 2D simulation. Comparing the results, 3D simulation shows better agreement with experimental data, specially in the maximum height position of the cylinder.

A Hardware Architecture for Retaining the Connectivity in Gray-Scale Image (그레이 레벨 연결성 복원 하드웨어 구조)

  • 김성훈;양영일
    • Journal of the Institute of Convergence Signal Processing
    • /
    • v.3 no.4
    • /
    • pp.23-28
    • /
    • 2002
  • In this paper, we have proposed the hardware architecture which implements the algorithm for retaining the connectivity which prevents the disconnection in the gray-scale image thinning. To extract the skeleton from the image in a real time, it is necessary to examine the connectivity of the skeleton in a real time. The proposed architecture finds the connectivity number in the 4-clock period. The architecture consists of three blocks, PS(Parallel to Serial) Converter and Stare Generator and Ridge Checker. The PS Converter changes the 3$\times$3 gray level image to four sets of image pixels. The State Generator examines the connectivity of the central pixel by searching the data from the PS Converter. The Ridge Checker determines whether the central pixel is on the skeleton or not. The proposed architecture finds the connectivity of the central pixel in a 3$\times$3 gray level image in the 4-clocks. The total circuits are verified by the design tools and operate correctly.

  • PDF

Real-time Hand Gesture Recognition System based on Vision for Intelligent Robot Control (지능로봇 제어를 위한 비전기반 실시간 수신호 인식 시스템)

  • Yang, Tae-Kyu;Seo, Yong-Ho
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.13 no.10
    • /
    • pp.2180-2188
    • /
    • 2009
  • This paper is study on real-time hand gesture recognition system based on vision for intelligent robot control. We are proposed a recognition system using PCA and BP algorithm. Recognition of hand gestures consists of two steps which are preprocessing step using PCA algorithm and classification step using BP algorithm. The PCA algorithm is a technique used to reduce multidimensional data sets to lower dimensions for effective analysis. In our simulation, the PCA is applied to calculate feature projection vectors for the image of a given hand. The BP algorithm is capable of doing parallel distributed processing and expedite processing since it take parallel structure. The BP algorithm recognized in real time hand gestures by self learning of trained eigen hand gesture. The proposed PCA and BP algorithm show improvement on the recognition compared to PCA algorithm.

Distribute Parallel Crawler Design and Implementation (분산형 병렬 크롤러 설계 및 구현)

  • Jang, Hyun Ho;jeon, kyung-sik;Lee, HooKi
    • Convergence Security Journal
    • /
    • v.19 no.3
    • /
    • pp.21-28
    • /
    • 2019
  • As the number of websites managed by organizations or organizations increases, so does the number of web application servers and containers. In checking the status of the web service of the web application server and the container, it is very difficult for the person to check the status of the web service after accessing the physical server at the remote site through the terminal or using other accessible software It. Previous research on crawler-related research is hard to find any reference to the processing of data from crawling. Data loss occurs when the crawler accesses the database and stores the data. In this paper, we propose a method to store the inspection data according to crawl - based web application server management without losing data.

Efficient CUDA Implementation of Multiple Planes Fitting Using RANSAC (RANSAC을 이용한 다중 평면 피팅의 효율적인 CUDA 구현)

  • Cho, Tai-Hoon
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.23 no.4
    • /
    • pp.388-393
    • /
    • 2019
  • As a fiiting method to data with outliers, RANSAC(RANdom SAmple Consensus) based algorithm is widely used in fitting of line, circle, ellipse, etc. CUDA is currently most widely used GPU with massive parallel processing capability. This paper proposes an efficient CUDA implementation of multiple planes fitting using RANSAC with 3d points data, of which one set of 3d points is used for one plane fitting. The performance of the proposed algorithm is demonstrated compared with CPU implementation using both artificially generated data and real 3d heights data of a PCB. The speed-up of the algorithm over CPU seems to be higher in data with lower inlier ratio, more planes to fit, and more points per plane fitting. This method can be easily applied to a wide variety of other fitting applications.

Smart City Governance Logic Model Converging Hub-and-spoke Data Management and Blockchain Technology (허브 앤 스포크형 데이터 관리 및 블록체인 기술 융합 스마트도시 거버넌스 로직모델)

  • Choi, Sung-Jin
    • Journal of KIBIM
    • /
    • v.14 no.1
    • /
    • pp.30-38
    • /
    • 2024
  • This study aims to propose a smart city governance logic model that can accommodate more diverse information service systems by mixing hub-and-spoke and blockchain technologies as a data management model. Specifically, the research focuses on deriving the logic of an operating system that can work across smart city planning based on the two data governance technologies. The first step of the logic is the generation and collection of information, which is first divided into information that requires information protection and information that can be shared with the public, and the information that requires privacy is blockchainized, and the shared information is integrated and aggregated in a data hub. The next step is the processing and use of the information, which can actively use the blockchain technology, but for the information that can be shared other than the protected information, the governance logic is built in parallel with the hub-and-spoke type. Next is the logic of the distribution stage, where the key is to establish a service contact point between service providers and beneficiaries. Also, This study proposes the establishment of a one-to-one data exchange relationship between information providers, information consumers, and information processors. Finally, in order to expand and promote citizen participation opportunities through a reasonable compensation system in the operation of smart cities, we developed virtual currency as a local currency and designed an open operation logic of local virtual currency that can operate in the compensation dimension of information.