• Title/Summary/Keyword: parallel data collection

Search Result 55, Processing Time 0.024 seconds

Implementation of high-speed parallel data transfer for MCG signal acquisition (심자도 신호 획득을 위한 고속 병렬 데이터 전송 구현)

  • Lee, Dong-Ha;Yoo, Jae-Tack
    • Proceedings of the Korean Institute of Electrical and Electronic Material Engineers Conference
    • /
    • 2004.11a
    • /
    • pp.444-447
    • /
    • 2004
  • A heart diagnosis system adopts hundreds of Superconducting Quantum Interface Device(SQUID) sensors for precision MCG(Magnetocardiogram) or MEG(Magnetoencephalogram) signal acquisitions. This system requires correct and real-time data acquisition from the sensors in a required sampling interval, i.e., 1 mili-second. This paper presents our hardware design and test results, to acquire data from 256 channel analog signal with 1-ksample/sec speed, using 12-bit 8-channel ADC devices, SPI interfaces, parallel interfaces, and 8-bit microprocessors. We chose to implement parallel data transfer between microprocessors as an effective way of achieving such data collection. Our result concludes that the data collection can be done in 250 ${\mu}sec$ time-interval.

  • PDF

Optimization of Data Acquisition System with Parallel Collection for PET

  • Yoshida, Eiji;Shimizu, Keiji;Murayama, Hideo
    • Proceedings of the Korean Society of Medical Physics Conference
    • /
    • 2002.09a
    • /
    • pp.311-313
    • /
    • 2002
  • We are under development of a 3D PET scanner with depth of interaction (DOI) capable of high sensitivity and high resolution. In this scanner, a maximum data transfer rate of coincidence pair's event information is 10 Mcps and one event is a 64-bit data format. This maximum data transfer rate corresponds by 10 times a conventional PET scanner. A data acquisition system, which fulfills the specification of this scanner, is considered for parallel collection with banks including several coincidence units. Data transfer rate is improved by optimizing parameters of a message size, and so on.

  • PDF

High-quality data collection for machine learning using block chain (블록체인을 활용한 양질의 기계학습용 데이터 수집 방안 연구)

  • Kim, Youngrang;Woo, Junghoon;Lee, Jaehwan;Shin, Ji Sun
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.23 no.1
    • /
    • pp.13-19
    • /
    • 2019
  • The accuracy of machine learning is greatly affected by amount of learning data and quality of data. Collecting existing Web-based learning data has danger that data unrelated to actual learning can be collected, and it is impossible to secure data transparency. In this paper, we propose a method for collecting data directly in parallel by blocks in a block - chain structure, and comparing the data collected by each block with data in other blocks to select only good data. In the proposed system, each block shares data with each other through a chain of blocks, utilizes the All-reduce structure of Parallel-SGD to select only good quality data through comparison with other block data to construct a learning data set. Also, in order to verify the performance of the proposed architecture, we verify that the original image is only good data among the modulated images using the existing benchmark data set.

Parallelization of Recursive Functions for Recursive Data Structures (재귀적 자료구조에 대한 재귀 함수의 병렬화)

  • An, Jun-Seon;Han, Tae-Suk
    • Journal of KIISE:Software and Applications
    • /
    • v.26 no.12
    • /
    • pp.1542-1552
    • /
    • 1999
  • 자료 병렬성이란 자료 집합의 원소들에 대하여 동일한 작업을 동시에 수행하므로써 얻어지는 병렬성을 말한다. 함수형 언어에서 자료 집합에 대한 반복 수행은 재귀적 자료형에 대한 재귀 함수에 의하여 표현된다. 본 논문에서는 이러한 재귀 함수를 자료 병렬 프로그램으로 변환하기 위한 병렬화 방법을 제시한다. 생성되는 병렬 프로그램의 병렬 수행 구조로는 일반적인 형태의 재귀적 자료형에 대하여 정의되는 다형적인 자료 병렬 연산을 사용하여 트리, 리스트 등과 같은 일반적인 재귀적 자료 집합에 대한 자료 병렬 수행이 가능하도록 하였다. 재귀 함수의 병렬화를 위해서는, 함수를 이루는 각각의 계산들의 병렬성을 재귀 호출에 의해 존재하는 의존성에 기반하여 분류하고, 이에 기반하여 각각의 계산들에 대한 적절한 자료 병렬 연산을 사용하는 병렬 프로그램을 생성하였다.Abstract Data parallelism is obtained by applying the same operations to each element of a data collection. In functional languages, iterative computations on data collections are expressed by recursions on recursive data structures. We propose a parallelization method for data-parallel implementation of such recursive functions. We employ polytypic data-parallel primitives to represent the parallel execution structure of the object programs, which enables data parallel execution with general recursive data structures, such as trees and lists. To transform sequential programs to their parallelized versions, we propose a method to classify the types of parallelism in subexpressions, based on the dependencies of the recursive calls, and generate the data-parallel programs using data-parallel primitives appropriately.

Experiment on Collection Characteristics of Sub micron Particles in Two-Stage Parallel-Plate Electrostatic Precipitators (2단 평행판 전기집진기의 서브마이크론 입자 포집특성 실험)

  • Oh, M.D.;Yoo, K.H.
    • Korean Journal of Air-Conditioning and Refrigeration Engineering
    • /
    • v.6 no.3
    • /
    • pp.237-246
    • /
    • 1994
  • Experimental data are reported for charging and collection of NaCl aerosols in the 0.03- to $0.2{\mu}m$-geometric-mean-diameter range in 2-stage parallel-plate electrostatic precipitators. The NaCl aerosols are generated with geometric standard deviation of about 1.74 and particle generation rate of about 10^9 particles/see by the constant output atomizer and injected into the air flow in the clean wind-tunnel. The 2-stage parallel-plate electrostatic precipitator installed in the test section of the wind-tunnel is operated with a positive corona discharge. The NaCl aerosols in the channel flow are sampled and transported to the aerosol particle number concentration measurement system by using the isoaxial sampling and transport system constructed based on the Okazaki and Willeke design. The aerosol particle number concentration measurement system measures the size distribution of submicrometer aerosols by an electrical mobility detection technique. It is confirmed from comparing the measured collection efficiencies in this study and the predicted ones by our previous theoretical analysis that the predicted collection efficiencies agree well with the experimental ones. It is also found from the comparison that below about $0.02{\mu}m$ all particles are not charged and the uncharged particles are not collected, and consequently 2-stage parallel-plate electrostatic precipitators are not suitable for that particle size range.

  • PDF

Deep Web and MapReduce

  • Tao, Yufei
    • Journal of Computing Science and Engineering
    • /
    • v.7 no.3
    • /
    • pp.147-158
    • /
    • 2013
  • This invited paper introduces results on Web science and technology obtained during work with the Korea Advanced Institute of Science and Technology. In the first part, we discuss algorithms for exploring the deep Web, which refers to the collection of Web pages that cannot be reached by conventional Web crawlers. In the second part, we discuss sorting algorithms on the MapReduce system, which has become a dominant paradigm for massive parallel computing.

Development of Big-data Management Platform Considering Docker Based Real Time Data Connecting and Processing Environments (도커 기반의 실시간 데이터 연계 및 처리 환경을 고려한 빅데이터 관리 플랫폼 개발)

  • Kim, Dong Gil;Park, Yong-Soon;Chung, Tae-Yun
    • IEMEK Journal of Embedded Systems and Applications
    • /
    • v.16 no.4
    • /
    • pp.153-161
    • /
    • 2021
  • Real-time access is required to handle continuous and unstructured data and should be flexible in management under dynamic state. Platform can be built to allow data collection, storage, and processing from local-server or multi-server. Although the former centralize method is easy to control, it creates an overload problem because it proceeds all the processing in one unit, and the latter distributed method performs parallel processing, so it is fast to respond and can easily scale system capacity, but the design is complex. This paper provides data collection and processing on one platform to derive significant insights from various data held by an enterprise or agency in the latter manner, which is intuitively available on dashboards and utilizes Spark to improve distributed processing performance. All service utilize dockers to distribute and management. The data used in this study was 100% collected from Kafka, showing that when the file size is 4.4 gigabytes, the data processing speed in spark cluster mode is 2 minute 15 seconds, about 3 minutes 19 seconds faster than the local mode.

A Study on the Data Collection Methods based Hadoop Distributed Environment (하둡 분산 환경 기반의 데이터 수집 기법 연구)

  • Jin, Go-Whan
    • Journal of the Korea Convergence Society
    • /
    • v.7 no.5
    • /
    • pp.1-6
    • /
    • 2016
  • Many studies have been carried out for the development of big data utilization and analysis technology recently. There is a tendency that government agencies and companies to introduce a Hadoop of a processing platform for analyzing big data is increasing gradually. Increased interest with respect to the processing and analysis of these big data collection technology of data has become a major issue in parallel to it. However, study of the collection technology as compared to the study of data analysis techniques, it is insignificant situation. Therefore, in this paper, to build on the Hadoop cluster is a big data analysis platform, through the Apache sqoop, stylized from relational databases, to collect the data. In addition, to provide a sensor through the Apache flume, a system to collect on the basis of the data file of the Web application, the non-structured data such as log files to stream. The collection of data through these convergence would be able to utilize as a basic material of big data analysis.

256-channel 1ks/s MCG Signal Acquisition System (256-channel 1 ksamples/sec 심자도 신호획득 시스템)

  • Lee, Dong-Ha;Yoo, Jae-Tack;Huh, Young
    • Proceedings of the KIEE Conference
    • /
    • 2004.11c
    • /
    • pp.538-540
    • /
    • 2004
  • Electrical currents generated by human heart activities create magnetic fields represented by MCG(MagnetoCardioGram). Since an MCG signal acquisition system requires precise and stable operation, the system adopts hundreds of SQUID(Superconducting QUantum Interface Device) sensors for signal acquisition. Such a system requires fast real-time data acquisition in a required sampling interval, i.e., 1 mili-second for each sensor. This paper presents designed hardware to acquire data from 256-channel analog signal with 1 ksamples/sec speed, using 12-bit 8-channel ADC devices, SPI interfaces, parallel interfaces, 8-bit microprocessors, and a DSP processor. We implemented SPI interface between ADCs and a microprocessor, parallel interfaces between microprocessors. Our result concludes that the data collection can be done in $168{\mu}sec$ time-interval for 256 SQUID sensors, which can be interpreted to 6 ksamples/sec speed.

  • PDF

Real-time 256-channel 12-bit 1ks/s Hardware for MCG Signal Acquisition (심자도 신호획득을 위한 실시간 256-채널 12-bit 1ks/s 하드웨어)

  • Yoo, Jae-Tack
    • The Transactions of the Korean Institute of Electrical Engineers D
    • /
    • v.54 no.11
    • /
    • pp.643-649
    • /
    • 2005
  • A heart diagnosis system adopts Superconducting Quantum Interface Device(SQUD) sensors for precise MCG(MagnetoCardioGram) signal acquisitions. Such system needs to deal with hundreds of sensors, requiring fast signal sampling md precise analog-to-digital conversions(ADC). Our development of hardware board, processing 64-channel 12-bit in 1 ks/s speed, is built by using 8-channel ADC chips, 8-bit microprocessors, SPI interfaces, and specially designed parallel data transfers between microprocessors to meet the 1ks/s, i.e. 1 mili-second sampling interval. We extend the design into 256-channel hardware and analyze the speed .using the measured data from the 64-channel hardware. Since our design exploits full parallel processing, Assembly level coding, and NOP(No Operation) instruction for timing control, the design provides expandability and lowest system timing margin. Our result concludes that the data collection with 256-channel analog input signals can be done in 201.5us time-interval which is much shorter than the required 1 mili-second period.