• Title/Summary/Keyword: Massive Data Processing

Search Result 231, Processing Time 0.025 seconds

EHT data processing and BH shadow imaging techniques

  • Cho, Ilje
    • The Bulletin of The Korean Astronomical Society
    • /
    • v.44 no.1
    • /
    • pp.59.2-59.2
    • /
    • 2019
  • Event Horizon Telescope (EHT) aims to resolve the innermost region to the super massive black hole (SMBH) with its extremely high angular resolution (~20-25 uas) and enhanced sensitivity (down to 1-10 mJy) in concert with the Atacama Large Millimeter/submillimeter Array (ALMA) at 1.3 mm wavelength. This has a great importance as the first observational probe of the black hole shadow which has been theoretically predicted as a ring-like emission affected by the general relativistic effect under a strong gravitational field of SMBH. During the 2017 April 5-11, four nights of EHT observing campaign were carried out towards its primary targets, M87 and $SgrA{\ast}$. To robustly ensure the data processing, independent pipelines for various radio data calibration softwares (e.g., AIPS, HOPS, CASA) have been developed and cross-compared each other. The EHT has also been developing newer interferometric imaging techniques (e.g., eht-imaging-library, SMILI, dynamical imaging), as well as using an established method (CLEAN). With these, the EHT has designed various strategies which will be adopted for convincing imaging results. In this talk, I review how the robustness of EHT data processing and imaging will be validated so that the results can be ensured against well known uncertainties or biases in the interferometric data calibration and imaging.

  • PDF

Fault Detection System Using Spatial Index Structure (공간자료구조를 활용한 단층인식 시스템)

  • Bang, Kap-San
    • Proceedings of the IEEK Conference
    • /
    • 2005.11a
    • /
    • pp.1205-1208
    • /
    • 2005
  • By adding user interface to the usual router, an improved functional router is implemented in this paper. Due to the massive amount of spatial data processing, spatial information processing area has been rapidly grown up in recent years based on powerful computer hardware and software development. Spatial index structures are the core engine of geographic information system(GIS). Analyzing and processing of spatial information using GIS has a lot of applications and the number application will be increased in the future. However, study on the under ground is in its infancy due to invisible characteristic of this information. This paper proposes the sub-surface fault detection system using the sub-surface layer information gathered from elastic wave. Detection of sub-surface fault provides very important information to the safety of above and sub-surface man made structures. Development of sub-surface fault detection system will serve as a pre-processing system assisting the interpretation of the geologist.

  • PDF

A Peak Load Control-Based Worker-Linker Pattern for Stably Processing Massive I/O Transactions (안정적인 대용량 I/O거래 처리를 위한 Peak Load Control(PLC) 기반의 Worker-Linker 패턴)

  • Lee, Yong-Hwan;Min, Dug-Ki
    • Journal of KIISE:Computing Practices and Letters
    • /
    • v.12 no.5
    • /
    • pp.312-325
    • /
    • 2006
  • Integration applications, such as EAI, B2Bi, need stable massive data processing systems during overload state cause by service request congestion in a short period time. In this paper, we propose the PLC (Peak Load Control)-based Worker-Linker pattern, which can effectively and stably process massive I/O transactions in spite of overload state generated by service request congestion. This pattern uses the delay time algorithm for the PLC mechanism. In this paper, we also show the example of applying the pattern to business-business integration framework and the experimental result for proving the stability of performance. According to our experiment result, the proposed delay time algorithm can stably control the heavy overload after the saturation point and has an effect on the controlling peak load.

Improvement of OPW-TR Algorithm for Compressing GPS Trajectory Data

  • Meng, Qingbin;Yu, Xiaoqiang;Yao, Chunlong;Li, Xu;Li, Peng;Zhao, Xin
    • Journal of Information Processing Systems
    • /
    • v.13 no.3
    • /
    • pp.533-545
    • /
    • 2017
  • Massive volumes of GPS trajectory data bring challenges to storage and processing. These issues can be addressed by compression algorithm which can reduce the size of the trajectory data. A key requirement for GPS trajectory compression algorithm is to reduce the size of the trajectory data while minimizing the loss of information. Synchronized Euclidean distance (SED) as an important error measure is adopted by most of the existing algorithms. In order to further reduce the SED error, an improved algorithm for open window time ratio (OPW-TR) called local optimum open window time ratio (LO-OPW-TR) is proposed. In order to make SED error smaller, the anchor points are selected by calculating point's accumulated synchronized Euclidean distance (ASED). A variety of error metrics are used for the algorithm evaluation. The experimental results show that the errors of our algorithm are smaller than the existing algorithms in terms of SED and speed errors under the same compression ratio.

Twostep Clustering of Environmental Indicator Survey Data

  • Park, Hee-Chang
    • Journal of the Korean Data and Information Science Society
    • /
    • v.17 no.1
    • /
    • pp.1-11
    • /
    • 2006
  • Data mining technique is used to find hidden knowledge by massive data, unexpectedly pattern, relation to new rule. The methods of data mining are decision tree, association rules, clustering, neural network and so on. Clustering is the process of grouping the data into clusters so that objects within a cluster have high similarity in comparison to one another. It has been widely used in many applications, such that pattern analysis or recognition, data analysis, image processing, market research on off-line or on-line and so on. We analyze Gyeongnam social indicator survey data by 2001 using twostep clustering technique for environment information. The twostep clustering is classified as a partitional clustering method. We can apply these twostep clustering outputs to environmental preservation and improvement.

  • PDF

Twostep Clustering of Environmental Indicator Survey Data

  • Park, Hee-Chang
    • 한국데이터정보과학회:학술대회논문집
    • /
    • 2005.10a
    • /
    • pp.59-69
    • /
    • 2005
  • Data mining technique is used to find hidden knowledge by massive data, unexpectedly pattern, relation to new rule. The methods of data mining are decision tree, association rules, clustering, neural network and so on. Clustering is the process of grouping the data into clusters so that objects within a cluster have high similarity in comparison to one another. It has been widely used in many applications, such that pattern analysis or recognition, data analysis, image processing, market research on off-line or on-line and so on. We analyze Gyeongnam social indicator survey data by 2001 using twostep clustering technique for environment information. The twostep clustering is classified as a partitional clustering method. We can apply these twostep clustering outputs to environmental preservation and improvement.

  • PDF

The Method of Analyzing Firewall Log Data using MapReduce based on NoSQL (NoSQL기반의 MapReduce를 이용한 방화벽 로그 분석 기법)

  • Choi, Bomin;Kong, Jong-Hwan;Hong, Sung-Sam;Han, Myung-Mook
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.23 no.4
    • /
    • pp.667-677
    • /
    • 2013
  • As the firewall is a typical network security equipment, it is usually installed at most of internal/external networks and makes many packet data in/out. So analyzing the its logs stored in it can provide important and fundamental data on the network security research. However, along with development of communications technology, the speed of internet network is improved and then the amount of log data is becoming 'Massive Data' or 'BigData'. In this trend, there are limits to analyze log data using the traditional database model RDBMS. In this paper, through our Method of Analyzing Firewall log data using MapReduce based on NoSQL, we have discovered that the introducing NoSQL data base model can more effectively analyze the massive log data than the traditional one. We have demonstrated execellent performance of the NoSQL by comparing the performance of data processing with existing RDBMS. Also the proposed method is evaluated by experiments that detect the three attack patterns and shown that it is highly effective.

Securing the Information using Improved Modular Encryption Standard in Cloud Computing Environment

  • A. Syed Ismail;D. Pradeep;J. Ashok
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.17 no.10
    • /
    • pp.2822-2843
    • /
    • 2023
  • All aspects of human life have become increasingly dependent on data in the last few decades. The development of several applications causes an enormous issue on data volume in current years. This information must be safeguarded and kept in safe locations. Massive volumes of data have been safely stored with cloud computing. This technology is developing rapidly because of its immense potentials. As a result, protecting data and the procedures to be handled from attackers has become a top priority in order to maintain its integrity, confidentiality, protection, and privacy. Therefore, it is important to implement the appropriate security measures in order to prevent security breaches and vulnerabilities. An improved version of Modular Encryption Standard (IMES) based on layered modelling of safety mechanisms is the major focus of this paper's research work. Key generation in IMES is done using a logistic map, which estimates the values of the input data. The performance analysis demonstrates that proposed work performs better than commonly used algorithms against cloud security in terms of higher performance and additional qualitative security features. The results prove that the proposed IMES has 0.015s of processing time, where existing models have 0.017s to 0.022s of processing time for a file size of 256KB.

A Survey on Massive Data Processing Model in Cloud Computing (클라우드 컴퓨팅에서의 대용량 데이터 처리 모델에 관한 조사)

  • Jin, Ah-Yeon;Park, Young-Ho
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2011.11a
    • /
    • pp.145-146
    • /
    • 2011
  • 클라우드 컴퓨팅은 세계적인 시장조사기관인 가트너사의 10대전략기술에서 2년 연속 1위를 할 정도로 많은 각광을 받고 있다. 클라우드 컴퓨팅이란 인터넷 기술을 활용하여 가상화된 컴퓨팅 자원을 서비스로 제공하는 것으로, 사용자는 IT자원을 필요한 만큼 빌려서 사용하고 사용한 만큼 비용을 지불하는 컴퓨팅을 지칭한다. 이러한 클라우드 컴퓨팅 상에서 폭발적으로 증가하고 있는 데이터를 효율적으로 병렬 처리할 수 있는 방법에 대하여 많은 연구가 활발히 이루어지고 있다. 이러한 대용량 데이터 처리를 위한 대표적인 모델에는 MapReduce와 Dryad가 있으며, 서로간에 많은 공통점이 있지만 MapReduce는 범용 프로그래밍 언어를 기반으로 쉬운 병렬 프로그래밍을 가능하게 했다는 점에서 많이 사용되고 있으며 Dryad는 재사용이 쉽고 데이터 처리 흐름을 유연하게 작성할 수 있다는 점에서 장점을 가지고 있다.

Design of a SIMT architecture GP-GPU Using Tile based on Graphic Pipeline Structure (타일 기반 그래픽 파이프라인 구조를 사용한 SIMT 구조 GP-GPU 설계)

  • Kim, Do-Hyun;Kim, Chi-Yong
    • Journal of IKEEE
    • /
    • v.20 no.1
    • /
    • pp.75-81
    • /
    • 2016
  • This paper proposes a design of the tile based on graphic pipeline to improve the graphic application performance in SIMT based GP-GPU. The proposed Tile based on graphics pipeline avoids unnecessary graphic processing operation, and processes the rasterization step in parallel. The massive data processing in parallel through SIMT architecture improve the computational performance, thereby improving the 3D graphic pipeline performance. The more vertex data of 3D model, the higher performance. The proposed structure was confirmed to improve processing performance of up to 3 times from about 1.18 times as compared to 'RAMP' and previous studies.