• Title/Summary/Keyword: Stream Processing System

Search Result 293, Processing Time 0.028 seconds

A Query Preprocessing Tool for Performance Improvement in Complex Event Stream Query Processing (복합 이벤트 스트림 질의 처리 성능 개선을 위한 질의 전처리 도구)

  • Choi, Joong-Hyun;Cho, Eun-Sun;Lee, Kang-Woo
    • KIISE Transactions on Computing Practices
    • /
    • v.21 no.8
    • /
    • pp.513-523
    • /
    • 2015
  • A complex event processing system, becoming useful in real life domains, efficiently processes stream of continuous events like sensor data from IoT systems. However, those systems do not work well on some types of queries yet, so that programmers should be careful about that. For instance, they do not sufficiently provide detailed guide to choose efficient queries among the almost same meaning queries. In this paper, we propose an query preprocessing tool for event stream processing systems, which helps programmers by giving them the hints to improve performance whenever their queries fall in any possible bad formats in the performance sense. We expect that our proposed module would be a big help to increases productivity of writing programs where debugging, testing, and performance tuning are not straightforward.

The XP-table: Runtime-efficient Region-based Structure for Collective Evaluation of Multiple Continuous XPath Queries (The XP-table: 다중 연속 XPath 질의의 집단 처리를 위한 실행시간 효율적인 영역 기반 구조체)

  • Lee, Hyun-Ho;Lee, Won-Suk
    • Journal of KIISE:Databases
    • /
    • v.35 no.4
    • /
    • pp.307-318
    • /
    • 2008
  • One of the primary issues confronting XML message brokers is the difficulty associated with processing a large set of continuous XPath queries over incoming XML seams. This paper proposes a novel system designed to present an effective solution to this problem. The proposed system transforms multiple XPath queries before their run-time into a new region-based data structure, called an XP-table, by sharing their common constraints. An XP-table is matched with a stream relation (SR) transformed from a target XML stream by a SAX parser. This arrangement is intended to minimize the runtime workload of continuous query processing. Also, system performance is estimated and verified through a variety of experiments, including comparisons with previous approaches such as YFilter and LazyDFA. The proposed system is practically linear- scalable and stable for evaluating a set of XPath queries in a continuous and timely fashion.

Fast Visualization Technique and Visual Analytics System for Real-time Analyzing Stream Data (실시간 스트림 데이터 분석을 위한 시각화 가속 기술 및 시각적 분석 시스템)

  • Jeong, Seongmin;Yeon, Hanbyul;Jeong, Daekyo;Yoo, Sangbong;Kim, Seokyeon;Jang, Yun
    • Journal of the Korea Computer Graphics Society
    • /
    • v.22 no.4
    • /
    • pp.21-30
    • /
    • 2016
  • Risk management system should be able to support a decision making within a short time to analyze stream data in real time. Many analytical systems consist of CPU computation and disk based database. However, it is more problematic when existing system analyzes stream data in real time. Stream data has various production periods from 1ms to 1 hour, 1day. One sensor generates small data but tens of thousands sensors generate huge amount of data. If hundreds of thousands sensors generate 1GB data per second, CPU based system cannot analyze the data in real time. For this reason, it requires fast processing speed and scalability for analyze stream data. In this paper, we present a fast visualization technique that consists of hybrid database and GPU computation. In order to evaluate our technique, we demonstrate a visual analytics system that analyzes pipeline leak using sensor and tweet data.

Rate-Controlled Data-Driven Real-Time Stream Processing for an Autonomous Machine (자율 기기를 위한 속도가 제어된 데이터 기반 실시간 스트림 프로세싱)

  • Noh, Soonhyun;Hong, Seongsoo;Kim, Myungsun
    • The Journal of Korea Robotics Society
    • /
    • v.14 no.4
    • /
    • pp.340-347
    • /
    • 2019
  • Due to advances in machine intelligence and increased demands for autonomous machines, the complexity of the underlying software platform is increasing at a rapid pace, overwhelming the developers with implementation details. We attempt to ease the burden that falls onto the developers by creating a graphical programming framework we named Splash. Splash is designed to provide an effective programming abstraction for autonomous machines that require stream processing. It also enables programmers to specify genuine, end-to-end timing constraints, which the Splash framework automatically monitors for violation. By utilizing the timing constraints, Splash provides three key language semantics: timing semantics, in-order delivery semantics, and rate-controlled data-driven stream processing semantics. These three semantics together collectively serve as a conceptual tool that can hide low-level details from programmers, allowing developers to focus on the main logic of their applications. In this paper, we introduce the three-language semantics in detail and explain their function in association with Splash's language constructs. Furthermore, we present the internal workings of the Splash programming framework and validate its effectiveness via a lane keeping assist system.

Attribute-based Approach for Multiple Continuous Queries over Data Streams (데이터 스트림 상에서 다중 연속 질의 처리를 위한 속성기반 접근 기법)

  • Lee, Hyun-Ho;Lee, Won-Suk
    • The KIPS Transactions:PartD
    • /
    • v.14D no.5
    • /
    • pp.459-470
    • /
    • 2007
  • A data stream is a massive unbounded sequence of data elements continuously generated at a rapid rate. Query processing for such a data stream should also be continuous and rapid, which requires strict time and space constraints. In most DSMS(Data Stream Management System), the selection predicates of continuous queries are grouped or indexed to guarantee these constraints. This paper proposes a new scheme tailed an ASC(Attribute Selection Construct) that collectively evaluates selection predicates containing the same attribute in multiple continuous queries. An ASC contains valuable information, such as attribute usage status, partially pre calculated matching results and selectivity statistics for its multiple selection predicates. The processing order of those ASC's that are corresponding to the attributes of a base data stream can significantly influence the overall performance of multiple query evaluation. Consequently, a method of establishing an efficient evaluation order of multiple ASC's is also proposed. Finally, the performance of the proposed method is analyzed by a series of experiments to identify its various characteristics.

The Study of Video Transcoding and Streaming System Based on Prediction Period

  • Park, Seong-Ho;Kim, Sung-Min;Lee, Hwa-Sei
    • Journal of information and communication convergence engineering
    • /
    • v.5 no.4
    • /
    • pp.339-345
    • /
    • 2007
  • Video transcoding is a technique used to convert a compressed input video stream with an arbitrary format, size, and bitrate into a different attribute video stream different attributes to provide a efficient video streaming service for the customers is dispersed in the heterogeneous networks. Specifically, frames deletion occur in a transcoding scheme that exploits the adjustment of frame rate, and at this time, the loss in temporal relation among frames due to frame deletion is compensated for the prediction of motion estimation by reusing motion vectors in the would-be deleted frames. But the processing time for transcoding don't have an improvement as much as our expectation because transcoding is done only within the transcoder. So in this paper, we propose a new transcoding algorithm based on prediction period to improve transcoding-related processing time. For this, we also modify the existing encoder so as to adjust dynamically frame rate based on the prediction period and deletion period of frames. To check how the proposed algorithm works nicely, we implement a video streaming system with the new transcoder and encoder to which it is applied. The result of the performance test shows that the streaming system with proposed algorithm improve 60% above in processing time and also PSNR have a good performance while the quality of pictures is preserved.

Optimal Economic Load Dispatch using Parallel Genetic Algorithms in Large Scale Power Systems (병렬유전알고리즘을 응용한 대규모 전력계통의 최적 부하배분)

  • Kim, Tae-Kyun;Kim, Kyu-Ho;Yu, Seok-Ku
    • The Transactions of the Korean Institute of Electrical Engineers A
    • /
    • v.48 no.4
    • /
    • pp.388-394
    • /
    • 1999
  • This paper is concerned with an application of Parallel Genetic Algorithms(PGA) to optimal econmic load dispatch(ELD) in power systems. The ELD problem is to minimize the total generation fuel cost of power outputs for all generating units while satisfying load balancing constraints. Genetic Algorithms(GA) is a good candidate for effective parallelization because of their inherent principle of evolving in parallel a population of individuals. Each individual of a population evaluates the fitness function without data exchanges between individuals. In application of the parallel processing to GA, it is possible to use Single Instruction stream, Multiple Data stream(SIMD), a kind of parallel system. The architecture of SIMD system need not data communications between processors assigned. The proposed ELD problem with C code is implemented by SIMSCRIPT language for parallel processing which is a powerfrul, free-from and versatile computer simulation programming language. The proposed algorithms has been tested for 38 units system and has been compared with Sequential Quadratic programming(SQP).

  • PDF

Historical Sensor Data Management Using Temporal Information (센서 데이터의 시간 정보를 이용한 이력 정보 관리)

  • Lee, Yang-Koo;Ryu, Keun-Ho
    • Journal of Korea Spatial Information System Society
    • /
    • v.10 no.4
    • /
    • pp.97-102
    • /
    • 2008
  • A wireless sensor network consists of many sensors that collect and transmit physical or environmental conditions at different locations to a server continuously. Many researches mainly focus on processing continuous queries on real-time data stream. However, they do not concern the problem of storing the historical data, which is mandatory to the historical queries. In this paper, we propose two time-based storage methods to store the sensor data stream and reduce the managed tuples without any loss of information, which lead to the improvement of the accuracy of query results.

  • PDF

Dynamic Load Management Method for Spatial Data Stream Processing on MapReduce Online Frameworks (맵리듀스 온라인 프레임워크에서 공간 데이터 스트림 처리를 위한 동적 부하 관리 기법)

  • Jeong, Weonil
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.19 no.8
    • /
    • pp.535-544
    • /
    • 2018
  • As the spread of mobile devices equipped with various sensors and high-quality wireless network communications functionsexpands, the amount of spatio-temporal data generated from mobile devices in various service fields is rapidly increasing. In conventional research into processing a large amount of real-time spatio-temporal streams, it is very difficult to apply a Hadoop-based spatial big data system, designed to be a batch processing platform, to a real-time service for spatio-temporal data streams. This paper extends the MapReduce online framework to support real-time query processing for continuous-input, spatio-temporal data streams, and proposes a load management method to distribute overloads for efficient query processing. The proposed scheme shows a dynamic load balancing method for the nodes based on the inflow rate and the load factor of the input data based on the space partition. Experiments show that it is possible to support efficient query processing by distributing the spatial data stream in the corresponding area to the shared resources when load management in a specific area is required.

Causality join query processing for data stream by spatio-temporal sliding window (시공간 슬라이딩윈도우기법을 이용한 데이터스트림의 인과관계 결합질의처리방법)

  • Kwon, O-Je;Li, Ki-Joune
    • Spatial Information Research
    • /
    • v.16 no.2
    • /
    • pp.219-236
    • /
    • 2008
  • Data stream collected from sensors contain a large amount of useful information including causality relationships. The causality join query for data stream is to retrieve a set of pairs (cause, effect) from streams of data. A part of causality pairs may however be lost from the query result, due to the delay from sensors to a data stream management system, and the limited size of sliding windows. In this paper, we first investigate spatial, temporal, and spatio-temporal aspects of the causality join query for data stream. Second, we propose several strategies for sliding window management based on these observations. The accuracy of the proposed strategies is studied by intensive experiments, and the result shows that we improve the accuracy of causality join query in data stream from simple FIFO strategy.

  • PDF