• Title/Summary/Keyword: stream data processing

Search Result 447, Processing Time 0.022 seconds

A Human Movement Stream Processing System for Estimating Worker Locations in Shipyards

  • Duong, Dat Van Anh;Yoon, Seokhoon
    • International Journal of Internet, Broadcasting and Communication
    • /
    • v.13 no.4
    • /
    • pp.135-142
    • /
    • 2021
  • Estimating the locations of workers in a shipyard is beneficial for a variety of applications such as selecting potential forwarders for transferring data in IoT services and quickly rescuing workers in the event of industrial disasters or accidents. In this work, we propose a human movement stream processing system for estimating worker locations in shipyards based on Apache Spark and TensorFlow serving. First, we use Apache Spark to process location data streams. Then, we design a worker location prediction model to estimate the locations of workers. TensorFlow serving manages and executes the worker location prediction model. When there are requirements from clients, Apache Spark extracts input data from the processed data for the prediction model and then sends it to TensorFlow serving for estimating workers' locations. The worker movement data is needed to evaluate the proposed system but there are no available worker movement traces in shipyards. Therefore, we also develop a mobility model for generating the workers' movements in shipyards. Based on synthetic data, the proposed system is evaluated. It obtains a high performance and could be used for a variety of tasksin shipyards.

GeoSensor Data Stream Processing System for u-GIS Computing (u-GIS 컴퓨팅을 위한 GeoSensor 데이터 스트림 처리 시스템)

  • Chung, Weon-Il;Shin, Soong-Sun;Back, Sung-Ha;Lee, Yeon;Lee, Dong-Wook;Kim, Kyung-Bae;Lee, Chung-Ho;Kim, Ju-Wan;Bae, Hae-Young
    • Journal of Korea Spatial Information System Society
    • /
    • v.11 no.1
    • /
    • pp.9-16
    • /
    • 2009
  • In ubiquitous spatial computing environments, GeoSensor generates sensor data streams including spatial information as well as various conventional sensor data from RFID, WSN, Web CAM, Digital Camera, CCTV, and Telematics units. This GeoSensor enables the revitalization of various ubiquitous USN technologies and services on geographic information. In order to service the u-GIS applications based on GeoSensors, it is indispensable to efficiently process sensor data streams from GeoSensors of a wide area. In this paper, we propose a GeoSensor data stream processing system for u-GIS computing over real-time stream data from GeoSensors with geographic information. The proposed system provides efficient gathering, storing, and continuous query processing of GeoSensor data stream, and also makes it possible to develop diverse u-GIS applications meet each user requirements effectively.

  • PDF

Analysis of Output Stream Characteristics Processing in Digital Hardware Random Number Generator (디지털 하드웨어 난수 발생기에서 출력열 특성 처리 분석)

  • Hong, Jin-Keun
    • Journal of the Korea Academia-Industrial cooperation Society
    • /
    • v.13 no.3
    • /
    • pp.1147-1152
    • /
    • 2012
  • In this paper, it is key issue about analysis of characteristics processing of digital random output stream of hardware random number generator, which is applied in medical area. The output stream of random number generator based on hardware binary random number is effected from factors such as delay, jitter, temperature, and so on. In this paper, it presents about major factor, which effects hardware output random number stream, and the randomness of output stream data, which are combined output stream and postprocessing data such as encryption algorithm, encoding algorithm, is analyzed. the analyzed results are evaluated by major test items of randomness.

Design and Implementation of Advanced Traffic Monitoring System based on Integration of Data Stream Management System and Spatial DBMS

  • Xia, Ying;Gan, Hongmei;Kim, Gyoung-Bae
    • Journal of Korea Spatial Information System Society
    • /
    • v.11 no.2
    • /
    • pp.162-169
    • /
    • 2009
  • The real-time traffic data is generated continuous and unbounded stream data type while intelligent transport system (ITS) needs to provide various and high quality services by combining with spatial information. Traditional database techniques in ITS has shortage for processing dynamic real-time stream data and static spatial data simultaneously. In this paper, we design and implement an advanced traffic monitoring system (ATMS) with the integration of existed data stream management system (DSMS) and spatial DBMS using IntraMap. Besides, the developed ATMS can deal with the stream data of DSMS, the trajectory data of relational DBMS, and the spatial data of SDBMS concurrently. The implemented ATMS supports historical and one time query, continuous query and combined query. Application programmer can develop various intelligent services such as moving trajectory tracking, k-nearest neighbor (KNN) query and dynamic intelligent navigation by using components of the ATMS.

  • PDF

Techniques of XML Fragment Stream Organization for Efficient XML Query Processing in Mobile Clients (이동 클라이언트에서 효율적인 XML 질의 처리를 위한 XML 조각 스트림 구성 기법)

  • Ryu, Jeong-Hoon;Kang, Hyun-Chul
    • The Journal of Society for e-Business Studies
    • /
    • v.14 no.4
    • /
    • pp.75-94
    • /
    • 2009
  • Since XML emerged as a standard for data exchange on the web, it has been established as a core component in e-Commerce and efficient query processing over XML data in ubiquitous computing environment has been also receiving much attention. Recently, the techniques were proposed whereby an XML document is fragmented into XML fragments to be streamed and the mobile clients receive the stream while processing queries over it. In processing queries over an XML fragment stream, the average access time significantly depends on the order of fragments in the stream. As such, for query performance, an efficient organization of XML fragment stream is required as well as the indexing for energy-efficient query processing due to the reduction of tuning time. In this paper, a technique of XML fragment stream organization based on query frequencies, fragment size, fragment access frequencies, and an active XML-based indexing scheme are proposed. Through implementation and performance experiments, our techniques were shown to be efficient compared with the conventional XML fragment stream organizations.

  • PDF

In-Memory Based Incremental Processing Method for Stream Query Processing in Big Data Environments (빅데이터 환경에서 스트림 질의 처리를 위한 인메모리 기반 점진적 처리 기법)

  • Bok, Kyoungsoo;Yook, Misun;Noh, Yeonwoo;Han, Jieun;Kim, Yeonwoo;Lim, Jongtae;Yoo, Jaesoo
    • The Journal of the Korea Contents Association
    • /
    • v.16 no.2
    • /
    • pp.163-173
    • /
    • 2016
  • Recently, massive amounts of stream data have been studied for distributed processing. In this paper, we propose an incremental stream data processing method based on in-memory in big data environments. The proposed method stores input data in a temporary queue and compare them with data in a master node. If the data is in the master node, the proposed method reuses the previous processing results located in the node chosen by the master node. If there are no previous results of data in the node, the proposed method processes the data and stores the result in a separate node. We also propose a job scheduling technique considering the load and performance of a node. In order to show the superiority of the proposed method, we compare it with the existing method in terms of query processing time. Our experimental results show that our method outperforms the existing method in terms of query processing time.

Real-Time IoT Big-data Processing for Stream Reasoning (스트림-리즈닝을 위한 실시간 사물인터넷 빅-데이터 처리)

  • Yun, Chang Ho;Park, Jong Won;Jung, Hae Sun;Lee, Yong Woo
    • Journal of Internet Computing and Services
    • /
    • v.18 no.3
    • /
    • pp.1-9
    • /
    • 2017
  • Smart Cities intelligently manage numerous infrastructures, including Smart-City IoT devices, and provide a variety of smart-city applications to citizen. In order to provide various information needed for smart-city applications, Smart Cities require a function to intelligently process large-scale streamed big data that are constantly generated from a large number of IoT devices. To provide smart services in Smart-City, the Smart-City Consortium uses stream reasoning. Our stream reasoning requires real-time processing of big data. However, there are limitations associated with real-time processing of large-scale streamed big data in Smart Cities. In this paper, we introduce one of our researches on cloud computing based real-time distributed-parallel-processing to be used in stream-reasoning of IoT big data in Smart Cities. The Smart-City Consortium introduced its previously developed smart-city middleware. In the research for this paper, we made cloud computing based real-time distributed-parallel-processing available in the cloud computing platform of the smart-city middleware developed in the previous research, so that we can perform real-time distributed-parallel-processing with them. This paper introduces a real-time distributed-parallel-processing method and system for stream reasoning with IoT big data transmitted from various sensors of Smart Cities and evaluate the performance of real-time distributed-parallel-processing of the system where the method is implemented.

Stream Data Processing based on Sliding Window at u-Health System (u-Health 시스템에서 슬라이딩 윈도우 기반 스트림 데이터 처리)

  • Kim, Tae-Yeun;Song, Byoung-Ho;Bae, Sang-Hyun
    • The Journal of Korea Institute of Information, Electronics, and Communication Technology
    • /
    • v.4 no.2
    • /
    • pp.103-110
    • /
    • 2011
  • It is necessary to accurate and efficient management for measured digital data from sensors in u-health system. It is not efficient that sensor network process input stream data of mass storage stored in database the same time. We propose to improve the processing performance of multidimensional stream data continuous incoming from multiple sensor. We propose process query based on sliding window for efficient input stream and found multiple query plan to Mjoin method and we reduce stored data using backpropagation algorithm. As a result, we obtained to efficient result about 18.3% reduction rate of database using 14,324 data sets.

Stream-based Biomedical Classification Algorithms for Analyzing Biosignals

  • Fong, Simon;Hang, Yang;Mohammed, Sabah;Fiaidhi, Jinan
    • Journal of Information Processing Systems
    • /
    • v.7 no.4
    • /
    • pp.717-732
    • /
    • 2011
  • Classification in biomedical applications is an important task that predicts or classifies an outcome based on a given set of input variables such as diagnostic tests or the symptoms of a patient. Traditionally the classification algorithms would have to digest a stationary set of historical data in order to train up a decision-tree model and the learned model could then be used for testing new samples. However, a new breed of classification called stream-based classification can handle continuous data streams, which are ever evolving, unbound, and unstructured, for instance--biosignal live feeds. These emerging algorithms can potentially be used for real-time classification over biosignal data streams like EEG and ECG, etc. This paper presents a pioneer effort that studies the feasibility of classification algorithms for analyzing biosignals in the forms of infinite data streams. First, a performance comparison is made between traditional and stream-based classification. The results show that accuracy declines intermittently for traditional classification due to the requirement of model re-learning as new data arrives. Second, we show by a simulation that biosignal data streams can be processed with a satisfactory level of performance in terms of accuracy, memory requirement, and speed, by using a collection of stream-mining algorithms called Optimized Very Fast Decision Trees. The algorithms can effectively serve as a corner-stone technology for real-time classification in future biomedical applications.

A Review of Window Query Processing for Data Streams

  • Kim, Hyeon Gyu;Kim, Myoung Ho
    • Journal of Computing Science and Engineering
    • /
    • v.7 no.4
    • /
    • pp.220-230
    • /
    • 2013
  • In recent years, progress in hardware technology has resulted in the possibility of monitoring many events in real time. The volume of incoming data may be so large, that monitoring all individual data might be intractable. Revisiting any particular record can also be impossible in this environment. Therefore, many database schemes, such as aggregation, join, frequent pattern mining, and indexing, become more challenging in this context. This paper surveys the previous efforts to resolve these issues in processing data streams. The emphasis is on specifying and processing sliding window queries, which are supported in many stream processing engines. We also review the related work on stream query processing, including synopsis structures, plan sharing, operator scheduling, load shedding, and disorder control.