• 제목/요약/키워드: Data stream mining

검색결과 97건 처리시간 0.024초

ADA: Advanced data analytics methods for abnormal frequent episodes in the baseline data of ISD

  • Biswajit Biswal;Andrew Duncan;Zaijing Sun
    • Nuclear Engineering and Technology
    • /
    • 제54권11호
    • /
    • pp.3996-4004
    • /
    • 2022
  • The data collected by the In-Situ Decommissioning (ISD) sensors are time-specific, age-specific, and developmental stage-specific. Research has been done on the stream data collected by ISD testbed in the recent few years to seek both frequent episodes and abnormal frequent episodes. Frequent episodes in the data stream have confirmed the daily cycle of the sensor responses and established sequences of different types of sensors, which was verified by the experimental setup of the ISD Sensor Network Test Bed. However, the discovery of abnormal frequent episodes remained a challenge because these abnormal frequent episodes are very small signals and may be buried in the background noise of voltage and current changes. In this work, we proposed Advanced Data Analytics (ADA) methods that are applied to the baseline data to identify frequent episodes and extended our approach by adding more features extracted from the baseline data to discover abnormal frequent episodes, which may lead to the early indicators of ISD system failures. In the study, we have evaluated our approach using the baseline data, and the performance evaluation results show that our approach is able to discover frequent episodes as well as abnormal frequent episodes conveniently.

Novel Push-Front Fibonacci Windows Model for Finding Emerging Patterns with Better Completeness and Accuracy

  • Akhriza, Tubagus Mohammad;Ma, Yinghua;Li, Jianhua
    • ETRI Journal
    • /
    • 제40권1호
    • /
    • pp.111-121
    • /
    • 2018
  • To find the emerging patterns (EPs) in streaming transaction data, the streaming is first divided into some time windows containing a number of transactions. Itemsets are generated from transactions in each window, and then the emergence of itemsets is evaluated between two windows. In the tilted-time windows model (TTWM), it is assumed that people need support data with finer accuracy from the most recent windows, while accepting coarser accuracy from older windows. Therefore, a limited array's elements are used to maintain all support data in a way that condenses old windows by merging them inside one element. The capacity of elements that accommodates the windows inside is modeled using a particular number sequence. However, in a stream, as new data arrives, the current array updating mechanisms lead to many null elements in the array and cause data incompleteness and inaccuracy problems. Two models derived from TTWM, logarithmic TTWM and Fibonacci windows model, also inherit the same problems. This article proposes a novel push-front Fibonacci windows model as a solution, and experiments are conducted to demonstrate its superiority in finding more EPs compared to other models.

데이터마이닝 기법을 활용한 상수 이용현황 분석 및 단기 물 수요예측 방법 비교 (The Comparison Among Prediction Methods of Water Demand And Analysis of Data on Water Services Using Data Mining Techniques)

  • 안지훈;김진화
    • 한국빅데이터학회지
    • /
    • 제1권1호
    • /
    • pp.9-17
    • /
    • 2016
  • 본 연구에서는 상수관망에 설치된 유량, 압력 센서를 통해 취득한 빅데이터에 대해 데이터마이닝 기법을 활용하여 해당 공급권역의 특성을 파악하고 그 정보에 기반하여 상수 공급에 있어서 유의할 점 등을 도출해보고자 하였다. 또한, 상수 사용에 대한 단기 수요예측을 수행하는데 있어서도 통계적 방법인 다중회귀분석과 데이터마이닝의 인공신경망 기법을 비교하여 좀 더 정확한 수요예측을 할 수 있는 모델을 제시해보고자 하였다. 데이터 수집과 테스트를 위하여 지자체 한 군의 소블록 지역을 대상으로 선정하였다. 해당 지역은 가정용 수요 외에도 관공서, 병원 등의 대형 업무용 수요도 일부 존재하고 있는 지역이다. 해당 지역의 센서를 통해 취득되는 연속 발생 데이터를 수집하였다. 이런 방식을 통해 취득된 데이터는 총 2,728건이며 이 중 2,632건은 예측모델을 생성하는데 96건은 예측모델의 예측력을 테스트 하는 데에 활용하였다. 이러한 테스트를 수행한 결과 상수 수요예측에 있어서 인공신경망이 다중회귀분석에 비교하여 더 좋은 예측율을 보였다.

  • PDF

Smart-Walk 시스템에서 스트림 빅데이터 분석을 통한 최적화 기법 (An Optimization Technique for Smart-Walk Systems Using Big Stream Log Data)

  • 조완섭;양경은;이중엽
    • 한국산업정보학회논문지
    • /
    • 제17권3호
    • /
    • pp.105-114
    • /
    • 2012
  • 본 논문에서는 장애인의 보행을 지원하는 Smart-Walk 시스템에서 하나의 시스템으로 여러 유형의 장애인을 지원할 수 있도록 하는 유니버설 디자인개념의 데이터베이스 구축방안을 제시한다. 또한, 운행로그를 분석하여 사용자의 사용현황과 이탈비율을 계산함으로써 시스템의 최적운영을 지원하는 방안을 제시한다. 다양한 사용자 유형과 그에 적합한 사용방법들을 데이터베이스에 저장하고 관리함으로써 간단하게 다양한 유형의 사용자들에게 지원할 수 있는 방법은 진정한 유니버설디자인 이념의 실현이라 할 수 있다. 사용자의 운행로그를 데이터웨어하우스 형태로 저장하고 온라인 분석 기법을 적용함으로써 시스템의 최적 운영에 유용한 정보를 실시간으로 추출해 낼 수 있게 된다.

그래프를 이용한 빈발 서비스 탐사 (Mining Frequent Service Patterns using Graph)

  • 황정희
    • 디지털콘텐츠학회 논문지
    • /
    • 제19권3호
    • /
    • pp.471-477
    • /
    • 2018
  • 시간의 변화에 따라 사용자의 관심도는 변화한다. 이 논문에서는 유비쿼터스 환경에서 연령, 시기, 계절 등에 따라 변화하는 사용자의 서비스 관심도를 고려하기 위하여 서비스에 대한 관심도를 동적 가중치로 부여하여 사용자에게 적합한 서비스를 추천하기 위한 방법을 제안한다. 사용자에게 제공한 서비스 이력 데이터를 기준으로 시기나 연령에 따른 일반적인 서비스 규칙을 저장하고, 실시간으로 변화하는 서비스의 관심도를 고려한 최신의 서비스 규칙을 지속적으로 추가하여 사용자의 관심 변화를 반영하는 서비스를 제공하기 위한 방법이다. 이를 위해 사용자에게 제공하는 일련의 서비스는 트랜잭션으로 고려하고 서비스는 항목으로 고려하여 서비스의 연관관계를 그래프로 표현하고, 이를 기반으로 빈발 서비스 항목을 발견한다. 발견된 빈발 서비스 항목은 사용자에게 유용한 최신의 정보 서비스를 의미한다.

EXTENDED ONLINE DIVISIVE AGGLOMERATIVE CLUSTERING

  • Musa, Ibrahim Musa Ishag;Lee, Dong-Gyu;Ryu, Keun-Ho
    • 대한원격탐사학회:학술대회논문집
    • /
    • 대한원격탐사학회 2008년도 International Symposium on Remote Sensing
    • /
    • pp.406-409
    • /
    • 2008
  • Clustering data streams has an importance over many applications like sensor networks. Existing hierarchical methods follow a semi fuzzy clustering that yields duplicate clusters. In order to solve the problems, we propose an extended online divisive agglomerative clustering on data streams. It builds a tree-like top-down hierarchy of clusters that evolves with data streams using geometric time frame for snapshots. It is an enhancement of the Online Divisive Agglomerative Clustering (ODAC) with a pruning strategy to avoid duplicate clusters. Our main features are providing update time and memory space which is independent of the number of examples on data streams. It can be utilized for clustering sensor data and network monitoring as well as web click streams.

  • PDF

The Big Data Analytics Regarding the Cadastral Resurvey News Articles

  • Joo, Yong-Jin;Kim, Duck-Ho
    • 한국측량학회지
    • /
    • 제32권6호
    • /
    • pp.651-659
    • /
    • 2014
  • With the popularization of big data environment, big data have been highlighted as a key information strategy to establish national spatial data infrastructure for a scientific land policy and the extension of the creative economy. Especially interesting from our point of view is the cadastral information is a core national information source that forms the basis of spatial information that leads to people's daily life including the production and consumption of information related to real estate. The purpose of our paper is to suggest the scheme of big data analytics with respect to the articles of cadastral resurvey project in order to approach cadastral information in terms of spatial data integration. As specific research method, the TM (Text Mining) package from R was used to read various formats of news reports as texts, and nouns were extracted by using the KoNLP package. That is, we searched the main keywords regarding cadastral resurvey, performing extraction of compound noun and data mining analysis. And visualization of the results was presented. In addition, new reports related to cadastral resurvey between 2012 and 2014 were searched in newspapers, and nouns were extracted from the searched data for the data mining analysis of cadastral information. Furthermore, the approval rating, reliability, and improvement of rules were presented through correlation analyses among the extracted compound nouns. As a result of the correlation analysis among the most frequently used ones of the extracted nouns, five groups of data consisting of 133 keywords were generated. The most frequently appeared words were "cadastral resurvey," "civil complaint," "dispute," "cadastral survey," "lawsuit," "settlement," "mediation," "discrepant land," and "parcel." In Conclusions, the cadastral resurvey performed in some local governments has been proceeding smoothly as positive results. On the other hands, disputes from owner of land have been provoking a stream of complaints from parcel surveying for the cadastral resurvey. Through such keyword analysis, various public opinion and the types of civil complaints related to the cadastral resurvey project can be identified to prevent them through pre-emptive responses for direct call centre on the cadastral surveying, Electronic civil service and customer counseling, and high quality services about cadastral information can be provided. This study, therefore, provides a stepping stones for developing an account of big data analytics which is able to comprehensively examine and visualize a variety of news report and opinions in cadastral resurvey project promotion. Henceforth, this will contribute to establish the foundation for a framework of the information utilization, enabling scientific decision making with speediness and correctness.

Model Development for Specific Degradation Using Data Mining and Geospatial Analysis of Erosion and Sedimentation Features

  • Kang, Woochul;Kang, Joongu;Jang, Eunkyung;Julien, Piere Y.
    • 한국수자원학회:학술대회논문집
    • /
    • 한국수자원학회 2020년도 학술발표회
    • /
    • pp.85-85
    • /
    • 2020
  • South Korea experiences few large scale erosion and sedimentation problems, however, there are numerous local sedimentation problems. A reliable and consistent approach to modelling and management for sediment processes are desirable in the country. In this study, field measurements of sediment concentration from 34 alluvial river basins in South Korea were used with the Modified Einstein Procedure (MEP) to determine the total sediment load at the sampling locations. And then the Flow Duration-Sediment Rating Curve (FD-SRC) method was used to estimate the specific degradation for all gauging stations. The specific degradation of most rivers were found to be typically 50-300 tons/㎢·yr. A model tree data mining technique was applied to develop a model for the specific degradation based on various watershed characteristics of each watershed from GIS analysis. The meaningful parameters are: 1) elevation at the middle relative area of the hypsometric curve [m], 2) percentage of wetland and water [%], 3) percentage of urbanized area [%], and 4) Main stream length [km]. The Root Mean Square Error (RMSE) of existing models is in excess of 1,250 tons/㎢·yr and the RMSE of the proposed model with 6 additional validations decreased to 65 tons/㎢·yr. Erosion loss maps from the Revised Universal Soil Loss Equation (RUSLE), satellite images, and aerial photographs were used to delineate the geospatial features affecting erosion and sedimentation. The results of the geospatial analysis clearly shows that the high risk erosion area (hill slopes and construction sites at urbanized area) and sedimentation features (wetlands and agricultural reservoirs). The result of physiographical analysis also indicates that the watershed morphometric characteristic well explain the sediment transport. Sustainable management with the data mining methodologies and geospatial analysis could be helpful to solve various erosion and sedimentation problems under different conditions.

  • PDF

Velest를 이용한 남한 지역의 P파 속도구조 분석 (P-wave velocity structure in Southern Korea by using Velest program)

  • 전정수
    • 한국지진공학회:학술대회논문집
    • /
    • 한국지진공학회 2000년도 춘계 학술발표회 논문집 Proceedings of EESK Conference-Spring
    • /
    • pp.49-54
    • /
    • 2000
  • Korea Institute of Geology Mining and Materials(KIGAM) has been operating Korean Earthquake Monitoring System(KEMS) to archive the real-time data stream and to determine event parameters (epicenter origin time and magnitude)by the automatic processing and analyst review. To do this KEMS uses the Vindel Hue's velocity model which was derived from Wonju KSRS data. Because KIGAM now receives the real-time data from many stations including Wonju KSRS Cholwon seismo-acoustic array Uljin Wolsung Youngkwang Taejon Seoul Kimcheon Taegu etc. the proper velocity model should be established around the Korean peninsula, In this study P were velocity structures was derived from VELEST program using 69 events among the 835 events determined by KEMS in 1999 which were recorded by at least 5 stations. General trend of velocity structure was similar to Sang Jo Kim's model but velocity value was low in crust and high in upper mantle. Due to the sensitivity of inversion results to the initial input model the artificial short and blast data might be added.

  • PDF

그래프 스트림 처리를 위한 점진적 빈발 패턴 기반 인-메모리 압축 기법 (In-memory Compression Scheme Based on Incremental Frequent Patterns for Graph Streams)

  • 이현병;신보경;복경수;유재수
    • 한국콘텐츠학회논문지
    • /
    • 제22권1호
    • /
    • pp.35-46
    • /
    • 2022
  • 최근 네트워크 기술 발전과 함께 IoT 및 소셜 네트워크 서비스의 활성화로 인해 많은 그래프 스트림 데이터가 생성되고 있다. 본 논문에서는 압축률 및 압축 시간에 대해 중점적으로 연구되던 기존의 압축 기법에 그래프 마이닝을 적용하여 스트림 그래프 환경을 함께 고려한 그래프 압축 기술을 제안한다. 또한, 최신 패턴을 유지하여 실시간으로 변화하는 스트림 그래프에서 압축 효율 및 처리속도를 향상시킨다. 본 논문에서는 그래프 스트림 처리를 위한 점진적 빈발 패턴 기반 압축 기법을 제안하였다. 제안하는 기법의 우수성을 보이기 위해 압축률과 처리시간을 기존기법과 비교하여 성능평가를 수행한다. 제안하는 기법은 그래프 데이터의 크기가 커질 때 중복되는 데이터가 많아져 기존 기법보다 빠른 처리속도를 보인다. 따라서, 빠른 처리가 요구되는 스트림 환경에서 제안하는 기법을 활용할 수 있다.