• Title/Summary/Keyword: Distributed Data Mining

Search Result 110, Processing Time 0.027 seconds

A mathematical spatial interpolation method for the estimation of convective rainfall distribution over small watersheds

  • Zhang, Shengtang;Zhang, Jingzhou;Liu, Yin;Liu, Yuanchen
    • Environmental Engineering Research
    • /
    • v.21 no.3
    • /
    • pp.226-232
    • /
    • 2016
  • Rainfall is one of crucial factors that impact on our environment. Rainfall data is important in water resources management, flood forecasting, and designing hydraulic structures. However, it is not available in some rural watersheds without rain gauges. Thus, effective ways of interpolating the available records are needed. Despite many widely used spatial interpolation methods, few studies have investigated rainfall center characteristics. Based on the theory that the spatial distribution of convective rainfall event has a definite center with maximum rainfall, we present a mathematical interpolation method to estimate convective rainfall distribution and indicate the rainfall center location and the center rainfall volume. We apply the method to estimate three convective rainfall events in Santa Catalina Island where reliable hydrological data is available. A cross-validation technique is used to evaluate the method. The result shows that the method will suffer from high relative error in two situations: 1) when estimating the minimum rainfall and 2) when estimating an external site. For all other situations, the method's performance is reasonable and acceptable. Since the method is based on a continuous function, it can provide distributed rainfall data for distributed hydrological model sand indicate statistical characteristics of given areas via mathematical calculation.

TeT: Distributed Tera-Scale Tensor Generator (분산 테라스케일 텐서 생성기)

  • Jeon, ByungSoo;Lee, JungWoo;Kang, U
    • Journal of KIISE
    • /
    • v.43 no.8
    • /
    • pp.910-918
    • /
    • 2016
  • A tensor is a multi-dimensional array that represents many data such as (user, user, time) in the social network system. A tensor generator is an important tool for multi-dimensional data mining research with various applications including simulation, multi-dimensional data modeling/understanding, and sampling/extrapolation. However, existing tensor generators cannot generate sparse tensors like real-world tensors that obey power law. In addition, they have limitations such as tensor sizes that can be processed and additional time required to upload generated tensor to distributed systems for further analysis. In this study, we propose TeT, a distributed tera-scale tensor generator to solve these problems. TeT generates sparse random tensor as well as sparse R-MAT and Kronecker tensor without any limitation on tensor sizes. In addition, a TeT-generated tensor is immediately ready for further tensor analysis on the same distributed system. The careful design of TeT facilitates nearly linear scalability on the number of machines.

A FCA-based Classification Approach for Analysis of Interval Data (구간데이터분석을 위한 형식개념분석기반의 분류)

  • Hwang, Suk-Hyung;Kim, Eung-Hee
    • Journal of the Korea Society of Computer and Information
    • /
    • v.17 no.1
    • /
    • pp.19-30
    • /
    • 2012
  • Based on the internet-based infrastructures such as various information devices, social network systems and cloud computing environments, distributed and sharable data are growing explosively. Recently, as a data analysis and mining technique for extracting, analyzing and classifying the inherent and useful knowledge and information, Formal Concept Analysis on binary or many-valued data has been successfully applied in many diverse fields. However, in formal concept analysis, there has been little research conducted on analyzing interval data whose attributes have some interval values. In this paper, we propose a new approach for classification of interval data based on the formal concept analysis. We present the development of a supporting tool(iFCA) that provides the proposed approach for the binarization of interval data table, concept extraction and construction of concept hierarchies. Finally, with some experiments over real-world data sets, we demonstrate that our approach provides some useful and effective ways for analyzing and mining interval data.

A MapReduce-Based Workflow BIG-Log Clustering Technique (맵리듀스기반 워크플로우 빅-로그 클러스터링 기법)

  • Jin, Min-Hyuck;Kim, Kwanghoon Pio
    • Journal of Internet Computing and Services
    • /
    • v.20 no.1
    • /
    • pp.87-96
    • /
    • 2019
  • In this paper, we propose a MapReduce-supported clustering technique for collecting and classifying distributed workflow enactment event logs as a preprocessing tool. Especially, we would call the distributed workflow enactment event logs as Workflow BIG-Logs, because they are satisfied with as well as well-fitted to the 5V properties of BIG-Data like Volume, Velocity, Variety, Veracity and Value. The clustering technique we develop in this paper is intentionally devised for the preprocessing phase of a specific workflow process mining and analysis algorithm based upon the workflow BIG-Logs. In other words, It uses the Map-Reduce framework as a Workflow BIG-Logs processing platform, it supports the IEEE XES standard data format, and it is eventually dedicated for the preprocessing phase of the ${\rho}$-Algorithm that is a typical workflow process mining algorithm based on the structured information control nets. More precisely, The Workflow BIG-Logs can be classified into two types: of activity-based clustering patterns and performer-based clustering patterns, and we try to implement an activity-based clustering pattern algorithm based upon the Map-Reduce framework. Finally, we try to verify the proposed clustering technique by carrying out an experimental study on the workflow enactment event log dataset released by the BPI Challenges.

A Distributed Domain Document Object Management using Semantic Reference Relationship (SRR을 이용한 분산 도메인 문서 객체 관리)

  • Lee, Chong-Deuk
    • Journal of Digital Convergence
    • /
    • v.10 no.5
    • /
    • pp.267-273
    • /
    • 2012
  • The semantic relationship structures hierarchically the huge amount of document objects which is usually not formatted. However, it is very difficult to structure relevant data from various distributed application domains. This paper proposed a new object management method to service the distributed domain objects by using semantic reference relationship. The proposed mechanism utilized the profile structure in order to extract the semantic similarity from application domain objects and utilized the joint matrix to decide the semantic relationship of the extracted objects. This paper performed the simulation to show the performance of the proposed method, and simulation results show that the proposed method has better retrieval performance than the existing text mining method and information extraction method.

Design of agent intrusion detection system applying data mining (데이터 마이닝을 적용한 에이전트 침입 탐지 시스템 설계)

  • Jeong Jong Kun;Lee Sung Tae;Kim Yong Ho;Lee Yun Bae
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2001.05a
    • /
    • pp.676-679
    • /
    • 2001
  • As network security is coning up with significant problem after the major Internet sites were hacked nowadays, IDS(Intrusion Detection System) is considered as a next generation security solution for more reliable network and system security rather than firewall. In this paper, we propose the new IDS model which tan detect intrusion in different systems as well as which ran make real-time detection of intrusion in the expanded distributed environment in host level of drawback of existing IDS. We implement its prototype and verify its validity. We use pattern extraction agent so that we can extract automatically audit file needed in distributed intrusion detection even in other platforms.

  • PDF

Development of Practical Data Mining Methods for Database Summarization

  • Lee, Do-Heon
    • The Journal of Information Technology and Database
    • /
    • v.4 no.2
    • /
    • pp.33-45
    • /
    • 1998
  • Database summarization is the procedure to obtain generalized and representative descriptions expressing the content of a large amount of database at a glance. We present a top-down summary refinement procedure to discover database summaries. The procedure exploits attribute concept hierarchies that represent ISA relationships among domain concepts. It begins with the most generalized summary and proceeds to find more specialized ones by stepwise refinements. This top-down paradigm reveals at least two important advantages compared to the previous bottom-up methods. Firstly, it provides a natural way of reflecting the user's own discovery preference interactively. Secondly, it does not produce too large intermediate result that makes it hard for the bottom-up approach to be applied in practical environment. The proposed procedure can also be easily extended for distributed databases. Information content measure of a database summary is derived in order to identify more informative summaries among the discovered results.

Analyzing Operation Deviation in the Deasphalting Process Using Multivariate Statistics Analysis Method

  • Park, Joo-Hwang;Kim, Jong-Soo;Kim, Tai-Suk
    • Journal of Korea Multimedia Society
    • /
    • v.17 no.7
    • /
    • pp.858-865
    • /
    • 2014
  • In the case of system like MES, various sensors collect the data in real time and save it as a big data to monitor the process. However, if there is big data mining in distributed computing system, whole processing process can be improved. In this paper, system to analyze the cause of operation deviation was built using the big data which has been collected from deasphalting process at the two different plants. By applying multivariate statistical analysis to the big data which has been collected through MES(Manufacturing Execution System), main cause of operation deviation was analyzed. We present the example of analyzing the operation deviation of deasphalting process using the big data which collected from MES by using multivariate statistics analysis method. As a result of regression analysis of the forward stepwise method, regression equation has been found which can explain 52% increase of performance compare to existing model. Through this suggested method, the existing petrochemical process can be replaced which is manual analysis method and has the risk of being subjective according to the tester. The new method can provide the objective analysis method based on numbers and statistic.

WebPR : A Dynamic Web Page Recommendation Algorithm Based on Mining Frequent Traversal Patterns (WebPR :빈발 순회패턴 탐사에 기반한 동적 웹페이지 추천 알고리즘)

  • Yoon, Sun-Hee;Kim, Sam-Keun;Lee, Chang-Hoon
    • The KIPS Transactions:PartB
    • /
    • v.11B no.2
    • /
    • pp.187-198
    • /
    • 2004
  • The World-Wide Web is the largest distributed Information space and has grown to encompass diverse information resources. However, although Web is growing exponentially, the individual's capacity to read and digest contents is essentially fixed. From the view point of Web users, they can be confused by explosion of Web information, by constantly changing Web environments, and by lack of understanding needs of Web users. In these Web environments, mining traversal patterns is an important problem in Web mining with a host of application domains including system design and Information services. Conventional traversal pattern mining systems use the inter-pages association in sessions with only a very restricted mechanism (based on vector or matrix) for generating frequent k-Pagesets. We develop a family of novel algorithms (termed WebPR - Web Page Recommend) for mining frequent traversal patterns and then pageset to recommend. Our algorithms provide Web users with new page views, which Include pagesets to recommend, so that users can effectively traverse its Web site. The main distinguishing factors are both a point consistently spanning schemes applying inter-pages association for mining frequent traversal patterns and a point proposing the most efficient tree model. Our experimentation with two real data sets, including Lady Asiana and KBS media server site, clearly validates that our method outperforms conventional methods.

Features of the Rural Revitalization Projects in Jang-su County Using LDA Topic Analysis of News Data - Focused on Keyword of Tourism and Livelihood - (뉴스데이터의 LDA 토픽 분석을 통한 장수군 농촌지역 활성화 사업의 특징 - 관광·생활 키워드를 중심으로 -)

  • Kim, Young-Jin;Son, Yong-hoon
    • Journal of Korean Society of Rural Planning
    • /
    • v.24 no.4
    • /
    • pp.69-80
    • /
    • 2018
  • In this study, we typified the project for revitalizing the rural area through text analysis using news data, and analyzed the main direction and characteristics of the project. In order to examine the factors emphasized among the issues related to the revitalization of rural areas, we used news data related to 'tourism' and 'livelihood', which are the main keyword of the project to promote rural areas. In the analysis, text mining techniques were used. Topic modeling was conducted on LDA techniques for major projects in 'tourism' and 'livelihood' keyword. Based on this, this study typified the projects that are carried out for the activation of rural areas by topic. As a result of the analysis, it was fount that the topics included in the project were distributed in 11 sub-types(Tourism Promotion, Regional Specialization, Local Festival, Development of Regional Scale, Urban and Rural Exchange, Agricultural Support, Community Forest Management, Improve the Settlement Environment, General Welfare Service, Low Class Support, Others). The characteristics of the rural revitalization projects were examined, and it was confirmed that domestic projects were carried out by tourism-oriented projects. To summarize, the government is making projects to revitalize rural areas through related ministries. Within the structure where the project is spreading to the region, a lot of projects are being carried out. It is understood that the tourism and welfare oriented projects are being carried out in the revitalization project of the domestic rural area. Therefore, in order to achieve the goal of rural revitalization, it is believed that it will be effective to carry out a balanced project to improve the settlement environment of the residents.