• Title/Summary/Keyword: Distributed Data Mining

Search Result 110, Processing Time 0.026 seconds

Privacy-Preserving in the Context of Data Mining and Deep Learning

  • Altalhi, Amjaad;AL-Saedi, Maram;Alsuwat, Hatim;Alsuwat, Emad
    • International Journal of Computer Science & Network Security
    • /
    • v.21 no.6
    • /
    • pp.137-142
    • /
    • 2021
  • Machine-learning systems have proven their worth in various industries, including healthcare and banking, by assisting in the extraction of valuable inferences. Information in these crucial sectors is traditionally stored in databases distributed across multiple environments, making accessing and extracting data from them a tough job. To this issue, we must add that these data sources contain sensitive information, implying that the data cannot be shared outside of the head. Using cryptographic techniques, Privacy-Preserving Machine Learning (PPML) helps solve this challenge, enabling information discovery while maintaining data privacy. In this paper, we talk about how to keep your data mining private. Because Data mining has a wide variety of uses, including business intelligence, medical diagnostic systems, image processing, web search, and scientific discoveries, and we discuss privacy-preserving in deep learning because deep learning (DL) exhibits exceptional exactitude in picture detection, Speech recognition, and natural language processing recognition as when compared to other fields of machine learning so that it detects the existence of any error that may occur to the data or access to systems and add data by unauthorized persons.

Grid-based Biological Data Mining using Dynamic Load Balancing (동적 로드 밸런싱을 이용한 그리드 기반의 생물학 데이터 마이닝)

  • Ma, Yong-Beom;Kim, Tae-Young;Lee, Jong-Sik
    • Journal of the Korea Society for Simulation
    • /
    • v.19 no.2
    • /
    • pp.81-89
    • /
    • 2010
  • Biological data mining has been noticed as an issue as the volume of biological data is increasing extremely. Grid technology can share and utilize computing data and resources. In this paper, we propose a hybrid system that combines biological data mining with grid technology. Especially, we propose a decision range adjustment algorithm for processing efficiency of biological data mining. We obtain a reliable data mining recognition rate automatically and rapidly through this algorithm. And communication loads and resource allocation are key issues in grid environment because the resources are geographically distributed and interacted with themselves. Therefore, we propose a dynamic load balancing algorithm and apply it to the grid-based biological data mining method. For performance evaluation, we measure average processing time, average communication time, and average resource utilization. Experimental results show that this method provides many advantages in aspects of processing time and cost.

A Parallel HDFS and MapReduce Functions for Emotion Analysis (감성분석을 위한 병렬적 HDFS와 맵리듀스 함수)

  • Back, BongHyun;Ryoo, Yun-Kyoo
    • Journal of the Korea society of information convergence
    • /
    • v.7 no.2
    • /
    • pp.49-57
    • /
    • 2014
  • Recently, opinion mining is introduced to extract useful information from SNS data and to evaluate the true intention of users. Opinion mining are required several efficient techniques to collect and analyze a large amount of SNS data and extract meaningful data from them. Therefore in this paper, we propose a parallel HDFS(Hadoop Distributed File System) and emotion functions based on Mapreduce to extract some emotional information of users from various unstructured big data on social networks. The experiment results have verified that the proposed system and functions perform faster than O(n) for data gathering time and loading time, and maintain stable load balancing for memory and CPU resources.

  • PDF

System Modeling for Analysing Exercises Using Data Mining (운동량 분석을 위한 데이터 마이닝 시스템 모델)

  • Lee, Sun-Geun;Im, Yeong-Mun
    • Proceedings of the Safety Management and Science Conference
    • /
    • 2013.11a
    • /
    • pp.393-400
    • /
    • 2013
  • Globally, smart phones have been rapidly distributed, which has led to changes in people's life cycle. Most people who are under 60 are supposed to use smart phones. Additionally, as the ratio of people who are interested in physical exercise is increasing, some applications for smart phones can manage dividual's exercise with the web servers. However, most of them can only check how much individual works out and cannot compare other's body type and life environment. Moreover, users cannot share their own data with others. This paper proposed the system which can resolve those kinds of problems through data mining techniques. The suggested model will have ability to figure out the relation between body type and the amount of exercise, find out if his work is proper from the result of classification and can pick out the features which is common to people who have similar body type and the amount of workout by applying data mining techiques. This model also will be able to recommend the proper amount of workout to each individual in order that they keep good health state efficiently.

  • PDF

Secure Multi-Party Computation of Correlation Coefficients (상관계수의 안전한 다자간 계산)

  • Hong, Sun-Kyong;Kim, Sang-Pil;Lim, Hyo-Sang;Moon, Yang-Sae
    • Journal of KIISE
    • /
    • v.41 no.10
    • /
    • pp.799-809
    • /
    • 2014
  • In this paper, we address the problem of computing Pearson correlation coefficients and Spearman's rank correlation coefficients in a secure manner while data providers preserve privacy of their own data in distributed environment. For a data mining or data analysis in the distributed environment, data providers(data owners) need to share their original data with each other. However, the original data may often contain very sensitive information, and thus, data providers do not prefer to disclose their original data for preserving privacy. In this paper, we formally define the secure correlation computation, SCC in short, as the problem of computing correlation coefficients in the distributed computing environment while preserving the data privacy (i.e., not disclosing the sensitive data) of multiple data providers. We then present SCC solutions for Pearson and Spearman's correlation coefficients using secure scalar product. We show the correctness and secure property of the proposed solutions by presenting theorems and proving them formally. We also empirically show that the proposed solutions can be used for practical applications in the performance aspect.

Analysis of Traffic Card Big Data by Hadoop and Sequential Mining Technique (하둡과 순차패턴 마이닝 기술을 통한 교통카드 빅데이터 분석)

  • Kim, Woosaeng;Kim, Yong Hoon;Park, Hee-Sung;Park, Jin-Kyu
    • Journal of Information Technology Applications and Management
    • /
    • v.24 no.4
    • /
    • pp.187-196
    • /
    • 2017
  • It is urgent to prepare countermeasures for traffic congestion problems of Korea's metropolitan area where central functions such as economic, social, cultural, and education are excessively concentrated. Most users of public transportation in metropolitan areas including Seoul use the traffic cards. If various information is extracted from traffic big data produced by the traffic cards, they can provide basic data for transport policies, land usages, or facility plans. Therefore, in this study, we extract valuable information such as the subway passengers' frequent travel patterns from the big traffic data provided by the Seoul Metropolitan Government Big Data Campus. For this, we use a Hadoop (High-Availability Distributed Object-Oriented Platform) to preprocess the big data and store it into a Mongo database in order to analyze it by a sequential pattern data mining technique. Since we analysis the actual big data, that is, the traffic cards' data provided by the Seoul Metropolitan Government Big Data Campus, the analyzed results can be used as an important referenced data when the Seoul government makes a plan about the metropolitan traffic policies.

A Design of XML-Based Distributed MDR Retrieval System for Data Preparation (데이터준비를 위한 XML 기반의 분산 MDR 검색 시스템 설계)

  • Ko Sucbum;Youn Sungdae
    • Journal of Korea Multimedia Society
    • /
    • v.7 no.9
    • /
    • pp.1329-1338
    • /
    • 2004
  • The purpose of data mining is to extract multi-dimensional information from a large database. The only information that we can extract from a large database is the column name, data type or simple comments included in the columns of database tables. With such unstructured and scarce information, it is very difficult and time taking to collect and to cleanse data by analyzing the purpose, characteristic and schema of the column during the data preparation step. In order to solve this problem, we propose solutions for reducing the time spent data preparation step in a relational database environment in this paper. That is, we propose useful elements to be considered during the data preparation step and then these elements are organized to constitute MDR(Metadata Registry) which is becoming the international standard of ISO/IEC : ll179. Finally, we propose a XML-based distributed MDR retrieval system that is convertible among heterogeneous systems and heterogeneous DBMSS.

  • PDF

User Access Patterns Discovery based on Apriori Algorithm under Web Logs (웹 로그에서의 Apriori 알고리즘 기반 사용자 액세스 패턴 발견)

  • Ran, Cong-Lin;Joung, Suck-Tae
    • The Journal of Korea Institute of Information, Electronics, and Communication Technology
    • /
    • v.12 no.6
    • /
    • pp.681-689
    • /
    • 2019
  • Web usage pattern discovery is an advanced means by using web log data, and it's also a specific application of data mining technology in Web log data mining. In education Data Mining (DM) is the application of Data Mining techniques to educational data (such as Web logs of University, e-learning, adaptive hypermedia and intelligent tutoring systems, etc.), and so, its objective is to analyze these types of data in order to resolve educational research issues. In this paper, the Web log data of a university are used as the research object of data mining. With using the database OLAP technology the Web log data are preprocessed into the data format that can be used for data mining, and the processing results are stored into the MSSQL. At the same time the basic data statistics and analysis are completed based on the processed Web log records. In addition, we introduced the Apriori Algorithm of Web usage pattern mining and its implementation process, developed the Apriori Algorithm program in Python development environment, then gave the performance of the Apriori Algorithm and realized the mining of Web user access pattern. The results have important theoretical significance for the application of the patterns in the development of teaching systems. The next research is to explore the improvement of the Apriori Algorithm in the distributed computing environment.

Data Mining mechanism using Data Cube and Neural Network in distributed environment (분산환경에서 데이터 큐브와 신경망을 이용한 데이터마이닝기법)

  • 박민기;바비제라도;이재완
    • Proceedings of the Korean Institute of Information and Commucation Sciences Conference
    • /
    • 2003.10a
    • /
    • pp.188-191
    • /
    • 2003
  • In this paper, we proposed data generalization and data cube mechanism for efficient data mining in distribute environment. We also proposed active Self Organization Map applying traditional Self Organization Map of Neural network for searching the most Informative data created from data cube after the generalization procedure and designed the system architecture for that.

  • PDF

Clustering Algorithm using the DFP-Tree based on the MapReduce (맵리듀스 기반 DFP-Tree를 이용한 클러스터링 알고리즘)

  • Seo, Young-Won;Kim, Chang-soo
    • Journal of Internet Computing and Services
    • /
    • v.16 no.6
    • /
    • pp.23-30
    • /
    • 2015
  • As BigData is issued, many applications that operate based on the results of data analysis have been developed, typically applications are products recommend service of e-commerce application service system, search service on the search engine service and friend list recommend system of social network service. In this paper, we suggests a decision frequent pattern tree that is combined the origin frequent pattern tree that is mining similar pattern to appear in the data set of the existing data mining techniques and decision tree based on the theory of computer science. The decision frequent pattern tree algorithm improves about problem of frequent pattern tree that have to make some a lot's pattern so it is to hard to analyze about data. We also proposes to model for a Mapredue framework that is a programming model to help to operate in distributed environment.