• Title/Summary/Keyword: Big data Processing

Search Result 1,063, Processing Time 0.035 seconds

Distributed Edge Computing for DNA-Based Intelligent Services and Applications: A Review (딥러닝을 사용하는 IoT빅데이터 인프라에 필요한 DNA 기술을 위한 분산 엣지 컴퓨팅기술 리뷰)

  • Alemayehu, Temesgen Seyoum;Cho, We-Duke
    • KIPS Transactions on Computer and Communication Systems
    • /
    • v.9 no.12
    • /
    • pp.291-306
    • /
    • 2020
  • Nowadays, Data-Network-AI (DNA)-based intelligent services and applications have become a reality to provide a new dimension of services that improve the quality of life and productivity of businesses. Artificial intelligence (AI) can enhance the value of IoT data (data collected by IoT devices). The internet of things (IoT) promotes the learning and intelligence capability of AI. To extract insights from massive volume IoT data in real-time using deep learning, processing capability needs to happen in the IoT end devices where data is generated. However, deep learning requires a significant number of computational resources that may not be available at the IoT end devices. Such problems have been addressed by transporting bulks of data from the IoT end devices to the cloud datacenters for processing. But transferring IoT big data to the cloud incurs prohibitively high transmission delay and privacy issues which are a major concern. Edge computing, where distributed computing nodes are placed close to the IoT end devices, is a viable solution to meet the high computation and low-latency requirements and to preserve the privacy of users. This paper provides a comprehensive review of the current state of leveraging deep learning within edge computing to unleash the potential of IoT big data generated from IoT end devices. We believe that the revision will have a contribution to the development of DNA-based intelligent services and applications. It describes the different distributed training and inference architectures of deep learning models across multiple nodes of the edge computing platform. It also provides the different privacy-preserving approaches of deep learning on the edge computing environment and the various application domains where deep learning on the network edge can be useful. Finally, it discusses open issues and challenges leveraging deep learning within edge computing.

PPFP(Push and Pop Frequent Pattern Mining): A Novel Frequent Pattern Mining Method for Bigdata Frequent Pattern Mining (PPFP(Push and Pop Frequent Pattern Mining): 빅데이터 패턴 분석을 위한 새로운 빈발 패턴 마이닝 방법)

  • Lee, Jung-Hun;Min, Youn-A
    • KIPS Transactions on Software and Data Engineering
    • /
    • v.5 no.12
    • /
    • pp.623-634
    • /
    • 2016
  • Most of existing frequent pattern mining methods address time efficiency and greatly rely on the primary memory. However, in the era of big data, the size of real-world databases to mined is exponentially increasing, and hence the primary memory is not sufficient enough to mine for frequent patterns from large real-world data sets. To solve this problem, there are some researches for frequent pattern mining method based on disk, but the processing time compared to the memory based methods took very time consuming. There are some researches to improve scalability of frequent pattern mining, but their processes are very time consuming compare to the memory based methods. In this paper, we present PPFP as a novel disk-based approach for mining frequent itemset from big data; and hence we reduced the main memory size bottleneck. PPFP algorithm is based on FP-growth method which is one of the most popular and efficient frequent pattern mining approaches. The mining with PPFP consists of two setps. (1) Constructing an IFP-tree: After construct FP-tree, we assign index number for each node in FP-tree with novel index numbering method, and then insert the indexed FP-tree (IFP-tree) into disk as IFP-table. (2) Mining frequent patterns with PPFP: Mine frequent patterns by expending patterns using stack based PUSH-POP method (PPFP method). Through this new approach, by using a very small amount of memory for recursive and time consuming operation in mining process, we improved the scalability and time efficiency of the frequent pattern mining. And the reported test results demonstrate them.

The Method of Analyzing Firewall Log Data using MapReduce based on NoSQL (NoSQL기반의 MapReduce를 이용한 방화벽 로그 분석 기법)

  • Choi, Bomin;Kong, Jong-Hwan;Hong, Sung-Sam;Han, Myung-Mook
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.23 no.4
    • /
    • pp.667-677
    • /
    • 2013
  • As the firewall is a typical network security equipment, it is usually installed at most of internal/external networks and makes many packet data in/out. So analyzing the its logs stored in it can provide important and fundamental data on the network security research. However, along with development of communications technology, the speed of internet network is improved and then the amount of log data is becoming 'Massive Data' or 'BigData'. In this trend, there are limits to analyze log data using the traditional database model RDBMS. In this paper, through our Method of Analyzing Firewall log data using MapReduce based on NoSQL, we have discovered that the introducing NoSQL data base model can more effectively analyze the massive log data than the traditional one. We have demonstrated execellent performance of the NoSQL by comparing the performance of data processing with existing RDBMS. Also the proposed method is evaluated by experiments that detect the three attack patterns and shown that it is highly effective.

Rapid Management Mechanism Against Harmful Materials of Agri-Food Based on Big Data Analysis (빅 데이터 분석 기반 농 식품 위해인자 신속관리 방법)

  • Park, Hyeon;Kang, Sung-soo;Jeong, Hoon;Kim, Se-Han
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.40 no.6
    • /
    • pp.1166-1174
    • /
    • 2015
  • There were the attempts to prevent the spread of harmful materials of the agri-food through the record tracking of the products with the bar code, the partial information tracking of the agri-food storage and the delivery vehicle, or the control of the temperature by intuition. However, there were many problems in the attempts because of the insufficient information, the information distortion and the independent information network of each distribution company. As a result, it is difficult to prevent the spread over the life-cycle of the agri-food using the attempts. To solve the problems, we propose the mechanism mainly to do context awareness, predict, and track the harmful materials of agri-food using big data processing.

Analysis of Vocational Training Needs Using Big Data Technique (빅데이터 기법을 활용한 직업훈련 요구분석)

  • Sung, Bo-Kyoung;You, Yen-Yoo
    • Journal of the Korea Convergence Society
    • /
    • v.9 no.5
    • /
    • pp.21-26
    • /
    • 2018
  • In this study, HRD-NET (http://hrd.go.kr), a vocational and training integrated computer network operated by the Ministry of Employment and Labor, is used to confirm whether job training information required by job seekers is being provided smoothly The question bulletin board was extracted using 'R' program which is optimized for big data technique. Therefore, the effectiveness, appropriateness, visualization, frequency analysis and association analysis of the vocational training system were conducted through this, The results of the study are as follows. First, the issue of vocational training card, video viewing, certificate issue, registration error, Second, management and processing procedures of learning cards for tomorrow 's learning cards are complicated and difficult. In addition, it was analyzed that the training cost system and the refund structure differentiated according to the training occupation, the process, and the training institution in the course of the training. Based on this paper, we will study not only the training system of the Ministry of Employment and Labor but also the improvement of the various training computer system of the government department through the analysis of big data.

Subway Congestion Prediction and Recommendation System using Big Data Analysis (빅데이터 분석을 이용한 지하철 혼잡도 예측 및 추천시스템)

  • Kim, Jin-su
    • Journal of Digital Convergence
    • /
    • v.14 no.11
    • /
    • pp.289-295
    • /
    • 2016
  • Subway is a future-oriented means of transportation that can be safely and quickly mass transport many passengers than buses and taxis. Congestion growth due to the increase of the metro users is one of the factors that hinder citizens' rights to comfortably use the subway. Accordingly, congestion prediction in the subway is one of the ways to maximize the use of passenger convenience and comfort. In this paper, we monitor the level of congestion in real time via the existing congestion on the metro using multiple regression analysis and big data processing, as well as their departure station and arrival station information More information about the transfer stations offer a personalized congestion prediction system. The accuracy of the predicted congestion shows about 81% accuracy, which is compared to the real congestion. In this paper, the proposed prediction and recommendation application will be a help to prediction of subway congestion and user convenience.

Smart Space based on Platform using Big Data for Efficient Decision-making (효율적 의사결정을 위한 빅데이터 활용 스마트 스페이스 플랫폼 연구)

  • Lee, Jin-Kyung
    • Informatization Policy
    • /
    • v.25 no.4
    • /
    • pp.108-120
    • /
    • 2018
  • With the rise of the Fourth Industrial Revolution and I-Korea 4.0, both of which pursue strategies for industrial innovation and for the solution to social problems, the real estate industry needs to change in order to make effective use of available space in smart environments. The implementation of smart spaces is a promising solution for this. The smart space is defined as a good use of space, whether it be a home, office, or retail store, within a smart environment. To enhance the use of smart spaces, efficient decision-making and well-timed and accurate interaction are required. This paper proposes a smart space based on platform which takes advantage of emerging technologies for the efficient storage, processing, analysis, and utilization of big data. The platform is composed of six layers - collection, transfer, storage, service, application, and management - and offers three service frameworks: activity-based, market-based, and policy-based. Based on these smart space services, decision-makers, consumers, clients, and social network participants can make better decisions, respond more quickly, exhibit greater innovation, and develop stronger competitive advantages.

A Study on System and Application Performance Monitoring System Using Mass Processing Engine(ElasticSearch) (대량 처리 엔진(ElasticSearch)을 이용한 시스템 및 어플리케이션 성능 모니터링 시스템에 관한 연구)

  • Kim, Seung-Cheon;Jang, Hee-Don
    • Journal of Digital Convergence
    • /
    • v.17 no.9
    • /
    • pp.147-152
    • /
    • 2019
  • Infrastructure is rapidly growing as Internet business grows with the latest IT technologies such as IoT, BigData, and AI. However, in most companies, a limited number of people need to manage a lot of hardware and software. Therefore, Polestar Enterprise Management System(PEMS) is applied to monitor the system operation status, IT service and key KPI monitoring. Real-time monitor screening prevents system malfunctions and quick response. With PEMS, you can see configuration information related to IT hardware and software at a glance, and monitor performance throughout the entire end-to-end period to see when problems occur in real time.

The Analysis of Priority Output Queuing Model by Short Bus Contention Method (Short Bus contention 방식의 Priority Output Queuing Model의 분석)

  • Jeong, Yong-Ju
    • The Transactions of the Korea Information Processing Society
    • /
    • v.6 no.2
    • /
    • pp.459-466
    • /
    • 1999
  • I broadband ISDN every packet will show different result if it would be processed according to its usage by the server. That is, normal data won't show big differences if they would be processed at normal speed. But it will improve the quality of service to process some kinds of data - for example real time video or voice type data or some data for a bid to by something through the internet - more fast than the normal type data. solution for this problem was suggested - priority packets. But the analyses of them are under way. Son in this paper a switching system for an output queuing model in a single server was assumed and some packets were given priorities and analysed. And correlation, simulating real life situation, was given too. These packets were analysed through three cases, first packets having no correlation, second packets having only correlation and finally packets having priority three cases, first packets having no correlation, second packets having only correlation and finally packets having priority and correlation. The result showed that correlation doesn't affect the mean delay time and the high priority packets have improved mean delay time regardless of the arrival rate. Those packets were assumed to be fixed-sized like ATM fixed-sized cell and the contention strategy was assumed to be short bus contention method for the output queue, and the mean delay length and the maximum 버퍼 length not to lose any packets were analysed.

  • PDF

An Analysis of Causes of Marine Incidents at sea Using Big Data Technique (빅데이터 기법을 활용한 항해 중 준해양사고 발생원인 분석에 관한 연구)

  • Kang, Suk-Young;Kim, Ki-Sun;Kim, Hong-Beom;Rho, Beom-Seok
    • Journal of the Korean Society of Marine Environment & Safety
    • /
    • v.24 no.4
    • /
    • pp.408-414
    • /
    • 2018
  • Various studies have been conducted to reduce marine accidents. However, research on marine incidents is only marginal. There are many reports of marine incidents, but the main content of existing studies has been qualitative, which makes quantitative analysis difficult. However, quantitative analysis of marine accidents is necessary to reduce marine incidents. The purpose of this paper is to analyze marine incident data quantitatively by applying big data techniques to predict marine incident trends and reduce marine accident. To accomplish this, about 10,000 marine incident reports were prepared in a unified format through pre-processing. Using this preprocessed data, we first derived major keywords for the Marine incidents at sea using text mining techniques. Secondly, time series and cluster analysis were applied to major keywords. Trends for possible marine incidents were predicted. The results confirmed that it is possible to use quantified data and statistical analysis to address this topic. Also, we have confirmed that it is possible to provide information on preventive measures by grasping objective tendencies for marine incidents that may occur in the future through big data techniques.