• Title/Summary/Keyword: Big data Processing

Search Result 1,063, Processing Time 0.029 seconds

Cascaded-Hop For DeepFake Videos Detection

  • Zhang, Dengyong;Wu, Pengjie;Li, Feng;Zhu, Wenjie;Sheng, Victor S.
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.16 no.5
    • /
    • pp.1671-1686
    • /
    • 2022
  • Face manipulation tools represented by Deepfake have threatened the security of people's biological identity information. Particularly, manipulation tools with deep learning technology have brought great challenges to Deepfake detection. There are many solutions for Deepfake detection based on traditional machine learning and advanced deep learning. However, those solutions of detectors almost have problems of poor performance when evaluated on different quality datasets. In this paper, for the sake of making high-quality Deepfake datasets, we provide a preprocessing method based on the image pixel matrix feature to eliminate similar images and the residual channel attention network (RCAN) to resize the scale of images. Significantly, we also describe a Deepfake detector named Cascaded-Hop which is based on the PixelHop++ system and the successive subspace learning (SSL) model. By feeding the preprocessed datasets, Cascaded-Hop achieves a good classification result on different manipulation types and multiple quality datasets. According to the experiment on FaceForensics++ and Celeb-DF, the AUC (area under curve) results of our proposed methods are comparable to the state-of-the-art models.

Modeling and Implementation of Public Open Data in NoSQL Database

  • Min, Meekyung
    • International Journal of Internet, Broadcasting and Communication
    • /
    • v.10 no.3
    • /
    • pp.51-58
    • /
    • 2018
  • In order to utilize various data provided by Korea public open data portal, data should be systematically managed using a database. Since the range of open data is enormous, and the amount of data continues to increase, it is preferable to use a database capable of processing big data in order to analyze and utilize the data. This paper proposes data modeling and implementation method suitable for public data. The target data is subway related data provided by the public open data portal. Schema of the public data related to Seoul metro stations are analyzed and problems of the schema are presented. To solve these problems, this paper proposes a method to normalize and structure the subway data and model it in NoSQL database. In addition, the implementation result is shown by using MongDB which is a document-based database capable of processing big data.

Big Data Patent Analysis Using Social Network Analysis (키워드 네트워크 분석을 이용한 빅데이터 특허 분석)

  • Choi, Ju-Choel
    • Journal of the Korea Convergence Society
    • /
    • v.9 no.2
    • /
    • pp.251-257
    • /
    • 2018
  • As the use of big data is necessary for increasing business value, the size of the big data market is getting bigger. Accordingly, it is important to apply competitive patents in order to gain the big data market. In this study, we conducted the patent analysis based keyword network to analyze the trend of big data patents. The analysis procedure consists of big data collection and preprocessing, network construction, and network analysis. The results of the study are as follows. Most of big data patents are related to data processing and analysis, and the keywords with high degree centrality and between centrality are "analysis", "process", "information", "data", "prediction", "server", "service", and "construction". we expect that the results of this study will offer useful information in applying big data patent.

In-Memory Based Incremental Processing Method for Stream Query Processing in Big Data Environments (빅데이터 환경에서 스트림 질의 처리를 위한 인메모리 기반 점진적 처리 기법)

  • Bok, Kyoungsoo;Yook, Misun;Noh, Yeonwoo;Han, Jieun;Kim, Yeonwoo;Lim, Jongtae;Yoo, Jaesoo
    • The Journal of the Korea Contents Association
    • /
    • v.16 no.2
    • /
    • pp.163-173
    • /
    • 2016
  • Recently, massive amounts of stream data have been studied for distributed processing. In this paper, we propose an incremental stream data processing method based on in-memory in big data environments. The proposed method stores input data in a temporary queue and compare them with data in a master node. If the data is in the master node, the proposed method reuses the previous processing results located in the node chosen by the master node. If there are no previous results of data in the node, the proposed method processes the data and stores the result in a separate node. We also propose a job scheduling technique considering the load and performance of a node. In order to show the superiority of the proposed method, we compare it with the existing method in terms of query processing time. Our experimental results show that our method outperforms the existing method in terms of query processing time.

Design and Implementation of Big Data Cluster for Indoor Environment Monitering (실내 환경 모니터링을 위한 빅데이터 클러스터 설계 및 구현)

  • Jeon, Byoungchan;Go, Mingu
    • Journal of Korea Society of Digital Industry and Information Management
    • /
    • v.13 no.2
    • /
    • pp.77-85
    • /
    • 2017
  • Due to the expansion of accommodation space caused by increase of population along with lifestyle changes, most of people spend their time indoor except for the travel time. Because of this, environmental change of indoor is very important, and it affects people's health and economy in resources. But, most of people don't acknowledge the importance of indoor environment. Thus, monitoring system for sustaining and managing indoor environment systematically is needed, and big data clusters should be used in order to save and manage numerous sensor data collected from many spaces. In this paper, we design a big data cluster for the indoor environment monitoring in order to store the sensor data and monitor unit of the huge building Implementation design big data cluster-based system for the analysis, and a distributed file system and building a Hadoop, HBase for big data processing. Also, various sensor data is saved for collection, and effective indoor environment management and health enhancement through monitoring is expected.

Enhanced Regular Expression as a DGL for Generation of Synthetic Big Data

  • Kai, Cheng;Keisuke, Abe
    • Journal of Information Processing Systems
    • /
    • v.19 no.1
    • /
    • pp.1-16
    • /
    • 2023
  • Synthetic data generation is generally used in performance evaluation and function tests in data-intensive applications, as well as in various areas of data analytics, such as privacy-preserving data publishing (PPDP) and statistical disclosure limit/control. A significant amount of research has been conducted on tools and languages for data generation. However, existing tools and languages have been developed for specific purposes and are unsuitable for other domains. In this article, we propose a regular expression-based data generation language (DGL) for flexible big data generation. To achieve a general-purpose and powerful DGL, we enhanced the standard regular expressions to support the data domain, type/format inference, sequence and random generation, probability distributions, and resource reference. To efficiently implement the proposed language, we propose caching techniques for both the intermediate and database queries. We evaluated the proposed improvement experimentally.

A Method for Selective Storing and Visualization of Public Big Data Using XML Structure (XML구조를 이용한 공공 빅데이터의 선별 저장 및 시각화 방법)

  • Back, BongHyun;Ha, Il-Kyu
    • Journal of the Korea Institute of Information and Communication Engineering
    • /
    • v.21 no.12
    • /
    • pp.2305-2311
    • /
    • 2017
  • In recent years, there have been tries to open public data from various government agencies along with publicization of public information for the public interest. In other words, various kinds of electronic data generated and collected by the public institutions as a result of their work are opened in the public portal sites. However, users who use it are limited in their use of big data due to lack of understanding of data format, lack of data processing knowledge, difficulty in accessing and managing data, and lack of visualization data to understand collected and stored data. Therefore, in this study, we propose a big data collection, storing and visualization platform that can collect big data provided by various public sites using data set URL and API regardless of data format, re-process collected data using XML structure.

Real-Time IoT Big-data Processing for Stream Reasoning (스트림-리즈닝을 위한 실시간 사물인터넷 빅-데이터 처리)

  • Yun, Chang Ho;Park, Jong Won;Jung, Hae Sun;Lee, Yong Woo
    • Journal of Internet Computing and Services
    • /
    • v.18 no.3
    • /
    • pp.1-9
    • /
    • 2017
  • Smart Cities intelligently manage numerous infrastructures, including Smart-City IoT devices, and provide a variety of smart-city applications to citizen. In order to provide various information needed for smart-city applications, Smart Cities require a function to intelligently process large-scale streamed big data that are constantly generated from a large number of IoT devices. To provide smart services in Smart-City, the Smart-City Consortium uses stream reasoning. Our stream reasoning requires real-time processing of big data. However, there are limitations associated with real-time processing of large-scale streamed big data in Smart Cities. In this paper, we introduce one of our researches on cloud computing based real-time distributed-parallel-processing to be used in stream-reasoning of IoT big data in Smart Cities. The Smart-City Consortium introduced its previously developed smart-city middleware. In the research for this paper, we made cloud computing based real-time distributed-parallel-processing available in the cloud computing platform of the smart-city middleware developed in the previous research, so that we can perform real-time distributed-parallel-processing with them. This paper introduces a real-time distributed-parallel-processing method and system for stream reasoning with IoT big data transmitted from various sensors of Smart Cities and evaluate the performance of real-time distributed-parallel-processing of the system where the method is implemented.

Hadoop System Design for Big data Processing of RFID Distribution (RFID/NFC 물류의 빅 데이터 처리를 위한 하둡 시스템의 설계)

  • Kim, Nam-Ho;Noh, Jin-Heon;Jeong, Hee-Ja
    • Smart Media Journal
    • /
    • v.2 no.3
    • /
    • pp.47-53
    • /
    • 2013
  • Recently convergence of IT in logistics system as a typical application RFID/NFC technology is being used, such as, according to the distribution of the flow is generated by a lot of big data. The Hadoop distributed system to collect data items produced by the parallel processing capabilities of logistics information and logistics information for the record management can create. Hadoop system to support the design and development of prototypes were approaching the possibility of its utilization.

  • PDF

A Study on Development Project Management using Open Source Configuration Management and Defect Tracking Tools (오픈소스 형상도구와 결함추적도구를 이용한 개발 프로젝트 관리에 관한 연구)

  • Lee, Jihyun;Park, Youngsik
    • Annual Conference of KIPS
    • /
    • 2012.11a
    • /
    • pp.1445-1447
    • /
    • 2012
  • 팀으로 구성된 개발 조직은 주어진 시간 안에 가용할 수 있는 인력으로 고객에게 전달될 최종 산출물인 소프트웨어를 개발한다. 개발과 통합을 거친 소프트웨어는 테스트를 거치며 발생되는 결함에 대해 수정 조치한다. 하지만 결함 수정과 병행하여 초기 요구사항은 변경되고 추가되어 개발되는 소프트웨어에 반영되야 함으로 팀에서는 메일, 전화, 또는 게시판 방식으로 결함 사항을 알리고 최종 소프트웨어를 개발해 나간다. 본 논문은 이러한 환경에 결함추적도구를 사용하여 결함 발견 시 등록하고, 해결된 소프트웨어의 버전을 관리하기 위한 방법에 대해 오픈소스 기반의 형상관리도구와 결함추적도구를 활용한 환경을 연구해 나가고자 한다.