• Title/Summary/Keyword: Big data Processing

Search Result 1,063, Processing Time 0.024 seconds

A method for optimizing lifetime prediction of a storage device using the frequency of occurrence of defects in NAND flash memory (낸드 플래시 메모리의 불량 발생빈도를 이용한 저장장치의 수명 예측 최적화 방법)

  • Lee, Hyun-Seob
    • Journal of Internet of Things and Convergence
    • /
    • v.7 no.4
    • /
    • pp.9-14
    • /
    • 2021
  • In computing systems that require high reliability, the method of predicting the lifetime of a storage device is one of the important factors for system management because it can maximize usability as well as data protection. The life of a solid state drive (SSD) that has recently been used as a storage device in several storage systems is linked to the life of the NAND flash memory that constitutes it. Therefore, in a storage system configured using an SSD, a method of accurately and efficiently predicting the lifespan of a NAND flash memory is required. In this paper, a method for optimizing the lifetime prediction of a flash memory-based storage device using the frequency of NAND flash memory failure is proposed. For this, we design a cost matrix to collect the frequency of defects that occur when processing data in units of Drive Writes Per Day (DWPD). In addition, a method of predicting the remaining cost to the slope where the life-long finish occurs using the Gradient Descent method is proposed. Finally, we proved the excellence of the proposed idea when any defect occurs with simulation.

An Efficient SLC Transition Method for Improving Defect Rate and Longer Lifetime on Flash Memory (플래시 메모리 상에서 불량률 개선 및 수명 연장을 위한 효율적인 단일 비트 셀 전환 기법)

  • Hyun-Seob Lee
    • Journal of Internet of Things and Convergence
    • /
    • v.9 no.3
    • /
    • pp.81-86
    • /
    • 2023
  • SSD (solid state disk), which is flash memory-based storage device, has the advantages of high density and fast data processing. Therefore, it is being utilized as a storage device for high-capacity data storage systems that manage rapidly increasing big data. However, flash memory, a storage media, has a physical limitation that when the write/erase operation is repeated more than a certain number of times, the cells are worn out and can no longer be used. In this paper, we propose a method for converting defective multi-bit cells into single-bit cells to reduce the defect rate of flash memory and extend its lifetime. The proposed idea distinguishes the defects and treatment methods of multi-bit cells and single-bit cells, which have different physical characteristics but are treated as the same defect, and converts the expected defective multi-bit cells into single-bit cells to improve the defect rate and extend the overall lifetime. Finally, we demonstrate the effectiveness of our proposed idea by measuring the increased lifetime of SSD through simulations.

An Effectiveness Verification for Evaluating the Amount of WTCI Tongue Coating Using Deep Learning (딥러닝을 이용한 WTCI 설태량 평가를 위한 유효성 검증)

  • Lee, Woo-Beom
    • Journal of the Institute of Convergence Signal Processing
    • /
    • v.20 no.4
    • /
    • pp.226-231
    • /
    • 2019
  • A WTCI is an important criteria for evaluating an mount of patient's tongue coating in tongue diagnosis. However, Previous WTCI tongue coating evaluation methods is a most of quantitatively measuring ration of the extracted tongue coating region and tongue body region, which has a non-objective measurement problem occurring by exposure conditions of tongue image or the recognition performance of tongue coating. Therefore, a WTCI based on deep learning is proposed for classifying an amount of tonger coating in this paper. This is applying the AI deep learning method using big data. to WTCI for evaluating an amount of tonger coating. In order to verify the effectiveness performance of the deep learning in tongue coating evaluating method, we classify the 3 types class(no coating, some coating, intense coating) of an amount of tongue coating by using CNN model. As a results by testing a building the tongue coating sample images for learning and verification of CNN model, proposed method is showed 96.7% with respect to the accuracy of classifying an amount of tongue coating.

Delay Analysis for Seamless Connections in Interworking between MPOA Networks and MPLS Networks (MPOA망과 MPLS 망 연동시 심리스 연결에 대한 지연 분석)

  • Kim, Dong-Ho;Lee, Soong-Hee;Jeon, Hyung-Goo
    • Journal of KIISE:Information Networking
    • /
    • v.29 no.2
    • /
    • pp.147-155
    • /
    • 2002
  • Seamless connection can be a good solution to reduce processing delays in interworking points, a possible big burden to the data transmission in Internet. This paper, therefore, will present several interworking methods which can minimize the delays through seamless connections in interworking between MPOA networks, usually ATM-LANs, and MPLS networks, mainly Internet backbones. We compare characteristics of those networks, propose requirements for interworking, and describe three Interworking methods. Proposed Interworking methods reduce end-to-end transmission delay for seamless connection between ATM VC and LSP, as decreases the number of IP lookup processing in LER. We describe detailed process and characteristic of each interworking method, And we analyze and compare end-to-end delay, using Jackson's network theorem, of proposed interworking methods.

Deobfuscation Processing and Deep Learning-Based Detection Method for PowerShell-Based Malware (파워쉘 기반 악성코드에 대한 역난독화 처리와 딥러닝 기반 탐지 방법)

  • Jung, Ho-jin;Ryu, Hyo-gon;Jo, Kyu-whan;Lee, Sangkyun
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.32 no.3
    • /
    • pp.501-511
    • /
    • 2022
  • In 2021, ransomware attacks became popular, and the number is rapidly increasing every year. Since PowerShell is used as the primary ransomware technique, the need for PowerShell-based malware detection is ever increasing. However, the existing detection techniques have limits in that they cannot detect obfuscated scripts or require a long processing time for deobfuscation. This paper proposes a simple and fast deobfuscation method and a deep learning-based classification model that can detect PowerShell-based malware. Our technique is composed of Word2Vec and a convolutional neural network to learn the meaning of a script extracting important features. We tested the proposed model using 1400 malicious codes and 8600 normal scripts provided by the AI-based PowerShell malicious script detection track of the 2021 Cybersecurity AI/Big Data Utilization Contest. Our method achieved 5.04 times faster deobfuscation than the existing methods with a perfect success rate and high detection performance with FPR of 0.01 and TPR of 0.965.

Real2Animation: A Study on the application of deepfake technology to support animation production (Real2Animation:애니메이션 제작지원을 위한 딥페이크 기술 활용 연구)

  • Dongju Shin;Bongjun Choi
    • Journal of the Institute of Convergence Signal Processing
    • /
    • v.23 no.3
    • /
    • pp.173-178
    • /
    • 2022
  • Recently, various computing technologies such as artificial intelligence, big data, and IoT are developing. In particular, artificial intelligence-based deepfake technology is being used in various fields such as the content and medical industry. Deepfake technology is a combination of deep learning and fake, and is a technology that synthesizes a person's face or body through deep learning, which is a core technology of AI, to imitate accents and voices. This paper uses deepfake technology to study the creation of virtual characters through the synthesis of animation models and real person photos. Through this, it is possible to minimize various cost losses occurring in the animation production process and support writers' work. In addition, as deepfake open source spreads on the Internet, many problems emerge, and crimes that abuse deepfake technology are prevalent. Through this study, we propose a new perspective on this technology by applying the deepfake technology to children's material rather than adult material.

Design and Implementation of Distributed Cluster Supporting Dynamic Down-Scaling of the Cluster (노드의 동적 다운 스케일링을 지원하는 분산 클러스터 시스템의 설계 및 구현)

  • Woo-Seok Ryu
    • The Journal of the Korea institute of electronic communication sciences
    • /
    • v.18 no.2
    • /
    • pp.361-366
    • /
    • 2023
  • Apache Hadoop, a representative framework for distributed processing of big data, has the advantage of increasing cluster size up to thousands of nodes to improve parallel distributed processing performance. However, reducing the size of the cluster is limited to the extent of permanently decommissioning nodes with defects or degraded performance, so there are limitations to operate multiple nodes flexibly in small clusters. In this paper, we discuss the problems that occur when removing nodes from the Hadoop cluster and propose a dynamic down-scaling technique to manage the distributed cluster more flexibly. To do this, we design and implement a modified Hadoop system and interfaces to support dynamic down-scaling of the cluster which supports temporary pause of a node and reconnection of it when necessary, rather than decommissioning the node when removing a node from the Hadoop cluster. We have verified that effective downsizing can be performed without performance degradation based on experimental results.

A Study on Drone Nozzle Design for Greenhouse Shading (온실차광을 위한 드론 전용노즐 설계에 관한 연구)

  • Ungjin Oh;Jin-Taek Lim
    • Journal of the Institute of Convergence Signal Processing
    • /
    • v.24 no.4
    • /
    • pp.249-254
    • /
    • 2023
  • Recently, the distribution of drones is being activated by saving farmers' working time and protecting them from harmful human bodies from pesticides due to the mission of spraying pesticides using drones. It is possible to compensate for various shortcomings derived from the existing pesticide spraying method, wide-area control and helicopter control. Recently, the smart farm expansion policy has actively used it to generate profits for farmers by increasing harvests by monitoring growth information of various crops based on IoT in real time and collecting big data on key variables, and related drone industry technologies are also being developed. In this study, drones were applied to the work of shading greenhouses to secure diversity in agricultural application fields, and basic research on the greenhouse environment was conducted to materialize the technology related to shading. In order to provide high-quality light in consideration of the internal and external environment of the green house, basic research was conducted to enable light-shielding missions using drones through nozzle design for uniform spraying of nozzles of drones, light-transmitting rate analysis of green houses, and light-shielding agent application experiments.

The Development of a Energy Monitoring System based on Data Collected from Food Factories (식품공장 수집 데이터 기반 에너지 모니터링 시스템 개발)

  • Chae-Eun Yeo;Woo-jin Cho;Jae-Hoi Gu
    • The Journal of the Convergence on Culture Technology
    • /
    • v.9 no.6
    • /
    • pp.1001-1006
    • /
    • 2023
  • Globally, rising energy costs and increased energy demand are important issues for the food processing and manufacturing industries, which consume significant amounts of energy throughout the supply chain. Accordingly, there is a need for the development of a real-time energy monitoring and analysis system that can optimize energy use. In this study, a food factory energy monitoring system was proposed based on IoT installed in a food factory, including monitoring of each facility, energy supply and usage monitoring for the heat treatment process, and search functions. The system is based on the IoT sensor of the food processing plant and consists of PLC, database server, OPC-UA server, UI server, API server, and CIMON's HMI. The proposed system builds big data for food factories and provides facility-specific monitoring through collection functions, as well as energy supply and usage monitoring and search service functions for the heat treatment process. This data collection-based energy monitoring system will serve as a guide for the development of a small and medium-sized factory energy monitoring and management system for energy savings. In the future, this system can be used to identify and analyze energy usage to create quantitative energy saving measures that optimize process work.

Using noise filtering and sufficient dimension reduction method on unstructured economic data (노이즈 필터링과 충분차원축소를 이용한 비정형 경제 데이터 활용에 대한 연구)

  • Jae Keun Yoo;Yujin Park;Beomseok Seo
    • The Korean Journal of Applied Statistics
    • /
    • v.37 no.2
    • /
    • pp.119-138
    • /
    • 2024
  • Text indicators are increasingly valuable in economic forecasting, but are often hindered by noise and high dimensionality. This study aims to explore post-processing techniques, specifically noise filtering and dimensionality reduction, to normalize text indicators and enhance their utility through empirical analysis. Predictive target variables for the empirical analysis include monthly leading index cyclical variations, BSI (business survey index) All industry sales performance, BSI All industry sales outlook, as well as quarterly real GDP SA (seasonally adjusted) growth rate and real GDP YoY (year-on-year) growth rate. This study explores the Hodrick and Prescott filter, which is widely used in econometrics for noise filtering, and employs sufficient dimension reduction, a nonparametric dimensionality reduction methodology, in conjunction with unstructured text data. The analysis results reveal that noise filtering of text indicators significantly improves predictive accuracy for both monthly and quarterly variables, particularly when the dataset is large. Moreover, this study demonstrated that applying dimensionality reduction further enhances predictive performance. These findings imply that post-processing techniques, such as noise filtering and dimensionality reduction, are crucial for enhancing the utility of text indicators and can contribute to improving the accuracy of economic forecasts.