• Title/Summary/Keyword: anomaly detection algorithm

Search Result 163, Processing Time 0.026 seconds

A Pre-processing Process Using TadGAN-based Time-series Anomaly Detection (TadGAN 기반 시계열 이상 탐지를 활용한 전처리 프로세스 연구)

  • Lee, Seung Hoon;Kim, Yong Soo
    • Journal of Korean Society for Quality Management
    • /
    • v.50 no.3
    • /
    • pp.459-471
    • /
    • 2022
  • Purpose: The purpose of this study was to increase prediction accuracy for an anomaly interval identified using an artificial intelligence-based time series anomaly detection technique by establishing a pre-processing process. Methods: Significant variables were extracted by applying feature selection techniques, and anomalies were derived using the TadGAN time series anomaly detection algorithm. After applying machine learning and deep learning methodologies using normal section data (excluding anomaly sections), the explanatory power of the anomaly sections was demonstrated through performance comparison. Results: The results of the machine learning methodology, the performance was the best when SHAP and TadGAN were applied, and the results in the deep learning, the performance was excellent when Chi-square Test and TadGAN were applied. Comparing each performance with the papers applied with a Conventional methodology using the same data, it can be seen that the performance of the MLR was significantly improved to 15%, Random Forest to 24%, XGBoost to 30%, Lasso Regression to 73%, LSTM to 17% and GRU to 19%. Conclusion: Based on the proposed process, when detecting unsupervised learning anomalies of data that are not actually labeled in various fields such as cyber security, financial sector, behavior pattern field, SNS. It is expected to prove the accuracy and explanation of the anomaly detection section and improve the performance of the model.

Development of Security Anomaly Detection Algorithms using Machine Learning (기계 학습을 활용한 보안 이상징후 식별 알고리즘 개발)

  • Hwangbo, Hyunwoo;Kim, Jae Kyung
    • The Journal of Society for e-Business Studies
    • /
    • v.27 no.1
    • /
    • pp.1-13
    • /
    • 2022
  • With the development of network technologies, the security to protect organizational resources from internal and external intrusions and threats becomes more important. Therefore in recent years, the anomaly detection algorithm that detects and prevents security threats with respect to various security log events has been actively studied. Security anomaly detection algorithms that have been developed based on rule-based or statistical learning in the past are gradually evolving into modeling based on machine learning and deep learning. In this study, we propose a deep-autoencoder model that transforms LSTM-autoencoder as an optimal algorithm to detect insider threats in advance using various machine learning analysis methodologies. This study has academic significance in that it improved the possibility of adaptive security through the development of an anomaly detection algorithm based on unsupervised learning, and reduced the false positive rate compared to the existing algorithm through supervised true positive labeling.

Keyed learning: An adversarial learning framework-formalization, challenges, and anomaly detection applications

  • Bergadano, Francesco
    • ETRI Journal
    • /
    • v.41 no.5
    • /
    • pp.608-618
    • /
    • 2019
  • We propose a general framework for keyed learning, where a secret key is used as an additional input of an adversarial learning system. We also define models and formal challenges for an adversary who knows the learning algorithm and its input data but has no access to the key value. This adversarial learning framework is subsequently applied to a more specific context of anomaly detection, where the secret key finds additional practical uses and guides the entire learning and alarm-generating procedure.

Effective Dimensionality Reduction of Payload-Based Anomaly Detection in TMAD Model for HTTP Payload

  • Kakavand, Mohsen;Mustapha, Norwati;Mustapha, Aida;Abdullah, Mohd Taufik
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.10 no.8
    • /
    • pp.3884-3910
    • /
    • 2016
  • Intrusion Detection System (IDS) in general considers a big amount of data that are highly redundant and irrelevant. This trait causes slow instruction, assessment procedures, high resource consumption and poor detection rate. Due to their expensive computational requirements during both training and detection, IDSs are mostly ineffective for real-time anomaly detection. This paper proposes a dimensionality reduction technique that is able to enhance the performance of IDSs up to constant time O(1) based on the Principle Component Analysis (PCA). Furthermore, the present study offers a feature selection approach for identifying major components in real time. The PCA algorithm transforms high-dimensional feature vectors into a low-dimensional feature space, which is used to determine the optimum volume of factors. The proposed approach was assessed using HTTP packet payload of ISCX 2012 IDS and DARPA 1999 dataset. The experimental outcome demonstrated that our proposed anomaly detection achieved promising results with 97% detection rate with 1.2% false positive rate for ISCX 2012 dataset and 100% detection rate with 0.06% false positive rate for DARPA 1999 dataset. Our proposed anomaly detection also achieved comparable performance in terms of computational complexity when compared to three state-of-the-art anomaly detection systems.

A New Semantic Kernel Function for Online Anomaly Detection of Software

  • Parsa, Saeed;Naree, Somaye Arabi
    • ETRI Journal
    • /
    • v.34 no.2
    • /
    • pp.288-291
    • /
    • 2012
  • In this letter, a new online anomaly detection approach for software systems is proposed. The novelty of the proposed approach is to apply a new semantic kernel function for a support vector machine (SVM) classifier to detect fault-suspicious execution paths at runtime in a reasonable amount of time. The kernel uses a new sequence matching algorithm to measure similarities among program execution paths in a customized feature space whose dimensions represent the largest common subpaths among the execution paths. To increase the precision of the SVM classifier, each common subpath is given weights according to its ability to discern executions as correct or anomalous. Experiment results show that compared with the known kernels, the proposed SVM kernel will improve the time overhead of online anomaly detection by up to 170%, while improving the precision of anomaly alerts by up to 140%.

Research on Data Tuning Methods to Improve the Anomaly Detection Performance of Industrial Control Systems (산업제어시스템의 이상 탐지 성능 개선을 위한 데이터 보정 방안 연구)

  • JUN, SANGSO;Lee, Kyung-ho
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.32 no.4
    • /
    • pp.691-708
    • /
    • 2022
  • As the technology of machine learning and deep learning became common, it began to be applied to research on anomaly(abnormal) detection of industrial control systems. In Korea, the HAI dataset was developed and published to activate artificial intelligence research for abnormal detection of industrial control systems, and an AI contest for detecting industrial control system security threats is being conducted. Most of the anomaly detection studies have been to create a learning model with improved performance through the ensemble model method, which is applied either by modifying the existing deep learning algorithm or by applying it together with other algorithms. In this study, a study was conducted to improve the performance of anomaly detection with a post-processing method that detects abnormal data and corrects the labeling results, rather than the learning algorithm and data pre-processing process. Results It was confirmed that the results were improved by about 10% or more compared to the anomaly detection performance of the existing model.

An Online Response System for Anomaly Traffic by Incremental Mining with Genetic Optimization

  • Su, Ming-Yang;Yeh, Sheng-Cheng
    • Journal of Communications and Networks
    • /
    • v.12 no.4
    • /
    • pp.375-381
    • /
    • 2010
  • A flooding attack, such as DoS or Worm, can be easily created or even downloaded from the Internet, thus, it is one of the main threats to servers on the Internet. This paper presents an online real-time network response system, which can determine whether a LAN is suffering from a flooding attack within a very short time unit. The detection engine of the system is based on the incremental mining of fuzzy association rules from network packets, in which membership functions of fuzzy variables are optimized by a genetic algorithm. The incremental mining approach makes the system suitable for detecting, and thus, responding to an attack in real-time. This system is evaluated by 47 flooding attacks, only one of which is missed, with no false positives occurring. The proposed online system belongs to anomaly detection, not misuse detection. Moreover, a mechanism for dynamic firewall updating is embedded in the proposed system for the function of eliminating suspicious connections when necessary.

Network Anomaly Detection using Association Rule Mining in Network Packets (네트워크 패킷에 대한 연관 마이닝 기법을 적용한 네트워크 비정상 행위 탐지)

  • Oh, Sang-Hyun;Chang, Joong-Hyuk
    • Journal of Korea Society of Industrial Information Systems
    • /
    • v.14 no.3
    • /
    • pp.22-29
    • /
    • 2009
  • In previous work, anomaly-based intrusion detection techniques have been widely used to effectively detect various intrusions into a computer. This is because the anomaly-based detection techniques can effectively handle previously unknown intrusion methods. However, most of the previous work assumed that the normal network connections are fixed. For this reason, a new network connection may be regarded as an anomalous event. This paper proposes a new anomaly detection method based on an association-mining algorithm. The proposed method is composed of two phases: intra-packet association mining and inter-packet association mining. The performances of the proposed method are comparatively verified with JAM, which is a conventional representative intrusion detection method.

Modeling of Positive Selection for the Development of a Computer Immune System and a Self-Recognition Algorithm

  • Sim, Kwee-Bo;Lee, Dong-Wook
    • International Journal of Control, Automation, and Systems
    • /
    • v.1 no.4
    • /
    • pp.453-458
    • /
    • 2003
  • The anomaly-detection algorithm based on negative selection of T cells is representative model among self-recognition methods and it has been applied to computer immune systems in recent years. In immune systems, T cells are produced through both positive and negative selection. Positive selection is the process used to determine a MHC receptor that recognizes self-molecules. Negative selection is the process used to determine an antigen receptor that recognizes antigen, or the nonself cell. In this paper, we propose a novel self-recognition algorithm based on the positive selection of T cells. We indicate the effectiveness of the proposed algorithm by change-detection simulation of some infected data obtained from cell changes and string changes in the self-file. We also compare the self-recognition algorithm based on positive selection with the anomaly-detection algorithm.

ANOMALY DETECTION FOR AN ORAL HEALTH CARE APPLICATION USING ONE CLASS YOLOV3

  • JAEHUN, BAEK;SEUNGWON, KIM;DONGWOOK, SHIN
    • Journal of the Korean Society for Industrial and Applied Mathematics
    • /
    • v.26 no.4
    • /
    • pp.310-322
    • /
    • 2022
  • In this report, we apply an anomaly detection algorithm to a mobile oral health care application. In particular, we have investigated one class YOLOv3 as an anomaly detection model to classify pictures of mouths which will be used as inputs in the following machine learning model. We have achieved outstanding performances by proposing appropriate annotation strategies for our data sets and modifying the loss function. Moreover, the model can classify not only oral and non-oral pictures but also output preprocessed pictures that only contain the area around the lips by using the predicted bounding box. Thus, the model performs prediction and preprocessing simultaneously.