• Title/Summary/Keyword: anomaly detection algorithm

Search Result 163, Processing Time 0.027 seconds

Density-based Outlier Detection for Very Large Data (대용량 자료 분석을 위한 밀도기반 이상치 탐지)

  • Kim, Seung;Cho, Nam-Wook;Kang, Suk-Ho
    • Journal of the Korean Operations Research and Management Science Society
    • /
    • v.35 no.2
    • /
    • pp.71-88
    • /
    • 2010
  • A density-based outlier detection such as an LOF (Local Outlier Factor) tries to find an outlying observation by using density of its surrounding space. In spite of several advantages of a density-based outlier detection method, the computational complexity of outlier detection has been one of major barriers in its application. In this paper, we present an LOF algorithm that can reduce computation time of a density based outlier detection algorithm. A kd-tree indexing and approximated k-nearest neighbor search algorithm (ANN) are adopted in the proposed method. A set of experiments was conducted to examine performance of the proposed algorithm. The results show that the proposed method can effectively detect local outliers in reduced computation time.

A Real-Time Network Traffic Anomaly Detection Scheme Using NetFlow Data (NetFlow 데이터를 이용한 실시간 네트워크 트래픽 어노멀리 검출 기법)

  • Kang Koo-Hong;Jang Jong-Soo;Kim Ki-Young
    • The KIPS Transactions:PartC
    • /
    • v.12C no.1 s.97
    • /
    • pp.19-28
    • /
    • 2005
  • Recently, it has been sharply increased the interests to detect the network traffic anomalies to help protect the computer network from unknown attacks. In this paper, we propose a new anomaly detection scheme using the simple linear regression analysis for the exported LetFlow data, such as bits per second and flows per second, from a border router at a campus network. In order to verify the proposed scheme, we apply it to a real campus network and compare the results with the Holt-Winters seasonal algorithm. In particular, we integrate it into the RRDtooi for detecting the anomalies in real time.

Flow-based Anomaly Detection Using Access Behavior Profiling and Time-sequenced Relation Mining

  • Liu, Weixin;Zheng, Kangfeng;Wu, Bin;Wu, Chunhua;Niu, Xinxin
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • v.10 no.6
    • /
    • pp.2781-2800
    • /
    • 2016
  • Emerging attacks aim to access proprietary assets and steal data for business or political motives, such as Operation Aurora and Operation Shady RAT. Skilled Intruders would likely remove their traces on targeted hosts, but their network movements, which are continuously recorded by network devices, cannot be easily eliminated by themselves. However, without complete knowledge about both inbound/outbound and internal traffic, it is difficult for security team to unveil hidden traces of intruders. In this paper, we propose an autonomous anomaly detection system based on behavior profiling and relation mining. The single-hop access profiling model employ a novel linear grouping algorithm PSOLGA to create behavior profiles for each individual server application discovered automatically in historical flow analysis. Besides that, the double-hop access relation model utilizes in-memory graph to mine time-sequenced access relations between different server applications. Using the behavior profiles and relation rules, this approach is able to detect possible anomalies and violations in real-time detection. Finally, the experimental results demonstrate that the designed models are promising in terms of accuracy and computational efficiency.

Sensitivity Analysis and Optimization of Design Variables Related to an Algorithm for Loss of Balance Detection (균형상살 검출 알고리즘에서 검출과 관련된 설계변수의 민감도 해석 몇 최적화)

  • Ko, B.K.;Kim, K.H.;Son, K.
    • Journal of Biomedical Engineering Research
    • /
    • v.32 no.1
    • /
    • pp.7-14
    • /
    • 2011
  • This study suggested an optimized algorithm for detecting the loss of balance(LOB) in the seated position. And the sensitivity analysis was performed in order to identify the role of each design variable in the algorithm. The LOB algorithm consisted of data processing of measured signals, an internal model of the central nervous system and a control error anomaly(CEA) detector. This study optimized design variables of a CEA detector to obtain improved values of the success rate(SR) of detecting the LOB and the margin time(MT) provided for preventing the falling. Nine healthy adult volunteers were involved in the experiments. All the subjects were asked to balance their body in a predescribed seated posture with the rear legs of a four-legged wooden chair. The ground reaction force from the right leg was measured from the force plate while the accelerations of the chair and the head were measured from a couple of piezoelectric accelerometers. The measured data were processed to predict the LOB using a detection algorithm. Variables S2, h2 and hd are related to the detector: S2 represents a data selecting window, h2 a time shift and hd an operating period of the LOB detection algorithm. S2 was varied from 0.1 to 10 sec with an increment of 0.1 sec, and both h2 and hd were varied from 0.01 to 1.0 sec with an increment of 0.01 sec. It was found that the SR and MT were increased by up to 9.7% and 0.497 sec comparing with the previously published case when the values of S2, h2 and hd were set to 4.5, 0.3 and 0.2 sec, respectively. Also the results of sensitivity analysis showed that S2 and h2 had considerable influence on the SR while these variables were not so sensitive to the MT.

Course Variance Clustering for Traffic Route Waypoint Extraction

  • Onyango Shem Otoi
    • Proceedings of the Korean Institute of Navigation and Port Research Conference
    • /
    • 2022.06a
    • /
    • pp.277-279
    • /
    • 2022
  • Rapid Development and adoption of AIS as a survailance tool has resulted in widespread application of data analysis technology, in addition to AIS ship trajectory clustering. AIS data-based clustering has become an increasingly popular method for marine traffic pattern recognition, ship route prediction and anomaly detection in recent year. In this paper we propose a route waypoint extraction by clustering ships CoG variance trajectory using Density-Based Spatial Clustering of Application with Noise (DBSCAN) algorithm in both port approach channel and coastal waters. The algorithm discovers route waypoint effectively. The result of the study could be used in traffic route extraction, and more-so develop a maritime anomaly detection tool.

  • PDF

Anomaly-Based Network Intrusion Detection: An Approach Using Ensemble-Based Machine Learning Algorithm

  • Kashif Gul Chachar;Syed Nadeem Ahsan
    • International Journal of Computer Science & Network Security
    • /
    • v.24 no.1
    • /
    • pp.107-118
    • /
    • 2024
  • With the seamless growth of the technology, network usage requirements are expanding day by day. The majority of electronic devices are capable of communication, which strongly requires a secure and reliable network. Network-based intrusion detection systems (NIDS) is a new method for preventing and alerting computers and networks from attacks. Machine Learning is an emerging field that provides a variety of ways to implement effective network intrusion detection systems (NIDS). Bagging and Boosting are two ensemble ML techniques, renowned for better performance in the learning and classification process. In this paper, the study provides a detailed literature review of the past work done and proposed a novel ensemble approach to develop a NIDS system based on the voting method using bagging and boosting ensemble techniques. The test results demonstrate that the ensemble of bagging and boosting through voting exhibits the highest classification accuracy of 99.98% and a minimum false positive rate (FPR) on both datasets. Although the model building time is average which can be a tradeoff by processor speed.

Anomaly Detection Model Using THRE-KBANN (THRE-KBANN을 이용한 이상현상탐지모델)

  • Shim, Dong-Hee
    • Journal of the Institute of Electronics Engineers of Korea CI
    • /
    • v.38 no.5
    • /
    • pp.37-43
    • /
    • 2001
  • Since Internet has been used anywhere, illegal intrusion to a certain host or network become the ciritical factor in security. Although many anomaly detection models have been proposed using the statistical analysis, data mining, genetic algorithm/programming to detect illegal intrusions, these models has defects to detect new types of intrusions. THRE-KBANN (theory-refinement knowledge-based artificial neural network) which can learn continuously based on KBANN, is proposed for the anomaly detection model in this paper. The performance of this model is compared with that of the model based on data mining using the experimental data. The ability of continual learning for the detection of new types of intrusions is also evaluated.

  • PDF

Design and Implementation of Machine Learning System for Fine Dust Anomaly Detection based on Big Data (빅데이터 기반 미세먼지 이상 탐지 머신러닝 시스템 설계 및 구현)

  • Jae-Won Lee;Chi-Ho Lin
    • The Journal of the Institute of Internet, Broadcasting and Communication
    • /
    • v.24 no.1
    • /
    • pp.55-58
    • /
    • 2024
  • In this paper, we propose a design and implementation of big data-based fine dust anomaly detection machine learning system. The proposed is system that classifies the fine dust air quality index through meteorological information composed of fine dust and big data. This system classifies fine dust through the design of an anomaly detection algorithm according to the outliers for each air quality index classification categories based on machine learning. Depth data of the image collected from the camera collects images according to the level of fine dust, and then creates a fine dust visibility mask. And, with a learning-based fingerprinting technique through a mono depth estimation algorithm, the fine dust level is derived by inferring the visibility distance of fine dust collected from the monoscope camera. For experimentation and analysis of this method, after creating learning data by matching the fine dust level data and CCTV image data by region and time, a model is created and tested in a real environment.

Anomaly Detection Mechanism based on the Session Patterns and Fuzzy Cognitive Maps (퍼지인식도와 세션패턴 기반의 비정상 탐지 메커니즘)

  • Ryu Dae-Hee;Lee Se-Yul;Kim Hyeock-Jin;Song Young-Deog
    • Journal of the Korea Society of Computer and Information
    • /
    • v.10 no.6 s.38
    • /
    • pp.9-16
    • /
    • 2005
  • Recently, since the number of internet users is increasing rapidly and, by using the Public hacking tools, general network users can intrude computer systems easily, the hacking problem is setting more serious. In order to prevent the intrusion. it is needed to detect the sign in advance of intrusion in a Positive Prevention by detecting the various forms of hackers intrusion trials to know the vulnerability of systems. The existing network-based anomaly detection algorithms that cope with port-scanning and the network vulnerability scans have some weakness in intrusion detection. they can not detect slow scans and coordinated scans. therefore, the new concept of algorithm is needed to detect effectively the various. In this Paper, we propose a detection algorithm for session patterns and FCM.

  • PDF

A Study on Anomaly Detection based on User's Command Analysis (사용자 명령어 분석을 통한 비정상 행위 판정에 관한 연구)

  • 윤정혁;오상현;이원석
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.10 no.4
    • /
    • pp.59-71
    • /
    • 2000
  • Due to the advance of computer and communication technology, intrusions or crimes using a computer have been increased rapidly while various information has been provided to users conveniently. As a results, many studies are necessary to detect the activities of intruders effectively. In this paper, a new association algorithm for the anomaly detection model is proposed in the process of generating user\`s normal patterns. It is that more recently observed behavior get more affection on the process of data mining. In addition, by clustering generated normal patterns for each use or a group of similar users, it is possible to identify the usual frequency of programs or command usage for each user or a group of uses. The performance of the proposed anomaly detection system has been tested on various system Parameters in order to identify their practical ranges for maximizing its detection rate.