• Title/Summary/Keyword: DATA PRE-PROCESSING

Search Result 801, Processing Time 0.034 seconds

Effective Payload-based Anomaly Detection Method Using Pre-trained Model (사전학습 모델을 활용한 효과적인 Http Payload 이상 탐지 방법)

  • LEE, Unggi;KIM, Wonchul
    • Proceedings of the Korea Information Processing Society Conference
    • /
    • 2022.11a
    • /
    • pp.228-230
    • /
    • 2022
  • 딥러닝 기반의 인공지능 기술이 발달함에 따라 이상 탐지 방법에도 딥러닝이 적용되었다. 네트워크 트래픽으로부터 요약 및 집계된 Feature 를 학습하는 방법과 Packet 자체를 학습하는 등의 방법이 있었다. 그러나 모두 정보의 제한적으로 사용한다는 단점이 있었다. 본 연구에서는 Http Request에 대한 사전학습 기반의 효과적인 이상 탐지 방법을 제안한다. 사전학습에 고려되는 토큰화 방법, Padding 방법, Feature 결합 방법, Feature 선택 방법과 전이학습 시 Numerical 정보를 추가하는 방법을 소개하고 각 실험을 통해 최적의 방법을 제안한다.

Development of LiDAR Simulator for Backpack-mounted Mobile Indoor Mapping System

  • Chung, Minkyung;Kim, Changjae;Choi, Kanghyeok;Chung, DongKi;Kim, Yongil
    • Journal of the Korean Society of Surveying, Geodesy, Photogrammetry and Cartography
    • /
    • v.35 no.2
    • /
    • pp.91-102
    • /
    • 2017
  • Backpack-mounted mapping system is firstly introduced for flexible movement in indoor spaces where satellite-based localization is not available. With the achieved advances in miniaturization and weight reduction, use of LiDAR (Light Detection and Ranging) sensors in mobile platforms has been increasing, and indeed, they have provided high-precision information on indoor environments and their surroundings. Previous research on the development of backpack-mounted mapping systems, has concentrated mostly on the improvement of data processing methods or algorithms, whereas practical system components have been determined empirically. Thus, in the present study, a simulator for a LiDAR sensor (Velodyne VLP-16), was developed for comparison of the effects of diverse conditions on the backpack system and its operation. The simulated data was analyzed by visual inspection and comparison of the data sets' statistics, which differed according to the LiDAR arrangement and moving speed. Also, the data was used as input to a point-cloud registration algorithm, ICP (Iterative Closest Point), to validate its applicability as pre-analysis data. In fact, the results indicated centimeter-level accuracy, thus demonstrating the potentials of simulation data to be utilized as a tool for performance comparison of pointdata processing methods.

Establishment Status of the Korea Ocean Satellite Center and GOCI-Data Distribution System (해양위성센터 구축 현황 및 GOCI 자료배포시스템 소개)

  • Yang, Chan-Su;Bae, Sang-Soo;Han, Hee-Jeong;Cho, Seong-Ick;Ahn, Yu-Hwan
    • Proceedings of the KSRS Conference
    • /
    • 2009.03a
    • /
    • pp.367-370
    • /
    • 2009
  • 한국해양연구원에서는 2009년 발사 예정인 통신해양기상위성(COMS: Communication, Ocean and Meteorological Satellite)의 해색센서인 정지궤도 해양위성(GOCI: Geostationary Ocean Color Imager) 데이터의 수신, 처리, 배포를 위한 해양위성센터(KOSC: Korea Ocean Satellite Center)를 구축하고 있다. 2005년 "해양위성센터 구축사업"의 시작으로, 전파 수신 환경 등의 조건을 고려하여, 안산에 위치한 한국해양연구원 본원으로 해양위성센터의 위치를 최종 확정하여 구축을 진행하고 있다. 2009년 3월 현재 수신시스템(GDAS: GOCI Data Aquisition System), 자료전처리시스템(IMPS: Image Pre-processing System), 자료처리시스템(GDPS: GOCI Data Processing System), 자료관리 시스템(DMS: Data Management System), 통합감시제어시스템(TMC: Total Management & Controlling System), 기관간 자료교환시스템(EDES: External Data Exchange System) 등이 구축 완료되었고, 위성자료 배포시스템(DDS: Data Distribution System)을 구축하고 있다. 고용량 데이터의 원활한 전송을 위한 데이터센터를 비롯하여 사용자관점에서의 시스템 구축을 추진하고 있으며, 위성 발사 후 사용자 등록을 시작할 계획이다.

  • PDF

Implementation of CNN-based Masking Algorithm for Post Processing of Aerial Image

  • CHOI, Eunsoo;QUAN, Zhixuan;JUNG, Sangwoo
    • Korean Journal of Artificial Intelligence
    • /
    • v.9 no.2
    • /
    • pp.7-14
    • /
    • 2021
  • Purpose: To solve urban problems, empirical research is being actively conducted to implement a smart city based on various ICT technologies, and digital twin technology is needed to effectively implement a smart city. A digital twin is essential for the realization of a smart city. A digital twin is a virtual environment that intuitively visualizes multidimensional data in the real world based on 3D. Digital twin is implemented on the premise of the convergence of GIS and BIM, and in particular, a lot of time is invested in data pre-processing and labeling in the data construction process. In digital twin, data quality is prioritized for consistency with reality, but there is a limit to data inspection with the naked eye. Therefore, in order to improve the required time and quality of digital twin construction, it was attempted to detect a building using Mask R-CNN, a deep learning-based masking algorithm for aerial images. If the results of this study are advanced and used to build digital twin data, it is thought that a high-quality smart city can be realized.

Implementation of Slaving Data Processing Function for Mission Control System in Space Center (우주센터 발사통제시스템의 추적연동정보 처리기능 구현)

  • Choi, Yong-Tae;Ra, Sung-Woong
    • Journal of Korea Society of Industrial Information Systems
    • /
    • v.19 no.3
    • /
    • pp.31-39
    • /
    • 2014
  • In KSLV-I launch mission, real-time data from the tracking stations are acquired, processed and distributed by the Mission Control System to the user group who needed to monitor processed data for safety and flight monitoring purposes. The processed trajectory data by the mission control system is sent to each tracking system for target designation in case of tracking failure. Also, the processed data are used for decision making for flight termination when anomalies occur during flight of the launch vehicle. In this paper, we propose the processing mechanism of slaving data which plays a key role of launch vehicle tracking mission. The best position data is selected by predefined logic and current status after every available position data are acquired and pre-processed. And, the slaving data is distributed to each tracking stations through time delay is compensated by extrapolation. For the accurate processing, operation timing of every procesing modules are triggered by time-tick signal(25ms period) which is driven from UTC(Universial Time Coordinates) time. To evaluate the proposed method, we compared slaving data to the position data which received by tracking radar. The experiments show the average difference value is below 0.01 degree.

Comparative Analysis of CNN Deep Learning Model Performance Based on Quantification Application for High-Speed Marine Object Classification (고속 해상 객체 분류를 위한 양자화 적용 기반 CNN 딥러닝 모델 성능 비교 분석)

  • Lee, Seong-Ju;Lee, Hyo-Chan;Song, Hyun-Hak;Jeon, Ho-Seok;Im, Tae-ho
    • Journal of Internet Computing and Services
    • /
    • v.22 no.2
    • /
    • pp.59-68
    • /
    • 2021
  • As artificial intelligence(AI) technologies, which have made rapid growth recently, began to be applied to the marine environment such as ships, there have been active researches on the application of CNN-based models specialized for digital videos. In E-Navigation service, which is combined with various technologies to detect floating objects of clash risk to reduce human errors and prevent fires inside ships, real-time processing is of huge importance. More functions added, however, mean a need for high-performance processes, which raises prices and poses a cost burden on shipowners. This study thus set out to propose a method capable of processing information at a high rate while maintaining the accuracy by applying Quantization techniques of a deep learning model. First, videos were pre-processed fit for the detection of floating matters in the sea to ensure the efficient transmission of video data to the deep learning entry. Secondly, the quantization technique, one of lightweight techniques for a deep learning model, was applied to reduce the usage rate of memory and increase the processing speed. Finally, the proposed deep learning model to which video pre-processing and quantization were applied was applied to various embedded boards to measure its accuracy and processing speed and test its performance. The proposed method was able to reduce the usage of memory capacity four times and improve the processing speed about four to five times while maintaining the old accuracy of recognition.

Mining Quantitative Association Rules using Commercial Data Mining Tools (상용 데이타 마이닝 도구를 사용한 정량적 연관규칙 마이닝)

  • Kang, Gong-Mi;Moon, Yang-Sae;Choi, Hun-Young;Kim, Jin-Ho
    • Journal of KIISE:Databases
    • /
    • v.35 no.2
    • /
    • pp.97-111
    • /
    • 2008
  • Commercial data mining tools basically support binary attributes only in mining association rules, that is, they can mine binary association rules only. In general, however. transaction databases contain not only binary attributes but also quantitative attributes. Thus, in this paper we propose a systematic approach to mine quantitative association rules---association rules which contain quantitative attributes---using commercial mining tools. To achieve this goal, we first propose an overall working framework that mines quantitative association rules based on commercial mining tools. The proposed framework consists of two steps: 1) a pre-processing step which converts quantitative attributes into binary attributes and 2) a post-processing step which reconverts binary association rules into quantitative association rules. As the pre-processing step, we present the concept of domain partition, and based on the domain partition, we formally redefine the previous bipartition and multi-partition techniques, which are mean-based or median-based techniques for bipartition, and are equi-width or equi-depth techniques for multi-partition. These previous partition techniques, however, have the problem of not considering distribution characteristics of attribute values. To solve this problem, in this paper we propose an intuitive partition technique, named standard deviation minimization. In our standard deviation minimization, adjacent attributes are included in the same partition if the change of their standard deviations is small, but they are divided into different partitions if the change is large. We also propose the post-processing step that integrates binary association rules and reconverts them into the corresponding quantitative rules. Through extensive experiments, we argue that our framework works correctly, and we show that our standard deviation minimization is superior to other partition techniques. According to these results, we believe that our framework is practically applicable for naive users to mine quantitative association rules using commercial data mining tools.

PBFiltering: An Energy Efficient Skyline Query Processing Method using Priority-based Bottom-up Filtering in Wireless Sensor Networks (PBFiltering: 무선 센서 네트워크에서 우선순위 기반 상향식 필터링을 이용한 에너지 효율적인 스카이라인 질의 처리 기법)

  • Seong, Dong-Ook;Park, Jun-Ho;Kim, Hak-Sin;Park, Hyoung-Soon;Roh, Kyu-Jong;Yeo, Myung-Ho;Yoo, Jae-Soo
    • Journal of KIISE:Databases
    • /
    • v.36 no.6
    • /
    • pp.476-485
    • /
    • 2009
  • In sensor networks, many methods have been proposed to process in-network aggregation effectively. Unlike general aggregation queries, skyline query processing compares multi-dimensional data for the result. Therefore, it is very difficult to process the skyline queries in sensor networks. It is important to filter unnecessary data for energy-efficient skyline query processing. Existing approach like MFTAC restricts unnecessary data transitions by deploying filters to whole sensors. However, network lifetime is reduced by energy consumption for many false positive data and filters transmission. In this paper, we propose a bottom up filtering-based skyline query processing algorithm of in-network for reducing energy consumption by filters transmission and a PBFiltering technique for improving performance of filtering. The proposed algorithm creates the skyline filter table (SFT) in the data gathering process which sends from sensor nodes to the base station and filters out unnecessary transmissions using it. The experimental results show that our algorithm reduces false positives and improves the network lifetime over the existing method.

A synchronized processing algorithm of asynchronous data with trigger (트리거를 이용한 비동기 데이터의 동기화 처리 알고리즘 연구)

  • 박성진;유지상
    • The Journal of Korean Institute of Communications and Information Sciences
    • /
    • v.28 no.12A
    • /
    • pp.1002-1008
    • /
    • 2003
  • In terrestrial data broadcasting, we are just on the beginning stage in all aspects including implementation and design techniques and only asynchronous data processing has been receiving a little study. In this paper, we therefore propose an efficient processing algorithm for synchronization of asynchronous data by using trigger information to make more diverse service possible with a variety of contents. In the proposed algorithm, trigger data is encapsulated in DSM-CC section and transmitted in a form of MPEG-2 TS. The data is then separated in PC type set-top box and detached asynchronous data and trigger data are stored by the proposed algorithm. Pre-loaded asynchronous data is displayed when STC(system time clock) has the same value as PTS(presentation time stamp). Proper operation of the proposed algorithm was verified by using a content of asynchronous data with extensible markup language(XML) and a declarative application(DA) browser.

DEVELOPMENT STATUS OF THE DOTIFS DATA SIMULATOR AND THE REDUCTION PACKAGE

  • CHUNG, HAEUN;RAMAPRAKASH, A.N.;PARK, CHANGBOM
    • Publications of The Korean Astronomical Society
    • /
    • v.30 no.2
    • /
    • pp.675-677
    • /
    • 2015
  • A data simulator and reduction package for the Devasthal Optical Telescope Integral Field Spectrograph (DOTIFS) has been developed. Since data reduction for the Integral Field Spectrograph (IFS) requires complicated procedures due to the complex nature of the integral spectrograph, common reduction procedures are usually not directly applicable for such an instrument. Therefore, the development of an optimized package for the DOTIFS is required. The data simulator observes artificial object and simulates CCD images for the instrument considering various effects; e.g. atmosphere, sky background, transmission, spectrograph optics aberration, and detector noise. The data reduction package has been developed based on the outcomes from the DOTIFS data simulator. The reduction package includes the entire processes for the reduction; pre-processing, flat-fielding, and sky subtraction. It generates 3D data cubes as a final product, which users can use for science directly.