• Title/Summary/Keyword: large data visualization

Search Result 240, Processing Time 0.022 seconds

Big Data Smoothing and Outlier Removal for Patent Big Data Analysis

  • Choi, JunHyeog;Jun, Sunghae
    • Journal of the Korea Society of Computer and Information
    • /
    • v.21 no.8
    • /
    • pp.77-84
    • /
    • 2016
  • In general statistical analysis, we need to make a normal assumption. If this assumption is not satisfied, we cannot expect a good result of statistical data analysis. Most of statistical methods processing the outlier and noise also need to the assumption. But the assumption is not satisfied in big data because of its large volume and heterogeneity. So we propose a methodology based on box-plot and data smoothing for controling outlier and noise in big data analysis. The proposed methodology is not dependent upon the normal assumption. In addition, we select patent documents as target domain of big data because patent big data analysis is a important issue in management of technology. We analyze patent documents using big data learning methods for technology analysis. The collected patent data from patent databases on the world are preprocessed and analyzed by text mining and statistics. But the most researches about patent big data analysis did not consider the outlier and noise problem. This problem decreases the accuracy of prediction and increases the variance of parameter estimation. In this paper, we check the existence of the outlier and noise in patent big data. To know whether the outlier is or not in the patent big data, we use box-plot and smoothing visualization. We use the patent documents related to three dimensional printing technology to illustrate how the proposed methodology can be used for finding the existence of noise in the searched patent big data.

Anomaly Detection Analysis using Repository based on Inverted Index (역방향 인덱스 기반의 저장소를 이용한 이상 탐지 분석)

  • Park, Jumi;Cho, Weduke;Kim, Kangseok
    • Journal of KIISE
    • /
    • v.45 no.3
    • /
    • pp.294-302
    • /
    • 2018
  • With the emergence of the new service industry due to the development of information and communication technology, cyber space risks such as personal information infringement and industrial confidentiality leakage have diversified, and the security problem has emerged as a critical issue. In this paper, we propose a behavior-based anomaly detection method that is suitable for real-time and large-volume data analysis technology. We show that the proposed detection method is superior to existing signature security countermeasures that are based on large-capacity user log data according to in-company personal information abuse and internal information leakage. As the proposed behavior-based anomaly detection method requires a technique for processing large amounts of data, a real-time search engine is used, called Elasticsearch, which is based on an inverted index. In addition, statistical based frequency analysis and preprocessing were performed for data analysis, and the DBSCAN algorithm, which is a density based clustering method, was applied to classify abnormal data with an example for easy analysis through visualization. Unlike the existing anomaly detection system, the proposed behavior-based anomaly detection technique is promising as it enables anomaly detection analysis without the need to set the threshold value separately, and was proposed from a statistical perspective.

Comparative Evaluation on Geotechnical Information 3D Visualization Program for Dredging Quantity Estimation (준설 물량 산출을 위한 지반정보 3차원 가시화 프로그램 비교 평가)

  • Lee, Boyoung;Hwang, Bumsik;Kim, Han-Saem;Cho, Wanjei
    • Journal of the Korean GEO-environmental Society
    • /
    • v.17 no.7
    • /
    • pp.35-42
    • /
    • 2016
  • There are many reclamation projects domestically and internationally which requires large quantity of reclaimable materials. To provide enough reclaimable soils which are limited in land, there have been various research focusing on the dredged soils in the marine environments. As a part of this research, a GIS based 3D dredging reclamation visualization program was developed for the volume estimation of dredged soils in 2015. The developed program is based on the digitized spatial information of the site investigation data with a consideration of the reliability of the data. Prior to the validation with the comparisons with the actual dredged volume measurement data, the developed program was compared with the commercial 3D visualization program with 3D visualized results from the test site near the Gunjang harbor. The validation of the developed program was performed in terms of the degree of visualized precision, the sectional and profiling of soil layers and the dredged volume estimation results. Based on the comparisons, both commercial and developed program show similar dredged volume with minor discrepancies in soil layers.

Development of an Automatic Generation Methodology for Digital Elevation Models using a Two-Dimensional Digital Map (수치지형도를 이용한 DEM 자동 생성 기법의 개발)

  • Park, Chan-Soo;Lee, Seong-Kyu;Suh, Yong-Cheol
    • Journal of the Korean Association of Geographic Information Studies
    • /
    • v.10 no.3
    • /
    • pp.113-122
    • /
    • 2007
  • The rapid growth of aerial survey and remote sensing technology has enabled the rapid acquisition of very large amounts of geographic data, which should be analyzed using real-time visualization technology. The level of detail(LOD) algorithm is one of the most important elements for realizing real-time visualization. We chose the triangulated irregular network (TIN) method to generate normalized digital elevation model(DEM) data. First, we generated TIN data using contour lines obtained from a two-dimensional(2D) digital map and created a 2D grid array fitting the size of the area. Then, we generated normalized DEM data by calculating the intersection points between the TIN data and the points on the 2D grid array. We used constrained Delaunay triangulation(CDT) and ray-triangle intersection algorithms to calculate the intersection points between the TIN data and the points on the 2D grid array in each step. In addition, we simulated a three-dimensional(3D) terrain model based on normalized DEM data with real-time visualization using a Microsoft Visual C++ 6.0 program in the DirectX API library and a quad-tree LOD algorithm.

  • PDF

A Study to Hierarchical Visualization of Firewall Access Control Policies (방화벽 접근정책의 계층적 가시화 방법에 대한 연구)

  • Kim, Tae-yong;Kwon, Tae-woong;Lee, Jun;Lee, Youn-su;Song, Jung-suk
    • Journal of the Korea Institute of Information Security & Cryptology
    • /
    • v.30 no.6
    • /
    • pp.1087-1101
    • /
    • 2020
  • Various security devices are used to protect internal networks and valuable information from rapidly evolving cyber attacks. Firewall, which is the most commonly used security device, tries to prevent malicious attacks based on a text-based filtering rule (i.e., access control policy), by allowing or blocking access to communicate between inside and outside environments. However, in order to protect a valuable internal network from large networks, it has no choice but to increase the number of access control policy. Moreover, the text-based policy requires time-consuming and labor cost to analyze various types of vulnerabilities in firewall. To solve these problems, this paper proposes a 3D-based hierarchical visualization method, for intuitive analysis and management of access control policy. In particular, by providing a drill-down user interface through hierarchical architecture, Can support the access policy analysis for not only comprehensive understanding of large-scale networks, but also sophisticated investigation of anomalies. Finally, we implement the proposed system architecture's to verify the practicality and validity of the hierarchical visualization methodology, and then attempt to identify the applicability of firewall data analysis in the real-world network environment.

DNA Sequence Visualization with k-convex Hull (k-convex hull을 이용한 DNA 염기 배열의 가시화)

  • Kim, Min Ah;Lee, Eun Jeong;Cho, Hwan Gyu
    • Journal of the Korea Computer Graphics Society
    • /
    • v.2 no.2
    • /
    • pp.61-68
    • /
    • 1996
  • In this paper we propose a new visualization technique to characterize qualitative information of a large DNA sequence. While a long DNA sequence has huge information, it is not easy to obtain genetic information from the DNA sequence. We transform DNA sequences into a polygon to compute their homology in image domain rather than text domain. Our program visualizes DNA sequences with colored random walk plots and simplify them k-convex hulls. A random walk plot represents DNA sequence as a curve in a plane. A k-convex hull simplifies a random work plot by removing some parts of its insignificant information. This technique gives a biologist an insight to detect and classify DNA sequences with easy. Experiments with real genome data proves our approach gives a good visual forms for long DNA sequences for homology analysis.

  • PDF

A Terrain Rendering Method using Roughness Map and Bias Map (거칠기맵과 편향맵을 이용한 지형 렌더링 가법)

  • Lee, Eun-Seok;Jo, In-Woo;Shin, Byeong-Seok
    • Journal of the Korea Computer Graphics Society
    • /
    • v.17 no.2
    • /
    • pp.1-9
    • /
    • 2011
  • In recent researches, several LOD techniques are used for real-time visualization of large sized terrain data. However, during mesh simplification, geometry popping may occur in consecutive frames, because of the geometric error. We propose an efficient method for reducing the geometry popping using roughness map and bias map. A roughness map and a bias map are used to move vertices of the terrain mesh to appropriate position where they minimize the geometry errors. A roughness map and a bias map are represented as a texture suitable for GPU processing. Moving vertices using bias map is processed on the GPU, so the high-speed visualization can be possible.

GIS Application for Planning Roadway Construction (도로 공사의 시공계획을 위한 GIS의 적용)

  • Kang Sang-Hyeok;Seo Jong-Won
    • Proceedings of the Korean Institute Of Construction Engineering and Management
    • /
    • autumn
    • /
    • pp.565-568
    • /
    • 2003
  • Roadway construction planning processes involve a large amount of information on design, construction methods, quantities, unit costs, and production rates. GIS (Geographic Information System) is a strong tool for integration and managing various types of information such as spatial and non-spatial data required for roadway construction planning. This paper proposes a GIS-based system for improving roadway construction planning with its 'Spatial Analysis' and 'Visualization' functions. The proposed system cail help construction planner make a proper decision in a unique way by integrating design information and construction information within the system and creating design element modules for space scheduling purposes in real-time with its 'Interactive Planning' function.

  • PDF

PIV Measurements of Ventilation Flow from the Air Vent of a Real Passenger Car (거대 화상용 PIV 시스템을 이용한 실차 내부 공기벨트 토출흐름의 속도장 측정 연구)

  • Lee, Jin-Pyung;Kim, Hak-Lim;Lee, Sang-Joon
    • Journal of the Korean Society of Visualization
    • /
    • v.7 no.1
    • /
    • pp.3-8
    • /
    • 2009
  • Most vehicles have a heating, ventilating and air conditioning (HVAC) device to control the thermal condition and to make comfortable environment in the passenger compartment. The improvement of ventilation flow inside the passenger compartment is crucial for providing comfortable environment. For this, better understanding on the variation of flow characteristics of ventilation air inside the passenger compartment with respect to various ventilation modes is strongly required. Most previous studies on the ventilation flow in a car cabin were carried out using computational fluid dynamics (CFD) analysis or scale-down water-model experiments. In this study, whole ventilation flow discharged from the air vent of a real passenger car was measured using a special PIV (particle image velocimetry) system for large-size FOV (field of view). Under real recirculation ventilation condition, the spatial distributions of stream-wise turbulence intensity and mean velocity were measured in the vortical panel-duct center plane under the panel ventilation mode. These experimental data would be useful for understanding the detailed flow structure of real ventilation flow and validating numerical predictions.

Effects of Synthetic Turbulent Boundary Layer on Fluctuating Pressure on the Wall (합성난류경계층이 벽면에서의 변동압력에 미치는 영향)

  • Yi, Y.W.;Lee, D.S.;Shin, K.K.;Hong, C.S.;Lim, H.C.
    • Journal of the Korean Society of Visualization
    • /
    • v.19 no.3
    • /
    • pp.92-98
    • /
    • 2021
  • Large Eddy Simulation (LES) has been popularly applied and used in the last several decades to simulate turbulent boundary layer in the numerical domain. A fully developed turbulent boundary layer has also been applied to predict the complicated wake flow behind bluff bodies. In this study we aimed to generate an artificial turbulent boundary layer, which is based on an exponential correlation function, and generates a series of realistic three-dimensional velocity data in two-dimensional inlet section which are correlated both in space and in time. The results suggest its excellent capability for high Reynolds number flows. To make an effective generation, a hexahedral mesh has been used and Cholesky decomposition was applied to possess suitable turbulent statistics such as the randomness and correlation of turbulent flow. As a result, the flow characteristics in the domain and fluctuating pressure near the wall are very close to those of fully developed turbulent boundary layers.