• 제목/요약/키워드: Data filtering

검색결과 1,777건 처리시간 0.028초

Handling Incomplete Data Problem in Collaborative Filtering System

  • Noh, Hyun-ju;Kwak, Min-jung;Han, In-goo
    • 한국산학기술학회:학술대회논문집
    • /
    • 한국산학기술학회 2003년도 Proceeding
    • /
    • pp.105-110
    • /
    • 2003
  • Collaborative filtering is one of the methodologies that are most widely used for recommendation system. It is based on a data matrix of each customer's preferences of products. There could be a lot of missing values in such preference. data matrix. This incomplete data is one of the reasons to deteriorate the accuracy of recommendation system. Multiple imputation method imputes m values for each missing value. It overcomes flaws of single imputation approaches through considering the uncertainty of missing values.. The objective of this paper is to suggest multiple imputation-based collaborative filtering approach for recommendation system to improve the accuracy in prediction performance. The experimental works show that the proposed approach provides better performance than the traditional Collaborative filtering approach, especially in case that there are a lot of missing values in dataset used for recommendation system.

  • PDF

Efficient Geographical Information-Based En-route Filtering Scheme in Wireless Sensor Networks

  • Yi, Chuanjun;Yang, Geng;Dai, Hua;Liu, Liang;Chen, Yunhua
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • 제12권9호
    • /
    • pp.4183-4204
    • /
    • 2018
  • The existing en-route filtering schemes only consider some simple false data injection attacks, which results in lower safety performance. In this paper, we propose an efficient geographical information-based en-route filtering scheme (EGEFS), in which each forwarding node verifies not only the message authentication codes (MACs), but also the report identifier and the legitimacy and authenticity of locations carried in a data report. Thus, EGEFS can defend against not only the simple false data injection attacks and the replay attack, but also the collusion attack with forged locations proposed in this paper. In addition, we propose a new method for electing the center-of-stimulus (CoS) node, which can ensure that only one detecting node will be elected as the CoS node to generate one data report for an event. The simulation results show that, compared to the existing en-route filtering schemes, EGEFS has higher safety performance, because it can resist more types of false data injection attacks, and it also has higher filtering efficiency and lower energy expenditure.

무선 센서 네트워크 모니터링 시스템을 위한 데이터 통계 분석 기반 데이터 필터링 기법 (Data Statical Analysis based Data Filtering Scheme for Monitoring System on Wireless Sensor Network)

  • 이현조;최영호;장재우
    • 한국콘텐츠학회논문지
    • /
    • 제10권3호
    • /
    • pp.53-63
    • /
    • 2010
  • 최근 무선 센서 네트워크를 활용한 다양한 모니터링 시스템 구축이 활발히 이루어지고 있다. 무선 센서네트워크 기반 모니터링 시스템 구축을 위해서, 세 가지 사항을 고려해야 한다. 첫째, 지속적인 모니터링을 위해서, 노드 실패 감지 기법이 요구된다. 둘째, 센서노드는 제한된 배터리 용량을 지니기 때문에, 에너지 소모량 감소를 위한 효율적인 데이터 필터링 기법이 요구된다. 마지막으로 데이터 필터링 수행 시, 계산오버헤드를 감소시키는 기법이 필요하다. 기존 칼만 데이터 필터링 기법은 우수한 필터링 성능을 나타내는 반면, 데이터 예측값 계산과정이 복잡하여 센서 노드에서의 계산 오버헤드가 증가하는 단점이 존재한다. 이러한 문제점들을 해결하기 위하여 본 논문에서는 데이터 통계 분석 기반 데이터 필터링 기법을 제안한다. 제안하는 기법은 첫째, 노드 실패 감지를 지원하기 위해, 주기적으로 노드 생존 메시지를 수집한다. 둘째, 불필요한 전송 메시지 수 감소를 위하여 샘플 데이터를 노드 생존 메시지에 포함하여 전송하고, 수집된 샘플 데이터 집합을 바탕으로 데이터 필터링을 수행한다. 마지막으로, 서버에서 데이터 통계 분석을 이용한 데이터 필터링 범위를 계산하기 때문에, 센서에서는 단순 비교연산만을 수행함으로써 센서 노드에서의 계산 오버헤드를 감소시킨다. 아울러 성능 분석을 통해 제안하는 기법이 기존 칼만 필터링 기법보다 전송 메시지 수 측면에서 성능이 우수함을 보인다.

Privacy-Preserving Two-Party Collaborative Filtering on Overlapped Ratings

  • Memis, Burak;Yakut, Ibrahim
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • 제8권8호
    • /
    • pp.2948-2966
    • /
    • 2014
  • To promote recommendation services through prediction quality, some privacy-preserving collaborative filtering solutions are proposed to make e-commerce parties collaborate on partitioned data. It is almost probable that two parties hold ratings for the same users and items simultaneously; however, existing two-party privacy-preserving collaborative filtering solutions do not cover such overlaps. Since rating values and rated items are confidential, overlapping ratings make privacy-preservation more challenging. This study examines how to estimate predictions privately based on partitioned data with overlapped entries between two e-commerce companies. We consider both user-based and item-based collaborative filtering approaches and propose novel privacy-preserving collaborative filtering schemes in this sense. We also evaluate our schemes using real movie dataset, and the empirical outcomes show that the parties can promote collaborative services using our schemes.

Analysis of Homomorphic Filtered Remotely Sensed Imagery and Multiple Geophysical Images

  • Ryu Hee-Young;Lee Kiwon;Kwon Byung-Doo
    • 대한원격탐사학회:학술대회논문집
    • /
    • 대한원격탐사학회 2004년도 Proceedings of ISRS 2004
    • /
    • pp.237-240
    • /
    • 2004
  • In this study, the digital image processing with image enhancement based on homomorphic filtering was performed using geophysical imaging data such as gravity, magnetic data and sub-scenes of satellite images such as LANDSAT, IKONOS, and KOMPSAT. Windows application program for executing homomorphic filtering was designed and newly implemented. In general, homomorphic filtering is technique that is based on Fourier transform, which enhances the contrast of image by removing the low frequencies and amplifying the high frequencies in frequency domain. We can enhance the image selectively using homomorphic filtering as compared with the existing method, which enhance the image totally. Through several experiment using remotely sensed imagery and geophysical image with this program, it is concluded that homomorphic filtering is more effective to reveal distinct characteristics for some complicated and multi-associated features on image data.

  • PDF

개선된 스케일 스페이스 필터링과 함수연결연상 신경망을 이용한 화학공정 감시 (Monitoring of Chemical Processes Using Modified Scale Space Filtering and Functional-Link-Associative Neural Network)

  • 최중환;김윤식;장태석;윤인섭
    • 제어로봇시스템학회논문지
    • /
    • 제6권12호
    • /
    • pp.1113-1119
    • /
    • 2000
  • To operate a process plant safely and economically, process monitoring is very important. Process monitoring is the task to identify the state of the system from sensor data. Process monitoring includes data acquisition, regulatory control, data reconciliation, fault detection, etc. This research focuses on the data recon-ciliation using scale-space filtering and fault detection using functional-link associative neural networks. Scale-space filtering is a multi-resolution signal analysis method. Scale-space filtering can extract highest frequency factors(noise) effectively. But scale-space filtering has too large calculation costs and end effect problems. This research reduces the calculation cost of scale-space filtering by applying the minimum limit to the gaussian kernel. And the end-effect that occurs at the end of the signal of the scale-space filtering is overcome by using extrapolation related with the clustering change detection method. Nonlinear principal component analysis methods using neural network have been reviewed and the separately expanded functional-link associative neural network is proposed for chemical process monitoring. The separately expanded functional-link associative neural network has better learning capabilities, generalization abilities and short learning time than the exiting-neural networks. Separately expanded functional-link associative neural network can express a statistical model similar to real process by expanding the input data separately. Combining the proposed methods-modified scale-space filtering and fault detection method using the separately expanded functional-link associative neural network-a process monitoring system is proposed in this research. the usefulness of the proposed method is proven by its application a boiler water supply unit.

  • PDF

A Strategy Study on Sensitive Information Filtering for Personal Information Protect in Big Data Analyze

  • Koo, Gun-Seo
    • 한국컴퓨터정보학회논문지
    • /
    • 제22권12호
    • /
    • pp.101-108
    • /
    • 2017
  • The study proposed a system that filters the data that is entered when analyzing big data such as SNS and BLOG. Personal information includes impersonal personal information, but there is also personal information that distinguishes it from personal information, such as religious institution, personal feelings, thoughts, or beliefs. Define these personally identifiable information as sensitive information. In order to prevent this, Article 23 of the Privacy Act has clauses on the collection and utilization of the information. The proposed system structure is divided into two stages, including Big Data Processing Processes and Sensitive Information Filtering Processes, and Big Data processing is analyzed and applied in Big Data collection in four stages. Big Data Processing Processes include data collection and storage, vocabulary analysis and parsing and semantics. Sensitive Information Filtering Processes includes sensitive information questionnaires, establishing sensitive information DB, qualifying information, filtering sensitive information, and reliability analysis. As a result, the number of Big Data performed in the experiment was carried out at 84.13%, until 7553 of 8978 was produced to create the Ontology Generation. There is considerable significan ce to the point that Performing a sensitive information cut phase was carried out by 98%.

ENHANCEMENT AND SMOOTHING OF HYPERSPECTAL REMOTE SENSING DATA BY ADVANCED SCALE-SPACE FILTERING

  • Konstantinos, Karantzalos;Demetre, Argialas
    • 대한원격탐사학회:학술대회논문집
    • /
    • 대한원격탐사학회 2006년도 Proceedings of ISRS 2006 PORSEC Volume II
    • /
    • pp.736-739
    • /
    • 2006
  • While hyperspectral data are very rich in information, their processing poses several challenges such as computational requirements, noise removal and relevant information extraction. In this paper, the application of advanced scale-space filtering to selected hyperspectral bands was investigated. In particular, a pre-processing tool, consisting of anisotropic diffusion and morphological leveling filtering, has been developed, aiming to an edge-preserving smoothing and simplification of hyperspectral data, procedures which are of fundamental importance during feature extraction and object detection. Two scale space parameters define the extent of image smoothing (anisotropic diffusion iterations) and image simplification (scale of morphological levelings). Experimental results demonstrated the effectiveness of the developed scale space filtering for the enhancement and smoothing of hyperspectral remote sensing data and their advantage against watershed over-segmentation problems and edge detection.

  • PDF

Adaptive Filtering Processing for Target Signature Enhancement in Monostatic Borehole Radar Data

  • Hyun, Seung-Yeup;Kim, Se-Yun
    • Journal of electromagnetic engineering and science
    • /
    • 제14권2호
    • /
    • pp.79-81
    • /
    • 2014
  • In B-scan data measured by a pulse-type monostatic borehole radar, target signatures are seriously obscured by two clutters that differ in orientation and intensity. The primary clutter appears as a nearly constant time delay, which is caused by internal ringing between antenna and transceiver in the radar system. The secondary clutter occurs as an oblique time delay due to the guided borehole wave along the logging cable of the radar antenna. This issue led us to perform adaptive filtering processing for orientation-based clutter removal. This letter describes adaptive filtering processing consisting of a combination of edge detection, data rotation, and eigenimage filtering. We show that the hyperbolic signatures of a dormant air-filled tunnel target can be more distinctly enhanced by applying the proposed approach to the B-scan data, which are measured in a well-suited test site for underground tunnel detection.

Adaptive Gaussian Mechanism Based on Expected Data Utility under Conditional Filtering Noise

  • Liu, Hai;Wu, Zhenqiang;Peng, Changgen;Tian, Feng;Lu, Laifeng
    • KSII Transactions on Internet and Information Systems (TIIS)
    • /
    • 제12권7호
    • /
    • pp.3497-3515
    • /
    • 2018
  • Differential privacy has broadly applied to statistical analysis, and its mainly objective is to ensure the tradeoff between the utility of noise data and the privacy preserving of individual's sensitive information. However, an individual could not achieve expected data utility under differential privacy mechanisms, since the adding noise is random. To this end, we proposed an adaptive Gaussian mechanism based on expected data utility under conditional filtering noise. Firstly, this paper made conditional filtering for Gaussian mechanism noise. Secondly, we defined the expected data utility according to the absolute value of relative error. Finally, we presented an adaptive Gaussian mechanism by combining expected data utility with conditional filtering noise. Through comparative analysis, the adaptive Gaussian mechanism satisfies differential privacy and achieves expected data utility for giving any privacy budget. Furthermore, our scheme is easy extend to engineering implementation.