• Title/Summary/Keyword: Data Bias

Search Result 1,738, Processing Time 0.027 seconds

Analysis of Relationship Between Meteorological Parameters and Solar Radiation at Cheongju (청주지역의 기상요소와 일사량과의 상관관계 분석)

  • Baek, Shin Chul;Shin, Hyoung Sub;Park, Jong Hwa
    • KCID journal
    • /
    • v.19 no.1
    • /
    • pp.87-96
    • /
    • 2012
  • Information of local solar radiation is essential for many field, including water resources management, crop yield estimation, crop growth model, solar energy systems and irrigation and drainage design. Unfortunately, solar radiation measurements are not easily available due to the cost and maintenance and calibration requirements of the measuring equipment and station. Therefore, it is important to elaborate methods to estimate the solar radiation based on readily available meteorological data. In this study, two empirical equations are employed to estimate daily solar radiation using Cheongju Regional Meteorological Office data. Two scenarios are considered: (a) sunshine duration data are available for a given location, or (b) only daily cloudiness index records exist. Simple linear regression with daily sunshine duration and cloudiness index as the dependent variable accounted for 91% and 80%, respectively of the variation of solar radiation(H) at 2011. Daily global solar radiation is highly correlated with sunshine duration. In order to indicate the performance of the models, the statistical test methods of the mean bias error(MBE), root mean square error(RMSE) and correlation coefficient(r) are used. Sunshine duration and cloudiness index can be easily and reliably measured and data are widely available.

  • PDF

A Study on the Construction of Stable Clustering by Minimizing the Order Bias (순서 바이어스 최소화에 의한 안정적 클러스터링 구축에 관한 연구)

  • Lee, Gye-Seong
    • The Transactions of the Korea Information Processing Society
    • /
    • v.6 no.6
    • /
    • pp.1571-1580
    • /
    • 1999
  • When a hierarchical structure is derived from data set for data mining and machine learning, using a conceptual clustering algorithm, one of the unsupervised learning paradigms, it is not unusual to have a different set of outcomes with respect to the order of processing data objects. To overcome this problem, the first classification process is proceeded to construct an initial partition. The partition is expected to imply the possible range in the number of final classes. We apply center sorting to the data objects in the classes of the partition for new data ordering and build a new partition using ITERATE clustering procedure. We developed an algorithm, REIT that leads to the final partition with stable and best partition score. A number of experiments were performed to show the minimization of order bias effects using the algorithm.

  • PDF

Bias-correction of Dual Polarization Radar rainfall using Convolutional Autoencoder

  • Jung, Sungho;Le, Xuan Hien;Oh, Sungryul;Kim, Jeongyup;Lee, GiHa
    • Proceedings of the Korea Water Resources Association Conference
    • /
    • 2020.06a
    • /
    • pp.166-166
    • /
    • 2020
  • Recently, As the frequency of localized heavy rains increases, the use of high-resolution radar data is increasing. The produced radar rainfall has still gaps of spatial and temporal compared to gauge observation rainfall, and in many studies, various statistical techniques are performed for correct rainfall. In this study, the precipitation correction of the S-band Dual Polarization radar in use in the flood forecast was performed using the ConvAE algorithm, one of the Convolutional Neural Network. The ConvAE model was trained based on radar data sets having a 10-min temporal resolution: radar rainfall data, gauge rainfall data for 790minutes(July 2017 in Cheongju flood event). As a result of the validation of corrected radar rainfall were reduced gaps compared to gauge rainfall and the spatial correction was also performed. Therefore, it is judged that the corrected radar rainfall using ConvAE will increase the reliability of the gridded rainfall data used in various physically-based distributed hydrodynamic models.

  • PDF

Fusion of Aerosol Optical Depth from the GOCI and the AHI Observations (GOCI와 AHI 자료를 활용한 에어로졸 광학두께 합성장 산출 연구)

  • Kang, Hyeongwoo;Choi, Wonei;Park, Jeonghyun;Kim, Serin;Lee, Hanlim
    • Korean Journal of Remote Sensing
    • /
    • v.37 no.5_1
    • /
    • pp.861-870
    • /
    • 2021
  • In this study, fused Aerosol Optical Depth (AOD) data were produced using AOD products from the Geostationary Ocean Color Imager (GOCI) onboard Communication, Oceanography and Meteorology Satellite (COMS)satellite and the Advanced Himawari Imager (AHI) onboard Himawari-8. Since the spatial resolution and the coordinate system between the satellite sensors are different, a preprocessing was first preceded. After that, using the level 1.5 AOD dataset of AErosol RObotic NETwork (AERONET), which is ground-based observation, correlations and trends between each satellite AOD and AERONET AOD were utilized to produce more accurate satellite AOD data than the originalsatellite AODs. The fused AOD were found to be more accurate than the originalsatellite AODs. Root Mean Square Error (RMSE) and mean bias of the fused AODs were calculated to be 0.13 and 0.05, respectively. We also compared errors of the fused AODs against those of the original GOCI AOD (RMSE: 0.15, mean bias: 0.11) and the original AHI AOD (RMSE: 0.15, mean bias: 0.05). It was confirmed that the fused AODs have betterspatial coverage than the original AODsin areas where there are no observations due to the presence of cloud from a single satellite.

Compatibility of DOAS and Conventional Point Monitoring System Through an Evaluation of Bias Structures Using Long-term Measurement Data in Seoul (장기관측자료를 이용한 DOAS와 점측정 분석시스템의 바이어스 구조에 대한 평가)

  • 김기현;김민영
    • Journal of Korean Society for Atmospheric Environment
    • /
    • v.17 no.5
    • /
    • pp.395-405
    • /
    • 2001
  • To make an assessment of the compatibility between DOAS and conventional point monitoring system (MCSAM-2: MS2), we investigated the concentrations of three criteria pollutants which include S $O_2$, N $O_2$, and $O_3$from a national monitoring station in Seoul during the periods of June 1999~August 2000. The average concentration values for the whole study period derived from hourly concentration data sets of those three species indicated that the mean differences between the two methods can be approximated as 18%. When the bias structure of two systems was evaluated through the computation of percent difference(PD) between the two such as ( $C_{DOAS}$- $C_{conventional}$ $C_{DOAS}$*100, differences between the two systems appeared to be quite systematic among different compounds. While the mode of bias peaked at 0~20% or 20~40% in terms of PD values, the cause of such positive bias mainly arised from generally enhanced concentration values of DOAS system. The structure of bias among different species was further assessed through linear regression analysis. Results of the analysis indicated that the dominant portions of differences observed from two monitoring systems can be accounted for by the systematic differences in their spanning and zeroing systems. S $O_2$(MS2)=0.6385 S $O_2$(DOAS)+2.0985($r^2$=0.7894) N $O_2$(MS2)=0.6548 N $O_2$(DOAS)+7.437($r^2$=0.7687) $O_3$(MS2)=1.0359 $O_3$(DOAS)-7.7885($r^2$=0.7944) The findings of slope values at around 0.64~0.65 from two species suggest that DOAS should respond more sensitively in upper bound concentration range. The offset values apart from zero indicate that more deliberate comparison needs to be made between these monitoring systems. However, based on the existence of strong correlations from at least 8,000 data points for each species of comparison, we were able to conclude that the compatibility of two monitoring systems is highly significant. With the improvement of calibration techniques for the DOAS system. its applicability for routine monitoring of airborne pollutant species is expected to be quite extendable.

  • PDF

A selection of optimal method for bias-correction in Global Seasonal Forecast System version 5 (GloSea5) (전지구 계절예측시스템 GloSea5의 최적 편의보정기법 선정)

  • Son, Chanyoung;Song, Junghyun;Kim, Sejin;Cho, Younghyun
    • Journal of Korea Water Resources Association
    • /
    • v.50 no.8
    • /
    • pp.551-562
    • /
    • 2017
  • In order to utilize 6-month precipitation forecasts (6 months at maximum) of Global Seasonal Forecast System version 5 (GloSea5), which is being provided by KMA (Korea Meteorological Administration) since 2014, for water resources management as well as other applications, it is needed to correct the forecast model's quantitative bias against observations. This study evaluated applicability of bias-correction skill in GloSea5 and selected an optimal method among 11 techniques that include probabilistic distribution type based, parametric, and non-parametric bias-correction to fix GloSea5's bias in precipitation forecasts. Non-parametric bias-correction provided the most similar results with observed data compared to other techniques in hindcast for the past events, yet relatively generated some discrepancies in forecast. On the contrary, parametric bias-correction produced the most reliable results in both hindcast and forecast periods. The results of this study are expected to be applicable to various applications using seasonal forecast model such as water resources operation and management, hydropower, agriculture, etc.

Development of robust Calibration for Determination Sweetness of Fuji Apple fruit using Near Infrared Reflectance Spectroscopy

  • Sohn, Mi-Ryeong;Kwon, Young-Kill;Cho, Rae-Kwang
    • Near Infrared Analysis
    • /
    • v.2 no.1
    • /
    • pp.55-58
    • /
    • 2001
  • The object of this work was to investigate the influence of growing district and harvest year on calibration for sweetness (Brix) determination of Fuji apple fruit using near infrared (NIR) reflectance spectroscopy, and to develop the robust calibration across these variation. The calibration models was based on wavelength range of 1100∼2500 nm using a stepwise multiple linear regression. A calibration model by sample set of one growing district was not transferable to other growing districts. The combined calibration (data of three growing districts) predicted reasonable well against a population set drawn from all growing districts (SEP=0.69, Bias=0.075). A calibration model by sample set of one harvest year was not also transferable to other harvest years. The combined calibration (data of three harvest years) predicted well against a population set drawn from all harvest years (SEP=0.53, Bias=0.004).

Sensitivity analysis in Bayesian nonignorable selection model for binary responses

  • Choi, Seong Mi;Kim, Dal Ho
    • Journal of the Korean Data and Information Science Society
    • /
    • v.25 no.1
    • /
    • pp.187-194
    • /
    • 2014
  • We consider a Bayesian nonignorable selection model to accommodate the selection bias. Markov chain Monte Carlo methods is known to be very useful to fit the nonignorable selection model. However, sensitivity to prior assumptions on parameters for selection mechanism is a potential problem. To quantify the sensitivity to prior assumption, the deviance information criterion and the conditional predictive ordinate are used to compare the goodness-of-fit under two different prior specifications. It turns out that the 'MLE' prior gives better fit than the 'uniform' prior in viewpoints of goodness-of-fit measures.

An Analysis on Efficiency for the Environmental Friendly Agricultural Product of Strawberry in GyeongBuk Province (경북지역 친환경딸기 농가의 인증유형에 따른 효율성 분석)

  • Lee, Sang-Ho;Song, Kyung-Hwan
    • Korean Journal of Organic Agriculture
    • /
    • v.21 no.4
    • /
    • pp.487-500
    • /
    • 2013
  • The purpose of this study is to estimate efficiency of environmental-friendly agricultural product by using Data Envelopment Analysis. A proposed method employs a bootstrapping approach to generating efficiency estimates through Monte Carlo simulation resampling process. The technical efficiency, pure technical efficiency, and scale efficiency measure of strawberry by pesticide-free certification is 0.967, 0.995, 0.968 respectively. However those of bias-corrected estimates are 0.918, 0.983, 0.934. We know that the DEA estimator is an upward biased estimator. In technical efficiency, average lower and upper confidence bounds of 0.807 and 0.960. According to these results, the DEA bootstrapping model used here provides bias-corrected and confidence intervals for the point estimates, it is more preferable.

Extraction of Bias and Gate Length dependent data of Substrate Parameters for RF CMOS Devices (RF CMOS 소자 기판 파라미터의 바이어스 및 게이트 길이 종속데이터 추출)

  • Lee, Yong-Taek;Choi, Mun-Sung;Lee, Seong-Hearn
    • Proceedings of the IEEK Conference
    • /
    • 2004.06b
    • /
    • pp.347-350
    • /
    • 2004
  • The substrate parameters of Si MOSFET equivalent circuit model were directly extracted from measured S-Parameters in the GHz region by using simple 2-port parameter equations. Using the above extract ion method, bias and gate length dependent curves of substrate parameters in the RF region are obtained by varying drain voltage at several short channel devices with various gate lengths. These extract ion data will greatly contribute to scalable RF nonlinear substrate modeling.

  • PDF