Journal of The Korean Society of Agricultural Engineers (한국농공학회논문집)

The Korean Society of Agricultural Engineers (한국농공학회)
권호 표지
DOI QR코드

DOI QR Code

Outlier Detection of Real-Time Reservoir Water Level Data Using Threshold Model and Artificial Neural Network Model

임계치 모형과 인공신경망 모형을 이용한 실시간 저수지 수위자료의 이상치 탐지

  • Kim, Maga (Department of Rural Systems Engineering, Seoul National University) ;
  • Choi, Jin-Yong (Department of Rural Systems Engineering, Research Institute of Agriculture and Life Sciences, Seoul National University) ;
  • Bang, Jehong (Department of Rural Systems Engineering, Seoul National University) ;
  • Lee, Jaeju (Rural Research Institute, Korea Rural Community Corporation)
  • Received : 2018.11.06
  • Accepted : 2018.12.06
  • Published : 2019.01.31
Download PDF
⟨ Previous Next ⟩

Abstract

Reservoir water level data identify the current water storage of the reservoir, and they are utilized as primary data for management and research of agricultural water. For the reservoir storage management, Korea Rural Community Corporation (KRC) installed water level stations at around 1,600 agricultural reservoirs and has been collecting the water level data every 10 minutes. However, various kinds of outliers due to noise and erroneous problems are frequently appearing because of environmental and physical causes. Therefore, it is necessary to detect outlier and improve the quality of reservoir water level data to utilize the water level data in purpose. This study was conducted to detect and classify outlier and normal data using two different models including the threshold model and the artificial neural network (ANN) model. The results were compared to evaluate the performance of the models. The threshold model identifies the outlier by setting the upper/lower bound of water level data and variation data and by setting bandwidth of water level data as a threshold of regarding erroneous water level. The ANN model was trained with prepared training dataset as normal data (T) and outlier (F), and the ANN model operated for identifying the outlier. The models are evaluated with reference data which were collected reservoir water level data in daily by KRC. The outlier detection performance of the threshold model was better than the ANN model, but ANN model showed better detection performance for not classifying normal data as outlier.

Keywords

NGHHCI_2019_v61n1_107_f0001.png 이미지

Fig. 1 Flow chart of the study

NGHHCI_2019_v61n1_107_f0002.png 이미지

Fig. 2 10-minute interval raw water level data of the Gaeun reservoir

NGHHCI_2019_v61n1_107_f0003.png 이미지

Fig. 3 General structure of multi-layer perceptron

NGHHCI_2019_v61n1_107_f0004.png 이미지

Fig. 4 T dataset after data pre-process

NGHHCI_2019_v61n1_107_f0005.png 이미지

Fig. 5 The structure of the artificial neural network model for outlier detection

NGHHCI_2019_v61n1_107_f0006.png 이미지

Fig. 6 The results of the threshold model according to value of α

NGHHCI_2019_v61n1_107_f0007.png 이미지

Fig. 7 Daily mean value of (a) raw data, (b) threshold data, (c) reference data after threshold model application

NGHHCI_2019_v61n1_107_f0008.png 이미지

Fig. 8 Daily mean value of (a) raw data, (b) target data, (c) ANN data, (d) reference data after ANN model application

NGHHCI_2019_v61n1_107_f0009.png 이미지

Fig. 9 The scatter plot of (a) raw data, (b) threshold data, (c) target data, (d) ANN data compared with reference data

NGHHCI_2019_v61n1_107_f0010.png 이미지

Fig. 10 Daily mean value of (a) raw data, (b) threshold data, (c) ANN data, (d) reference data (2016. 01. 01.~2016. 12. 31.)

Table 1 Properties of the Gaeun reservoir

NGHHCI_2019_v61n1_107_t0001.png 이미지

Table 2 Properties of water level data

NGHHCI_2019_v61n1_107_t0002.png 이미지

Table 3 Input data of the artificial neural network model for outlier detection

NGHHCI_2019_v61n1_107_t0003.png 이미지

Table 4 The errors by input data in application of the artificial neural network model for outlier detection

NGHHCI_2019_v61n1_107_t0004.png 이미지

Table 5 The statistical parameters (R2, MAE, RMSE) compared to reference data

NGHHCI_2019_v61n1_107_t0005.png 이미지

References

  1. Ahn, J. H., M. S. Kang, I. H. Song, K. D. Lee, J. H. Song, and J. R. Jang, 2012. Estimation of surface runoff from paddy plots using an artificial neural network. Journal of the Korean Society of Agricultural Engineers 54(4): 66-71 (in Korean). doi:10.5389/KSAE.2012.54.4.065.
  2. Ahn, S. J., I. S. Yeon, and K. I. Kim, 2000a. Rainfall forecasting Using neural network. Journal of The Korean Society of Civil Engineers 20(5B): 711-722 (in Korean).
  3. Ahn, S. J., K. W. Jun, and K. I. Kim, 2000b. Forecasting of runoff hydrograph using neural network algorithms. Journal of Korean Water Resources Association 33(4): 505-515 (in Korean).
  4. Bang, J. H., Y. H. Lee, S. Y. Jeong, and J. Y. Choi, 2017. A study of outlier detection on time series of water-level in agricultural reservoir. In Proceedings of the Korean Society of Agricultural Engimeers Conefrence 60 (in Korean).
  5. Choi, J. K., and M. S. Kang, 2000. Application of neural network to water resources. Journal of Korean National Comittee on Irrigation and Drainage 7(2): 248-258 (in Korean).
  6. Choi, J. Y., 2018. Development of quality control methods for water level data and irrigation water supply estimation, 4-10. Korea Rural Community Corporation.
  7. Feng, S., Q. Hu, and W. Qian, 2004. Quality control of daily meteorological data in China, 195-2000: a new dataset. International Journal of Climatology 24: 853-870. doi:10.1002/joc.1047.
  8. Gonzalez-Rouco, J. F., J. L. Jimenez, V. Quesada, and F. Valero, 1999. Quality control and homogeneity of precipitation data in the southwest of Europe. Journal of Climate 14: 964-978. doi:10.1175/1520-0442(2001)014<0964:QCAHOP>2.0.CO;2.
  9. Günther, F., and S. Fritsch, 2010. Neuralnet: training of neural networks. The R Journal 2(1): 30-38. https://doi.org/10.32614/RJ-2010-006
  10. Harrison, D. L., S. J. Driscoll, and M. Kitchen, 2000. Improving precipitation estimates from weather radar using quality control and correction techniques. Meteorological Applications 7(2): 135-144. doi:10.1017/S1350482700001468.
  11. Horsburgh, J. S., D. G. Tarboton, D. R. Maidment, amd I. Zaslavsky, 2008. A relational model for environmental and water resources data. Water Resources Research 44(5). doi:10.1029/2007WR006392.
  12. Jang, S. H., J. Y. Yoon, S. D. Kim, and Y. N. Yoon, 2007. An establishment of operation and management system for flood control and conservation in reservoir with date: I. establishment of real-time inflow prediction model using recorded rainfall data. Journal of The Korean Society of Civil Engineers 27(2B): 133-140 (in Korean).
  13. Jeong, G. H., and T. W. Kim, 2007. Comparing water distribution model with reservoir and water system management. Water and Future 40(10): 38-43 (in Korean).
  14. Kang, B. S., and B. K. Lee, 2008. Predicting probability of precipitation using artificial neural network and mesoscale numerical weather prediction. Journal of The Korean Society of Civil Engineers 28(5B): 485-493 (in Korean).
  15. Kang, M. G., H. S. Jeong, and J. T. Kim, 2010. Efficient management of agricultural canal systems through quality management of water level and water quantity data. Rural Resources 52(2): 87-96 (in Korean).
  16. Kilonsky, B. J., and P. Caldwell, 1991. In the pursuit of high-quality sea level data. In Oceans 91 Proceedings (2): 669-675. doi:10.1109/OCEANS.1991.627921.
  17. Kim, C. H., J. A. Ryu, D. G. Kim, and G. B. Kim, 2016. Analysis of the effects of drainage systems in wetlands based on changes in groundwater level, soil moisture content, and water quality. The Journal of Engineering Geology 26(2): 251-260 (in Korean). doi:/10.9720/kseg.2016.2.251.
  18. Kim, H. G., M. I. Kim, M. S. Lee, Y. S. Park, and J. H. Kwak, 2017. Correlation of deep landside occurrence and variation of groundwater level. Journal of The Korea Society of Forest Engineering and technology 15(1): 1-12 (in Korean).
  19. Kim, H. K., S. M. Kim, and S. W. Park, 2006. Development of hydrologic data management system based on relational database. Journal of Korean Water Resources Association 39(10): 855-866 (in Korean). https://doi.org/10.3741/JKWRA.2006.39.10.855
  20. Kim, S. J., Y. S. Kwon, K. H. Lee, and H. S. Kim, 2010. Radar rainfall adjustment by artificial neural network and runoff analysis. Journal of The Korean Society of Civil Engineers 30(2B): 159-167 (in Korean).
  21. Kim, S. W., 2000. A study on the forecasting of daily streamflow using the multilayer neural networks model. Journal of Korean Water Resources Association 33(5): 537-550 (in Korean).
  22. Kim, S. W., and J. D. Salas, 2000. The flood water stage prediction based on neural networks method in stream gauge station. Journal of Korean Water Resources Association 33(2): 247-262 (in Korean).
  23. Lee, C. W., Y. S. Mang, and Y. S. Kim, 2014. Behavior of fill dam subjected to continuous water level change and overflow. Journal of the Korean Geo-Environmental Society 15(6): 41-48 (in Korean). doi:10.14481/jkges.2014.15.6.41.
  24. Mounce, S. R., R. B. Mounce, and J. B. Boxall, 2011. Novelty detection for time series data analysis in water distribution systems using support vector machines. Journal of Hydroinformatics 13(4): 672-686. doi:10.2166/hydro.2010.144.
  25. Mourad, M., and J. L. Bertrand-Krajewski, 2002. A method for automatic validation of long time series of data in urban hydrology. Water Science & Technology 45(4-5): 263-270. doi:10.2166/wst.2002.0601.
  26. Oh, C. R., S. C. Park, H. M. Lee, and Y. P. Pyo, 2002. A forecasting of water quality in the Youngsan river using neural network. Journal of The Korean Society of Civil Engineers 22(3B): 372-382 (in Korean).
  27. Oh, J. W., J. H. Park, and Y. K. Kim, 2008. Missing hydrological data estimation using neural network and real time data reconciliation. Journal of Korean Water Resources Association (10): 1059-1065 (in Korean). doi:10.3741/JKWRA.2008.41.10.1059.
  28. Park, J. H., M. S. Kang, J. H. Song, and S. M. Jun, 2015. Design and implementation of IoT-based intelligent platform for water level monitoring. Journal of the Korean Society of Rural Planning 21(4): 177-186 (in Korean). doi:10.7851/ksrp.2015.21.4.177.
  29. Schneider, U., A. Becker, P. Finger, A. Meyer-Christoffer, M. Ziese, and B. Rudolf, 2014. GPCC's new land surface precipitation climatology based on quality-controlled in situ data its role in quantifying the global water cycle. Theoretical and Applied Climatology 115(1-2): 15-40. doi:10.1007/s00704-013-0860-x.
  30. Seo, Y. M., E. H. Choi, and W. K. Yeo, 2017. Reservoir water level forecasting using machine learning models. Journal of the Korean Society of Agricultural Engineers 59(3): 97-110 (in Korean). doi:10.5389/KSAE.2017.59.3.097.
  31. Shim, S. B., S. K. Kim, R. H. Park, and D. K. Hoh, 1997. Decision support system for real-time reservoir operation during flood period. Journal of Korean Water Resources Association 30(5): 431-439 (in Korean).
  32. Steiner, M., J. A. Smith, S. J. Burges, C. V. Alonso, and R. W. Darden, 1999. Effect of bias adjustment and rain gauge data quality control on radar rainfall estimation. Water Resources Research 35(8): 2487-2503. doi:10.1029/1999WR900142.
  33. Yeo, W. K., Y. M. Seo, S. Y. Lee, and H. K. Ji, 2010. Study on water stage prediction using hybrid model of artificial neural network and genetic algorithm. Journal of Korean Water Resources Association 43(8): 721-731 (in Korean). doi:10.3741/JKWRA.2010.43.8.721.
  34. Yoon, K. H., B. C. Seo, and H. S. Shin, 2004. Dam inflow forecasting for short term flood based on neural networks in Nakdong river basin. Journal of Korean Water Resources Association 37(1): 67-75 (in Korean). https://doi.org/10.3741/JKWRA.2004.37.1.067