Korean Chemical Engineering Research

The Korean Institute of Chemical Engineers (한국화학공학회)
권호 표지
DOI QR코드

DOI QR Code

Hyperparameter Optimization and Data Augmentation of Artificial Neural Networks for Prediction of Ammonia Emission Amount from Field-applied Manure

토양에 살포된 축산 분뇨로부터 암모니아 방출량 예측을 위한 인공신경망의 초매개변수 최적화와 데이터 증식

  • Pyeong-Gon, Jung (CoSPE, Dept. Chemical Engineering, Hankyong National University) ;
  • Young-Il, Lim (CoSPE, Dept. Chemical Engineering, Hankyong National University)
  • 정평곤 (한경대학교 화학공학과 CoSPE 연구센터) ;
  • 임영일 (한경대학교 화학공학과 CoSPE 연구센터)
  • Received : 2022.05.30
  • Accepted : 2022.07.14
  • Published : 2023.02.01
Download PDF
⟨ Previous Next ⟩

Abstract

A sufficient amount of data with quality is needed for training artificial neural networks (ANNs). However, developing ANN models with a small amount of data often appears in engineering fields. This paper presented an ANN model to improve prediction performance of the ammonia emission amount with 83 data. The ammonia emission rate included eleven inputs and two outputs (maximum ammonia loss, Nmax and time to reach half of Nmax, Km). Categorical input variables were transformed into multi-dimensional equal-distance variables, and 13 data were added into 66 training data using a generative adversarial network. Hyperparameters (number of layers, number of neurons, and activation function) of ANN were optimized using Gaussian process. Using 17 test data, the previous ANN model (Lim et al., 2007) showed the mean absolute error (MAE) of Km and Nmax to 0.0668 and 0.1860, respectively. The present ANN outperformed the previous model, reducing MAE by 38% and 56%.

인공신경망을 이용한 모델 개발에서 데이터의 품질은 모델 성능에 큰 영향을 주고, 양질의 충분한 데이터가 인공신경망 훈련을 위해 필요하다. 하지만, 공학 분야에서는 적은 양의 데이터로 모델을 개발해야 하는 경우가 자주 발생한다. 본 논문은 토양에 살포된 축산 분뇨로부터 암모니아 방출량에 대한 적은 수의 데이터(83 개)를 사용하여 인공신경망 모델의 예측 성능을 향상할 수 있는 방안을 제시하였다. Michaelis-Menten 식으로 표현되는 암모니아 방출량 문제는 11개 입력변수에 대하여 2개 출력변수로 구성되었다. 출력변수는 최대 질소 발생량(Nmax, kg/ha)과 Nmax의 절반에 도달하는 시간(Km, h) 이다. 범주형 입력변수에 대해 다차원 등간격 기법인 one-hot encoding 을 이용하여 데이터 전처리를 수행하였고, 훈련데이터 66개에 대하여 generative adversarial network (GAN)을 이용하여 13개 데이터를 추가로 보강하였다. 또한, 인공신경망의 초매개변수인 은닉층 수, 각 은닉층 내 뉴런 수, 활성화 함수의 최적 조합을 찾기 위하여 Gaussian process (GP)를 사용하였다. 기존의 인공신경망 구조(Lim et al., 2007) 는 17개 평가데이터에 대하여 mean absolute error (MAE)는 Km에서 0.0668, Nmax에서 0.1860이었다. 본 연구에서 제시된 인공신경망 모델은 Km에서 0.0414, Nmax에서 0.0818로 MAE 가 기존 모델 대비 각각 38%, 56% 감소하였다. 본 연구에서 제시된 방법은 적은 양의 데이터를 갖는 문제에서 인공신경망 성능을 향상하기 위하여 활용할 수 있을 것이다.

Keywords

Acknowledgement

This work was supported by a National Research Foundation of Korea (NRF) grant funded by the Korean government (MEST) (Grant number: NRF-2021R1A2C1011618).

References

  1. Lim, Y., Moon, Y.-S. and Kim, T.-W., "Artificial Neural Network Approach for Prediction of Ammonia Emission from Field-Applied Manure and Relative Significance Assessment of Ammonia Emission Factors," Eur. J. Agron., 26(4), 425-434(2007). https://doi.org/10.1016/j.eja.2007.01.008
  2. Sintermann, J., Neftel, A., Ammann, C., Hani, C., Hensen, A., Loubet, B. and Flechard, C. R., "Are Ammonia Emissions from Field-Applied Slurry Substantially over-Estimated in European Emission Inventories?," Biogeosciences, 9(5), 1611-1632(2012). https://doi.org/10.5194/bg-9-1611-2012
  3. Pedersen, J., Andersson, K., Feilberg, A., Delin, S., Hafner, S. and Nyord, T., "Effect of Exposed Surface Area on Ammonia Emissions from Untreated, Separated, and Digested Cattle Manure," Biosyst. Eng., 202, 66-78(2021). https://doi.org/10.1016/j.biosystemseng.2020.12.005
  4. Moon, Y. S., Lim, Y. and Kim, T. W., "Prediction of Ammonia Emission Rate from Field-Applied Animal Manure Using the Artificial Neural Network," Korean Chem. Eng. Res., 45(2), 133-142(2007).
  5. Jain, S., Shukla, S. and Wadhvani, R., "Dynamic Selection of Normalization Techniques Using Data Complexity Measures," Expert Syst. Appl., 106, 252-262(2018). https://doi.org/10.1016/j.eswa.2018.04.008
  6. Jackson, J. E., A User's Guide to Principal Components, 1st ed., John Wiley and Sons, New York(1991).
  7. Ren, J., Chen, J., Shi, D., Li, Y., Li, D., Wang, Y. and Cai, D., "Online Multi-Fault Power System Dynamic Security Assessment Driven by Hybrid Information of Anticipated Faults and Pre-Fault Power Flow," Int. J. Electr. Power Energy Syst., 136, 107651(2022).
  8. Hovden, I. T., "Optimizing Artificial Neural Network Hyperparameters and Architecture," (2019).
  9. Islam, A., Belhaouari, S. B., Rehman, A. U. and Bensmail, H., "Knnor: An Oversampling Technique for Imbalanced Datasets," Appl. Soft Comput., 115, 108288(2022).
  10. Ngo, S. I. and Lim, Y.-I., "Solution and Parameter Identification of a Fixed-Bed Reactor Model for Catalytic Co2 Methanation Using Physics-Informed Neural Networks," Catalysts, 11(11), 1304(2021).
  11. Raissi, M., Perdikaris, P. and Karniadakis, G. E., "Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations," J. Comput. Phys., 378, 686-707(2019). https://doi.org/10.1016/j.jcp.2018.10.045
  12. Yang, Y., Zha, K., Chen, Y.-C., Wang, H. and Katabi, D., "Delving into Deep Imbalanced Regression," arXiv preprint arXiv:2102.09554, (2021).
  13. Zhuo, Y. and Brgoch, J., "Opportunities for Next-Generation Luminescent Materials through Artificial Intelligence," J. Phys. Chem. Lett., 12(2), 764-772(2021). https://doi.org/10.1021/acs.jpclett.0c03203
  14. Coomans, D. and Massart, D. L., "Alternative K-Nearest Neighbour Rules in Supervised Pattern Recognition: Part 1. K-Nearest Neighbour Classification by Using Alternative Voting Rules," Anal. Chim. Acta, 136, 15-27(1982). https://doi.org/10.1016/S0003-2670(01)95359-0
  15. Chawla, N. V., Bowyer, K. W., Hall, L. O. and Kegelmeyer, W. P., "Smote: Synthetic Minority over-Sampling Technique," J. Artif. Intell. Res., 16, 321-357(2002). https://doi.org/10.1613/jair.953
  16. Torgo, L., Ribeiro, R., Pfahringer, B. and Branco, P., Smote for Regression, ed., Springer, Berlin, Heidelberg, Correia(2013).
  17. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., WardeFarley, D., Ozair, S., Courville, A. and Bengio, Y., "Generative Adversarial Nets," Adv. Neural Inf. Process. Syst., 27, (2014).
  18. Asadi, M. and McPhedran, K. N., "Greenhouse Gas Emission Estimation from Municipal Wastewater Using a Hybrid Approach of Generative Adversarial Network and Data-Driven Modelling," Sci. Total Environ., 800, 149508(2021).
  19. Du, X., Xu, H. and Zhu, F., "Understanding the Effect of Hyperparameter Optimization on Machine Learning Models for Structure Design Problems," Comput. Aided Des, 135, 103013(2021).
  20. Bergstra, J., Bardenet, R., Bengio, Y. and Kegl, B., "Algorithms for Hyper-Parameter Optimization," Adv. Neural Inf. Process. Syst., (2011).
  21. Yang, Z. and Zhang, A., "Hyperparameter Optimization Via Sequential Uniform Designs," J. Mach. Learn. Res., 22(149), 1-47(2021).
  22. Yang, L. and Shami, A., "On Hyperparameter Optimization of Machine Learning Algorithms: Theory and Practice," Neurocomputing, 415, 295-316(2020). https://doi.org/10.1016/j.neucom.2020.07.061
  23. Jankovic, A., Chaudhary, G. and Goia, F., "Designing the Design of Experiments (Doe) - an Investigation on the Influence of Different Factorial Designs on the Characterization of Complex Systems," Energy Build, 250, 111298(2021).
  24. Misselbrook, T. H., Nicholson, F. A. and Chambers, B. J., "Predicting Ammonia Losses Following the Application of Livestock Manure to Land," Bioresour. Technol., 96(2), 159-168(2005). https://doi.org/10.1016/j.biortech.2004.05.004
  25. Rodriguez, P., Bautista, M. A., Gonzalez, J. and Escalera, S., "Beyond One-Hot Encoding: Lower Dimensional Target Embedding," Image Vis. Comput., 75, 21-31(2018). https://doi.org/10.1016/j.imavis.2018.04.004
  26. Demir, S., Mincev, K., Kok, K. and Paterakis, N. G., "Data Augmentation for Time Series Regression: Applying Transformations, Autoencoders and Adversarial Networks to Electricity Price Forecasting," Appl. Energ., 304, 117695(2021).
  27. Ramachandran, P., Zoph, B. and Le, Q. V., "Searching for Activation Functions," arXiv preprint arXiv:1710.05941, (2017).
  28. LeCun, Y., Bengio, Y. and Hinton, G., "Deep Learning," Nature, 521(7553), 436-444(2015). https://doi.org/10.1038/nature14539
  29. Kohavi, R., "A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection," Proceedings of the 14th International Joint Conference on Artificial Intelligence - Volume 2, Morgan Kaufmann Publishers Inc., Montreal, Quebec, Canada, pp. 1137-1143(1995).
  30. li, W. and Liu, Z., "A Method of Svm with Normalization in Intrusion Detection," Procedia Environ. Sci., 11, 256-262(2011). https://doi.org/10.1016/j.proenv.2011.12.040
  31. Jahirul, M. I., Rasul, M. G., Brown, R. J., Senadeera, W., Hosen, M. A., Haque, R., Saha, S. C. and Mahlia, T. M. I., "Investigation of Correlation between Chemical Composition and Properties of Biodiesel Using Principal Component Analysis (Pca) and Artificial Neural Network (Ann)," Renew. Energy, 168, 632-646 (2021). https://doi.org/10.1016/j.renene.2020.12.078
  32. Liu, Y., Zhao, S., Wang, Q. and Gao, Q., "Learning More Distinctive Representation by Enhanced Pca Network," Neurocomputing, 275, 924-931(2018). https://doi.org/10.1016/j.neucom.2017.09.041
  33. Berk, J., Nguyen, V., Gupta, S., Rana, S. and Venkatesh, S., "Exploration Enhanced Expected Improvement for Bayesian Optimization," Machine Learning and Knowledge Discovery in Databases, September, Cham(2018).
  34. Riedmiller, M. and Braun, H., "A Direct Adaptive Method for Faster Backpropagation Learning: The Rprop Algorithm," IEEE International Conference on Neural Networks, March, 1993.