Journal of Intelligence and Information Systems (지능정보연구)

Korea Intelligent Information System Society (한국지능정보시스템학회)
권호 표지
DOI QR코드

DOI QR Code

Resolving data imbalance through differentiated anomaly data processing based on verification data

검증데이터 기반의 차별화된 이상데이터 처리를 통한 데이터 불균형 해소 방법

  • Hwang, Chulhyun (Dept of Big Data, Hanyang Woman's University)
  • 황철현 (한양여자대학교 빅데이터과)
  • Received : 2022.10.16
  • Accepted : 2022.11.25
  • Published : 2022.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Data imbalance refers to a phenomenon in which the number of data in one category is too large or too small compared to another category. Due to this, it has been raised as a major factor that deteriorates performance in machine learning that utilizes classification algorithms. In order to solve the data imbalance problem, various ovrsampling methods for amplifying prime number distribution data have been proposed. Among them, SMOTE is the most representative method. In order to maximize the amplification effect of minority distribution data, various methods have emerged that remove noise included in data (SMOTE-IPF) or enhance only border lines (Borderline SMOTE). This paper proposes a method to ultimately improve classification performance by improving the processing method for anomaly data in the traditional SMOTE method that amplifies minority classification data. The proposed method consistently presented relatively high classification performance compared to the existing methods through experiments.

데이터 불균형은 한 분류의 데이터 수가 다른 분류에 비해 지나치게 크거나 작은 현상을 의미하며. 이로 인해 분류 알고리즘을 활용하는 기계학습에서 성능을 저하시키는 주요 요인으로 제기되고 있다. 데이터 불균형 문제 해결을 위해서 소수 분포 데이터를 증폭하는 다양한 오버 샘플링(Over Sampling) 방법들이 제안되고 있다. 이 가운데 SMOTE는 가장 대표적인 방법으로 소수 분포 데이터의 증폭 효과를 극대화하기 위해 데이터에 포함된 잡음을 제거(SMOTE-IPF)하거나, 경계선만을 강화(Borderline SMOTE) 시키는 다양한 방법들이 출현하였다. 이 논문은 소수분류 데이터를 증폭하는 전통적인 SMOTE 방법에서 이상데이터(Anomaly Data)에 대한 처리방법개선을 통해 궁극적으로 분류성능을 높이는 방법을 제안한다. 제안 방법은 실험을 통해 기존 방법에 비해 상대적으로 높은 분류성능을 일관성 있게 제시하였다.

Keywords

References

  1. Ali, H., Salleh, M. N. M., Saedudin, R., Hussain, K., & Mushtaq, M. F. (2019). Imbalance class problems in data mining: a review. Indonesian Journal of Electrical Engineering and Computer Science, 14(3), 1560-1571.
  2. Chawla, N. V., Bowyer, K. W., Hall, L. O., & Kegelmeyer, W. P. (2002). SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16, 321-357. https://doi.org/10.1613/jair.953
  3. Cheng, K., Zhang, C., Yu, H., Yang, X., Zou, H., & Gao, S. (2019). Grouped SMOTE with noise filtering mechanism for classifying imbalanced data. IEEE Access, 7, 170668-170681. https://doi.org/10.1109/access.2019.2955086
  4. Choi, N., & Kim, W. (2019). Anomaly Detection for User Action with Generative Adversarial Networks. Journal of Intelligence and Information Systems, 25(3), 43-62. https://doi.org/10.13088/JIIS.2019.25.3.043
  5. Cortez, P., & Silva, A. M. G. (2008). Using data mining to predict secondary school student performance.
  6. Ester, M., Kriegel, H. P., Sander, J., & Xu, X. (1996). A density-based algorithm for discovering clusters in large spatial databases with noise. in: Proceedings of Second International Conference on Knowledge Discovery and Data Mining, Portland, OR, 226-231.
  7. Fernandez, A., Garcia, S., Herrera, F., & Chawla, N. V. (2018). SMOTE for learning from imbalanced data: progress and challenges, marking the 15-year anniversary. Journal of artificial intelligence research, 61, 863-905. https://doi.org/10.1613/jair.1.11192
  8. Gazzah, S., & Amara, N. E. B. (2008, September). New oversampling approaches based on polynomial fitting for imbalanced data sets. In 2008 the eighth iapr international workshop on document analysis systems (pp. 677-684). IEEE.
  9. Ghorbani, R., & Ghousi, R. (2020). Comparing different resampling methods in predicting students' performance using machine learning techniques. IEEE Access, 8, 67899-67911. https://doi.org/10.1109/access.2020.2986809
  10. Han, H., Wang, W. Y., & Mao, B. H. (2005, August). Borderline-SMOTE: a new over-sampling method in imbalanced data sets learning. In International conference on intelligent computing (pp. 878-887). Springer, Berlin, Heidelberg.
  11. Krawczyk, B. (2016). Learning from imbalanced data: open challenges and future directions. Progress in Artificial Intelligence, 5(4), 221-232. https://doi.org/10.1007/s13748-016-0094-0
  12. Lee, D., & Kim, N. (2022). Anomaly Detection Methodology Based on Multimodal Deep Learning. Journal of Intelligence and Information Systems, 28(2), 101-125. https://doi.org/10.13088/JIIS.2022.28.2.101
  13. Nguyen, H. D., Tran, K. P., Thomassey, S., & Hamad, M. (2021). Forecasting and Anomaly Detection approaches using LSTM and LSTM Autoencoder techniques with the applications in supply chain management. International Journal of Information Management, 57, 102282. https://doi.org/10.1016/j.ijinfomgt.2020.102282
  14. Saez, J. A., Luengo, J., Stefanowski, J., & Herrera, F. (2015). SMOTE-IPF: Addressing the noisy and borderline examples problem in imbalanced classification by a re-sampling method with filtering. Information Sciences, 291, 184-203. https://doi.org/10.1016/j.ins.2014.08.051
  15. Serradilla, O., Zugasti, E., Ramirez de Okariz, J., Rodriguez, J., & Zurutuza, U. (2021). Adaptable and explainable predictive maintenance: Semi-supervised deep learning for anomaly detection and diagnosis in press machine data. Applied Sciences, 11(16), 7376. https://doi.org/10.3390/app11167376
  16. Shin, B., Lee, J., Han, S., & Park, C.-S. (2021). A Study of Anomaly Detection for ICT Infrastructure using Conditional Multimodal Autoencoder. Journal of Intelligence and Information Systems, 27(3), 57-73. https://doi.org/10.13088/JIIS.2021.27.3.057
  17. Wu, G., & Chang, E. Y. (2003, August). Class-boundary alignment for imbalanced dataset learning. In ICML 2003 workshop on learning from imbalanced data sets II, Washington, DC (pp. 49-56).