Journal of Intelligence and Information Systems (지능정보연구)

Korea Intelligent Information System Society (한국지능정보시스템학회)
권호 표지
DOI QR코드

DOI QR Code

The Effect of Meta-Features of Multiclass Datasets on the Performance of Classification Algorithms

다중 클래스 데이터셋의 메타특징이 판별 알고리즘의 성능에 미치는 영향 연구

  • Received : 2020.01.11
  • Accepted : 2020.03.14
  • Published : 2020.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

Big data is creating in a wide variety of fields such as medical care, manufacturing, logistics, sales site, SNS, and the dataset characteristics are also diverse. In order to secure the competitiveness of companies, it is necessary to improve decision-making capacity using a classification algorithm. However, most of them do not have sufficient knowledge on what kind of classification algorithm is appropriate for a specific problem area. In other words, determining which classification algorithm is appropriate depending on the characteristics of the dataset was has been a task that required expertise and effort. This is because the relationship between the characteristics of datasets (called meta-features) and the performance of classification algorithms has not been fully understood. Moreover, there has been little research on meta-features reflecting the characteristics of multi-class. Therefore, the purpose of this study is to empirically analyze whether meta-features of multi-class datasets have a significant effect on the performance of classification algorithms. In this study, meta-features of multi-class datasets were identified into two factors, (the data structure and the data complexity,) and seven representative meta-features were selected. Among those, we included the Herfindahl-Hirschman Index (HHI), originally a market concentration measurement index, in the meta-features to replace IR(Imbalanced Ratio). Also, we developed a new index called Reverse ReLU Silhouette Score into the meta-feature set. Among the UCI Machine Learning Repository data, six representative datasets (Balance Scale, PageBlocks, Car Evaluation, User Knowledge-Modeling, Wine Quality(red), Contraceptive Method Choice) were selected. The class of each dataset was classified by using the classification algorithms (KNN, Logistic Regression, Nave Bayes, Random Forest, and SVM) selected in the study. For each dataset, we applied 10-fold cross validation method. 10% to 100% oversampling method is applied for each fold and meta-features of the dataset is measured. The meta-features selected are HHI, Number of Classes, Number of Features, Entropy, Reverse ReLU Silhouette Score, Nonlinearity of Linear Classifier, Hub Score. F1-score was selected as the dependent variable. As a result, the results of this study showed that the six meta-features including Reverse ReLU Silhouette Score and HHI proposed in this study have a significant effect on the classification performance. (1) The meta-features HHI proposed in this study was significant in the classification performance. (2) The number of variables has a significant effect on the classification performance, unlike the number of classes, but it has a positive effect. (3) The number of classes has a negative effect on the performance of classification. (4) Entropy has a significant effect on the performance of classification. (5) The Reverse ReLU Silhouette Score also significantly affects the classification performance at a significant level of 0.01. (6) The nonlinearity of linear classifiers has a significant negative effect on classification performance. In addition, the results of the analysis by the classification algorithms were also consistent. In the regression analysis by classification algorithm, Naïve Bayes algorithm does not have a significant effect on the number of variables unlike other classification algorithms. This study has two theoretical contributions: (1) two new meta-features (HHI, Reverse ReLU Silhouette score) was proved to be significant. (2) The effects of data characteristics on the performance of classification were investigated using meta-features. The practical contribution points (1) can be utilized in the development of classification algorithm recommendation system according to the characteristics of datasets. (2) Many data scientists are often testing by adjusting the parameters of the algorithm to find the optimal algorithm for the situation because the characteristics of the data are different. In this process, excessive waste of resources occurs due to hardware, cost, time, and manpower. This study is expected to be useful for machine learning, data mining researchers, practitioners, and machine learning-based system developers. The composition of this study consists of introduction, related research, research model, experiment, conclusion and discussion.

기업의 경쟁력 확보를 위해 판별 알고리즘을 활용한 의사결정 역량제고가 필요하다. 하지만 대부분 특정 문제영역에는 적합한 판별 알고리즘이 어떤 것인지에 대한 지식은 많지 않아 대부분 시행착오 형식으로 최적 알고리즘을 탐색한다. 즉, 데이터셋의 특성에 따라 어떠한 분류알고리즘을 채택하는 것이 적합한지를 판단하는 것은 전문성과 노력이 소요되는 과업이었다. 이는 메타특징(Meta-Feature)으로 불리는 데이터셋의 특성과 판별 알고리즘 성능과의 연관성에 대한 연구가 아직 충분히 이루어지지 않았기 때문이며, 더구나 다중 클래스(Multi-Class)의 특성을 반영하는 메타특징에 대한 연구 또한 거의 이루어진 바 없다. 이에 본 연구의 목적은 다중 클래스 데이터셋의 메타특징이 판별 알고리즘의 성능에 유의한 영향을 미치는지에 대한 실증 분석을 하는 것이다. 이를 위해 본 연구에서는 다중 클래스 데이터셋의 메타특징을 데이터셋의 구조와 데이터셋의 복잡도라는 두 요인으로 분류하고, 그 안에서 총 7가지 대표 메타특징을 선택하였다. 또한, 본 연구에서는 기존 연구에서 사용하던 IR(Imbalanced Ratio) 대신 시장집중도 측정 지표인 허핀달-허쉬만 지수(Herfindahl-Hirschman Index, HHI)를 메타특징에 포함하였으며, 역ReLU 실루엣 점수(Reverse ReLU Silhouette Score)도 새롭게 제안하였다. UCI Machine Learning Repository에서 제공하는 복수의 벤치마크 데이터셋으로 다양한 변환 데이터셋을 생성한 후에 대표적인 여러 판별 알고리즘에 적용하여 성능 비교 및 가설 검증을 수행하였다. 그 결과 대부분의 메타특징과 판별 성능 사이의 유의한 관련성이 확인되었으며, 일부 예외적인 부분에 대한 고찰을 하였다. 본 연구의 실험 결과는 향후 메타특징에 따른 분류알고리즘 추천 시스템에 활용할 것이다.

Keywords

References

  1. Andric, K., D. Kalpic, and Z. Bohacek, "An Insight into the Effects of Class Imbalance and Sampling on Classification Accuracy in Credit Risk Assessment," Computer Science and Information Systems, Vol.16, No.1(2019), 155-178. https://doi.org/10.2298/CSIS180110037A
  2. Bader-El-Den, M. E. Teitei, and T. Perry, "Biased Random Forest for Dealing with the Class Imbalance Problem," IEEE Transactions on Neural Networks and Learning Systems, Vol.30, No.7(2018), 2163-2172. https://doi.org/10.1109/tnnls.2018.2878400
  3. Brazdil, P., J.Gama, and B.Henery, "Characterizing the Applicability of Classification Algorithms using Meta-Level Learning," In European Conference on Machine Learning (1994, April), 83-102.
  4. Brissaud, J. B, "The Meanings of Entropy," Entropy, Vol.7, No.1(2005), 68-96. https://doi.org/10.3390/e7010068
  5. Chaitra, P. C., and D. R. S. Kumar, "A Review of Multi-Class Classification Algorithms," International Journal of Pure and Applied Mathematics, Vol.118, No.14(2018), 17-26.
  6. Chan, P. P., Z. M. He, H. Li, and C. C.Hsu, "Data Sanitization Against Adversarial Label Contamination based on Data Complexity," International Journal of Machine Learning and Cybernetics, Vol.9, No.6(2018), 1039-1052. https://doi.org/10.1007/s13042-016-0629-5
  7. Chawla, N. V., N. Japkowicz, and A. Kotcz, "Special Issue on Learning from Imbalanced Data Sets," ACM Sigkdd Explorations Newsletter, Vol.6, No.1(2004), 1-6. https://doi.org/10.1145/1007730.1007733
  8. Del Rio, S., V. Lopez, J. M. Benitez, and F. Herrera, "On the use of MapReduce for Imbalanced Big Data using Random Forest," Information Sciences, Vol. 285(2014), 112-137. https://doi.org/10.1016/j.ins.2014.03.043
  9. Dogan, N., and Z. Tanrikulu, "A Comparative Analysis of Classification Algorithms in Data Mining for Accuracy, Speed and Robustness," Information Technology and Management, Vol.14, No.2(2013), 105-124. https://doi.org/10.1007/s10799-012-0135-8
  10. Domingos, P. and M. Pazzani, "Beyond Independence: Conditions for the Optimality of the Simple Bayesian Classifier," in Proceedings of the 13th International Conference on Machine Learning, (1996) 105-122.
  11. Galar, G., M. Fernandez, A. Barrenechea, E. Bustince, and F. Herrera, "A Review on Ensembles for the Class Imbalance Problem: Bagging-, Boosting-, and Hybrid-Based Approaches," IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), Vol.42, No.4(2011), 463-484. https://doi.org/10.1109/TSMCC.2011.2161285
  12. Garcia, L. P., A. C. de Carvalho, and A. C. Lorena, "Effect of Label Noise in the Complexity of Classification Problems," Neurocomputing, Vol.160(2015), 108-119. https://doi.org/10.1016/j.neucom.2014.10.085
  13. Garcia, L. P., A. C. Lorena, M. C.de Souto, and T. K. Ho, "Classifier Recommendation using Data Complexity Measures," In 2018 24th International Conference on Pattern Recognition (ICPR), (2018, August), 874-879.
  14. Guyon, I., and A. Elisseeff, "An Introduction to Variable and Feature Selection," Journal of Machine Learning Research, Vol.3(Mar, 2003), 1157-1182.
  15. Japkowicz, N., and S. Stephen, "The Class Imbalance Problem: A Systematic Study," Intelligent Data Analysis, Vol.6, No.5(2002), 429-449. https://doi.org/10.3233/IDA-2002-6504
  16. Khoshgoftaar, T. M., J. Van Hulse, and A. Napolitano, "Comparing Boosting and Bagging Techniques with Noisy and Imbalanced data," IEEE Transactions on Systems, Man, and Cybernetics-Part A: Systems and Humans, Vol.41, No.3(2010), 552-568. https://doi.org/10.1109/TSMCA.2010.2084081
  17. Kiang, M. Y., "A Comparative Assessment of Classification Methods," Decision Support Systems, Vol.35, No.4(2003), 441-454. https://doi.org/10.1016/S0167-9236(02)00110-0
  18. Krawczyk, B. "Learning from Imbalanced Data: Open Challenges and Future Directions," Progress in Artificial Intelligence, Vol.5, No.4(2016), 221-232. https://doi.org/10.1007/s13748-016-0094-0
  19. Krawczyk, B., M. Galar, Ł. Jelen, and F. Herrera, "Evolutionary Undersampling Boosting for Imbalanced Classification of Breast Cancer Malignancy," Applied Soft Computing, Vol.38 (2016), 714-726. https://doi.org/10.1016/j.asoc.2015.08.060
  20. Kwon, O., and J. M. Sim, "Effects of Data Set Features on the Performances of Classification Algorithms," Expert Systems with Applications, Vol.40, No.5(2013), 1847-1857. https://doi.org/10.1016/j.eswa.2012.09.017
  21. L'heureux, A., K. Grolinger, H. F. Elyamany, and M. A. Capretz, "Machine Learning with Big Data: Challenges and Approaches," IEEE Access, Vol.5(2017), 7776-7797. https://doi.org/10.1109/ACCESS.2017.2696365
  22. Lopez, V., A. Fernandez, S. Garcia, V. Palade, and F. Herrera, "An Insight into Classification with Imbalanced Data: Empirical Results and Current Trends on using Data Intrinsic Characteristics," Information Sciences, Vol.250 (2013), 113-141. https://doi.org/10.1016/j.ins.2013.07.007
  23. Lorena, A. C., A. I. Maciel, P. B. de Miranda, I. G. Costa, and R. B. Prudencio, "Data Complexity Meta-Features for Regression Problems," Machine Learning, Vol.107, No.1 (2018), 209-246. https://doi.org/10.1007/s10994-017-5681-1
  24. Lotte, F., L. Bougrain, A. Cichocki, M. Clerc, M. Congedo, A. Rakotomamonjy, and F. Yger, "A Review of Classification Algorithms for EEG-based Brain-Computer Interfaces: a 10 year update," Journal of Neural Engineering, Vol.15, No.3(2018), 031005. https://doi.org/10.1088/1741-2552/aab2f2
  25. Lu, C., J. Qiao, and J. Chang, "Herfindahl-Hirschman Index based Performance Analysis on the Convergence Development," Cluster Computing, Vol.20, No.1(2017), 121-129. https://doi.org/10.1007/s10586-017-0737-3
  26. Matsumoto, A., U. Merlone, and F. Szidarovszky, "Some Notes on Applying the Herfindahl-Hirschman Index," Applied Economics Letters, Vol.19, No.2(2012), 181-184. https://doi.org/10.1080/13504851.2011.570705
  27. Moran-Fernandez, L., V. Bolon-Canedo, and A. Alonso-Betanzos, "Can Classification Performance be Predicted by Complexity Measures? A Study using Microarray Data," Knowledge and Information Systems, Vol.51, No.3(2017), 1067-1090. https://doi.org/10.1007/s10115-016-1003-3
  28. Napierala, K., and J. Stefanowski, "Types of Minority Class Examples and their Influence on Learning Classifiers from Imbalanced Data," Journal of Intelligent Information Systems, Vol.46, No.3(2016), 563-597. https://doi.org/10.1007/s10844-015-0368-1
  29. Nemhauser, G., and L. Wolsey, "The Scope of Integer and Combinatorial Optimization," Integer and Combinatorial Optimization, (Eds.) G. Nemhauser, and L. Wolsey, John Wiley & Sons, Inc., 1988, 1-26.
  30. Nguyen, H. L., Y. K. Woon, and W. K. Ng, "A Survey on Data Stream Clustering and Classification," Knowledge and information systems, Vol.45, No.3(2015), 535-569. https://doi.org/10.1007/s10115-014-0808-1
  31. Prabakaran, S., R. Sahu, and S. Verma, "Classification of Multi Class Dataset using Wavelet Power Spectrum," Data Mining and Knowledge Discovery, Vol.15, No.3(2007), 297-319. https://doi.org/10.1007/s10618-007-0068-8
  32. Qiu, J., Q. Wu, G. Ding, Y. Xu, and S. Feng, "A Survey of Machine Learning for Big Data Processing," URASIP Journal on Advances in Signal Processing, No.1(2016), 67.
  33. Rok.B and L. Lusa, "Improved Shrunken Centroid Classifiers for High-Dimensional Class-Imbalanced Data," BMC bioinformatics, Vol.14, No.1(2013), 64. https://doi.org/10.1186/1471-2105-14-64
  34. Sen, P. C., M. Hajra, and M. Ghosh, "Supervised Classification Algorithms in Machine Learning: A Survey and Review," In Emerging Technology in Modelling and Graphics. Springer, Singapore, 2020.
  35. Shannon, C. E, "A Mathematical Theory of Communication," Bell System Technical Journal, Vol.27, No.3(1948), 379-423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
  36. Sun, A., E.P. Lim, and Y. Liu, "On Strategies for Imbalanced Text Classification using SVM: A Comparative Study," Decision Support Systems, Vol.48, No.1(2009), 191-201. https://doi.org/10.1016/j.dss.2009.07.011
  37. Sun, Y., A. K. Wong, and M. S. Kamel "Classification of Imbalanced Data: A Review," International Journal of Pattern Recognition and Artificial Intelligence, Vol.23, No.04 (2009), 687-719. https://doi.org/10.1142/S0218001409007326
  38. Weiss, G. M., and F. Provost, "Learning when Training Data are Costly: The Effect of Class Distribution on Tree Induction," Journal of Artificial Intelligence Research, Vol.19(2003), 315-354. https://doi.org/10.1613/jair.1199
  39. Wolpert, D. H., and W. G. Macready, "No Free Lunch Theorems for Optimization," IEEE Transactions on Evolutionary Computation, Vol.1, No.1 (1997), 67-82. https://doi.org/10.1109/4235.585893
  40. Wu, G., and E. Y. Chang, "Aligning Boundary in Kernel Space for Learning Imbalanced Dataset," In Fourth IEEE International Conference on Data Mining (ICDM'04), 2004, November, 265-272.
  41. Wu, X., V. Kumar, J. R. Quinlan, J. Ghosh, Q. Yang, H. Motoda, ... and Z. H. Zhou, "Top 10 Algorithms in Data Mining," Knowledge and Information Systems, Vol.14, No.1(2008), 1-37. https://doi.org/10.1007/s10115-007-0114-2
  42. Zhang, M. L., J. M. Pena, and V. Robles, "Feature Selection for Multi-Label naive Bayes Classification," Information Sciences, Vol.179, No.19(2009), 3218-3229. https://doi.org/10.1016/j.ins.2009.06.010
  43. Zieba, M., and J. M. Tomczak, "Boosted SVM with Active Learning Strategy for Imbalanced Data," Soft Computing, Vol.19, No.12(2015), 3357-3368. https://doi.org/10.1007/s00500-014-1407-5
  44. Zhang, X., R. Li, B. Zhang, Y. Yang, J. Guo, and X. Ji, "An Instance-based Learning Recommendation Algorithm of Imbalance Handling Methods," Applied Mathematics and Computation, Vol.351(2019), 204-218. https://doi.org/10.1016/j.amc.2018.12.020