The Korean Journal of Applied Statistics (응용통계연구)

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

Comparison of evaluation measures for classification models on binary data

이진자료 분류모형에 대한 평가측도의 특성 비교

  • Kim, Byungsoo (Department of Statistics, Inje University) ;
  • Kwon, Soyoung (Medical Device Policy Division, Ministry of Food and Drug Safety)
  • 김병수 (인제대학교 통계학과) ;
  • 권소영 (식품의약품안전처 의료기기정책과)
  • Received : 2019.01.18
  • Accepted : 2019.02.15
  • Published : 2019.04.30
Download PDF
⟨ Previous Next ⟩

Abstract

This study investigates the characteristics of evaluation measures for classification models on a binary response variable in order to evaluate their suitability for use. Six measures are considered: Accuracy, Sensitivity, Specificity, Precision, F-measure, and the Heidke's skill score (HSS). Evaluation measures are reformulated using x(ratio of actually 1), y(ratio predicted by 1), z(ratio of both actual and predicted by 1) from the confusion matrix. We suggest two necessary conditions to assess the suitability of the evaluation measures. The first condition is that the measure function is constant for x and y in the case of a random model. The second condition is that the measure function is increasing for z and decreasing for x and y. Since only HSS satisfies the two conditions, that is always appropriate as an evaluation measure for the classification model on the binary response variable, and the other measures should be used within a limited range.

본 논문에서는 반응변수가 이진형인 분류모형에 대한 평가측도들의 특성을 파악하고 사용하기 적합한 평가측도인가를 살펴보았다. 고려한 측도는 정분류율, 민감도, 특이도, 정밀도, F-measure, HSS (Heidke's skill score)의 6개이다. 각 측도들은 이원분할표에서 x(실제로 1인 비율), y(1로 예측되는 비율), z(실제와 예측이 모두 1인 비율)을 사용하여 표현하였다. 본 연구는 평가측도가 사용하기 적합한 측도가 되기 위한 조건으로 두 가지를 제안하였다. 제1조건은 랜덤모형인 경우에 평가측도는 x와 y에 대해 상수이고, 제2조건은 평가측도의 식이 세 변수들(x, y, z) 모두로 이루어지고 z에 대해서 증가함수이고 x와 y에 대해서 감소함수이어야 한다는 것이다. HSS는 두 조건을 모두 만족하므로 이진형 반응변수의 분류모형에 대한 평가측도로 항상 사용이 적합하고, 다른 측도들은 제한된 범위 내에서만 사용하는 것이 좋다.

Keywords

GCGHDE_2019_v32n2_291_f0001.png 이미지

Figure 3.1. Contour plot of measures for x and y in case of random model.

Table 1.1. Accuracy score in each random model

GCGHDE_2019_v32n2_291_t0001.png 이미지

Table 2.1. General components of confusion matrix

GCGHDE_2019_v32n2_291_t0002.png 이미지

Table 2.2. Measures for confusion matrix

GCGHDE_2019_v32n2_291_t0003.png 이미지

Table 2.3. Confusion matrix reformulated by ratios (x, y, z)

GCGHDE_2019_v32n2_291_t0004.png 이미지

Table 3.1. Measures in case of random model (z = xy)

GCGHDE_2019_v32n2_291_t0005.png 이미지

Table 2.4. Measures reformulated by ratios (x; y; z)

GCGHDE_2019_v32n2_291_t0006.png 이미지

Table 4.1. Measures in case x is constant c

GCGHDE_2019_v32n2_291_t0007.png 이미지

Table 4.2. Measures in case y is constant c

GCGHDE_2019_v32n2_291_t0008.png 이미지

Table 4.3. Measures in case the comparability ratio is 1 (y = x)

GCGHDE_2019_v32n2_291_t0009.png 이미지

References

  1. Bekkar, M., Djemaa, H. K., and Altiouche, T. A. (2013). Evaluation measures for models assessment over imbalanced data sets, Journal of Information Engineering and Applications, 3, 27-38.
  2. Kim, B., Bae, W., Seok, K., Cho, D., and Choi, K. (2018a). SAS EM 14.1 Data Mining Basis and Application, Kyowoo, 293-317.
  3. Kim, H., Shin, D., Shin, W., and Hwang, C. (2018b). Rating Information-Aided Denoising AutoEncoder for effective collaborative filtering, The Journal of Korean Institute of Communications and Information Sciences, 43, 357-1367.
  4. Kim, M., Kim, S., and Ock, C. (2015). A predictive model of problem drinking of workers using decision tree analysis, Journal of The Korean Society of Living Environmental System, 22, 460-468. https://doi.org/10.21086/ksles.2015.06.22.3.460
  5. Kim, S. Y. (2016). The comparison of analytical models for risk factors of colonic adenomatous polyp (Master Thesis), Graduate School, Chung-Ang University.
  6. Leem, Y. M. and Ryu, C. H. (2006). A comparison of data mining techniques for predicting model of industrial accidents. In Proceedings for the Spring Conference 2006, Society of Korea Industrial and Systems Engineering, 107-113.
  7. Park, I., Kim, Y., Choi, Y., Kim, S., Kim, E., Won, S., and Kang, S. (2013). Development of advanced TB case classification model using NHI claims data, The Journal of Digital Policy & Management, 11, 289-299.
  8. Sakong, J. H. (2012). A study on predicting stock price based on data mining techniques (Master Thesis), Graduate School, Inje University.
  9. Sohn, K., Lee, J., Lee, S., and Ryu, C. (2005). Statistical models for prediction of heavy rain in Honam area, Asia-Pacific Journal of Atmospheric Sciences, 41, 897-907.
  10. Sung, O. (2013). A empirical study on the relevance of technology finance supporting business for technologically innovative SMEs, Journal of Korea Technology Innovation Society, 16, 303-322.