The Korean Journal of Applied Statistics (응용통계연구)

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

Standard Criterion of VUS for ROC Surface

ROC 곡면에서 VUS의 판단기준

  • Hong, C.S. (Department of Statistics, Sungkyunkwan University) ;
  • Jung, E.S. (Department of Statistics, Sungkyunkwan University) ;
  • Jung, D.G. (Department of Statistics, Sungkyunkwan University)
  • Received : 2013.10.01
  • Accepted : 2013.10.28
  • Published : 2013.12.31
Download PDF
⟨ Previous Next ⟩

Abstract

Many situations are classified into more than two categories in real world. In this work, we consider ROC surface and VUS, which are graphical representation methods for classification models with three categories. The standard criteria of AUC for the probability of default based on Basel II is extended to the VUS for ROC surface; therefore, the standardized criteria of VUS for the classification model is proposed. The ranges of AUC, K-S and mean difference statistics corresponding to VUS values for each class of the standard criteria are obtained. The standard criteria of VUS for ROC surface can be established by exploring the relationships of these statistics.

현실세계에는 두 가지 범주 이상으로 분류되는 경우가 많이 존재한다. 본 논문은 분류범주가 세 종류인 분류모형을 시각적으로 표현하는 방법인 ROC 곡면과 이 곡면 아래의 체적을 나타내는 VUS 통계량을 고려한다. 바젤 II에 기반한 부도확률에 관한 AUC 통계량의 판단기준을 ROC 곡면에서의 VUS에 대하여 확장하여, VUS에 의한 판별력 판단기준 13단계를 제안한다. 제안한 판단기준 각 단계에서의 VUS값에 대응하는 AUC, K-S 통계량 그리고 세 분포의 평균차이에 대한 범위를 탐색하고, 이들의 관계를 살펴봄으로써 VUS 통계량의 판별력 판단기준을 설정한다.

Keywords

References

  1. Bradley, A. P. (1997). The use of the area under the ROC curve in the evaluation of machine learning algorithms, Pattern Recognition, 30, 1145-1159. https://doi.org/10.1016/S0031-3203(96)00142-2
  2. Dreiseitl, S., Ohno-Machado, L. and Binder, M. (2000). Comparing three-class diagnostic tests by three-way ROC analysis, Medical Decision Making, 20, 323-331. https://doi.org/10.1177/0272989X0002000309
  3. Engelmann, B., Hayden, E. and Tasche, D. (2003). Measuring the discriminative power of rating systems, Risk, 82-86.
  4. Fawcett, T. (2003). ROC graphs: notes and practical considerations for data mining researchers, HP Labs Tech Report HPL-2003-4.
  5. Hanley, J. A. and McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve, Radiology, 143, 29-36. https://doi.org/10.1148/radiology.143.1.7063747
  6. Heckerling, P. S. (2001). Parametric three-way receiver operating characteristic surface analysis using mathematica, Medical Decision Making, 21, 409-417. https://doi.org/10.1177/02729890122062703
  7. Hong, C. S. and Choi, J. S. (2009). Optimal threshold from ROC and CAP curves, The Korean Journal of Applied Statistics, 22, 911-921. https://doi.org/10.5351/KJAS.2009.22.5.911
  8. Hong, C. S., Joo, J. S. and Choi, J. S. (2010). Optimal thresholds from mixture distributions, The Korean Journal of Applied Statistics, 23, 13-28. https://doi.org/10.5351/KJAS.2010.23.1.013
  9. Hosmer, D. W. and Lemeshow, S. (2000). Applied Logistic Regression, John Wiley & Sons, New York.
  10. Joseph, M. P. (2005). A PD validation framework for Basel II internal ratings-based systems, Credit Scoring and Credit Control IV.
  11. Lim, C. K. (2005). Introduction of goodness-of-fit test methods for credit evaluation system, Financial Supervisory Service, Risk Review, 33-54.
  12. Mossman, D. (1999). Three-way ROCs, Medical Decision Making, 19, 78-89. https://doi.org/10.1177/0272989X9901900110
  13. Nakas, C. T., Alonzo, T. A. and Yiannoutsos, C. T. (2010). Accuracy and cut off point selection in three class classification problems using a generalization of the Youden index, Statistics in Medicine, 29, 2946-2955. https://doi.org/10.1002/sim.4044
  14. Nakas, C. T. and Yiannoutsos, C. T. (2004). Ordered multiple-class ROC analysis with continuous measurements, Statistics in Medicine, 23, 3437-3449. https://doi.org/10.1002/sim.1917
  15. Patel, A. C. and Markey, M. K. (2005). Comparison of three-class classification performance metrics: a case study in breast cancer CAD, International Society for Optical Engineering, 5749, 581-589.
  16. Provost, F. and Fawcett, T. (2001). Robust classification for imprecise environments, Machine Learning, 42, 203-231. https://doi.org/10.1023/A:1007601015854
  17. Scurfield, B. K. (1996). Multiple-event forced-choice tasks in the theory of signal detectability, Journal of Mathematical Psychology, 40, 253-269. https://doi.org/10.1006/jmps.1996.0024
  18. Sobehart, J. R. and Keenan, S. C. (2001). Measuring default accurately, Credit Risk Special Report, Risk, 14, 31-33.
  19. Tasche, D. (2006). Validation of internal rating systems and PD estimates, The Analytics of Risk Model Validation, 28, 169-196.
  20. Thomas, L. C., Edelman, D. B. and Crook, J. N. (2004). Readings in Credit Scoring: Foundations, Developments, and Aims, Oxford finance, Oxford University Press, New York.
  21. Wandishin, M. S. and Mullen, S. J. (2009). Multiclass ROC analysis, Weather and Forecasting, 24, 530-547. https://doi.org/10.1175/2008WAF2222119.1
  22. Wilkie, A. D. (2004). Measures for comparing scoring systems, in readings in credit scoring-recent decelopments, Advances, and Aims, Oxford Finance.
  23. Zou, K. H. (2002). Receiver operation characteristic literature research, On-line ibliography available from: http://www.spl.havard.edu/pages/ppl/zou/roc.html.

Cited by

  1. ROC curve and AUC for linear growth models vol.26, pp.6, 2015, https://doi.org/10.7465/jkdi.2015.26.6.1367
  2. Proposition of polytomous discrimination index and test statistics vol.27, pp.2, 2016, https://doi.org/10.7465/jkdi.2016.27.2.337
  3. Alternative Optimal Threshold Criteria: MFR vol.27, pp.5, 2014, https://doi.org/10.5351/KJAS.2014.27.5.773
  4. Standard criterion of hypervolume under the ROC manifold vol.25, pp.3, 2014, https://doi.org/10.7465/jkdi.2014.25.3.473
  5. VUS and HUM Represented with Mann-Whitney Statistic vol.22, pp.3, 2015, https://doi.org/10.5351/CSAM.2015.22.3.223
  6. Test Statistics for Volume under the ROC Surface and Hypervolume under the ROC Manifold vol.22, pp.4, 2015, https://doi.org/10.5351/CSAM.2015.22.4.377