The Korean Journal of Applied Statistics (응용통계연구)

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

A New Measure of Agreement to Resolve the Two Paradoxes of Cohen's Kappa

COHEN의 합치도의 두 가지 역설을 해결하기 위한 새로운 합치도의 제안

  • Park, Mi-Hee (Department of Biostatistics, The Catholic University of Korea) ;
  • Park, Yong-Gyu (Department of Biostatistics, The Catholic University of Korea)
  • 박미희 (가톨릭대학교 대학원 의학통계학과) ;
  • 박용규 (가톨릭대학교 대학원 의학통계학과)
  • Published : 2007.03.31
Download PDF
⟨ Previous Next ⟩

Abstract

In a $2\times2$ table showing binary agreement between two raters, it is known that Cohen's $\kappa$, a chance-corrected measure of agreement, has two paradoxes. $\kappa$ is substantially sensitive to raters' classification probabilities(marginal probabilities) and does not satisfy conditions as a chance-corrected measure of agreement. However, $\kappa$ and other established measures have a reasonable and similar value when each marginal distribution is close to 0.5. The objectives of this paper are to present a new measure of agreement, H, which resolves paradoxes of $\kappa$ by adjusting unbalanced marginal distributions and to compare the proposed measure with established measures through some examples.

두 평정자가 있는 이차원 분류표에서 우연에 의한 합치 비율을 보정한 Cohen의 합치도($\kappa$)는 문제점이 있는 측도로 알려져 있다. $\kappa$는 평정자의 분류 비율(주변확률)에 매우 민감하고, 합치도로서의 조건도 만족하지 못한다. 그러나 $\kappa$를 비롯한 기존 합치도들은 주변확률이 0.5에 가까울 때 안정되며 비슷한 값을 갖는다. 본 연구에서는 이차원 분류표의 불균형적 주변분포를 보정함으로써 $\kappa$의 역설을 해결하는 새로운 합치도 H를 제안한다. 또한 예제를 통해 기존의 합치도들과 제안된 합치도를 비교한다.

Keywords

References

  1. Aickin, M. (1990). Maximum likelihood estimation of agreement in the constant predictive probability model, and its relation to Cohen's kappa, Biometrics, 46, 293-302 https://doi.org/10.2307/2531434
  2. Andres, A. M. and Marzo, P. F. (2004). Delta: A new measure of agreement between two raters, The British Journal of Mathematical and Statistical Psychology, 57, 1-19 https://doi.org/10.1348/000711004849268
  3. Bennet, E. M., Alpert, R. and Goldstein, A. C. (1954). Communications through limited response questionning, Public Opinion Quarterly, 18, 303-308 https://doi.org/10.1086/266520
  4. Bishop, Y. M. M., Fienberg, S. E. and Holland, P. W. (1995). Discrete Multivariate Analysis, MIT Press, Cambridge, Mass
  5. Brennan, R. L. and Prediger, D. (1981). Coefficient kappa: Some uses, misuses, and alternatives, Educational and Psychological Measurement, 41, 687-699 https://doi.org/10.1177/001316448104100307
  6. Byrt, T., Bishop, J. and Carlin, J. B. (1993). Bias, prevalence and kappa, Journal of Clinical Epidemiology, 46, 423-429 https://doi.org/10.1016/0895-4356(93)90018-V
  7. Cicchetti, D. V. and Feinstein, A. R. (1990). High agreement but low kappa: 2. Resolving the paradoxes, Journal of Clinical Epidemiology, 43, 551-558 https://doi.org/10.1016/0895-4356(90)90159-M
  8. Cohen, J. (1960). A coefficient of agreement for nominal scales, Educational and Psychological Measurement, 20, 37-46 https://doi.org/10.1177/001316446002000104
  9. Feinstein, A. R. and Cicchetti, D. V. (1990). High agreement but low kappa: 1. The problems of two paradoxes, Journal of Clinical Epidemiology, 43, 543-549 https://doi.org/10.1016/0895-4356(90)90158-L
  10. Ferger, W. F. (1931). The nature and use of the harmonic mean, Journal of the American Statistical Association, 26, 36-40
  11. Gwet, K. (2001). Handbook of inter-rater reliability, STATAXIS Publishing company, Gaithersburg
  12. Holley, J. W. and Guilford, J. P. (1964). A note on the G index of agreement, Educational and Psychological Measurement, 24, 749-753 https://doi.org/10.1177/001316446402400402
  13. Janson, S. and Vegelius, J. (1979). On generalizations of the G index and the PHI coefficient to nominal scales, Multivariate Behavioral Research, 14, 255-269 https://doi.org/10.1207/s15327906mbr1402_9
  14. Lantz, C. A. and Nebenzahl, E. (1996). Behavior and interpretation of the K, statistic: Resolution of the two paradoxes, Journal of Clinical Epidemiology, 49, 431-434 https://doi.org/10.1016/0895-4356(95)00571-4
  15. Scott, W. A. (1955). Reliability of content analysis: The case of nominal scale coding, Public Opinion Quarterly, 19, 321-325 https://doi.org/10.1086/266577

Cited by

  1. A Study on Comparison of Generalized Kappa Statistics in Agreement Analysis vol.25, pp.5, 2012, https://doi.org/10.5351/KJAS.2012.25.5.719