The Korean Journal of Applied Statistics (응용통계연구)

The Korean Statistical Society (한국통계학회)
권호 표지
DOI QR코드

DOI QR Code

Reliability in longitudinal study

종단적 연구의 신뢰도

  • Jinuk Kim (Department of Sport and Exercise Sciences, Kunsan National University)
  • Received : 2023.09.03
  • Accepted : 2023.12.21
  • Published : 2024.02.29
Download PDF
⟨ Previous Next ⟩

Abstract

The purpose of this study is to investigate retest reliabilities in longitudinal study, the same test is administered repeatedly over time. Linear mixed models were used to establish various situations of tests occurred in longitudinal study. Combination of two types of true value and three types of systematic error was considered. In order to apply the models to real longitudinal data, height data from the Berkeley growth study and vocabulary score data from the University of Chicago experimental school were used. Using the mixed model, there is an advantage that the reliability can be determined by selecting the covariance structure of the true value and the error separately. However, in order to properly analyze the reliability, researchers need to consider variations that can occur in measurement, such as characteristics of subject, the test, and the the treatment applied in the study. And the proper model should be selected and the quality of the measurement should be evaluated for each trial.

본 연구의 목적은 동일 검사의 반복 시행이 이루어지는 종단적 연구의 재검사 신뢰도를 알아보는 것이다. 종단적 연구에서 나타날 수 있는 다양한 상황의 측정을 혼합모형을 이용하여 수립하였다. 참값은 안정성과 불안정성, 계통오차는 세 가지 형태의 조합에 의한 측정 모형을 고려하였으며 신뢰도를 정의하였다. 실제 종단적 자료의 적용과 신뢰도 분석을 위해 버클리 성장연구에서의 키 자료와 시카고 대학 실험학교에서의 어휘점수 자료를 이용했다. 혼합모형을 이용하면 참값과 오차의 공분산 구조를 따로 선택하여 신뢰도를 알아볼 수 있는 장점이 있다. 그러나 연구자는 종단적 연구의 적절한 신뢰도 분석을 위해 피험자와 검사, 그리고 연구에서 가해지는 처리의 특성과 같이 측정에서 발생가능한 변동 요인을 잘 살펴볼 필요가 있다. 그리고 이에 맞는 모형을 선택하고 시행마다 다른 신뢰도 계수로 측정의 질을 평가해야 한다.

Keywords

References

  1. Bentler PM (2017). Specificity-enhanced reliability coefficients, Psychological Methods, 22, 527-540.  https://doi.org/10.1037/met0000092
  2. Blok H and Saris WE (1983). Using longitudinal data to estimate reliability, Applied Psychological Measurement, 7, 295-301.  https://doi.org/10.1177/014662168300700305
  3. Bost JE (1995). The effects of correlated errors on generalizability and dependability coefficients, Applied Psychological Measurement, 19, 191-203.  https://doi.org/10.1177/014662169501900206
  4. Box GE (1954). Some theorems on quadratic forms applied in the study of analysis of variance problems: II. Effects of inequality of variance and of correlation between errors in the two-way classification, The Annals of Mathematical Statistics, 25, 484-498.  https://doi.org/10.1214/aoms/1177728717
  5. Briesch AM, Swaminathan H, Welsh M, and Chafouleas SM (2014). Generalizability theory: A practical guide to study design, implementation, and interpretation, Journal of School Psychology, 52, 13-35.  https://doi.org/10.1016/j.jsp.2013.11.008
  6. Cronbach LJ (1951). Coefficient alpha and the internal structure of tests, Psychometrika, 16, 297-334.  https://doi.org/10.1007/BF02310555
  7. Cronbach LJ, Rajaratnam N, and Gleser GC (1963). Theory of generalizability: A liberalization of reliability theory, British Journal of Statistical Psychology, 16, 137-163.  https://doi.org/10.1111/j.2044-8317.1963.tb00206.x
  8. Deshon RP, Ployhart RE, and Sacco JM (1998). The estimation of reliability in longitudinal models, International Journal of Behavioral Development, 22, 493-515.  https://doi.org/10.1080/016502598384243
  9. Fitzmaurice GM, Laird NM, and Ware JH (2011). Applied Longitudinal Analysis (2nd ed), John Wiley & Sons, Hoboken, NJ. 
  10. Guttman L (1945). A basis for analyzing test-retest reliability, Psychometrika, 10, 255-282.  https://doi.org/10.1007/BF02288892
  11. Heise DR (1969). Separating reliability and stability in test-retest correlation, American Sociological Review, 34, 93-101.  https://doi.org/10.2307/2092790
  12. Jagodzinski W and Kuhnel SM (1987). Estimation of reliability and stability in single-indicator multiple-wave models, Sociological Methods and Research, 15, 219-258.  https://doi.org/10.1177/0049124187015003003
  13. Jennrich RI and Schluchter MD (1986). Unbalanced repeated-measures models with structured covariance matrices, Biometrics, 42, 805-820.  https://doi.org/10.2307/2530695
  14. Joreskog KG (1970). Estimation and testing of simplex models, British Journal of Mathematical and Statistical Psychology, 23, 121-145.  https://doi.org/10.1111/j.2044-8317.1970.tb00439.x
  15. Joreskog KG (1971). Statistical analysis of sets of congeneric tests, Psychometrika, 36, 109-133.  https://doi.org/10.1007/BF02291393
  16. Kim J (2022). Reliability coefficients of measurement using linear mixed model, The Korean Journal of Physical Education, 61, 359-370.  https://doi.org/10.23949/kjpe.2022.9.61.6.26
  17. Littell RC, Pendergast J, and Natarajan R (2000). Modelling covariance structure in the analysis of repeated measures data, Statistics in Medicine, 19, 1793-1819.  https://doi.org/10.1002/1097-0258(20000715)19:13<1793::AID-SIM482>3.0.CO;2-Q
  18. Lord FM and Novick MR (2008). Statistical Theories of Mental Test Scores, Information Age Publishing (Original work published 1968). 
  19. Marcoulides KM (2019). Reliability estimation in longitudinal studies using latent growth curve modeling, Measurement: Interdisciplinary Research and Perspectives, 17, 67-77.  https://doi.org/10.1080/15366367.2018.1522169
  20. Marsh HW and Grayson D (1994). Longitudinal confirmatory factor analysis: Common, time-specific, itemspecific, and residual-error components of variance, Structural Equation Modeling, 1, 116-145.  https://doi.org/10.1080/10705519409539968
  21. Maxwell AE (1968). The effect of correlated errors on estimates of reliability coefficients, Educational and Psychological Measurement, 28, 803-811.  https://doi.org/10.1177/001316446802800309
  22. Miller MB (1995). Coefficient alpha: A basic introduction from the perspectives of classical test theory and structural equation modeling, Structural Equation Modeling, 2, 255-273.  https://doi.org/10.1080/10705519509540013
  23. Palmquist B and Green DP (1992). Estimation of models with correlated measurement errors from panel data, Sociological Methodology, 22, 119-146.  https://doi.org/10.2307/270994
  24. Raffalovich LE and Bohrnstedt GW (1987). Common, specific, and error variance components of factor models: Estimation with longitudinal data, Sociological Methods & Research, 15, 385-405.  https://doi.org/10.1177/0049124187015004003
  25. Schmidt FL, Le H, and Ilies R (2003). Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs, Psychological Methods, 8, 206-224.  https://doi.org/10.1037/1082-989X.8.2.206
  26. Sideridis GD, Tsaousis I, and Al-Sadaawi A (2019). An application of reliability estimation in longitudinal designs through modeling item-specific error variance, Educational and Psychological Measurement, 79, 1038-1063.  https://doi.org/10.1177/0013164419843162
  27. Smith KW (1974). On estimating the reliability of composite indexes through factor analysis, Sociological Methods & Research, 2, 485-510.  https://doi.org/10.1177/004912417400200405
  28. Smith PL and Luecht RM (1992). Correlated effects in generalizability studies, Applied Psychological Measurement, 16, 229-235.  https://doi.org/10.1177/014662169201600302
  29. Taylor JR (1997). An Introduction to Error Analysis (2nd ed), University Science Books. 
  30. Tisak J and Tisak MS (1996). Longitudinal models of reliability and validity: A latent curve approach, Applied Psychological Measurement, 20, 275-288.  https://doi.org/10.1177/014662169602000307
  31. Tuddenham RD and Snyder MM (1954). Physical growth of California boys and girls from birth to age 18, University of California Publications in Child Development, 1, 183-364. 
  32. Vispoel WP, Morris CA, and Kilinc M (2018). Applications of generalizability theory and their relations to classical test theory and structural equation modeling, Psychological Methods, 23, 1-26.  https://doi.org/10.1037/met0000107
  33. Werts CE, Breland HM, Grandy J, and Rock DR (1980). Using longitudinal data to estimate reliability in the presence of correlated measurement errors, Educational and Psychological Measurement, 40, 19-29.  https://doi.org/10.1177/001316448004000103
  34. Werts C, Linn RL, and Joreskog KG (1978). Reliability of college grades from longitudinal data, Educational and Psychological Measurement, 38, 89-95.  https://doi.org/10.1177/001316447803800113
  35. Wiley DE and Wiley JA (1970). The estimation of measurement error in panel data, American Sociological Review, 35, 112-117.  https://doi.org/10.2307/2093858
  36. Wolfinger R (1993). Covariance structure selection in general mixed models, Communications in Statistics: Simulation and Computation, 22, 1079-1106.  https://doi.org/10.1080/03610919308813143