Health Policy and Management (보건행정학회지)

Korean Academy of Health Policy and Management (한국보건행정학회)
권호 표지
DOI QR코드

DOI QR Code

Evaluating the Performance of the Emergency Medical Services Index

  • Eun, Sang Jun (Regional Cardiocerebrovascular Center, Seoul National University Bundang Hospital) ;
  • Lee, Jin-Seok (Department of Health Policy and Management, Seoul National University College of Medicine, Institute of Health Policy and Management, Seoul National University Medical Research Center) ;
  • Kim, Yoon (Department of Health Policy and Management, Seoul National University College of Medicine, Institute of Health Policy and Management, Seoul National University Medical Research Center) ;
  • Jung, Koo Young (Department of Emergency Medicine, Ewha Womans University School of Medicine) ;
  • Park, Sue Kyung (Department of Preventive Medicine, Seoul National University College of Medicine) ;
  • Lee, Jin Yong (Public Health Medical Service, Seoul Metropolitan Government Seoul National University Boramae Medical Center)
  • Received : 2013.03.08
  • Accepted : 2013.05.20
  • Published : 2013.06.30
Download PDF
⟨ Previous Next ⟩

Abstract

Background: In 2006 Emergency Medical Services Index (EMSI), which summarizes the performance of regional emergency medical services system, was developed. This study assesses the performance of the EMSI to help determine whether EMSI can be used as evaluation tool. Methods: To build a composite score of the EMSI from predefined 24 indicators, 3 normalized values were calculated for each indicator, the normalized values of each indicator were weighted using 4 weighting methods, and the weighted values were aggregated into the final composite score using 2 aggregation schemes. The performance of EMSI was evaluated using 3 criteria: discrimination, construct validity, and sensitivity. Discrimination was the proportion of regions that did not include the overall median rank in the 5th to 95th percentiles rank interval, which was calculated from Monte Carlo simulation. Construct validity was a correlation among the alternative EMSIs. Sensitivity of EMSIs was evaluated by total shift of quartile membership and changes of 5th to 95th percentile intervals. Results: The total discrimination performance of the EMSI was 50.0%. Correlation coefficients between EMSIs using standardized values and those using rescaled values ranged from 0.621 to 0.997. Variation of the quartile membership of regions ranged from 0.0% to 75.0%. The total change in the 5th to 95th percentile intervals ranged from -19 to +17 places. Conclusion: The results suggested that the EMSI could be used as a tool for evaluating quality of regional EMS system and for identifying the areas for quality improvement.

Keywords

References

  1. Holroyd BR, Nabors MD. Changing the system. In: Polsky SS; American College of Emergency Physicians, editors. Continuous quality improvement in EMS. Dallas (TX): American College of Emergency Physicians; 1992. p. 248-250
  2. Institute of Medicine; Committee on the Future of Emergency Care in the United States Health System. Emergency medical services at the crossroads. Washington (DC): National Academies Press; 2006.
  3. American College of Emergency Physicians. The national report card on the state of emergency medicine [Internet]. Dallas (TX): American College of Emergency Physicians; 2006 [cited 2006 May 8].
  4. Becher JW Jr. The national report card on the state of emergency medicine. J Am Osteopath Assoc 2006;106(5):254-255.
  5. Kang CH, Kim Y, Lee PS, Kwon YD, Kim CY, Shin YS. Performance evaluation of emergency medical center. Korean J Prev Med 1997;30(4): 884-892.
  6. Im JS, Lee G, Kang KH, Nah BJ, Im YS, Seo HJ, et al. Status of and plans for emergency medical services in rural regions. Seoul: National Emergency Medical Center, Gachon University of Medicine and Science; 2007.
  7. Kim Y, Jung KY, Kim H, Shin SD, Lee JS. Study on the development of the Emergency Medical Services Index and evaluation methodology. Seoul: Ministry of Health and Welfare; 2006.
  8. Nardo M, Saisana M, Saltelli A, Tarantola S. Tools for composite indicators building. Ispra: European Commission, Joint Research Centre; 2005.
  9. Nardo M, Saisana M, Saltelli A, Tarantola S, Hoffman A. Handbook on constructing composite indicators: methodology and user guide. Paris: Organization for Economic Cooperation and Development Publishing; 2005.
  10. Saisana M, Saltelli A, Tarantola S. Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. J R Stat Soc Series A 2005;168(2):307-323. https://doi.org/10.1111/j.1467-985X.2005.00350.x
  11. McGlynn EA, Asch SM. Developing a clinical performance measure. Am J Prev Med 1998;14(3 Suppl):14-21. https://doi.org/10.1016/S0749-3797(97)00032-9
  12. Reeves D, Campbell SM, Adams J, Shekelle PG, Kontopantelis E, Roland MO. Combining multiple indicators of clinical quality: an evaluation of different analytic approaches. Med Care 2007;45(6):489-496. https://doi.org/10.1097/MLR.0b013e31803bb479
  13. Cherchye L, Moesen W, Rogge N, Van Puyenbroeck T. An introduction to 'benefit of the doubt' composite indicators. Soc Indic Res 2007;82(1): 111-145. https://doi.org/10.1007/s11205-006-9029-7
  14. Mononen E. Uncertainty and sensitivity analysis on technology barometer composite indicator [Internet]. Helsinki: Helsinki University of Technology; 2007 [cited 2007 May 14].
  15. Jacobs R, Goddard M, Smith PC. How robust are hospital ranks based on composite performance measures? Med Care 2005;43(12):1177-1184. https://doi.org/10.1097/01.mlr.0000185692.72905.4a
  16. Agency for Healthcare Research and Quality. AHRQ quality indicators: patient safety indicators (PSI) composite measure draft technical report [Internet]. Rockville (MD): Agency for Healthcare Research and Quality; 2006 [cited 2007 Aug 28].
  17. Agency for Healthcare Research and Quality. AHRQ quality indicators: inpatient quality indicators (IQI) composite measure draft technical report [Internet]. Rockville (MD): Agency for Healthcare Research and Quality; 2006 [cited 2007 Aug 28].
  18. Daley J, Ash AS, Iezzoni LI. Validity and reliability of risk adjusters. In: Iezzoni LI, editor. Risk adjustment for measuring health care outcomes. 3rd ed. Chicago: Health Administration Press; 2003. p. 207-215.
  19. Lun G, Holzer D, Tappeiner G, Tappeiner U. The stability of rankings derived from composite indicators: analysis of the “Il Sole 24 Ore” quality of life report. Soc Indic Res 2006;77(2):307-331. https://doi.org/10.1007/s11205-005-4505-z
  20. Freudenberg M. Composite indicators of country performance: a critical assessment. OECD Science, Technology and Industry Working Papers 2003/16 [Internet]. Paris: Organization for Economic Cooperation and Development Publishing; 2003 [cited 2007 Aug 27].
  21. Graff L, Stevens C, Spaite D, Foody J. Measuring and improving quality in emergency medicine. Acad Emerg Med 2002;9(11):1091-1107. https://doi.org/10.1111/j.1553-2712.2002.tb01563.x
  22. Kizer KW. The emerging imperative for health care quality improvement. Acad Emerg Med 2002;9(11):1078-1084. https://doi.org/10.1111/j.1553-2712.2002.tb01561.x