• Journal of Korean Medical Science
• /
• 제33권43호
• /
• pp.239.1-239.12
• /
• 2018

Background: We described a novel multi-step retinal fundus image reading system for providing high-quality large data for machine learning algorithms, and assessed the grader variability in the large-scale dataset generated with this system. Methods: A 5-step retinal fundus image reading tool was developed that rates image quality, presence of abnormality, findings with location information, diagnoses, and clinical significance. Each image was evaluated by 3 different graders. Agreements among graders for each decision were evaluated. Results: The 234,242 readings of 79,458 images were collected from 55 licensed ophthalmologists during 6 months. The 34,364 images were graded as abnormal by at-least one rater. Of these, all three raters agreed in 46.6% in abnormality, while 69.9% of the images were rated as abnormal by two or more raters. Agreement rate of at-least two raters on a certain finding was 26.7%-65.2%, and complete agreement rate of all-three raters was 5.7%-43.3%. As for diagnoses, agreement of at-least two raters was 35.6%-65.6%, and complete agreement rate was 11.0%-40.0%. Agreement of findings and diagnoses were higher when restricted to images with prior complete agreement on abnormality. Retinal/glaucoma specialists showed higher agreements on findings and diagnoses of their corresponding subspecialties. Conclusion: This novel reading tool for retinal fundus images generated a large-scale dataset with high level of information, which can be utilized in future development of machine learning-based algorithms for automated identification of abnormal conditions and clinical decision supporting system. These results emphasize the importance of addressing grader variability in algorithm developments.

• 한국전문물리치료학회지
• /
• 제29권4호
• /
• pp.262-268
• /
• 2022

Background: Virtual reality (VR) programs based on motion capture camera are the most convenient and cost-effective approaches for remote rehabilitation. Assessment of physical function is critical for providing optimal VR rehabilitation training; however, direct muscle strength measurement using camera-based kinematic data is impracticable. Therefore, it is necessary to develop a method to indirectly estimate the muscle strength of users from the value obtained using a motion capture camera. Objects: The purpose of this study was to determine whether the pedaling speed converted using the VR engine from the captured foot position data in the VR environment can be used as an indirect way to evaluate knee muscle strength, and to investigate the validity and reliability of a camera-based VR program. Methods: Thirty healthy adults were included in this study. Each subject performed a 15-second maximum pedaling test in the VR and built-in speedometer modes. In the VR speedometer mode, a motion capture camera was used to detect the position of the ankle joints and automatically calculate the pedaling speed. An isokinetic dynamometer was used to assess the isometric and isokinetic peak torques of knee flexion and extension. Results: The pedaling speeds in VR and built-in speedometer modes revealed a significantly high positive correlation (r = 0.922). In addition, the intra-rater reliability of the pedaling speed in the VR speedometer mode was good (ICC [intraclass correlation coefficient] = 0.685). The results of the Pearson correlation analysis revealed a significant moderate positive correlation between the pedaling speed of the VR speedometer and the peak torque of knee isokinetic flexion (r = 0.639) and extension (r = 0.598). Conclusion: This study suggests the potential benefits of measuring the maximum pedaling speed using 3D depth camera in a VR environment as an indirect assessment of muscle strength. However, technological improvements must be followed to obtain more accurate estimation of muscle strength from the VR cycling test.

• 한국산학기술학회논문지
• /
• 제12권1호
• /
• pp.318-326
• /
• 2011

본 연구는 절차기술의 객관구조화 진료시험(Objective Structured Clinical Examination)에서 표준화환자가 평가자의 역할을 수행할 수 있는지 알아보기 위해 신뢰도를 평가하는데 그 목적이 있다. 시험의 주제는 남성 도뇨관삽관과 창상드레싱 2가지로 정하고, 2년 이상의 객관구조화 진료시험 채점 경력이 있는 표준화환자와 교수 각 4명을 2명씩 짝을 지워 한 주제 당 표준화환자 그룹과 교수 그룹이 동시에 채점하게 하였다. 표준화환자들에게는 술기의 정의, 방법, 주의점, 후유증에 대한 교육이 이루어졌으며 동영상이 포함된 강의, 교수의 시연 후 표준화환자가 직접 실습해보고 교수로부터 되먹임을 받는 순서로 총 8시간( 주제당 4시간)의 교육이 시행되었다. 8명의 평가자 모두 객관구조화 진료시험 전날 모여 기존의 동영상자료를 이용한 가상 채점으로 1시간동안 채점 표준화를 이루었다. 채점표는 체크리스트 14문항과 총괄평가 1문항으로 이루어졌다. 한 학생당, 주제당 5분간의 시험 후 2분간의 평가가 이루어졌다. 표준화환자와 교수간의 분석은 GENOVA program을 이용하였다. 연구 결과 주제 전체에서 G상수는 0.839, 평가자의 신뢰도는 0.946으로 매우 높았다. 표준화환자그룹과 교수그룹 사이의 평가자간 일치도는 체크리스트에서 0.949, 총괄평가에서 0.908이었다. 따라서 적절한 교육이 선행되어진다면 표준화환자도 절차기술의 객관화진료시험에 신뢰할 만한 평가자로 이용되어질 수 있을 것이다.

• 의학교육논단
• /
• 제25권2호
• /
• pp.139-158
• /
• 2023

Defining a competent doctor is important for educating and training doctors. However, competency frameworks have rarely been validated during the process of their development in Korea. The purpose of this study was to validate the patient-centered doctor's competency framework, which had been developed by our expert working group (EWG). Two rounds of Delphi questionnaire surveys were conducted among a panel of experts on medicine and medical education. The panel members were provided with six core competencies, 17 sub-competencies, and 53 enabling competencies, and were asked to rate the importance of these competencies on a 5-point Likert scale. Between April and July 2021, a total of 28 experts completed both rounds. The data of the Delphi study were analyzed for the mean, standard deviation, median, inter-rater agreement (IRA), and content validity ratio (CVR). A CVR >0.36 and IRA ≥0.75 were deemed to indicate validity and agreement. This study found that five enabling competencies were not valid, and agreement was not reached for three sub-competencies and two enabling competencies. In consideration of CVR and the individual opinions of panel members at each session, the final competencies were extracted through consensus meetings of the EWG. The competencies were modified into six core competencies, 16 sub-competencies, and 47 enabling competencies. This study is meaningful in that it proposes patient-centered doctor's competencies enabling the development of residents' milestone competencies, an assessment system, and educational programs.