Korean Journal of Radiology

  • 제24권12호
  • /
  • Pages.1179-1189
  • /
  • 2023
  • /
  • 1229-6929(pISSN)
  • /
  • 2005-8330(eISSN)
대한영상의학회 (The Korean Society of Radiology)
권호 표지
DOI QR코드

DOI QR Code

Reporting Quality of Research Studies on AI Applications in Medical Images According to the CLAIM Guidelines in a Radiology Journal With a Strong Prominence in Asia

  • Dong Yeong Kim (Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine) ;
  • Hyun Woo Oh (NAVER Inc.) ;
  • Chong Hyun Suh (Department of Radiology and Research Institute of Radiology, Asan Medical Center, University of Ulsan College of Medicine)
  • 투고 : 2023.10.20
  • 심사 : 2023.10.26
  • 발행 : 2023.12.01
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

Objective: We aimed to evaluate the reporting quality of research articles that applied deep learning to medical imaging. Using the Checklist for Artificial Intelligence in Medical Imaging (CLAIM) guidelines and a journal with prominence in Asia as a sample, we intended to provide an insight into reporting quality in the Asian region and establish a journal-specific audit. Materials and Methods: A total of 38 articles published in the Korean Journal of Radiology between June 2018 and January 2023 were analyzed. The analysis included calculating the percentage of studies that adhered to each CLAIM item and identifying items that were met by ≤ 50% of the studies. The article review was initially conducted independently by two reviewers, and the consensus results were used for the final analysis. We also compared adherence rates to CLAIM before and after December 2020. Results: Of the 42 items in the CLAIM guidelines, 12 items (29%) were satisfied by ≤ 50% of the included articles. None of the studies reported handling missing data (item #13). Only one study respectively presented the use of de-identification methods (#12), intended sample size (#19), robustness or sensitivity analysis (#30), and full study protocol (#41). Of the studies, 35% reported the selection of data subsets (#10), 40% reported registration information (#40), and 50% measured inter and intrarater variability (#18). No significant changes were observed in the rates of adherence to these 12 items before and after December 2020. Conclusion: The reporting quality of artificial intelligence studies according to CLAIM guidelines, in our study sample, showed room for improvement. We recommend that the authors and reviewers have a solid understanding of the relevant reporting guidelines and ensure that the essential elements are adequately reported when writing and reviewing the manuscripts for publication.

키워드

과제정보

This work was supported by the National Research Foundation of Korea (NRF- 2021R1C1C1014413).

참고문헌

  1. Park SH, Han K, Jang HY, Park JE, Lee JG, Kim DW, et al. Methods for clinical evaluation of artificial intelligence algorithms for medical diagnosis. Radiology 2023;306:20-31 
  2. Park SH, Sul AR, Ko Y, Jang HY, Lee JG. Radiologist's guide to evaluating publications of clinical research on ai: how we do it. Radiology 2023;308:e230288 
  3. Aggarwal R, Sounderajah V, Martin G, Ting DSW, Karthikesalingam A, King D, et al. Diagnostic accuracy of deep learning in medical imaging: a systematic review and metaanalysis. NPJ Digit Med 2021;4:65 
  4. Liu X, Faes L, Kale AU, Wagner SK, Fu DJ, Bruynseels A, et al. A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. Lancet Digit Health 2019;1:e271-e297 
  5. Kim DW, Jang HY, Kim KW, Shin Y, Park SH. Design characteristics of studies reporting the performance of artificial intelligence algorithms for diagnostic analysis of medical images: results from recently published papers. Korean J Radiol 2019;20:405-410 
  6. Cruz Rivera S, Liu X, Chan AW, Denniston AK, Calvert MJ. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI extension. Lancet Digit Health 2020;2:e549-e560 
  7. Liu X, Cruz Rivera S, Moher D, Calvert MJ, Denniston AK. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension. Lancet Digit Health 2020;2:e537-e548 
  8. Vasey B, Nagendran M, Campbell B, Clifton DA, Collins GS, Denaxas S, et al. Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. BMJ 2022;377:e070904 
  9. Sounderajah V, Ashrafian H, Golub RM, Shetty S, De Fauw J, Hooft L, et al. Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol. BMJ Open 2021;11:e047709 
  10. Collins GS, Dhiman P, Andaur Navarro CL, Ma J, Hooft L, Reitsma JB, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open 2021;11:e048008 
  11. Mongan J, Moy L, Kahn CE, Jr. Checklist for artificial intelligence in medical imaging (CLAIM): a guide for authors and reviewers. Radiol Artif Intell 2020;2:e200029 
  12. Tejani AS, Klontzas ME, Gatti AA, Mongan J, Moy L, Park SH, et al. Updating the checklist for artificial intelligence in medical imaging (CLAIM) for reporting AI research. Nat Mach Intell 2023;5:950-951 
  13. Belue MJ, Harmon SA, Lay NS, Daryanani A, Phelps TE, Choyke PL, et al. The low rate of adherence to checklist for artificial intelligence in medical imaging criteria among published prostate MRI artificial intelligence algorithms. J Am Coll Radiol 2023;20:134-145 
  14. Lans A, Pierik RJB, Bales JR, Fourman MS, Shin D, Kanbier LN, et al. Quality assessment of machine learning models for diagnostic imaging in orthopaedics: a systematic review. Artif Intell Med 2022;132:102396 
  15. Si L, Zhong J, Huo J, Xuan K, Zhuang Z, Hu Y, et al. Deep learning in knee imaging: a systematic review utilizing a checklist for artificial intelligence in medical imaging (CLAIM). Eur Radiol 2022;32:1353-1361 
  16. Bhandari A, Scott L, Weilbach M, Marwah R, Lasocki A. assessment of artificial intelligence (AI) reporting methodology in glioma MRI studies using the checklist for ai in medical imaging (CLAIM). Neuroradiology 2023;65:907-913 
  17. Zhong J, Hu Y, Zhang G, Xing Y, Ding D, Ge X, et al. An updated systematic review of radiomics in osteosarcoma: utilizing CLAIM to adapt the increasing trend of deep learning application in radiomics. Insights Imaging 2022;13:138 
  18. Tsang B, Gupta A, Takahashi MS, Baffi H, Ola T, Doria AS. Applications of artificial intelligence in magnetic resonance imaging of primary pediatric cancers: a scoping review and CLAIM score assessment. Jpn J Radiol 2023;41:1127-1147 
  19. Kouli O, Hassane A, Badran D, Kouli T, Hossain-Ibrahim K, Steele JD. Automated brain tumor identification using magnetic resonance imaging: a systematic review and meta-analysis. Neurooncol Adv 2022;4:vdac081 
  20. Alabed S, Maiter A, Salehi M, Mahmood A, Daniel S, Jenkins S, et al. Quality of reporting in AI cardiac MRI segmentation studies - a systematic review and recommendations for future studies. Front Cardiovasc Med 2022;9:956811 
  21. Guo Y, Hao Z, Zhao S, Gong J, Yang F. Artificial intelligence in health care: bibliometric analysis. J Med Internet Res 2020;22:e18228 
  22. Choi JS, Han BK, Ko ES, Bae JM, Ko EY, Song SH, et al. Effect of a deep learning framework-based computer-aided diagnosis system on the diagnostic performance of radiologists in differentiating between malignant and benign masses on breast ultrasonography. Korean J Radiol 2019;20:749-758 
  23. Lee SM, Lee JG, Lee G, Choe J, Do KH, Kim N, et al. CT image conversion among different reconstruction kernels without a sinogram by using a convolutional neural network. Korean J Radiol 2019;20:295-303 
  24. Park S, Lee SM, Do KH, Lee JG, Bae W, Park H, et al. Deep learning algorithm for reducing CT slice thickness: effect on reproducibility of radiomic features in lung cancer. Korean J Radiol 2019;20:1431-1440 
  25. Ahn Y, Yoon JS, Lee SS, Suk HI, Son JH, Sung YS, et al. Deep learning algorithm for automated segmentation and volume measurement of the liver and spleen using portal venous phase computed tomography images. Korean J Radiol 2020;21:987-997 
  26. Hong JH, Park EA, Lee W, Ahn C, Kim JH. Incremental image noise reduction in coronary CT angiography using a deep learning-based technique with iterative reconstruction. Korean J Radiol 2020;21:1165-1177 
  27. Hwang EJ, Kim H, Yoon SH, Goo JM, Park CM. Implementation of a deep learning-based computer-aided detection system for the interpretation of chest radiographs in patients suspected for COVID-19. Korean J Radiol 2020;21:1150-1160 
  28. Koo HJ, Lee JG, Ko JY, Lee G, Kang JW, Kim YH, et al. Automated segmentation of left ventricular myocardium on cardiac computed tomography using deep learning. Korean J Radiol 2020;21:660-669 
  29. Park HJ, Shin Y, Park J, Kim H, Lee IS, Seo DW, et al. Development and validation of a deep learning system for segmentation of abdominal muscle and fat on computed tomography. Korean J Radiol 2020;21:88-100 
  30. Shin YJ, Chang W, Ye JC, Kang E, Oh DY, Lee YJ, et al. Low-dose abdominal CT using a deep learning-based denoising algorithm: a comparison with CT reconstructed with filtered back projection or iterative reconstruction algorithm. Korean J Radiol 2020;21:356-364 
  31. Weikert T, Noordtzij LA, Bremerich J, Stieltjes B, Parmar V, Cyriac J, et al. Assessment of a deep learning algorithm for the detection of rib fractures on whole-body trauma computed tomography. Korean J Radiol 2020;21:891-899 
  32. Zhou QQ, Wang J, Tang W, Hu ZC, Xia ZY, Li XS, et al. Automatic detection and classification of rib fractures on thoracic CT using convolutional neural network: accuracy and feasibility. Korean J Radiol 2020;21:869-879 
  33. Hwang HJ, Seo JB, Lee SM, Kim EY, Park B, Bae HJ, et al. Content-based image retrieval of chest CT with convolutional neural network for diffuse interstitial lung disease: performance assessment in three major idiopathic interstitial pneumonias. Korean J Radiol 2021;22:281-290 
  34. Kim JH, Yoon HJ, Lee E, Kim I, Cha YK, Bak SH. Validation of deep-learning image reconstruction for low-dose chest computed tomography scan: emphasis on image quality and noise. Korean J Radiol 2021;22:131-138 
  35. Kim K, Kim S, Han K, Bae H, Shin J, Lim JS. Diagnostic performance of deep learning-based lesion detection algorithm in CT for detecting hepatic metastasis from colorectal cancer. Korean J Radiol 2021;22:912-921 
  36. Kim UH, Kim MY, Park EA, Lee W, Lim WH, Kim HL, et al. Deep learning-based algorithm for the detection and characterization of MRI safety of cardiac implantable electronic devices on chest radiographs. Korean J Radiol 2021;22:1918-1928 
  37. Lee JG, Kim H, Kang H, Koo HJ, Kang JW, Kim YH, et al. Fully automatic coronary calcium score software empowered by artificial intelligence technology: validation study using three CT cohorts. Korean J Radiol 2021;22:1764-1776 
  38. Lee KC, Lee KH, Kang CH, Ahn KS, Chung LY, Lee JJ, et al. Clinical validation of a deep learning-Based Hybrid (Greulich-Pyle and Modified Tanner-Whitehouse) method for bone age assessment. Korean J Radiol 2021;22:2017-2025 
  39. Park HS, Jeon K, Cho YJ, Kim SW, Lee SB, Choi G, et al. Diagnostic performance of a new convolutional neural network algorithm for detecting developmental dysplasia of the hip on anteroposterior radiographs. Korean J Radiol 2021;22:612-623 
  40. Purkayastha S, Xiao Y, Jiao Z, Thepumnoeysuk R, Halsey K, Wu J, et al. Machine learning-based prediction of COVID-19 severity and progression to critical illness using CT imaging and clinical data. Korean J Radiol 2021;22:1213-1224 
  41. Weikert T, Rapaka S, Grbic S, Re T, Chaganti S, Winkel DJ, et al. Prediction of patient management in COVID-19 using deep learning-based fully automated extraction of cardiothoracic CT metrics and laboratory findings. Korean J Radiol 2021;22:994-1004 
  42. Yan C, Lin J, Li H, Xu J, Zhang T, Chen H, et al. cycle-consistent generative adversarial network: effect on radiation dose reduction and image quality improvement in ultralow-dose CT for evaluation of pulmonary tuberculosis. Korean J Radiol 2021;22:983-993 
  43. Yang J, Chen Z, Liu W, Wang X, Ma S, Jin F, et al. Development of a malignancy potential binary prediction model based on deep learning for the mitotic count of local primary gastrointestinal stromal tumors. Korean J Radiol 2021;22:344-353 
  44. Yeoh H, Hong SH, Ahn C, Choi JY, Chae HD, Yoo HJ, et al. Deep learning algorithm for simultaneous noise reduction and edge sharpening in low-dose CT images: a pilot study using lumbar spine CT. Korean J Radiol 2021;22:1850-1857 
  45. Yoo SJ, Yoon SH, Lee JH, Kim KH, Choi HI, Park SJ, et al. Automated lung segmentation on chest computed tomography images with extensive lung parenchymal abnormalities using a deep neural network. Korean J Radiol 2021;22:476-488 
  46. Yu Y, Gao Y, Wei J, Liao F, Xiao Q, Zhang J, et al. A three-dimensional deep convolutional neural network for automatic segmentation and diameter measurement of type B aortic dissection. Korean J Radiol 2021;22:168-178 
  47. Bae K, Oh DY, Yun ID, Jeon KN. Bone suppression on chest radiographs for pulmonary nodule detection: comparison between a generative adversarial network and dual-energy subtraction. Korean J Radiol 2022;23:139-149 
  48. Chang S, Han K, Lee S, Yang YJ, Kim PK, Choi BW, et al. Automated measurement of native T1 and extracellular volume fraction in cardiac magnetic resonance imaging using a commercially available deep learning algorithm. Korean J Radiol 2022;23:1251-1259 
  49. Choi JW, Cho YJ, Ha JY, Lee YY, Koh SY, Seo JY, et al. deep learning-assisted diagnosis of pediatric skull fractures on plain radiographs. Korean J Radiol 2022;23:343-354 
  50. Kim YS, Jang MJ, Lee SH, Kim SY, Ha SM, Kwon BR, et al. Use of artificial intelligence for reducing unnecessary recalls at screening mammography: a simulation study. Korean J Radiol 2022;23:1241-1250 
  51. Lee JH, Kim KH, Lee EH, Ahn JS, Ryu JK, Park YM, et al. Improving the performance of radiologists using artificial intelligence-based detection support software for mammography: a multi-reader study. Korean J Radiol 2022;23:505-516 
  52. Otgonbaatar C, Ryu JK, Shin J, Woo JY, Seo JW, Shim H, et al. Improvement in image quality and visibility of coronary arteries, stents, and valve structures on CT angiography by deep learning reconstruction. Korean J Radiol 2022;23:1044-1054 
  53. Park HJ, Yoon JS, Lee SS, Suk HI, Park B, Sung YS, et al. Deep learning-based assessment of functional liver capacity using gadoxetic acid-enhanced hepatobiliary phase MRI. Korean J Radiol 2022;23:720-731 
  54. Park J, Shin J, Min IK, Bae H, Kim YE, Chung YE. Image quality and lesion detectability of lower-dose abdominopelvic CT obtained using deep learning image reconstruction. Korean J Radiol 2022;23:402-412 
  55. Park JH, Park I, Han K, Yoon J, Sim Y, Kim SJ, et al. Feasibility of deep learning-based analysis of auscultation for screening significant stenosis of native arteriovenous fistula for hemodialysis requiring angioplasty. Korean J Radiol 2022;23:949-958 
  56. Son W, Kim M, Hwang JY, Kim YW, Park C, Choo KS, et al. Comparison of a deep learning-based reconstruction algorithm with filtered back projection and iterative reconstruction algorithms for pediatric abdominopelvic CT. Korean J Radiol 2022;23:752-762 
  57. Yoo H, Kim EY, Kim H, Choi YR, Kim MY, Hwang SH, et al. Artificial intelligence-based identification of normal chest radiographs: a simulation study in a multicenter health screening cohort. Korean J Radiol 2022;23:1009-1018 
  58. Hwang EJ, Goo JM, Nam JG, Park CM, Hong KJ, Kim KH. Conventional versus artificial intelligence-assisted interpretation of chest radiographs in patients with acute respiratory symptoms in emergency department: a pragmatic randomized clinical trial. Korean J Radiol 2023;24:259-270 
  59. Lee SB, Hong Y, Cho YJ, Jeong D, Lee J, Yoon SH, et al. Deep learning-based computed tomography image standardization to improve generalizability of deep learning-based hepatic segmentation. Korean J Radiol 2023;24:294-304 
  60. Mnih V, Heess N, Graves A, Kavukcuoglu K. Recurrent models of visual attention. arXiv:1406.6247 [Preprint] 2014 [posted Jun 14 2014; cited Sep 15, 2023]. Available at: https://doi.org/10.48550/arXiv.1406.6247 
  61. Clarivate. Journal Citation Reports(TM) for Korean Journal of Radiology [accessed on October 19, 2023]. Available at: https://jcr.clarivate.com/jcr-jp/journalprofile?journal=KOREAN%20J%20RADIOL&year=2022 
  62. Simago Journal & Country Rank. Simago Journal & Country Rank for Korean Journal of Radiology [accessed on October 19, 2023]. Available at: https://www.scimagojr.com/journalsearch.php?q=17255&tip=sid&exact=no 
  63. Soyer P. Agreement and observer variability. Diagn Interv Imaging 2018;99:53-54 
  64. Fernandez JC, Mounier L, Pachon C. A model-based approach for robustness testing. In: Khendek F, Dssouli R, eds. Testing of Communicating Systems. Berlin, Heidelberg: Springer, 2005:333-348 
  65. Joel MZ, Umrao S, Chang E, Choi R, Yang DX, Duncan JS, et al. Using adversarial images to assess the robustness of deep learning models trained on diagnostic images in oncology. JCO Clin Cancer Inform 2022;6:e2100170 
  66. Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions. arXiv:1511.07122 [Preprint] 2016 [posted Nov 23 2015; revised Apr 30 2016; cited Sep 15, 2023]. Available at: https://doi.org/10.48550/arXiv.1511.07122 
  67. Camino R, Hammerschmidt CA, State R. Working with deep generative models and tabular data imputation. First Workshop on the Art of Learning with Missing Values (Artemiss) Hosted by the 37th International Conference on Machine Learning (ICML); 2020 Jul 12-18; Vienna, Austria 
  68. Radiological Society of North America. Scientific style guide: writing a manuscript for Radiology [accessed on October 16, 2023]. Available at: https://pubs.rsna.org/page/radiology/author-instructions/scientificediting 
  69. Silcox C, Dentzer S, Bates DW. AI-enabled clinical decision support software: a "Trust and Value Checklist" for clinicians. NEJM Catal Innov Care Deliv 2020;1