Korean Journal of Radiology

  • 제24권7호
  • /
  • Pages.698-714
  • /
  • 2023
  • /
  • 1229-6929(pISSN)
  • /
  • 2005-8330(eISSN)
대한영상의학회 (The Korean Society of Radiology)
권호 표지
DOI QR코드

DOI QR Code

Developing and Evaluating Deep Learning Algorithms for Object Detection: Key Points for Achieving Superior Model Performance

  • Jang-Hoon Oh (Department of Radiology, Kyung Hee University Hospital, Kyung Hee University College of Medicine) ;
  • Hyug-Gi Kim (Department of Radiology, Kyung Hee University Hospital, Kyung Hee University College of Medicine) ;
  • Kyung Mi Lee (Department of Radiology, Kyung Hee University Hospital, Kyung Hee University College of Medicine)
  • 투고 : 2022.10.07
  • 심사 : 2023.05.16
  • 발행 : 2023.07.01
PDF 다운로드
⟨ 이전 논문 다음 논문 ⟩

초록

In recent years, artificial intelligence, especially object detection-based deep learning in computer vision, has made significant advancements, driven by the development of computing power and the widespread use of graphic processor units. Object detection-based deep learning techniques have been applied in various fields, including the medical imaging domain, where remarkable achievements have been reported in disease detection. However, the application of deep learning does not always guarantee satisfactory performance, and researchers have been employing trial-and-error to identify the factors contributing to performance degradation and enhance their models. Moreover, due to the black-box problem, the intermediate processes of a deep learning network cannot be comprehended by humans; as a result, identifying problems in a deep learning model that exhibits poor performance can be challenging. This article highlights potential issues that may cause performance degradation at each deep learning step in the medical imaging domain and discusses factors that must be considered to improve the performance of deep learning models. Researchers who wish to begin deep learning research can reduce the required amount of trial-and-error by understanding the issues discussed in this study.

키워드

과제정보

This research was the result of a study on the "HPC Support" Project, supported by the 'Ministry of Science and ICT' and NIPA.

참고문헌

  1. Wang YE, Wei GY, Brooks D. Benchmarking TPU, GPU, and CPU platforms for deep learning. arXiv:1907.10701v4 [Preprint]. [posted July 24, 2019; revised October 22, 2019; cited October 4, 2022]. https://arxiv.org/abs/1907.10701 
  2. Girshick R. Fast R-CNN. In: 2015 IEEE International Conference on Computer Vision (ICCV); 2015 December 7-13; Santiago, Chile. Danvers: The Institute of Electrical and Electronics Engineers, Inc.; 2015. p. 1440-1448 
  3. Sun X, Wu P, Hoi SCH. Face detection using deep learning: an improved faster RCNN approach. Neurocomputing 2018;299:42-50 
  4. Balaban S. Deep learning and face recognition: the state of the art. In: Kakadiaris IA, Kumar A, Scheirer WJ, editors. Biometric Surveill Technol Hum Act Identif XII; 2015 April 20-24; Baltimore, United States. Bellingham: Society of PhotoOptical Instrumentation Engineers; 2015. p. 94570B 
  5. Brunetti A, Buongiorno D, Trotta GF, Bevilacqua V. Computer vision and deep learning techniques for pedestrian detection and tracking: a survey. Neurocomputing 2018;300:17-33 
  6. Feyzabadi S. Joint deep learning for car detection. arXiv:1412.7854v2 [Preprint]. [posted December 25, 2014; revised July 14, 2016; cited October 4, 2022]. https://arxiv.org/abs/1412.7854 
  7. Zhang M, Young GS, Chen H, Li J, Qin L, McFaline-Figueroa JR, et al. Deep-learning detection of cancer metastases to the brain on MRI. J Magn Reson Imaging 2020;52:1227-1236 
  8. Wang CJ, Hamm CA, Savic LJ, Ferrante M, Schobert I, Schlachter T, et al. Deep learning for liver tumor diagnosis part II: convolutional neural network interpretation using radiologic imaging features. Eur Radiol 2019;29:3348-3357 
  9. Ebsim R, Naqvi J, Cootes TF. Automatic detection of wrist fractures from posteroanterior and lateral radiographs: A deep learning-based approach. In: Vrtovec T, Yao J, Zheng G, Pozo JM, editors. Computational methods and clinical applications in musculoskeletal imaging. 6th International Workshop, MSKI 2018; 2018 September 16; Granada, Spain. Cham: Springer; 2019. p. 114-125 
  10. Weikert T, Noordtzij LA, Bremerich J, Stieltjes B, Parmar V, Cyriac J, et al. Assessment of a deep learning algorithm for the detection of rib fractures on whole-body trauma computed tomography. Korean J Radiol 2020;21:891-899 
  11. Zhou QQ, Wang J, Tang W, Hu ZC, Xia ZY, Li XS, et al. Automatic detection and classification of rib fractures on thoracic ct using convolutional neural network: accuracy and feasibility. Korean J Radiol 2020;21:869-879 
  12. Choi JW, Cho YJ, Ha JY, Lee YY, Koh SY, Seo JY, et al. Deep learning-assisted diagnosis of pediatric skull fractures on plain radiographs. Korean J Radiol 2022;23:343-354 
  13. Jaiswal AK, Tiwari P, Kumar S, Gupta D, Khanna A, Rodrigues JJPC. Identifying pneumonia in chest X-rays: a deep learning approach. Measurement 2019;145:511-518 
  14. Myung MJ, Lee KM, Kim HG, Oh J, Lee JY, Shin I, et al. Novel approaches to detection of cerebral microbleeds: single deep learning model to achieve a balanced performance. J Stroke Cerebrovasc Dis 2021;30:105886 
  15. von Eschenbach WJ. Transparency and the black box problem: why we do not trust AI. Philos Technol 2021;34:1607-1622 
  16. Zhao Z, Guo Y, Shen H, Ye J. Adaptive object detection with dual multi-label prediction. In: Vedaldi A, Bischof H, Brox T, Frahm JM, editors. Computer Vision - ECCV 2020. 16th European Conference; 2020 August 23-28; Glasgow, United Kingdom. Cham: Springer; 2020. p. 54-69 
  17. Zhou Z, Sanders JW, Johnson JM, Gule-Monroe MK, Chen MM, Briere TM, et al. Computer-aided detection of brain metastases in T1-weighted MRI for stereotactic radiosurgery using deep learning single-shot detectors. Radiology 2020;295:407-415 
  18. Kim K, Kim S, Han K, Bae H, Shin J, Lim JS. Diagnostic performance of deep learning-based lesion detection algorithm in CT for detecting hepatic metastasis from colorectal cancer. Korean J Radiol 2021;22:912-921 
  19. Oh JH, Kim HG, Lee KM, Ryu CW, Park S, Jang JH, et al. Effective end-to-end deep learning process in medical imaging using independent task learning: application for diagnosis of maxillary sinusitis. Yonsei Med J 2021;62:1125-1135 
  20. Chen H, Yu L, Dou Q, Shi L, Mok VCT, Heng PA. Automatic detection of cerebral microbleeds via deep learning based 3D feature representation. In: 2015 IEEE 12th International Symposium on Biomedical Imaging (ISBI); 2015 April 16-19; Brooklyn, United States. New York: The Institute of Electrical and Electronics Engineers, Inc; 2015. p. 764-767 
  21. Jain S, Sima DM, Ribbens A, Cambron M, Maertens A, Van Hecke W, et al. Automatic segmentation and volumetry of multiple sclerosis brain lesions from MR images. NeuroImage Clin 2015;8:367-375 
  22. Kang H, Witanto JN, Pratama K, Lee D, Choi KS, Choi SH, et al. Fully automated MRI segmentation and volumetric measurement of intracranial meningioma using deep learning. J Magn Reson Imaging 2023;57:871-881 
  23. Cao G, Xie X, Yang W, Liao Q, Shi G, Wu J. Feature-fused SSD: fast detection for small objects. In: Yu H, Dong J, editors. Ninth International Conference on Graphic and Image Processing (ICGIP 2017); 2017 October 14-16; Qingdao, China. Qingdao: International Society for Optics and Photonics (SPIE); 2018. p. 106151E 
  24. Tong K, Wu Y, Zhou F. Recent advances in small object detection based on deep learning: a review. Image Vis Comput 2020;97:103910 
  25. Liu Y, Sun P, Wergeles N, Shang Y. A survey and performance evaluation of deep learning methods for small object detection. Expert Syst Appl 2021;172:114602 
  26. Akselrod-Ballin A, Karlinsky L, Hazan A, Bakalo R, Horesh AB, Shoshan Y, et al. Deep learning for automatic detection of abnormal findings in breast mammography. In: Cardoso MJ, Arbel T, Carneiro G, Syeda-Mahmood T, Tavares JMRS, Moradi M, et al editors. Deep learning in medical image analysis and multimodal learning for clinical decision support. Third International Workshop, DLMIA 2017, and 7th International Workshop; 2017 September 14; Quebec, Canada. Cham: Springer; 2017. p. 321-329 
  27. Takao H, Amemiya S, Kato S, Yamashita H, Sakamoto N, Abe O. Deep-learning single-shot detector for automatic detection of brain metastases with the combined use of contrast-enhanced and non-enhanced computed tomography images. Eur J Radiol 2021;144:110015 
  28. Nam JG, Hwang EJ, Kim DS, Yoo SJ, Choi H, Goo JM, et al. Undetected lung cancer at posteroanterior chest radiography: potential role of a deep learning-based detection algorithm. Radiol Cardiothorac Imaging 2020;2:e190222 
  29. Meng Z, Fan X, Chen X, Chen M, Tong Y. Detecting small signs from large images. In: Zhang C, Palanisamy B, Khan L, Sarvestani SS, editors. 2017 IEEE International Conference on Information Reuse and Integration (IRI); 2017 August 4-6; San Diego, United States. Danvers: The Institute of Electrical and Electronics Engineers, Inc.; 2017. p. 217-224 
  30. Zhao Q, Sheng T, Wang Y, Tang Z, Chen Y, Cai L, et al. M2det: a single-shot object detector based on multi-level feature pyramid network. In: The Thirty-Third AAAI Conference on Artificial Intelligence, The Thirty-First Conference on Innovative Applications of Artificial Intelligence, The Ninth Symposium on Educational Advances in Artificial Intelligence; 2019 January 27-Feb 1; Honolulu, United States. Honolulu: Association for the Advancement of Artificial Intelligence (AAAI); 2019. p. 9259-9266 
  31. Cui L, Ma R, Lv P, Jiang X, Gao Z, Zhou B, et al. MDSSD: multi-scale deconvolutional single shot detector for small objects. Sci China Inf Sci 2020;63:120113 
  32. Cao C, Wang B, Zhang W, Zeng X, Yan X, Feng Z, et al. An improved faster R-CNN for small object detection. IEEE Access 2019;7:106838-106846 
  33. Cho J, Lee K, Shin E, Choy G, Do S. How much data is needed to train a medical image deep learning system to achieve necessary high accuracy? arXiv:1511.06348v2 [Preprint]. [posted November 19, 2015; revised January 7, 2016; cited October 4, 2022]. https://arxiv.org/abs/1511.06348 
  34. Fei-Fei L, Deng J, Li K. ImageNet: constructing a large-scale image database. J Vision 2009;9:1037 
  35. Sun Q, Yang Y, Sun J, Yang Z, Zhang J. Using deep learning for content-based medical image retrieval. In: Cook TS, Zhang J, editors. Medical Imaging 2017: Imaging Informatics Healthcare, Research, and Applications; 2017 February 11-16; Orlando, United States. Orlando: International Society for Optics and Photonics (SPIE); 2017. p. 1013812 
  36. Kim HG, Lee KM, Kim EJ, Lee JS. Improvement diagnostic accuracy of sinusitis recognition in paranasal sinus X-ray using multiple deep learning models. Quant Imaging Med Surg 2019;9:942-951 
  37. Kim Y, Lee KJ, Sunwoo L, Choi D, Nam CM, Cho J, et al. Deep Learning in diagnosis of maxillary sinusitis using conventional radiography. Invest Radiol 2019;54:7-15 
  38. Han S, Hwang SI, Lee HJ. The classification of renal cancer in 3-phase CT images using a deep learning method. J Digit Imaging 2019;32:638-643 
  39. Lee H, Tajmir S, Lee J, Zissen M, Yeshiwas BA, Alkasab TK, et al. Fully automated deep learning system for bone age assessment. J Digit Imaging 2017;30:427-441 
  40. Al-antari MA, Al-Masni MA, Choi MT, Han SM, Kim TS. A fully integrated computer-aided diagnosis system for digital X-ray mammograms via deep learning detection, segmentation, and classification. Int J Med Inform 2018;117:44-54 
  41. Jeon Y, Lee K, Sunwoo L, Choi D, Oh DY, Lee KJ, et al. Deep learning for diagnosis of paranasal sinusitis using multi-view radiographs. Diagnostics (Basel) 2021;11:250 
  42. Oh JH, Kim HG, Lee KM, Ryu CW. Reliable quality assurance of X-ray mammography scanner by evaluation the standard mammography phantom image using an interpretable deep learning model. Eur J Radiol 2022;154:110369 
  43. Janssens R, Zeng G, Zheng G. Fully automatic segmentation of lumbar vertebrae from CT images using cascaded 3D fully convolutional networks. In: 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018); 2018 April 4-7; Washington, DC, United States. Danvers: The Institute of Electrical and Electronics Engineers, Inc.; 2018. p. 893-897 
  44. Kern D, Mastmeyer A. 3D Bounding box detection in volumetric medical image data: a systematic literature review. J Image Graph 2022;10:17-27 
  45. Zhu X, Zhu L, D Song, Wang D, Wu F, Wu J. Comparison of single- and dual-energy CT combined with artificial intelligence for the diagnosis of pulmonary nodules. Clin Radiol 2023;78:e99-e105 
  46. Kurata Y, Nishio M, Moribata Y, Kido A, Himoto Y, Otani S, et al. Automatic segmentation of uterine endometrial cancer on multi-sequence MRI using a convolutional neural network. Sci Rep 2021;11:14440 
  47. Al-masni MA, Kim WR, Kim EY, Noh Y, Kim DH. Automated detection of cerebral microbleeds in MR images: a two-stage deep learning approach. NeuroImage Clin 2020;28:102464 
  48. Fei Y, Zhan B, Hong M, Wu X, Zhou J, Wang Y. Deep learning-based multi-modal computing with feature disentanglement for MRI image synthesis. Med Phys 2021;48:3778-3789 
  49. Feng H, Cao J, Wang H, Xie Y, Yang D, Feng J, et al. A knowledge-driven feature learning and integration method for breast cancer diagnosis on multi-sequence MRI. Magn Reson Imaging 2020;69:40-48 
  50. Chen H, Dou Q, Yu L, Qin J, Heng PA. VoxResNet: deep voxelwise residual networks for brain segmentation from 3D MR images. Neuroimage 2018;170:446-455 
  51. Park YW, Jun Y, Lee Y, Han K, An C, Ahn SS, et al. Robust performance of deep learning for automatic detection and segmentation of brain metastases using three-dimensional black-blood and three-dimensional gradient echo imaging. Eur Radiol 2021;31:6686-6695 
  52. Zhu Y, Man C, Gong L, Dong D, Yu X, Wang S, et al. A deep learning radiomics model for preoperative grading in meningioma. Eur J Radiol 2019;116:128-134 
  53. Rolnick D, Veit A, Belongie S, Shavit N. Deep learning is robust to massive label noise. arXiv:1705.10694v3 [Preprint]. [posted May 30, 2017; revised February 26, 2018; cited October 4, 2022]. https://arxiv.org/abs/1705.10694 
  54. Uhm KH, Jung SW, Choi MH, Shin HK, Yoo JI, Oh SW, et al. Deep learning for end-to-end kidney cancer diagnosis on multi-phase abdominal computed tomography. NPJ Precis Oncol 2021;5:54 
  55. Rezatofighi H, Tsoi N, Gwak J, Sadeghian A, Reid I, Savarese S. Generalized intersection over union: a metric and a loss for bounding box regression. In: The IEEE/CVF Conference on Computer Vision and Vision and Pattern ion (CVPR), 2019; 2019 June 16-20; Long Beach, United States. California: Institute of Electrical and Electronics Engineers (IEEE); 2019. p. 658-666 
  56. Zhao ZQ, Zheng P, Xu ST, Wu X. Object detection with deep learning: a review. IEEE Trans Neural Netw Learn Syst 2019;30:3212-3232 
  57. Liu L, Ouyang W, Wang X, Fieguth P, Chen J, Liu X, et al. Deep learning for generic object detection: a survey. Int J Comput Vis 2020;128:261-318 
  58. Hoeser T, Kuenzer C. Object detection and image segmentation with deep learning on Earth observation data: a review-part I: evolution and recent trends. Remote Sens 2020;12:1667 
  59. Girshick R, Donahue J, Darrell T, Malik J. Rich feature hierarchies for accurate object detection and semantic segmentation. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014; 2014 June 23-28; Ohio, United States. Colombus: Institute of Electrical and Electronics Engineers (IEEE); 2014. p. 580-587 
  60. Lin TY, Goyal P, Girshick R, He K, Dollar P. Focal loss for dense object detection. IEEE Trans Pattern Anal Mach Intell 2020;42:318-327 
  61. Liu W, Anguelov D, Erhan D, Szegedy C, Reed S, Fu CY, et al. SSD: single shot multibox detector. In: Leibe B, Matas J, Sebe N, Welling M, editors. Computer vision - ECCV 2016. 14th European Conference; 2016 October 11-14; Amsterdam, Netherlands. Cham: Springer; 2016. p. 21-37 
  62. Ren S, He K, Girshick R, Sun J. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Pattern Anal Mach Intell 2017;39:1137-1149 
  63. He K, Gkioxari G, Dollar P, Girshick R. Mask R-CNN. IEEE Trans Pattern Anal Mach Intell 2020;42:386-397 
  64. Lin TY, Dollar P, Girshick R, He K, Hariharan B, Belongie S. Feature pyramid networks for object detection. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017; 2017 July 21-26; Honolulu, United States. Honolulu: Institute of Electrical and Electronics Engineers (IEEE); 2017. p. 2117-2125 
  65. Dai J, Li Y, He K, Sun J. R-FCN: object detection via region-based fully convolutional networks. In: Lee D, Sugiyama M, Luxburg U, Guyon I, Garnett R, editors. 30th Conference on Neural Information Processing Systems (NIPS 2016); 2016 December 5-10; Barcelona, Spain. Barcelona: Neural Information Processing Systems Foundation, Inc. (NeurIPS); 2016. p. 379-387 
  66. Redmon J, Divvala S, Girshick R, Farhadi A. You only look once: unified, real-time object detection. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016; 2016 June 26-July 1; Las Vegas, United States. Las Vegas: Institute of Electrical and Electronics Engineers (IEEE); 2016. p. 779-788 
  67. Redmon J, Farhadi A. YOLO9000: Better, faster, stronger. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017; 2017 July 21-26; Honolulu, United States. Honolulu: Institute of Electrical and Electronics Engineers (IEEE); 2017. p. 7263-7271. 
  68. Redmon J, Farhadi A. YOLOv3: an incremental improvement. arXiv:1804.02767 [Preprint]. [posted April 8, 2018; cited October 4, 2022]. https://arxiv.org/abs/1804.02767 
  69. Bochkovskiy A, Wang CY, Mark Liao HY. YOLOv4: optimal speed and accuracy of object detection. arXiv:2004.10934 [Preprint]. [posted April 23, 2020; cited October 4, 2022]. https://arxiv.org/abs/2004.10934 
  70. Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition. arXiv:1409.1556v6 [Preprint]. [posted September 4, 2014; revised April 10, 2015; cited October 4, 2022]. https://arxiv.org/abs/1409.1556 
  71. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016; 2016 June 26-July 1; Las Vegas, United States. Las Vegas: Institute of Electrical and Electronics Engineers (IEEE); 2016. p. 770-778 
  72. Szegedy C, Vanhoucke V, Ioffe S, Shlens J, Wojna Z. Rethinking the inception architecture for computer vision. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016; 2016 June 26-July 1; Las Vegas, United States. Las Vegas: Institute of Electrical and Electronics Engineers (IEEE); 2016. p. 2818-2826 
  73. Stassa G, Evans JA. Radiology, anatomy, and the medical student. Radiology 1969;92:1562-1563 
  74. Yadav SS, Jadhav SM. Deep convolutional neural network based medical image classification for disease diagnosis. J Big Data 2019;6:113 
  75. Liu S, Utriainen D, Chai C, Chen Y, Wang L, Sethi SK, et al. Cerebral microbleed detection using susceptibility weighted imaging and deep learning. Neuroimage 2019;198:271-282 
  76. Yang H, Jo E, Kim HJ, Cha IH, Jung YS, Nam W, et al. Deep learning for automated detection of cyst and tumors of the jaw in panoramic radiographs. J Clin Med 2020;9:1839 
  77. Aszemi NM, Dominic PDD. Hyperparameter optimization in convolutional neural network using genetic algorithms. Int J Adv Comput Sci Appl 2019;10:269-278 
  78. Ait Amou MA, Xia K, Kamhi S, Mouhafid M. A novel MRI diagnosis method for brain tumor classification based on CNN and Bayesian Optimization. Healthcare (Basel) 2022;10:494 
  79. Bergstra J, Bardenet R, Bengio Y, Kegl B. Algorithms for hyper-parameter optimization. In: Shawe-Taylor J, Zemel R, Bartlett P, Pereira F, Weinberger KQ, editors. 25th Annual Conference on Neural Information Processing Systems (NIPS 2011); 2011 December 12-14; Granada, Spain. Granada: Neural Information Processing Systems Foundation, Inc. (NeurIPS); 2011. p. 2546-2554 
  80. Huang Q, Mao J, Liu Y. An improved grid search algorithm of SVR parameters optimization. In: Yang Y, editor. 2012 IEEE 14th International Conference on Communication Technology; 2012 November 9-11; Chengdu, China. Chengdu: The Institute of Electrical and Electronics Engineers, Inc.; 2012. p. 1022-1026 
  81. Bergstra J, Bengio Y. Random search for hyper-parameter optimization. J Mach Learn Res 2012;13:281-305 
  82. Zhang Y, Sohn K, Villegas R, Pan G, Lee H. Improving object detection with deep convolutional networks via Bayesian optimization and structured prediction. In: The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015; 2015 June 7-12; Boston, United States. Boston: Institute of Electrical and Electronics Engineers (IEEE); 2015. p. 249-258 
  83. Shahriari B, Swersky K, Wang Z, Adams RP, de Freitas N. Taking the human out of the loop: a review of Bayesian optimization. Proc IEEE 2016;104:148-175 
  84. Brochu E, Cora VM, de Freitas N. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv:1012.2599 [Preprint]. [posted December 12, 2010; cited October 4, 2022]. https://arxiv.org/abs/1012.2599 
  85. Snoek J, Larochelle H, Adams RP. Practical Bayesian optimization of machine learning algorithms. In: Pereira F, Burges CJ, Bottou L, Weinberger KQ, editors. 26th Annual Conference on Neural Information Processing Systems 2012; 2012 December 3-6; Lake Tahoe, United States. Lake Tahoe: Neural Information Processing Systems Foundation, Inc. (NeurIPS); 2012. p. 2951-2959 
  86. Everingham M, Van Gool L, Williams CKI, Winn J, Zisserman A. The Pascal Visual Object Classes (VOC) Challenge. Int J Comput Vis 2010;88:303-338 
  87. Lin TY, Maire M, Belongie S, Hays J, Perona P, Ramanan D, et al. Microsoft COCO: Common objects in context. In: Fleet D, Pajdla T, Schiele B, Tuytelaars T, editors. Computer Vision - ECCV 2014. 13th European Conference; 2014 September 6-12; Zurich, Switzerland. Cham: Springer; 2014. p. 740-755 
  88. Girshick R, Donahue J, Darrell T, Malik J. Region-based convolutional networks for accurate object detection and segmentation. IEEE Trans Pattern Anal Mach Intell 2016;38:142-158 
  89. Hardalac F, Uysal F, Peker O, Ciceklidag M, Tolunay T, Tokgoz N, et al. Fracture detection in wrist X-ray images using deep learning-based object detection models. Sensors (Basel) 2022;22:1285 
  90. Zhang E, Zhang Y. Average precision BT. In: Liu L, Ozsu T, eds. Encyclopedia of database systems. Boston: Springer, 2009:192-193 
  91. Beitzel SM, Jensen EC, Frieder O. MAP BT. In: Liu L, Ozsu T, eds. Encyclopedia of database systems. Boston: Springer, 2009:1691-1692