Journal of the Korean Society for Library and Information Science (한국문헌정보학회지)

Korean Society For Library And Information Science (한국문헌정보학회)
권호 표지
DOI QR코드

DOI QR Code

A Study on the Relationship between the Review Results of Articles and Impact Metrics in an Open Peer Review Platform

오픈 피어 리뷰 환경에서 학술 논문 심사 결과와 영향력 지표 간의 관련성에 관한 연구

  • 조재인 (인천대학교 문헌정보학과 ) ;
  • 박종도 (인천대학교 문헌정보학과 )
  • Received : 2023.04.21
  • Accepted : 2023.05.09
  • Published : 2023.05.31
Download PDF
⟨ Previous Next ⟩

Abstract

This study analyzed the open peer review results for 585 papers in the field of social sciences in F1000Research, a representative OPR(Open Peer Review) platform, and checked the relationship between the number of cited-by, altmetrics and review score. In addition, by verifying whether the review score shows a moderating effect between the relationship between the utilization of the paper and the cited-by, it was confirmed whether the paper evaluated as high quality in the open review platform can promote the number of cited-by. As a result of the analysis, first, there was no significant difference in the number of cited-by between the approved and conditionally approved paper groups, but the converted review score and the number of cited-by showed a weak positive correlation (r = 0.40 - 0.60). Second, the review score showed a weak correlation with the altmetrics, and it was analyzed that review result could weakly predict the number of cited-by and social impact. Finally, it was verified that the review score performed a significant positive moderating effect (B=1.69, P < 0.01) in making the use of the paper lead to citation. As a result of the conditional effect test, it was verified that it showed the greatest effect(B=11.32, 95% CI [10.57, 12.08]) in the group of papers rated as the highest quality. Therefore, it was analyzed that the open review scores can help researchers select high quality papers and induce citations.

본 연구는 대표적인 OPR(Open Peer Review) 플랫폼인 F1000Research에서 사회과학분야의 논문 585건을 대상으로 개방형 동료 심사 결과를 분석하고 피인용, 알트메트릭스와 어떠한 관련성을 보이는지 확인하였다. 더불어 논문의 활용이 피인용에 미치는 영향 관계 내에서 심사 점수가 조절효과(Moderating effect)를 나타내는지 검증함으로써, OPR 환경에서 고품질로 평가된 논문이 피인용을 촉진할 수 있는지 확인하였다. 분석 결과 첫 번째, 승인과 조건부 승인된 논문 그룹 간에 피인용 횟수에 유의미한 차이가 나타나지 않았지만, 환산된 심사 점수와 피인용 횟수는 유의한 정(+)의 상관성(r= 0.40 ~ 0.60)을 나타냈다. 두번째, 심사 점수는 알트메트릭스와도 약한 상관성을 나타내 심사자의 품질 평가 결과는 피인용과 사회적 영향을 약하게 예측할 수 있는 것으로 분석되었다. 마지막으로 심사 점수는 논문의 활용을 피인용으로 이어지게 하는데 유의한 양의 방향의 조절효과 (B=1.69, P < 0.01)를 수행하며, 조건부 효과 검사 결과 가장 고품질로 평가된 논문 집단에서 가장 큰 효과(B=11.32, 95% CI [10.57, 12.08])를 나타내는 것으로 검증되었다. 따라서 공개된 심사 결과는 연구자들의 우수 논문 선별을 도와 인용을 유도하는데 도움을 줄 수 있는 것으로 파악되었다.

Keywords

Acknowledgement

본 연구는 인천대학교 우수 연구소 집단 연구 지원 사업(2022)으로 수행되었음.

References

  1. Hur, Won-Moo (2013). How researchers estimate indirect effect using bootstrapping: the case of simple, multiple, and double mediation. Korea Business Review, 6(3), 43-59.
  2. Jeong, Yong-il, Ahn, Sungsoo, & Noh, Ji-yoon (2022). A study on awareness of open access practices for academic research information. Proceedings of the Korean Institute of Information and Commucation Sciences Conference, 18-21.
  3. Kim, Ji-Young, Kim, Hyun Soo, & Shim, Wonsik (2020). A study on open peer review perception of Korean authors in a mega OA journal. Journal of the Korean Society for Information Management, 37(4), 131-150.
  4. ASAPbio (2018). Transparency, recognition, and innovation in peer review in the life sciences: peer review survey results. Available: https://asapbio.org/peer-review/survey
  5. Belluz, J., Plumer, B., & Resnick, B. (2016). The 7 biggest problems facing science, according to 270 scientists. Vox. Available: https://www.vox.com/2016/7/14/12016710/science-challeges-research-funding-peer-review-process
  6. Bornmann, L. & Leydesdorff, L. (2013). The validation of (advanced) bibliometric indicators through peer assessments: a comparative study using data from InCites and F1000Research Journal of Informetrics, 7(2), 286-291. https://doi.org/10.1016/j.joi.2012.12.003
  7. Bornmann, L., Wolf, M., & Daniel, H. D. (2012). Closed versus open reviewing of journal manuscripts: how far do comments differ in language use? Scientometrics, 91, 843-856. http://doi.org/10.1007/s11192-011-0569-5
  8. European Commission (2020). European Commission awards contract for setting up an open access publishing platform. Available: https://research-and-innovation.ec.europa.eu/news/all-research-and-innovation-news/european-commission-awards-contract-setting-open-access-publishing-platform-2020-03-20_en
  9. F1000Research (2023). https://f1000research.com/
  10. Hayes, A. F. (2017). Introduction to Mediation, Moderation, and Conditional Process Analysis: A Regression-Based Approach (2nd ed.). New York: Guilford Press.
  11. Huggett, S. (2012) F1000Research journal rankings: an alternative way to evaluate the scientific impact of scholarly communications. Research Trends, 1(26), Article 3. Available: https://www.researchtrends.com/researchtrends/vol1/iss26/3
  12. Janowicz, K. & Hitzler, P. (2012). Open and transparent: the review process of the semantic web journal. Learned Publishing, 25(1), 48-55. http://doi.org/10.1087/20120107
  13. Khan, K. (2010). Is open peer review the fairest system? No. BMJ, 341, c6425. http://doi.org/10.1136/bmj.c6425
  14. Merton, R. K. (1968). Matthew effect in science: reward and communication systems of science are considered. Science, 159(3810), 56-63. https://doi.org/10.1126/science.159.3810.56
  15. Mulligan, A., Hall, L., & Raphael, E. (2013). Peer review in a changing world: an international study measuring the attitudes of researchers. Journal of American Society for Information Science and Technology, 64, 132-161. http://doi.org/10.1002/asi.22798
  16. Ross-Hellauer, T. (2017). What is open peer review? a systematic review. F1000Research Research, 6, 1-37. http://doi.org/10.12688/f1000research.11369.2
  17. Smith, R. (2006). Peer review: a flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99, 178-182. http://doi.org/10.1258/jrsm.99.4.178
  18. Thelwall, M., Allen, L., Papas, E.-R., Nyakoojo, Z., & Weigert, V. (2021). Does the use of open, non-anonymous peer review in scholarly publishing introduce bias? evidence from the F1000Research post-publication open peer review publishing model. Journal of Information Science, 47(6), 809-820. https://doi.org/10.1177/0165551520938678
  19. Tsakonas, G. (2021). Open science cannot succeed without open peer review. LIBER Quarterly: The Journal of the Association of European Research Libraries, 31(1), 1-6. https://doi.org/10.53377/lq.11114
  20. van Rooyen, S., Godlee, F., Evans, S., Black, N., & Savage, R. (1999). Effect of open peer review on quality of reviews and on reviewers' recommendations: a randomised trial. British Medical Journal, 318(7175), 23. https://doi.org/10.1136/bmj.318.7175.23
  21. Vines, T. (2013). How rigorous is the post-publication review process at F1000 research? Available: https://scholarlykitchen.sspnet.org/2013/03/27/how-rigorous-is-the-post-publication-review-process-at-f1000-research/
  22. Walsh, E., Rooney, M., Appleby, L., & Wilkinson, G. (2000). Open peer review: a randomised controlled trial. The British Journal of Psychiatry, 176(1), 47-51. https://doi.org/10.1192/bjp.176.1.47
  23. Waltman, L. & Costas, R. (2014). F1000Research recommendations as a potential new data source for research evaluation: a comparison with citations. Journal of the Association for Information Science & Technology, 65(3), 433-445. https://doi.org/10.1002/asi.23040
  24. Wang, P., Williams, J., Zhang, N., & Wu, Q. (2020). F1000Research Prime recommended articles and their citations: an exploratory study of four journals. Scientometrics 122, 933-955. https://doi.org/10.1007/s11192-019-03302-w
  25. Williams, A. E. (2017). F1000: an overview and evaluation. Information and Learning Sciences, 118(7-8), 364-371. https://doi-org-ssl.access.inu.ac.kr/10.1108/ILS-06-2017-0065
  26. Wolfram, D., Wang, P., Hembree, A., & Park, H. (2020). Open peer review: promoting transparency in open science. Scientometrics. https://doi.org/10.1007/s11192-020-03488-4
  27. Zong, Q., Fan, L., Xie, Y., & Huang, J. (2020). The relationship of polarity of post-publication peer review to citation count: evidence from publons. Online Information Review, 44(3), 583-602. https://doi.org/10.1108/OIR-01-2019-0027
  28. Zong, Q., Xie, Y., & Liang, J. (2020). Does open peer review improve citation count? evidence from a propensity score matching analysis of PeerJ. Scientometrics, 125, 607-623. https://doi.org/10.1007/s11192-020-03545-y