Assessment and Support Measures of Academic Journals in the National Open Access Platform AccessON

Abstract

This study aims to assess the maturity of Korean open access (OA) journals using the OA infrastructure provided by the Korea Institute of Science and Technology Information, and develop necessary strategies for future improvement. The assessment model consists of three dimensions, 12 items, and 24 sub-items. The importance of the three dimensions (A: OA policy establishment and disclosure, B: OA sustainability, and C: Journal openness quality) was differentiated by the Analytic Hierarchy Process, and the maturity stages were divided into five levels (Entry, Growth1, Growth2, Maturity1, and Maturity2). The assessment was carried out twice for 100 academic journals. The results indicated that the proportion of journals at or above the Growth1 level increased by 11% to reach 83% during the second assessment phase, which could be owing to the learnings of the first assessment. Following expert consultations on the assessment results, three support measures were identified to activate OA. The first includes OA promotion and education activities, which involve creating standard regulations and guidelines, and advancing educational activities for societies that are either preparing for or currently implementing OA. The second involves providing support for technical aspects, such as identifiers, XMLization, and copyright management, through peer review and OA publishing platforms. The third includes collaborative activities to enhance journal evaluations and the recognition criteria for researchers' achievements in OA journals, and fostering cooperation with national and research and development institutions for financial support.

1. BACKGROUND AND PURPOSE

The inception of open access (OA) involved activities where authors self-archived their research papers. The term was officially defined by the Budapest Open Access Initiative (BOAI) (Chan et al., 2002) as the practice of providing online access to scholarly research outputs without cost, in addition to minimizing copyright and other usage restrictions to enable free access.

OA acts as a significant driver in promoting knowledge sharing and advancing scholarly research. In the traditional publishing model, users face difficulties accessing the latest research findings due to high subscription fees and copyright restrictions. By allowing researchers and the general public easy access to scholarly materials, OA improves the quality of research and education and accelerates the dissemination of knowledge (Willinsky, 2009).

The importance of OA is evident in ensuring research transparency and the reproducibility of research results, which enhances scientific reliability. In addition, OA ensures equal access to information for researchers in developing countries and institutions facing budget constraints. This is critical for promoting global scholarly collaboration and bridging the international knowledge gap.

OA contributes to the rapid dissemination of research. Unlike traditional academic publishing, OA journals are quickly published on online platforms, allowing the latest research on urgent public health crises or environmental issues to be rapidly communicated. This increases the timeliness and social impact of academic research. In light of the recent COVID-19 pandemic, the general public has recognized the importance of quickly spreading research findings. However, OA is not achieved merely by making academic papers available online. It signifies a complete transformation of traditional academic publishing. This involves changes in related systems, reflecting diverse stakeholders’ perspectives, new publishing and service infrastructures for OA, and changes in how academic societies handle costs.

Successful OA realization requires the efforts of various stakeholders, including researchers, societies, funding organizations, and governments. These actors must work together organically and play complementary roles. Since it is necessary to summarize the efforts required to promote OA and evaluate the current level of activation, it is important to cumulatively evaluate various factors, assess problems, and establish development directions.

This study’s purpose is to assess the maturity stages of journals that have declared OA in Korea and identify the current status of OA journals and the factors necessary to revitalize them. For this purpose, an assessment model and maturity stages are needed. The assessment model utilizes the results of a study conducted by the Korea Institute of Science and Technology Information (KISTI) from 2021 to 2022 by Kim et al. (2023). Their study summarized various factors necessary for assessing OA journals and organized them into a model. The maturity stage utilized the criteria presented by KISTI. The assessment was limited to 100 journals using KISTI’s system—a Korean research funding organization that provides free support for OA platforms. The assessment model and maturity stages are not internationally agreed-upon standards, and the assessment is limited to Korean OA journals utilizing KISTI’s platform. These can be considered study limitations.

KISTI seeks to improve the quality of Korean OA journals and provides the technical infrastructure necessary for the maturation of OA. To sustain these services, it is necessary to accurately assess and analyze the status of journals currently utilizing KISTI’s platform. This assessment is not intended to evaluate the current level of OA journals, but instead to identify the support elements needed in the future. These support elements are anticipated to serve as a direct basis for KISTI to provide OA-related support services, and in the long run, may contribute to increasing the OA level in Korea.

2. LITERATURE REVIEW

Since the BOAI defined the term “open access” in 2002 as providing free online access to scholarly research outputs, various studies have explored its implications, benefits, and challenges. Early OA-related studies introduced the concept, highlighting its necessity and importance (Lee, 2008; Suber, 2005). There has been a significant emphasis on understanding OA trends and consolidating the positions of diverse stakeholders, in order to develop strategies to advance OA. For example, Yoon and Kim (2007) considered organizing the differing perspectives of the library community, research groups, and publishing sector as an essential task in the spread of OA. Accordingly, they conducted a study that compiled OA trends and limitations and arranged issues to make proposals. Similarly, Choi and Cho (2005) categorized numerous cases to define OA trends, proposing the roles of librarians, universities, publishers, funding organizations, societies, and governments.

Subsequent studies have explored the elements necessary for adopting OA, with Ko et al. (2009) investigating whether factors that influence OA adoption vary by the type of publishing institution or academic field. They pointed out OA promotion and the transition of excellent academic journals to OA as important steps for the future. Meanwhile, other studies have classified OA into gold, green, and bronze OA journals to better understand the current state of OA (Basson et al., 2022; Piwowar et al., 2018).

Addressing copyright issues is critical in the OA landscape. Hong (2008) highlighted the importance of resolving such disputes in the context of South Korean academic papers and investigated the copyright attribution problems of 906 journals. Kim (2020) discussed the policy and legal aspects of OA, emphasizing how amendments to copyright law could either facilitate or hinder the OA movement. Shavell (2010) argued for the complete removal of copyright from scholarly works.

Kim et al. (2016) examined OA policies for scholarly papers supported by public research funds, suggesting ways to enhance their accessibility through a comparison of policies from different countries. Other studies have investigated the effectiveness of OA, with the findings of Brody et al. (2004) and Antelman (2004) revealing that OA papers demonstrated higher citation rates and impact than non-OA papers.

Focusing on researchers, Park (2022) investigated their perceptions of establishing a healthy OA ecosystem. Shim (2021) explored the criteria by which South Korean researchers choose OA journals and found that they have less experience with OA publishing, with differing criteria for selecting OA and non-OA journals. This underscores the need for policy improvements and education in this field. Studies on OA implementation and acceptance in universities and academic libraries, which are primary centers of research, have also been conducted. Kim and Lee (2005) suggested that technical and policy factors should be considered when operating an OA repository in a university. Kim (2018) proposed ways to manage and share theses and dissertations through OA university repositories. Kang and Chang (2010) examined criteria for libraries to select and evaluate OA journals.

Recent studies have identified and sought to improve the current state and challenges of OA. Bosman et al. (2021) conducted a large-scale survey and focus group interviews with 1,619 diamond OA journals to determine their status, quality management practices, and sustainability challenges. About 67% of the journals managed quality through double-blind peer review, and 78% adhered to OA guidelines. However, the sustainability of OA journals faces challenges, including operating platforms and server limitations. Bosman et al. (2021) pointed to the need for efforts to expand visibility due to the difficulties of indexing OA content, noting that 20% of diamond OA journals are considering discontinuing their current OA publishing due to a lack of operational staff and financial constraints.

Paquet et al. (2022) analyzed the status of OA academic publishing in Canada, based on field and university, finding that the mandatory OA policies of research funding agencies impact the expansion of OA academic publishing. They suggested that a national-level OA publishing platform is necessary to stimulate OA publishing. Choi et al. (2012) investigated user satisfaction with South Korean OA services, and noted that the future development of OA requires policy support, OA journal publishing and repository operations, establishment of a governance system, fostering of researcher participation through OA promotional activities, and building of a global cooperation system. Kim (2014) suggested that academic societies explore various OA distribution methods and identify the methods and platforms suitable for each academic society, emphasizing the importance of disclosing each society’s OA policy on SHERPA/RoMEO, DOAJ, and PMC. Additionally, Duc et al. (2020) and Krawczyk and Kulczycki (2021) pointed out that well-intentioned OA has led to unintended consequences, such as the emergence of predatory journals, leading to confusion among scholars.

Numerous other studies, such as those by Antelman (2004) and Brody et al. (2004), have demonstrated the significant positive impacts of OA, including higher citation rates, improved accessibility to research, and increased dissemination of scholarly knowledge. However, several challenges are yet to be resolved, including changes in the publishing model, copyright issues, and the development of sustainable financial models.

3. METHOD

3.1. Assessment Model

In this study, we used the assessment model developed by KISTI (Kim et al., 2023) to assess the current state of South Korean academic journals that have declared themselves to be OA. The model simplifies reality, yet does not perfectly represent it. Moreover, not all target academic journals have the same level of preparation. Thus, while we made the assessment based on this model, we introduced the analytic hierarchy process (AHP) to construct different levels of importance for dimensions and items, accounting for their varying importance. In addition, we interpreted the results by dividing the maturity levels into Entry, Growth1, Growth2, Maturity1, and Maturity2, and assigning score ranges for each level. Table 1 outlines the assessment model.

Table 1. Assessment framework for open access (OA) journal publishing practices

This model is a modified version of the 2021 model. The revised model, which we adjusted based on the appropriateness of the dimensions, items, sub-items, and key questions, reflects the results of interviews held with editors of journals that publish OA articles (Kim et al., 2023).

The model consists of three dimensions, 12 items, and 24 sub-items. Each sub-item has specific key questions for assessment. In the present study, we used these questions for assessment. However, direct verification by internal stakeholders of the journals is necessary for items A1, B1, B2, B3, and B4. Other items can be verified through the journal’s website and the manuscript submission review system. Hence, we conducted surveys with internal stakeholders, such as editors and society presidents, for A1, B1, B2, B3, and B4, and directly investigated the other items.

In addition, in KISTI’s assessment model, A2.1 contains two key questions: Q4 and Q5. However, Q4 is designed to assess the three sub-items belonging to A1. For the convenience of quantifying future assessment outcomes, we divided A2’s Q4 into three questions during the actual assessment (i.e., we restructured it so that individual scores could be obtained for each question).

3.2. Weight Model

KISTI’s assessment model includes scoring during the actual assessment, with “not applicable” receiving 0, “partially applicable” receiving 0.5 and 0.75, and “applicable” receiving 1 point. We adhered to this standard; however, for ease of converting future measurement results, we converted 1 point into 100 points for calculation. In the original model, we gave all three dimensions the same importance. However, given that all OA promotion environments do not start from the same baseline, especially reflecting the situation of OA in South Korea, we performed an AHP analysis to apply different scores for the assessment.

AHP, developed by Saaty (1980), is used to find answers to complex decision-making problems. Saaty (1980) proposed an approach to solving such problems by classifying various options into components, determining the relative priorities of these components, and deciding on the final priorities. AHP has been utilized for decision-making in numerous fields.

In this study, we conducted an online survey to derive weights. The participants included internal staff at KISTI, researchers involved in previous studies, journal officials who had been interviewed for previous studies, and experts familiar with KISTI’s OA activities. We administered the survey to 18 participants from February 24 to March 5, 2023, over 10 days. We finally used 11 responses. As a result, we derived the weights for dimensions and items of the assessment model, as shown in Table 2.

Table 2. Results of AHP analysis

AHP, analytic hierarchy process; OA, open access.

According to Table 2, the weights of the three dimensions are as follows: A: OA policy establishment and disclosure=48%, B: OA sustainability=29%, and C: Journal openness quality=23%. It is evident that OA policy establishment and disclosure is the most important factor in the current Korean OA environment. In terms of each dimension, A1: OA policy establishment (66%), B1: Finance (39%), and C1: Submission and review (23%) were the most important items.

In this study, we converted the overall assessment outcomes into a 100-point scale by applying the weights to the dimensions based on the AHP results. The converted scores do not numerically represent the current state of OA journals. Hence, we interpreted the OA maturity levels as divided into the five levels used by KISTI (Table 3).

Table 3. Open access maturity levels

Table 3 is based on KISTI’s criteria, considering that the subjects of the assessment model, though varying in degree, aim for OA and utilize KISTI’s OA platform. Hence, the entry-level range is 50 or less, and we set the subsequent four levels equally.

4. ASSESSMENT OF OA JOURNALS AND THEIR PROGRESS

The 100 journals targeted for maturity assessment are located in South Korea and represent the entire set of academic journals that utilize the Journal Repository provided by KISTI’s OA platform, AccessON, or the OA-supporting peer review management system called ACOMS+. These journals were selected because they all currently use KISTI’s OA infrastructure, and hence comprise a comprehensive sample for evaluating the maturity of OA practices within the platform. The assessment was conducted twice, and Table 4 presents its progress.

Table 4. Assessment progress

OA, open access.

As shown in Table 4, items directly investigated by the researcher were re-verified by a separate reviewer to determine appropriateness before final analysis. We conducted the assessment twice. During the second assessment phase, individual survey forms were sent to journal officials who had responded in the first phase. For direct investigations, we utilized an online survey tool (Google Forms) to record evidence of survey results (screen captures or URLs).

5. ASSESSMENT RESULTS AND DERIVATION OF IMPROVEMENT MEASURES

5.1. Assessment Results

Table 5 displays descriptive statistics for the first and second assessment phases.

Table 5. Descriptive statistics of the first and second assessment results

As depicted in Table 5, all three dimensions improved in the second assessment phase. Specifically, the weighted overall average in the first survey was 58.92, which reached 67.34 in the second survey, an improvement of 8.42 points. By dimension, A: OA policy establishment and disclosure increased from 28.40 to 33.54, B: OA sustainability increased from 19.80 to 20.69, and C: Journal openness quality increased from 10.71 to 13.11. In particular, the improvement of 5.14 points for dimension A is significant compared to 0.89 for dimension B and 2.4 for dimension C. Hence, in dimension A, the contents related to the establishment of policies within the society were strengthened before the second survey.

The results, when mapped to maturity levels for comparison, are shown in Fig. 1.

Fig. 1. Maturity levels by dimension (left: first results; right: second results). OA, open access.

In the first assessment, dimension A was at the Growth1 level, dimension B was at the Growth2 level, and dimension C was at the Entry level. In the second assessment phase, dimension A improved to the Growth2 level, dimension B remained at the Growth2 level, and dimension C improved to the Growth1 level.

Table 6 presents the distribution of maturity levels for the 100 responding journals.

Table 6. Comparison of the first and second maturity levels

While 72% of the journals were at the Growth1 level or above in the first assessment phase, this increased to 83% in the second assessment phase, demonstrating a significant improvement of 11%. To assess the current situation of South Korean OA journals and construct a future support system, further detailed verification is needed. It is necessary to examine the assessment outcomes by dimension, focusing on the second assessment phase. The assessment results for dimension A are presented in Table 7.

Table 7. Second assessment results for dimension A​​​​​​​

As seen in Table 7, journals at the Entry level need support for formalization and public disclosure of overall OA policies. Even journals at the growth level require clarity on the policy of disclosing OA publishing finances (A1.2), formalization of the archival policy (A1.3), and transparency and registration of formalized OA regulations (A2.1). The assessment results show low figures, indicating the need for future educational activities.

Table 8 presents detailed results for dimension B.

Table 8. Second assessment results for dimension B​​​​​​​

As displayed in Table 8, regardless of the maturity level, all journals urgently need support for transparency in OA academic publishing finances (B1.3). Meanwhile, journals at the Entry level require support for transparency in OA publishing finances (B1.3), training and education for academic society staff (B2.1), and collaboration for publication and dissemination (B4.1). Hence, it is evident that South Korean OA journals lack awareness of transparent operations beyond the value of sharing academic works.

Table 9 outlines detailed results for dimension C.

Table 9. Second assessment results for dimension C​​​​​​​

Dimension C showed the lowest assessment scores. Regardless of the maturity level, active support is necessary for C1 and C2, which deal with openness of reviewers and opinions and depend mostly on the initiative of the academic societies. This indicates that awareness of the importance of process transparency, beyond sharing the final outputs, is low. Journals at the entry and growth levels exhibited low usage of identification systems for papers and authors. For journals at the growth level, providing machine-readable metadata for individual papers, as in C4.4, is a challenging issue to be solved by academic societies individually. Such problems could be addressed by platform operators, such as KISTI, through functional improvements or separate informatics projects.

The interval between the two assessment rounds was three months, which is short. Nonetheless, the improvement in assessment outcomes implies that the maturity model assessments served as a form of education and promotion. This demonstrates that academic societies declaring to be OA in South Korea do not have a deep understanding of OA. In particular, it is necessary to recognize that process transparency is important. As a government-led OA platform operating agency, KISTI needs to perform various activities beyond providing technical support to promote OA.

Table 10 depicts the support items necessary for OA journals based on the problems identified in the assessment results.

Table 10. Support measures for OA journals

OA, open access; ORCID, Open Researcher and Contributor ID; ISNI, international standard name identified.

5.2. Consultations with Experts

In this study, we shared the assessment results with OA experts, and upon consulting with them, we performed further assessments to identify crucial considerations for future OA support.

The experts included three researchers who conducted maturity model studies and three association representatives currently engaged in OA publishing. An advisory meeting was held using the online meeting tool Zoom on September 19, 2023 at 15:00 hours. The purpose of the advisory meeting was to share the results of the two assessment phases and develop the necessary content for future OA support. The participants shared their opinions about the assessment results and other aspects, summarized in Table 11.

Table 11. Summary of results from the advisory meeting with experts

OA, open access; KISTI, Korea Institute of Science and Technology Information.

The most discussed aspect was finance. The most significant challenge for academic societies aiming for OA was editorial, publishing, and infrastructure costs for academic paper services. In South Korea, institutions such as KISTI provide peer review, archiving, and service systems on a national scale free of charge. This is a prime example of public funds being directly invested in creating an OA environment. KISTI has built an OA platform and continues to manage and improve it, striving to develop the OA ecosystem. These efforts play a vital role in enhancing the openness of research results, promoting the sharing of scholarly knowledge, and contributing to scientific and technological advancement.

Nonetheless, the actual costs of editing, reviewing, and publishing remain a challenge. Despite declaring themselves to be OA, academic societies are often forced to rely on authors for review and publication costs. This is highlighted by maturity assessments and reaffirmed through the advisory meeting, underscoring the need for support.

The importance of forming a separate organization for OA activities, rather than individual academic societies acting alone, was highlighted to address issues collectively. As shown by the improved results of the second assessment phase, education and promotion are essential to accurately communicate what is necessary for OA and what academic societies need to prepare and implement.

Furthermore, there is a need to reconsider the assessment model and method, a point often made by previous researchers (Kim et al., 2023). It is important to note that respondents’ levels of OA awareness vary; hence, rather than interpreting survey results directly, conducting qualitative research on a few journals based on the assessment results was suggested. In particular, it may be necessary to reconsider the current weighting method and assessment criteria. Researchers have noted that despite containing the fundamental elements of OA journals, the lowest scores for dimension C indicate issues with transparency and accessibility. For technical support, preemptively preparing technical devices and encouraging societal participation in aspects where culture needs to change (such as open peer review) was identified as a potential promotion strategy.

5.3. Summary of Improvement Measures

The improvement measures derived from the journal assessments and consultations with experts include aspects that require the efforts of academic societies and external support. As an operator of a nationally supported OA platform, KISTI can support the following elements among the proposed improvements:

• OA promotion and education activities: Creation of standard regulations and guidelines and educational activities for societies preparing for or currently implementing OA.

• Technical support: Support for technical elements, such as identifiers, XMLization, and copyright management, through peer review and OA publishing platforms.

• Collaborative activities: Activities for improving journal evaluations and recognition criteria for researchers’ achievements regarding OA journals, and cooperation with national and research and development institutions for financial support.

As noted in the consultation with experts, financial support for editorial staff and other OA-related expenses is difficult for societies to manage and requires external support.

6. CONCLUSION

We used KISTI’s OA maturity model to assess OA journals and derive improvement measures. Although the period was short, the two assessment phases confirmed an improvement in the maturity of the target journals. This can be interpreted as an educational effect where the dimensions and elements of the maturity model were reviewed by the responding journals. These results are due to academic societies in South Korea becoming aware of the maturity model’s assessment items between the two surveys, correcting the errors, and supplementing the aspects they were unprepared for. While there is increasing interest in OA and quantitative growth, it is necessary to address qualitative aspects such as transparency in the peer review processes, quality of metadata, and adherence to ethical publishing standards, in order to fully realize OA’s positive effects.

The results indicate that while the maturity of OA journals has improved over time, some areas still require improvement. In particular, targeted support and interventions are needed to address specific weaknesses, such as financial transparency, policy documentation, and technological infrastructure.

Moreover, the maturity model used in the study needs to be updated and supplemented as new trends and issues emerge in the development of OA journals. The weights used in this study reflect the priorities and importance assigned by the stakeholders of OA in South Korea at the time of the survey. As the OA landscape evolves, these priorities may change, necessitating modifications to the weights. Continuous monitoring, support, and education are important for advancing the OA movement, with the goal of a more transparent, accessible, and high-quality scholarly communication ecosystem. This can be interpreted as the journals gaining a better understanding of the maturity model through the process of reviewing its dimensions and elements, which in turn led to improvements in their practices.

In this paper, we focus on the results of the first OA maturity assessment, using the maturity model developed by KISTI, and propose future improvement measures. This study was conducted on 100 journals utilizing KISTI’s platform; however, it is difficult to say that these journals are representative of all OA journals in Korea. Further research is needed on the assessment model and in various OA journals. In addition, it is necessary to analyze the differences in survey data while considering the specific characteristics of academic societies, such as their respective academic fields.