The Application of Delphi-AHP Method in the Priority of Policies for Expanding the Use of Artificial Intelligence

Abstract

Governments around the world are actively establishing strategies and initiatives to spread the use of artificial intelligence (AI), for AI is not a mere new technology, but is an innovative technology that brings about extensive changes in industrial and social structures and is a core engine that will lead the 4th Industrial Revolution. The South Korean government has also been paying attention to AI as a technology and tool for innovative growth, but its application to the industries is still rather sluggish. The government has prepared multifarious AI-related policies with the aim of constructing South Korea as an AI powerhouse, but there is no clear strategy on which detailed policies to implement first and which industries to apply AI preferentially. With these limitations of South Korea's AI policies in mind, this paper analyzed the priorities of industries in AI adoption and the priorities of AI-related national policies, using Delphi-AHP method for 30 top-level AI experts in South Korea. The results of analysis show that AI application is urgent and necessary in the fields of medical/healthcare, public and safety, and manufacturing, which seems to reflect the peak of the COVID-19 crisis in the second half of 2020 at the time of the investigation. And it turns out that policies related to AI talent cultivation, data, and R&D investment are important and urgent above all in order for organizations to apply AI. This suggests that strategies are required to focus limited national resources on these industries and policies first.

1. Introduction

AI is in the spotlight as a technology that will bring a revolution to various industrial sectors. For example, AI can be applied to traditional industries such as manufacturing in order to enhance process efficiency or provide customized goods/service. In the sector of education, AI can assist teachers, and in the sector of welfare, the AI chatbot can be a helpful companion with whom old people can chat. In addition, in the sector of public security, it can be used to respond to crimes, like the detection of abnormal behavior or illegal photo-taking. Besides, AI can support data-based decision making in various areas directly linked to our life, and it is expected that its roles in the areas of sensibility and creativity will become greater according as AI technology progresses.

AI, which is not a mere new technology, but an innovative technology that brings about a wide range of structural changes across society and industries, is very important in that it is a core engine to lead the Fourth Industrial Revolution that will enhance citizens' quality of life and national competitiveness. AI is set to shape global competitiveness over the coming decades, promising to grant early adopters a significant economic and strategic advantage [1]. And therefore, countries and regions around the world are establishing strategies and initiatives to guide and foster the development of AI. Now, according to the AI observatory maintained by the Organization for Economic Cooperation and Development (OECD), some 60 countries, including South Korea, have AI policies [2].

Despite the positive impacts of AI and the active efforts of governments, however, it has been revealed that the actual adoption and full-fledged use of AI in industries is more sluggish than expected. A survey¹⁾**conducted by the Korea Development Institute (KDI) from October to November in 2020 showed the aggregate AI adoption rate of 3.6% for companies [3].Similarly, a survey conducted by the Statistics Korea also revealed South Korea's low AI adoption rate (average 3% across industries) and industrial gaps [4].Now that KDI and Statistics Korea surveyed companies of considerable sizes, that is, companies with 20 employees or more and companies with 50 regular employees or more, respectively, (in general, companies of large sizes adopt AI more actively), the AI adoption rate among all companies including companies of very small sizes may be lower than their findings.

In the previous century, South Korea successfully strengthened her position as a fast follower in the world economy. In the 21st century, however, according as traditional industries have already entered a mature stage and late comers give a chase, South Korea faces the challenge of having to achieve an innovative growth as a fast mover that leads industrial changes and pioneers new areas. And AI, as a technology and tool for such an innovative growth, is becoming important. In particular, the expansion of AI adoption has become more important, according as global economic crisis and untact demand caused by the recent pandemic have heightened the need for digital transformation and the preoccupancy of AI technology has become part of national competitiveness. Therefore, the government needs to collect strategic ideas in order to expand AI adoption quickly across industries.

The South Korean government has announced several strategies* for AI-based national development since before the pandemic, and it seems that the government has been successful in showing that it is equipped with all-around policies that are not behind other countries. However, the strategy for prioritizing the numerous policies is unclear. In the ‘National Strategy for Artificial Intelligence, ’which set forth the Korean government's AI strategies, the vision of ‘AI for Everyone, AI of Everything' was established, and 9 major strategies and pan-governmental 100 major implementation tasks were suggested to achieve the vision [5]. The National Strategy is very excellent in that it contains almost all policy tasks in all fields required for the adoption and expansion of AI; however, it has the limitations that it does not clearly suggest which policy should be promoted in priority among the numerous policies and which industry requires more important and urgent support. Given these limitations of Korean AI strategies, this study is going to clarify to which area and policies the limited national resources should be allocated first, by analyzing the priorities of industries in AI adoption and the priorities of AI-related national policies, using the Delphi-AHP (analytic hierarchy process) method.

Therefore, this study set the following two research objectives: the selection of industries to which AI should be applied in priority, and the determination of the priorities of policies for expanding AI application. Identifying the priorities of industries in AI adoption and the priorities of policies will be of great help in administering limited national resources efficiently to expand AI.

2. Method

2.1 Review of literature on AI policies

Prior to the Delphi method, literature review was conducted. To identify which items were included in policies for expanding the use of AI, a number of strategies announced meantime by the South Korean government and policy plans suggested by public agencies were analyzed. National strategies subjected to literature review included “Data Industry Activation Strategy” (2018) [6], “Data AI Economy Activation Strategy” (2019), “National Strategy for Artificial Intelligence” (2019), “Strategy for Developing Cloud Industry in Preparation for Data Economy and the Era of AI” (2020), and “Strategy for the Innovative Growth of Digital-based Industries” (2020) [7]. And reports of public agencies on policy plans included “A Prospective Analysis of Artificial intelligence Technology and Innovation Policies” by the Science & Technology Policy Institute (2018) [8], “The Mid-to Long-term Direction of AI Data Construction Derived from AI Field Survey” by the National Information Society Agency (2019) [9], and “Policy for Activating the Convergence and Use of Artificial Intelligence (AI) in the Public and Private Sectors” by the Korea Information Society Development Institute (2019) [10].

Items collected from the literature review were used to provide illustrative examples to respondents in the first round of the Delphi. The examples were to help respondents' memory and understanding about various policy items. The expert panel was allowed to refer to this examples according to their own judgment, and to voice their own opinions freely.

2.1 Delphi method

The Delphi method was originally conceived in the 1950s by Olaf Helmer and Norman Dalkey of the Rand Corporation [11]. It was devised in order to obtain the most reliable opinion consensus of a group of experts by subjecting them to a series of questionnaires [12]. It is based on the assumption that group judgements are more valid than individual judgments, and it is a method for finding solutions to complicated problems from opinions of an expert group. While there is no prescribed number of panel members for the Delphi, most studies use 15~35 experts for the panel [13], and it is reported that 2-3 rounds are enough for Delphi method [14].

In this study, two rounds of the Delphi were conducted online. The first round consisted of unstructured open-ended questions, and respondents were allowed to refer to illustrative items, if necessary. And the second round consisted of structured closed-ended questions of a 5-point scale based on the results of responses in the first round. Items for the second round of the Delphi were prepared after 3 experts in economics, business administration, and information system analyzed free answers obtained from the first round Delphi, combined similar contents into groups, and reclassified them by removing minority opinions. Based on the results of the second round, policy items of above average importance were selected.

Under the agreement of 3 experts, policy items finally identified from the first and second round were classified into five categories by using the framework of STEEP analysis, a macro-perspective analysis technique. STEEP refers to five macro-environmental factors of social, technological, environmental, economic, and political aspects that can affect company and industry competitiveness. National policies can have direct and indirect effects on the macro environment of company activities and industry, and thus policy items were classified according to these five aspects and were used later for the AHP analysis. STEEP analysis is a framework that can be used to study various aspects of government policy, and Lee et al.(2017) have derived policy issues in each area based on STEEP analysis [15].

On the other hand, the experts were told to select from various industry types areas where AI application is expected to be easy (ease), areas where AI application is expected to be necessary (necessity), and areas where AI application is expected to be urgent (urgency), and they were asked to freely state reasons for their selection.

The Delphi method was conducted in November, 2020. For the method, High-level experts currently researching or dealing with AI in industry, academia, and research were first contacted via email and phone, and were asked to consent to participate in the research. A total of 67 experts were contacted; and in the first round, 31 experts responded, showing a response rate of 46.3%, and in the second round, one dropout occurred, and a total of 30 experts (9 from industries, 11 from academia, and 10 from research institutes) responded. Of these, 10 experts are the 3rd advisory members of the Presidential Committee on the 4th Industrial Revolution in Korea. The information of the respondents is shown in Table 1.

(Table 1) Experts List

2.3 AHP method

AHP (Analytic Hierarchy Process) is a multi-criteria decision-making method developed by Saaty (1977) [16]. It is a technique for evaluating mutually exclusively decision alternatives systematically and deriving their priorities. When a problem is complex and there are multiple criteria for evaluation, AHP is based on the principle of making judgment by using the process of analyzing the problem by stage or hierarchically similarly to human thinking system. The AHP technique is a method for deriving results by measuring the relative importance or preference of alternatives with a ratio scale, and comparing them quantitatively. Owing to its advantages of simplicity, clarity, convenience, and generality, it is actively used in the decision-making process of the public sector. In general, a questionnaire for the AHP analysis uses a 9-point scale to evaluate the importance of each alternative, but a 5-point scale is also used to enhance the reliability of responses. This study used a 5-point scale.

Due to the characteristics of AHP that performs pair-wise comparison of several alternatives, sometimes consistency in pair-wise comparison responses may be low. Therefore, whether the results of responses are consistent is judged by means of consistency ratio (CR). If the CR value of responses exceeds a reference value, the responses are considered inconsistent, and are excluded from the analysis. Saaty claims that a ratio of less than .20 is considered tolerable [17]. In this study, responses that exceeded this reference value were excluded from analysis.

In this study, the AHP analysis was carried out in order to compare the priorities of 5 perspectives and their items derived from the Delphi. And this evaluation was made in the two dimensions of 'the importance of policy' and 'the urgency of policy, ' which are representative evaluation indices of policies. That is, in the dimension of 'the importance of policy, ' the pair-wise comparison of the 5 policy perspectives and of items within each policy perspective was performed. And also the same procedure was carried out in the dimension of 'the urgency of policy.' All of the 30 experts in industry, academia, and research who previously responded in the Delphi submitted their responses in the pair-wise comparison for the AHP analysis, which was conducted in early December, 2020.

3. Result

3.1 Awareness of industries to adopt AI

In this study, the experts were asked to respond with 3 sectors prioritized among 10 industries with respect to each of the ease of AI adoption, the necessity for AI adoption, and the urgency of AI adoption.

First, in the category of ease, the manufacturing sector showed the highest response rate, followed by medical/healthcare and finance. The respondents said that they pointed out the manufacturing as the industry of the easiest AI adoption because basically it is easy to secure data in this sector due to systematical data accumulation and there is not any major problem in using the data safely. The medical/healthcare sector was cited because healthcare information, not an individual's medical information, has great potential for applicability, it is good to apply AI by using data accumulated through cooperative clusters between hospitals, and it is an area to which pattern recognition technology can be applied relatively easily. The finance sector was evaluated as an area of easy AI grafting because it is easy to collect data, there are already rich data, the secured data have often been cleaned, there is a high demand for intelligent service, and there are few legal restrictions and restrictions in personal information.

Second, in the category of necessity, the medical/ healthcare sector showed the highest rate of response, followed by the public and safety service and the transportation/logistics. The necessity for AI adoption was perceived most highly in the medical/healthcare sector because the purpose and benefits of the sector assume public characteristics and have great socioeconomic ripple effects, and we now face the challenge of overcoming the COVID-19 pandemic. As for the public and safety service, the experts responded that the sector needs the use of AI because AI can be used in areas to which people have difficulty in access, efficiency can be raised with less manpower, and the effects of AI can occur very immediately. As for the transportation/logistics sector, it is a cost-effective area, and it was found that this sector needs data-driven technology development for the sake of advance prediction and accident prevention.

Finally, in the category of urgency, medical/healthcare sector ranked first, followed by the manufacture and the public and safety service. As for the medical/healthcare sector, this sector can derive the maximum outcome of AI use most easily, while it affects citizens' life directly; it can complement and support insufficient medical manpower; and most of all, it was assessed that AI adoption in this sector is urgent in order to prepare against unexpected medical crisis like COVID-19. As for manufacturing, which is a representative industry of South Korea, it was pointed out that it is urgent to enhance its competitiveness and improve productivity by using AI. And the public and safety service, which is directly linked to citizens' life and safety, was assessed as urgently needing the use of AI, given the spread of COVID-19; and particularly, there was an opinion that it is urgent to construct AI-based social safety networks, for example, in disaster prevention and response, smart public security management, and traffic safety.

(Figure 1) Perception about industries in terms of the ease, necessity, and urgency of AI application

3.2 Identification of policy items based on Delphi method

A total of 22 policy items for activating the use of AI were identified from the Delphi method. And these policy items were categorized into 5 policy perspectives: social, technological, economic, environmental, and political (Table 2, Table 3). 3 policy items were categorized as a social perspective, 5 as a technological perspective, 4 as an economic perspective, 4 as an environmental perspective, and 6 as a political perspective.

(Table 2) STEEP: 5 perspectives of AI policy

(Table 3) AI policy items by perspectives

3.3 Prioritization of policy items through AHP analysis

When constructing a hierarchy for AHP analysis in this work, the ‘goal’ of ‘Expanding the use of AI’ was placed at the top, and 5 policy perspectives (STEEP) to achieve the goal were located at layer 1. And at layer 2 were placed ‘policy items’ identified from the Delphi (Figure 2). With this hierarchy of AHP, the pair-wise comparison of priorities was made for each layer in two dimensions of ‘the importance of policy’ and ‘the urgency of policy.’

(Figure 2) AHP hierarchy to determine the priorities of AI policies

3.3.1 Importance of policy

First, at layer 1, the priorities of 5 perspectives were analyzed in terms of the ‘importance of policy.’ As a result, the technological perspective showed the highest weight (0.312), followed by economic (0.216), political (0.196), environmental (0.166) and social (0.111) perspectives.

Next, policy items at layer 2 were subjected to pair-wise comparison according to the 5 perspectives, in terms of 'the importance of policy, ' and the results were analyzed to calculate the weight of each item. And this weight was multiplied by the weight of the relevant perspective calculated above, and thus the composite weight value was obtained. Then, a total of 22 policy items were prioritized.

As a result, among the 22 items, “Establishment of a system for fostering and securing AI talent” (0.081) was selected as the most important policy item, followed by “Expanding investment in next-generation AI research and fundamental research” (0.075), “Supporting the construction and sharing of quality dataset” (0.074), “Developing AI technology based on data specialized in industrial application for AI specialized companies” (0.061), and “Supplying data reflecting private-sector demand and expanding/improving data demand creation system” (0.058) (Table 4).

(Table 4) Priority ranking and weights in terms of policy importance

3.3.2 Urgency of policy

The analysis of ‘the urgency of policy’ was carried out with the same procedure and method as the above importance of policy. First, at layer 1, the priorities of the 5 perspectives were analyzed in terms of ‘the urgency of policy.’ As a result, the technological perspective showed the highest weight (0.303), followed by economic (0.213), political (0.188), environmental (0.171) and social (0.125) perspectives.

Next, policy items at layer 2 were subjected to pair-wise comparison according to the 5 perspectives, in terms of ‘the urgency of policy, ’ and the results were analyzed to calculate the weight of each item. And this weight was multiplied by the weight of the relevant perspective calculated above, and thus the composite weight value was obtained. Then, a total of 22 policy items were prioritized.

As a result, among the 22 items, “Establishment of a system for fostering and securing AI talent” (0.078) was selected as the most urgent policy item, followed by “Supporting the construction and sharing of quality dataset” (0.075), “Developing AI technology based on data specialized in industrial application for AI specialized companies” (0.069), “Expanding investment in next generation AI research and fundamental research” (0.067), and “Supplying data reflecting private-sector demand and expanding/improving data demand creation system” (0.065) (Table 5).

(Table 5) Priority ranking and weights in terms of policy urgency

3.3.3 Synthesis of the importance and urgency of policy

In the AHP method, experts were asked to make a pair-wise comparison of significance between ‘the importance of policy’ and ‘the urgency of policy’ as policy evaluation indices in case of establishing policies for expanding the use of AI, and then the weight of each evaluation index was calculated. ‘The importance of policy’ obtained the weight of 0.606, showing that the experts recognized it as a more important evaluation index than ‘the urgency of policy’(0.394).

Lastly, a procedure for determining the final priorities of the policy items was carried out in the combined consideration of policy importance and policy urgency. ①The composite weights calculated according to policy items at the above analysis of ‘policy importance’ were multiplied by the weight of ‘policy importance’ (0.606), ②the composite weights calculated according to policy items at the above analysis of ‘policy urgency’ were multiplied by the weight of ‘policy urgency’ (0.394), and then ③these two values were added (①+②) to calculate final weights, and the final ranking was decided on the basis thereof.

It was found that a policy item that showed the highest priority finally was “Establishment of a system for fostering and securing AI talent” (0.080), followed by “Supporting the construction and sharing of quality dataset” (0.074) and “Expanding investment in next-generation AI research and fundamental research” (0.072) (Table 6).

(Table 6) Priority ranking and weight : Synthesis of the importance and urgency of policy

4. Discussion

4.1 Perception about industries

It is expected that AI will greatly improve the productivity of the existing industries by achieving efficiency on the basis of automation and optimization. Therefore, attempt has been made across the world to achieve revolutionary change into intelligent industries by grafting AI onto the existing industries, thereby enhancing productivity and creating new value added. In 2018, McKinsey & Company estimated that AI could add around 1.2% every year, or $13 trillion, to global output by 2030 [18]. Thus, the task of raising competitiveness through turning into intelligent industries by grafting AI onto the existing industries is very important at the national level.

In the results of this study, the the medical/healthcare sector and the public and safety sector ranked high in terms of both necessity and urgency. This is presumed to be strongly related to the pandemic situation in the second half of 2020, when the research was conducted. Given that the issue of overcoming COVID-19 is a desperate and urgent task of the country and society, there are necessity and urgency to use AI in various ways in the medical/healthcare sector and the public and safety sector in order to promote citizens' health and safety. In this regard, the use of AI to develop vaccines and medicine, which has been carried out since the outbreak of COVID-19, and the use of AI at the stage of epidemic prevention is very desirable and needs to be encouraged further. Until the COVID-19 pandemic ends or even after the COVID-19 ends, it is necessary to continue to use AI in the medical/healthcare sector and the public and safety sector in order to prepare against other epidemics or national and social crises.

On the other hand, it is worth paying attention to the manufacturing industry in that it was chosen as a preferential AI adoption sector in terms of all of ease, necessity, and urgency. Obviously, the manufacturing industry is a preferential industry worth considering AI grafting positively, because it shows a high degree of ease for the use of AI in that data acquisition is relatively easy in the industry compared with other industries, and the industry is comparatively less sensitive to the privacy issue arising from the use of data. In addition, it is necessary to construct an industrial ecosystem that can maximize value added and secure national competitiveness by grafting innovative technologies like AI onto manufacturing, given that South Korea, which is characterized by the export manufacturing centered economic structure, has recently been in crisis situation due to the growth of Chinese manufacturing and the recent slowdown in domestic manufacturing.

4.2 Priorities of policies

In this study, the policy of “Establishment of a system for fostering and securing AI talent” was selected as the AI policy item of the highest priority. The experts showed such a consensus that they selected this as an item that should be given top priority in terms of both ‘importance of policy’ and ‘urgency of policy.’ In major foreign countries as well, the cultivation of AI talent is treated as the most important national policy task. These countries are making efforts to make a massive investment in the cultivation of AI talent and to create research networks at the level of national strategies. It is necessary to note that the US and China are promoting extensive investment in the sector of human resources in order to maintain their global leadership in AI, while in Canada and the UK also, their governments are taking the lead in mustering their AI research competence. South Korea, however, definitely lacks talent in this area, and so it is feared that its AI competitiveness will possibly lag far behind in the future. For this reason, fostering and securing AI talent is a critical national policy item, and is also one of core policy items emphasized in the 'National Strategy for Artificial Intelligence' (2019). In this national strategy, South Korea set the goal of fostering the world's best AI talent and realizing AI education for all the citizens, and established detailed tasks. Efforts should be concentrated on these policy tasks, for their successful implementation and outcome will be a very important foundation for South Korea to become an AI leader.

The policy of “Supporting the construction and sharing of quality dataset” was identified as the second priority. The Korean government has so actively engaged in the opening and use of public data that it placed first overall for the third consecutive years in 2015, 2017, and 2019 in OECD’s open government data policies evaluation [19]. Problems, however, have been raised that data sharing in the private sector is retarded and closed, and that data in the public sector are not equipped with satisfactory quality. For the construction of an active AI ecosystem and the promotion of the use of AI, it is necessary to review the role of the government in the data circulation structure (collection - processing - distribution - application). Substantial efforts are important for successfully promoting various policy tasks to support the construction and sharing of quality dataset, which the South Korean government unveiled in “Data Industry Activation Strategy” (2018), “Data AI Economy Activation Strategy” (2019), and “National Strategy for Artificial Intelligence” (2019).

An AI policy item of the third priority is “Expanding investment in next-generation AI research and fundamental research.” It is South Korea's important national task to secure technological competitiveness equal to AI leader countries. While the United States is the world leader in AI ecosystem, China is catching up fast based on large amounts of data and huge capital investments. Given the rapid development of AI technology, it is feared that a gap between South Korea and these leader countries may be widened further. For South Korea to leap into an AI powerhouse, national investment in AI R&D is important and urgent. Against this background, it is very fortunate that in the “National Strategy for Artificial Intelligence” (2019), the Korean government devised strategies for expanding support to AI fundamental research and software and reorganizing the AI R&D system overall, so that the basic foundation of AI can be strengthened.

On the other hand, it was found that an item of the lowest priority was establishing AI ethics, followed by improving the legal system in each area. Nonetheless, that's not to say that these police items are not important or urgent at all. Considering that the development and use of AI technology may cause unintended side effects and adverse effects, guidelines and ethics for reliable AI use are necessary, and the improvement of legal systems to facilitate AI adoption is also necessary and important. It may be understood that these items were pushed back in their priorities because the establishment of these policies involves lots of administrative expenses and long time for social discussion and consensus and the procedure for improvement in legal system. As competition among countries to secure AI competitiveness becomes fierce, in order to secure AI national competitiveness quickly, it is necessary to promote first the policies of technological and economic aspects, such as AI talent cultivation, data construction, and R&D, which are relatively less exposed to social and political controversy and are key to the securing of competitiveness.

5. Conclusion

Recently, the interest of companies in AI adoption has increased according as the development of AI-related technologies has been accelerated and untact economy has been established as a new trend due to COVID-19. For AI is emerging as a means to innovation as well as an essential factor in sustainable business operation, amidst intensifying competition and crisis situation. It is obvious that AI is now being recognized as an essential factor, not an option, in business operation. It is expected that it will be possible to innovate all ranges of industries by means of AI. But considering that at present it is the initial stage of AI grafting onto industries, it is necessary to support AI adoption in some industry sectors first to increase best use cases, and then to seek AI expansion to other industries. It is desirable to prioritize the introduction of AI in the medical/healthcare, public and safety, manufacturing sectors, considering the crisis situation of COVID-19 pandemic. In addition, to promote the adoption of AI by enterprises, talent cultivation, data, and R&D investments are important and urgent most of all, and thus the government should pay more attention to these policies continuously and concentrate on the implementation of these policies in priority.

South Korea is in a situation that she needs to secure a new growth momentum according as her global competitiveness even in the areas of her traditional strength, such as manufacturing, is lowered. In this context, AI is emerging as one of key factors in market competition and industrial production as the Fourth Industrial Revolution spreads. Therefore, it is necessary to create visible outcomes by using AI properly in the existing industries. This study revealed in which industry sectors among the existing various industries AI adoption is easy, necessary, and urgent. These results are a contribution of this study in that they can be beneficial when discussing AI policies to overcome the Covid-19 crisis. In addition, this study provides policy implications in that it suggests the ranking of policy items that should be prioritized by the government to expand the use of AI across industries. Until now, the government has promoted the active use of AI across industries in order to revitalize the economy through intelligent industries and to secure future growth engine. However, it has been difficult to determine the priorities of policy implementation, because the policy tasks have been enumerated fragmentarily and it is difficult to grasp their importance. This study is valuable in that it comprehensively identified, from experts' viewpoint, policy tasks for expanding the use of AI in South Korea, and suggested future policies to which attention should be given in priority. Excellent policy tasks for AI adoption have been suggested meantime in several national strategies; however, quantitative studies that compare and suggest priorities among the tasks are few. That is, there has been no study that empirically analyzes which policy should be implemented in priority. Thus, the policy contributions of this study can be found in that these deficiencies have been supplemented.

Despite these several contributions, this study has the following limitations. First, it concerns the rigor of categorization of policy items. The policy items obtained from Delphi were classified according to the STEEP analysis framework based on the consensus of 3 experts. Nevetheless, different opinions may exist on the interpretation of policy items, which hinder strict classification. In addition, although opinions of industry, academia, and research fields may differ, this study did not distinguish them and reveal any differences, which has a limitation. Therefore, it is suggested that future researchers should develop a research design for looking into the difference among them. Besides the analysis of differences in opinions among experts from these fields, it is also worth consideration to look into opinions of people involved in policy-making process. Meaningful results may be derived from confirming whether there is any difference in viewpoints between those who establish or implement policies and the industry-academia-research experts.