1. Introduction
The state involvement in society is of course a barometer of the government’s role in implementing their tasks and functions effectively and efficiently to achieve optimal service performance. In achieving the optimal service performance, the Indonesian government has issued various policies (Ferguson, 2019). One of them is the Financial Management Pattern of Public Service Agency (PPK-BLU) policy. In carrying out its duties and functions, the PPK-BLU policy has been implemented by the Rural Telecommunications and Information Center (BTIP) within the Ministry of Communication and Information Technology (Gasiea et al., 2010). The formation of the BTIP is a realization of the mandate of the International Telecommunication Union (ITU) (Amal, 2020).
The BTIP is constructed as an organization that has flexibility in all aspects such as organization, budget, and staffing (Allen et al., 2015). As an organization, BTIP is one of the technical implementation units as a service provider that has been established by the Minister of Finance as an organizational unit. A policy that is supposed to be in line with increasingly rapid technological advances is striving to provide infrastructure and public services through the use of Information and Communication Technology (Sarah, 2018). Therefore, BTIP is to carry out the task of managing the ICT Universal Service Obligation (USO) concept to seek equal distribution of ICT networks throughout Indonesia to be able to improve efficiency by facilitating interaction and communication between governments and communication through the provision of IT (Imami, 2019).
However, the implementation of PPK-BLU was not followed by a comprehensive change in aspects of organization, human resources, and finance (Pahala et al., 2016). These conditions cause the BTIP to not experience significant changes in its performance in providing universal services in the field of telecommunications and informatics to the public (Mir & Sutiyono, 2013).
Based on this background, this study aims to provide a comprehensive evaluation of the BTIP after the implementation of the PPK-BLU policy in telecommunications services and information to the public. This research integrates the McKinsey 7S method, Analytical Hierarchy Process, and Likert scale. The integration of these methods can provide quantitative values in providing analysis of evaluation results.
There are previous studies as references evaluating Regional Knowledge Innovation System (Chiu & Lin, 2019). There are research on assessing the impact of performance management on employees and private organizations in Tanzania (Samwel, 2018), research on factors influencing variables in evaluating organizational performance (Ali, 2018), research on identifying, evaluating, and measuring the performance of small and medium-sized businesses in Portugal (Felizardo et al., 2017). There are also research on performance evaluation and organizational unit tools to determine efficiency levels (Tavakoli et al., 2017), research on evaluating and analyzing the mediating role of employees’ organizational commitment to organizational performance (Nikpour, 2017), research on evaluation of economic level and potential (Nurlanova et al., 2017). There are research as well on gap analysis and development of non-financial organizational performance evaluation instruments (Crucke & Decramer, 2016), research on the analysis of the relationship between performance and three input factors (innovation, system, quality) in evaluating organizational performance (Mafini, 2015), research on evaluating industrial performance with a balanced scorecard framework in a long and short term perspective (Ondoro, 2015), and research on the analysis of the effect of performance assessment and the influence of bias and errors in evaluations at Sabzevar (Javidmehr & Ebrahimpour, 2015).
Research related to McKinsey 7S and AHP includes studies on identifying and analyzing priority factors of organizational readiness to implement ERP based on organizational agility using McKinsey 7S (9S) and AHP (Shiri et al., 2015). There is research on the business organizations’ strategic assessment within the context of the McKinsey 7S model (Gökdeniz et al., 2017) as well as research on evaluating the maturity of technology companies under the McKinsey 7S’scriteria and weighting using AHP method (Demir & Kocaoglu, 2019). Research has been conducted on evaluation of Non-Government Organizations, boards of directors, donors, and government using the McKinsey 7S approach (Simon & Ronoh, 2017). There are also research on the investigation and analysis of the Qeshm free zone organizational structure based on the McKinsey 7S (Ravanfar, 2015). Furthermore, there is research on the McKinsey 7S Model as a method for analyzing organizational effectiveness through Shared Values, Strategies, Structure, Systems, Style, Staff, and Skills (Alam, 2017).
This present research is limited to evaluating the implementation of PPK-BLU policies in BTIP. The work is expected to contribute to the development of the implementation of PPK-BLU policies, both in BTIP and in other agencies. This research is also expected to be used as material for determining the next strategy in improving BTIP performance.
This research consists of several sections. Section 2 discusses the PPK-BLU, BTIP, the use of the McKinsey 7S method, research stages, and research instruments. Section 3 presents the results of the assessment and discussion of the evaluation of the implementation of the PPK-BLU policy at BTIP. Section 4 provides the conclusions of the study.
2. Material/Methods
2.1. BTIP
The Rural Telecommunications and Information Center (BTIP) is a special agency that has the responsibility to manage the provision of universal telecommunications service (Adediran et al. 2016). Telecommunications here mean any transmitting, sending, and/or receiving of any information in the form of signs, signals, writings, pictures, sounds, and voice through a wire, optical, radio or other electromagnetic systems (Imami, 2019).
2.2. Financial Management Pattern of Public Service Agency (PPK-BLU)
The Government Regulation Number 23 of 2005 concerning Financial Management of BLU as a Financial Management Pattern of Public Service Agency (PPK-BLU), states as follows (Pahala et al., 2016): Financial Management Pattern of Public Service Agency, hereinafter referred to as PPK-BLU, is a financial management pattern that provides discretion in the form of flexibility to implement sound business practices to improve services to the public to advance public welfare and improve the lives of the nation, as regulated in this Government Regulation. As exceptions and provisions for managing public finances in general. (Juliani, 2018).
2.3. Evaluation
Evaluation means measurement and improvement in the activities carried out, such as comparing the results of activities made (Behmane et al., 2018). The goal is that plans made to achieve the objectives set can be implemented. Evaluation can also be interpreted as the activity of gathering information about humans, systems, or tools performance, which are then used to determine the best alternative in making decisions (Kusek & Rist, 2004). The results are intended for re-planning, and also functioned as the final administration and management by combining and collecting data with standard objectives (Stufflebeam, 2001).
Evaluation is needed in every activity that has been carried out (Dwi et al., 2018). This is to ensure that we are running by the objectives to be achieved. Evaluation is not only limited to technical activities, but it also covers nontechnical activities (Azhar, 2018). It is used in various fields, especially in the scope of companies, projects or other work related to the system (Gebrehiwet & Luo, 2019).
2.4. Policy
Policy is a series of concepts and principles that make up a line of implementation of a job, leadership or way to act (Itika, 2011). The policy also refers to the process of making important decisions in an organization (Ejimabo, 2015). Policies can also play a role as a political and financial mechanism or in any form (Hudson et al., 2018). Decision-making in a policy must always be carefully thought out (Panpatte & Takale, 2019). Therefore, by its definition, a policy is a set of decisions taken by politicians to choose goals and ways to achieve them (Viennet & Pont, 2017).
2.5. McKinsey 7S
The McKinsey 7S Framework is a tool used to analyze the internal aspects of a company’s organization using seven main elements, namely, Strategy, Structure, Systems, Share values, Style, Staff, and Skills (Jayakrishnana et al., 2018). The McKinsey 7S model was introduced by Tom Peters and Robert Waterman who worked as Consultants at McKinsey & Company in the 1980s (Gökdeniz et al., 2017). According to them, the alignment of the seven elements in the organization is a key factor for a company’s success (Liffler & Tschiesner, 2013). The McKinsey 7S model can be applied to a variety of situations and is an excellent tool in designing the shape of an organization, improving organizational performance, testing changes in organizational factors, harmonizing departments and processes during acquisitions and mergers and determining the best strategy for the organization (Alam, 2017).
McKinsey 7S consists of seven elements, which are divided into two big groups; the first is the “Hard” group which includes Strategy, Structure, and System. These three elements are categorized as “Hard” group because they are easier to identify and can be directly influenced by management (Demir & Kocaoglu, 2019). The second group is the “Soft” group, which includes Shared Values, Skills, Staff, and Style. The four elements are categorized as “Soft” because they are intangible and relatively difficult to describe, and are very influenced by the culture of an organization (Shiri et al., 2015).
2.6. Analytical Hierarchy Process (AHP)
AHP is a decision-making support model developed by Thomas L. Saaty. This decision-making support model will break down complex multi-factor or multi-criteria problems into a hierarchy (Saaty, 1980). Hierarchy is defined as a representation of a complex problem in a multi-level structure where the first level is the goal, followed by the level of factors, criteria, sub-criteria, and so on down to the last level of alternatives (Nguyen et al., 2020). Through the use of hierarchy, a complex problem can be broken down into groups, which are then arranged in a hierarchical form so that the problem will appear more structured and more systematic (Sharma et al., 2013).
The decision-making process is choosing the best alternative (Saaty, 1990). Such as structuring the problem, determining alternatives, determining the likelihood value for the aleatory variable, setting the value, requiring time preference, and specifying of the risk (Wang et al., 2014). No matter how wide the alternative can be determined or how detailed exploration of the possible value will be, the limitation cover is the basis of comparison in the form of a single criterion (Mutmainah, et al., 2017).
There are three main principles in problem-solving in AHP according to Saaty, namely, Decomposition, Comparative Judgment, and Logical Consistency. Generally, the SFCM method algorithm is formulated as follows (Suryaningkusuma et al., 2018):
(1) Decomposition of the problem.
(2) Assessment/weighting to compare elements.
(3) Preparing matrix and consistency test.
(4) Setting priority in each hierarchy.
(5) Synthesizing priorities.
(6) Making the decision.
The pairwise comparison assessment process in AHP refers to the assessment score developed by Saaty. It is described as follows (Saaty & Vasgas, 2006):
The steps and process of AHP are presented as follows (Mutmainah et al., 2017):
(1) Defining the problem and setting goals. Carrying out an alternative development, if, at this stage, AHP is used to choose alternatives or develop alternative priorities.
(2) Arranging problems into a hierarchy so that complex problems can be viewed in terms of its detail and can be measured.
(3) Arranging priority for each problematic element in the hierarchy. This process results in weight or contributing elements to the achievement of objectives so that the element with the highest weight has priority handling. Prior results from a pairwise comparison matrix between all elements at the same hierarchical level.
(4) Conducting consistency testing of comparisons between elements found at each level of the hierarchy.
2.7. Research Subjects and Research Objects
The research subject is the main source of research data, which provides data in the form of the variables or problems under study. The research subject will be the conclusion of the research results. Therefore, the subject of this study must be following the issues raised. The research object is a problem, issue or problem that is discussed, researched and investigated. The object is the problem under study, from people, objects or activities that have certain variations determined by researchers to be studied and then drawn conclusions.
In this study, the research subjects are several experts who are related to the BTIP and PPK BLU. The research subjects included personnel at the Ministry of Communication and Information, personnel at the Ministry of Finance, personnel at the Ministry of Administration and Bureaucratic Reform, and scholars. The object of research is the BTIP’s policy in carrying out PPK BLU. This research was done at Rural Telecommunications and Information Center of Ministry of Communication and Information Technology.
2.8. Research Objectives and Stages
The research objective is to assess the performance level of the BTIP after implementing the PPK-BLU policy. In determining the assessment, several stages were done as follows; the first stage, identifying criteria related to KDPBLU in the McKinsey 7S framework. The second stage is to give weight to these criteria. Next, providing a Likert scale assessment of the criteria that have been weighted. Multiplying the results of weighting by using the Likert scale value to determine the evaluation value of the BTIP after the PPK BLU was carried out and to explain the BTIP’s performance results.
2.9. Flowchart
3. Results and Discussion
To find out the results of organizational performance, weighting and score assessment were done. After the questionnaire results were collected, the next step was to process the data. The results of the analysis were explained as follows:
3.1. Identification and Criteria Weighting
Weighting and identification criteria are based on the development of the Pairwise Comparison method. Parameters calculated in weighting are according to Table 6. Based on data analysis, it can be seen that the results of the weighting questionnaire are following each table of criteria and sub-criteria.
Table 1: AHP’s Preference Value
Table 2: List of Experts in Study
Table 3: List of Numbers of Respondents in Study
Table 4: Performance Evaluation Research Analysis Score
Source: (Maksum, Luddin, & Idris, 2019).
Table 5: McKinsey 7S Criteria for BTIP Performance Evaluation
Source: (Demir & Kocaoglu, 2019); (Gökdeniz, Kartal, & Kömürcü, 2017); (Shiri, Anvari, & Soltani, 2015)
Table 6: BTIP Performance Assessment Results based on McKinsey 7S
3.2 Performance Evaluation of BTIP Performance based on McKinsey 7S
Based on the overall evaluation results (Figure 3), in carrying out the PPK BLU, BTIP has a performance value of 81.195%, which is included within Good category. The McKinsey 7S elements used as a whole have the following values: one aspect is categorized as Very Good (the strategy aspect with a value of 91.025%); two aspects are categorized as Good (structural aspects with a value of 86.857%, skill aspects with a value of 81.432%); three aspects are categorized as Adequate (style aspect with a value of 76.441%, staff aspect with a value of 78.605%, and share value aspect with a value of 76.331%); one aspect is categorized as Bad, that is the system aspect with a value of 67.503%.
Figure 3: Histogram of BTIP Performance Assessment Results
Considering the results of this research, to actualize and realize the activities carried out by BTIP, some improvements are needed, especially related to several aspects with Bad and Adequate categories towards the Good category. In the system aspect, it is necessary to carry out some performance improvements by improving data processing and increasing the use of technology. Besides, the SOP of the BTIP needs to be designed in detail to regulate every step of the work carried out, including measuring the level of success. On the implementation scope, the office network development is required to submit reports relating to possible risks. The system that regulates the steps of the organization’s work in detail must be carried out and obeyed by every operational unit so that things or activities out of the existing system will be seen as a violation which becomes the object of the audit.
Furthermore, in three aspects with sufficient categories, namely, the style aspect, the staff aspect, and the share value aspect, it is necessary to increase performance towards the Good category. In the style aspect, leadership improvement is needed to support BTIP’s organizational policies to motivate, interact, and controls subordinates who become their regular subordinates through intensive communication. In the staff aspect, it is necessary to increase the ability of the project team to support the needs of BTIP through structured and planned programs to improve employee competencies and capabilities when working in a team. To get competent employees, BTIP needs to establish a recruitment pattern by looking at the competencies and abilities of prospective employees who will occupy certain positions. In the aspect of share value, BTIP needs to have value guidelines that must be understood and implemented by each organization. The role of employee socialization and coaching must be continuously enhanced to support the work culture system to increase the value of employee confidence.
4. Conclusion
Based on the overall results, in carrying out the PPK BLU, BTIP has a performance value of 81.195%, which is included within Good category. The 7S McKinsey elements used as a whole have the following values: one aspect is categorized as Very Good (the strategy aspect with a value of 91.025%; two aspects are categorized as Good (structural aspects with a value of 86.857% and skill aspects with a value of 81.432%); three aspects are categorized as Adequate (style aspect with a value of 76.441%, staff aspect with a value of 78.605%, and share value aspect with a value of 76.331%); one aspect is categorized as Bad, that is the system aspect with a value of 67.503%.
5. Future Work
Based on the results of the study, it is necessary to implement a strategy to actualize and realize activities at every action taken by the BTIP by increasing the use of technology. For future work, it is necessary to make a modeling system for developing policy strategies to improve BTIP’s performance in implementing PPK BLU policy. In addition to assessing BTIP’s performance, in future work, studies need to be carried out to assess the ability of employees to identify the value of competence and ability.
References
- Adediran, Y., Opadiji, J. F., Faruk, N., & Bello, O. W. (2016). On Issues and Challenges of Rural Telecommunications Access in Nigeria. African Journal of Research in Mathematics, Science and Technology Education, 3(2), 16-26.
- Alam, P. A. (2017). Measuring Organizational Effectiveness through Performance Management System and Mckinsey's 7S Model. Asian Journal of Management, 8(4), 1280-1286. https://doi.org/10.5958/2321-5763.2017.00194.9
- Ali, B. O. (2018). Factors Influencing the Effectiveness of Internal Audit on Organizational Performance. International Journal of Advanced Engineering, Management and Science, 4(4), 219-226.
- Allen, R., Hurcan, Y., Murphy, P., Queyranne, M., & Ylaoutinen, S. (2015). The Evolving Functions and Organization of Finance Ministries. IMF Working Paper No. 15/232. https://www.imf.org/en/Publications/WP/Issues/2016/12/31/The-Evolving-Functions-and-Organization-of-Finance-Ministries-43382
- Amal, S. (2020). Analog Switch-Off Process Towards the Digitalization of Indonesian Broadcasting. Riau, Indonesia: Universitas Negeri Sultan Syarif Kasim.
- Azhar, F. (2018). Authentic Teaching and Assessment as the Solution to Educational Evaluation in Reference to ASEAN Economic Community in Indonesia. International Journal of Educational Best Practices, 2(1), 26-38. https://doi.org/10.31258/ijebp.v2n1.p26-38
- Behmane, D., Villerusa, A., Berkis, U., & Rutitis, D. (2018). Evaluation of External Factors for the Development of the Latvian Health. Humanities and Social Sciences, 26(2), 46-63.
- Chiu, S.-H., & Lin, T.-Y. (2019). Evaluation of Regional Knowledge Innovation System in China: An Economic Framework Based on Dynamic Slacks-based Approach. Journal of Asian Finance, Economics and Business, 6(3), 141-149. https://doi.org/10.13106/jafeb.2019.vol6.no3.141
- Crucke, S., & Decramer, A. (2016). The Development of a Measurement Instrument for the Organizational Performance of Social Enterprises. Sustainability, 161(8), 1-30.
- Demir, E., & Kocaoglu, B. (2019). Maturity Assessment in teh Technology Business within the McKinsey 7S Framework. Research Journal of Business and Management, 6(3), 158-166.
- Dwi, Y. R., Haryono, & Florentinus, T. S. (2018). The Evaluation of the CIPP Model in The Implementation of Character Education at Junior High School. Innovative Journal of Curriculum and Educational Technology, 7(2), 65-77.
- Ejimabo, N. O. (2015). The Influence of Decision Making in Organizational Leadership and Management Activities. Journal of Entrepreneurship & Organization Management, 4(2), 1-13.
- Felizardo, A., Felix, E. G., & Thomaz, J. P. (2017). Organizational Performance Measurement and Evaluation Systems in Smes: The Case of the Transforming Industry in Portugal. Evora, Portugal: Universidade de Evora.
- Ferguson, B. (2019). Competing for influence: The role of the public service in better government in Australia. Acton, Australia: ANU Press.
- Gasiea, Y., Emsley, M., & Mikhailov, L. (2010). Rural Telecommunications Infrastructure Selection Using the Analytic Network Process. Journal of Telecommunications and Information Technology, 2(2), 28-42.
- Gebrehiwet, T., & Luo, H. (2019). Risk Level Evaluation on Construction Project Lifecycle Using Fuzzy Comprehensive Evaluation and TOPSIS. Symmetry, 11(12), 1-15.
- Gokdeniz, I., Kartal, C., & Komurcu, K. (2017). Strategic Assessment based on 7S McKinsey Model for a Business by Using Analytic Network Process (ANP). International Journal of Academic Research in Business and Social Sciences, 7(6), 342-353.
- Hudson, B., Hunter, D., & Peckham, S. (2018). Policy Failure and the Policy-Implementation Gap: Can Policy Support Programs Help? Journal Policy Design and Practice, 2(1), 1-14.
- Imami, A. (2019). Pengelolahan Internet dalam Mewujudkan Desa Broadband terpadu di Desa Wonosari Kabupaten Bengkalis. JOM FISIP, 6(1), 1-13.
- Itika, J. S. (2011). Fundamentals of Human Resource Management. Morogoro, Tanzania: Mzumbe University.
- Javidmehr, M., & Ebrahimpour, M. (2015). Performance appraisal bias and errors: The influences and consequences. International Journal of Organizational Leadership, 4, 286-302. https://doi.org/10.33844/ijol.2015.60464
- Jayakrishnana, M., Mohamadb, A. K., Azmic, F. R., & Abdullahd, A. (2018). Implementation of Business Intelligence Framework for Malaysian Halal Food Manufacturing Industry towards Initiate Strategic Financial Performance Management. Management Science Letters, 8, 1059-1076.
- Juliani, H. (2018). Eksistensi Badan Layanan Umum Sebagai Penyelenggara Pelayanan Publik. Adminitrative Law & Governance Journal, 1(1), 47-61.
- Kusek, J. Z., & Rist, R. C. (2004). Ten Step to A Result-Based Monitoring and Evaluation System. Washington, DC: The World Bank.
- Liffler, M., & Tschiesner, A. (2013). The Internet of Things and the Future of Manufacturing. New York, NY: McKinsey & Company.
- Mafini, C. (2015). Predicting Organisational Performance Through Innovation, Quality And Inter-Organisational Systems: A Public Sector Perspective. The Journal of Applied Business Research, 31(3), 939-952. https://doi.org/10.19030/jabr.v31i3.9227
- Maksum, A., Luddin, M. R., & Idris, F. (2019). Evaluation Program for the Career Development of Indonesian Navy Civilian Personnel using the Cipp Evaluation Model and Analytical Hierarchy Process (AHP). International Journal of Recent Technology and Engineering, 8(4), 1309-1316.
- Mir, M., & Sutiyono, W. (2013). Public Sector Financial Management Reform: A Case Study of Local Government Agencies in Indonesia. Australasian Accounting, Business and Finance Journal, 7(4), 97-117. https://doi.org/10.14453/aabfj.v7i4.7
- Mutmainah, Marfuah, U., & Panudju, A. T. (2017). Employee Performance Appraisal Model Using Human Resources Scorecard And Analytical Hierarchy Process (AHP). Journal of Scientific & Technology Research, 6(11), 81-84.
- Nguyen, P.-H., Tsai, J.-F., Kumar G, V. A., & Hu, Y.-C. (2020). Stock Investment of Agriculture Companies in the Vietnam Stock Exchange Market: An AHP Integrated with GRA-TOPSISMOORA Approaches. Journal of Asian Finance, Economics and Business, 7(7), 113-121. https://doi.org/10.13106/jafeb.2020.vol7.no7.113
- Nikpour, A. (2017). The Impact of Organizational Culture on Organizational Performance: The Mediating Role of Employee's Organizational Commitment. International Journal of Organizational Leadership, 6, 65-72. https://doi.org/10.33844/ijol.2017.60432
- Nurlanova, N. K., Kireyeva, A. A., & Ruzanov, R. M. (2017). Evaluation of Economic Potential and Level of Concentration of the Regions of Kazakhstan. Journal of Asian Finance, Economics and Business, 4(2), 37-44. http://dx.doi.org/10.13106/jafeb.2017.vol4.no2.37
- Ondoro, C. O. (2015). "Measuring Organization Performance" From Balanced Scorecard to Balanced ESG Framework. International Journal of Economics, Commerce and Management, 3(11), 715-725.
- Pahala, I., Sudarma, M., Sutrisno, & Rosidi. (2016). Financial Accountability on Public Service Agency (BLU): Phenomenological study at the State University of Jakarta. Review of Integrative Business & Economic, 5(2), 11-27.
- Panpatte, S., & Takale, V. (2019). To Study the Decision Making Process in an Organization for its Effectiveness. The International Journal of Business Management and Technology, 3(1), 73-78.
- Ravanfar, M. M. (2015). Analyzing Organizational Structure Based on 7s Model of Mckinsey. Global Journal of Management and Business Research, 15(10-A). https://journalofbusiness.org/index.php/GJMBR/article/view/1792
- Saaty, T. L. (1980). The Analytical Hierarchy Process. New York, NY: McGraw Hill.
- Saaty, T. L. (1990). How to make a Decision : The Analytic Hierarchy Process. European Journal of Operation Research, 48(1), 9-26. https://doi.org/10.1016/0377-2217(90)90057-I
- Saaty, T. L., & Vasgas, L. G. (2006). Decision Making with the Analytical Hierarchy Process (AHP). New York, NY: Springer Science Bussiness Media.
- Samwel, J. O. (2018). An Assessment of the Impact of Performance Management on Employee and Organization Performance - Evidence from Selected Private Organizations in Tanzania. International Journal of Human Resource Studies, 8(2), 199-217. https://doi.org/10.5296/ijhrs.v8i3.13415
- Sharma, P., Thakar, G., & Gupta, R. (2013). Evaluation of Assembly Line Balancing Methods Using An Analytical Hierarchy Process (AHP) and Technique for Order Preferences by Similarity to Ideal Solution (TOPSIS) Based Approach. International Journal for Quality Research, 7(4), 523-544.
- Shiri, S., Anvari, A., & Soltani, H. (2015). Identifying and Prioritizing of Readiness Factors forImplementing ERP Based on Agility (Extension of McKinsey 7S model). European Online Journal of Natural and Social Sciences, 4(1), 56-74.
- Simon, K. S., & Ronoh, D. K. (2017). Strategy Implementation and Performance of Non Governmental Organizations in Kericho County, Kenya. Management and Organizational Studies, 4(3), 34-42. https://doi.org/10.5430/mos.v4n3p34
- Stufflebeam, D. L. (2001). Evaluation Model. Hoboken, NJ: John Wiley & Sons.
- Suryaningkusuma, R. P., Subyantoro, A., & Sabihaini. (2018). Combining Analytical Hierarchy Process and Simple Multi-Attribute Rating Technique for designing a Sustainable Balanced Scorecard as Strategic Performance Measurement System. International Journal of Computer Science and Network, 7(4), 279-288.
- Tavakoli, M. M., Molavi, B., & Shirouyehzad, H. (2017). Organizational Performance Evaluation Considering Human Capital Management Approach by Fuzzy-DEA: A Case Study. International Journal of Research in Industrial Engineering, 6(1), 1-16.
- Viennet, R., & Pont, B. (2017). Eucation Policy Implementation: A Literature Review and Proposed Framework. Paris, France: Organisation for Economic Co-operation and Development.
- Wang, J., Li, M., Liu, Y., Zhang, H., Zou, W., & Cheng, L. (2014). Safety assessment of shipping routes in the South China Sea based on the fuzzy analytic hierarchy process. Safety Science, 62, 46-57. https://doi.org/10.1016/j.ssci.2013.08.002