IE interfaces (산업공학)

Korean Institute of Industrial Engineers (대한산업공학회)
권호 표지
DOI QR코드

DOI QR Code

Empirical Analysis of DEA models Validity for R&D Project Performance Evaluation : Focusing on Rank Correlation with Normalization Index

R&D 프로젝트 성과평가를 위한 DEA모형의 타당성 실증분석 : 정규화지표와의 순위상관을 중심으로

  • Park, Sung-Min (Department of Business Administration, Baekseok University)
  • Received : 2011.07.09
  • Accepted : 2011.10.25
  • Published : 2011.12.01
Download PDF
⟨ Previous Next ⟩

Abstract

This study analyzes a relationship between Data Envelopment Analysis(DEA) efficiency scores and a normalization index in order to examine the validity of DEA models. A normalization index concerned in this study is 'sales per R&D project fund' which is regarded as a crucial R&D project performance evaluation index in practice. For this correlation analysis, three distinct DEA models are selected such as DEA basic model, DEA/AR-I revised model(i.e. DEA basic model with Acceptance Region Type I constraints) and Super-Efficiency(SE) model. Especially, SE model is adopted where efficient R&D projects(i.e. Decision Making Units, DMU's) with DEA efficiency score of unity from DEA basic model can be further differentiated in ranks. Considering the non-normality and outliers, two rank correlation coefficients such as Spearman's ${\rho}_s$ and Kendall's ${\tau}_B$ are investigated in addition to Pearson's ${\gamma}$. With an up-to-date empirical massive dataset of n = 482 R&D projects associated with R&D Loan Program of Korea Information Communication Promotion Fund in the year of 2011, statistically significant (+) correlations are verified between the normalization index and every model's DEA efficiency scores with all three correlation coefficients. Especially, the congruence verified in this empirical analysis can be a useful reference for enhancing the practitioner's acceptability onto DEA efficiency scores as a real-world R&D project performance evaluation index.

Keywords

References

  1. Andersen, P. and Petersen, N. (1993), A Procedure for Ranking Efficient Units in Data Envelopment Analysis, Management Science, 39(10), 1261-1264. https://doi.org/10.1287/mnsc.39.10.1261
  2. Bae, Y., Kim, J., and Kim, S. (2006), Assessment of Ammunition Companies Using the IDEA Model, IE Interfaces, 19(4), 291-299.
  3. Banker, R. D., Charnes, A., and Cooper, W. W. (1984), Some Models for Estimating Technical and Scale Inefficiencies in Data Envelopment Analysis, Management Science, 30(9), 1078-1092. https://doi.org/10.1287/mnsc.30.9.1078
  4. Banker, R. D., Conrad, R. F., and Strauss, R. P. (1986), A Comparative Application of Data Envelopment Analysis and Translog Methods : An Illustrative Study of Hospital Production, Management Science, 32(1), 30-44. https://doi.org/10.1287/mnsc.32.1.30
  5. Bessent, A., Bessent, W., Kennington, J., and Reagan, B. (1982), An Application of Mathematical Programming to Assess Productivity in the Houston Independent School District, Management Science, 28(12), 1355-1367. https://doi.org/10.1287/mnsc.28.12.1355
  6. Bitman, W. R. and Sharif, N. (2008), A Conceptual Framework for Ranking R&D Projects, IEEE Transactions on Engineering Management, 55(2), 267-278.
  7. Callen, J. L. (1991), Data Envelopment Analysis : Partial Survey and Applications for Management Accounting, Journal of Management Accounting Research, 3, 35-56.
  8. Charnes, A. and Cooper, W. W. (1980), Auditing and Accounting for Program Efficiency and Management Efficiency in Not-for-Profit Entities, Accounting, Organizations and Society, 5(1), 87-107. https://doi.org/10.1016/0361-3682(80)90025-2
  9. Charnes, A., Cooper, W. W., and Rhodes, E. (1978), Measuring the Efficiency of Decision Making Units, European Journal of Operational Research, 2(6), 429- 444. https://doi.org/10.1016/0377-2217(78)90138-8
  10. Charnes, A., Cooper, W. W., and Rhodes, E. (1981), Evaluating Program and Managerial Efficiency : An Application of Data Envelopment Analysis to Program Follow Through, Management Science, 27(6), 668-697. https://doi.org/10.1287/mnsc.27.6.668
  11. Cooper, W. W., Seiford, L. M., and Tone, K. (2007), Data Envelopment Analysis : A Comprehensive Text With Models, Applications, References and DEA-Solver Software, 2nd edn, Springer, New York.
  12. Cooper, W. W., Seiford, L. M., and Zhu, J. (2004), Handbook on Data Envelopment Analysis, Springer, Boston.
  13. Farris, J. A., Groesbeck, R. L., Aken, E. M. V., and Letens, G. (2006), Evaluating the Relative Performance of Engineering Design Projects : A Case Study Using Data Envelopment Analysis, IEEE Transactions on Engineering Management, 53(3), 471-482. https://doi.org/10.1109/TEM.2006.878100
  14. Frontline Systems, Inc. (2007), Premium Solver Platform Version 7.1 for Microsoft Excel, http://www.solver.com/Default.htm.
  15. Gregoriou, G. N. and Zhu, J. (2005), Evaluating Hedge Fund and CTA Performance, John Wiley and Sons, New Jersey.
  16. Henriksen, A. D. and Traynor, A. J. (1999), A Practical R&D Project-Selection Scoring Tool, IEEE Transactions on Engineering Management, 46(2), 158-170.
  17. Hsu, F. M. and Hsueh, C. C. (2009), Measuring Relative Efficiency of Government-Sponsored R&D Projects : A Three-Stage Approach, Evaluation and Program Planning, 32(2), 178-186. https://doi.org/10.1016/j.evalprogplan.2008.10.005
  18. Hwang, S. (2006), STEPI Policy Study 2006-12, Methodology of Economic Assessment for Classified R&D Programs, STEPI, Korea.
  19. Hwang, Y. and Hwang, S. (2005), STEPI Policy Study 2004-20, An Assessment of the Performance Evaluation System for Government R&D, STEPI, Korea.
  20. Kellogg, W. K. Foundation (2004), W. K. Kellogg Foundation Logic Development Guide, MI Battle Creek.
  21. Kim, J. and Kim, S. (2007), A Real Estate Price Appraisal Model Based on the Data Envelopment Analysis-Assurance Region(DEA-AR), Housing Studies Review, 15(1), 29-61.
  22. Ko, Y., Yoon, H., and Lee, J. (2004), KDI Research Report 2004-02 : Performance Management of Public Sector, KDI, Korea.
  23. Korea Institute of Science and Technology Information(KISTI) (2008), Law- Enforcement Ordinance-Enforcement Regulations, http://www.ntis.go.kr/ThMain.do, KISTI-NTIS, Korea.
  24. Lee, H., Park, Y., and Choi, H. (2009), Comparative Evaluation of Performance of National R&D Programs With Heterogeneous Objectives : A DEA Approach, European Journal of Operational Research, 196(3), 847-855. https://doi.org/10.1016/j.ejor.2008.06.016
  25. Lee, D. and Yang, W. (2004), Performance Evaluations of Professional Baseball Players Using DEA/OERA, IE Interfaces, 17(4), 440-449.
  26. McLaughlin, J. A. and Jordan, G. B. (1999), Logic Models : A Tool for Telling Your Programʼs Performance Story, Evaluation and Program Planning, 22(1), 65-72. https://doi.org/10.1016/S0149-7189(98)00042-1
  27. Ministry of Information and Communication (MIC) and Korea Association of Information and Telecommunication (KAIT) (2007), A Study on the Performance Evaluation Method of Public Policy for the Small and Medium Size IT Firms, MIC and KAIT, Korea.
  28. Ministry of Knowledge Economy (MKE) and Korea Evaluation Institute of Industrial Technology (KEIT) (2009), Industry and Technology Development Program Performance Investigation and Analysis Report, MKE and KEIT, Korea.
  29. Ministry of Knowledge Economy (MKE) and Korea Evaluation Institute of Industrial Technology (KEIT) (2010), Knowledge and Economy and Technology Innovation Program Performance Investigation and Analysis Report, MKE and KEIT, Korea
  30. Ministry of Knowledge Economy (MKE) and National IT Industry Promotion Agency (NIPA) (2011), Information and Communication Promotion Fund-Loan Program Performance Investigation and Analysis, MKE and NIPA, Korea.
  31. MinitabR (2005), MinitabR Release 14.20 StatGuide, Minitab Inc., State College.
  32. Montgomery, D. C. and Runger, G. C. (1999), Applied Statistics and Probability for Engineers, 2nd edn, John Wiley and Sons, New York.
  33. Oral, M., Kettani, O., and Lang, P. (1991), A Methodology for Collective Evaluation and Selection of Industrial R&D Projects, Management Science, 37(7), 871-885. https://doi.org/10.1287/mnsc.37.7.871
  34. Park, S. (2010), Two-Staged DEA/AR-I Performance Evaluation Model for R&D Projects Efficiency Correlation Analysis and Programs Positioning Investigation, DAEHAN Journal of Business, 23(6), 3285-3303.
  35. Park, S., Kim, H., and Baek, D. (2008), Design of DEA/(AR-I, ARGM) Models and Sensitivity Analysis for Performance Evaluation on Governmental Funding Projects for IT Small and Medium-sized Enterprises, Journal of the Korean Institute of Industrial Engineers, 34(2), 190-204.
  36. Park, S., Kim, H., Sul, W., Baek, D., and Khoe, K. (2011), Handbook on R&D Performance Evaluation, Korean Studies Information, Korea.
  37. Parks, R. B. (1983), Technical Efficiency of Public Decision Making Units, Policy Studies Journal, 12(2), 337-346. https://doi.org/10.1111/j.1541-0072.1983.tb00275.x
  38. PASW Statistics (2009), PASW Statistics Release 18, IBM SPSS.
  39. Rhim, H., Yoo, S., and Kim, Y. (1999), A DEA/AHP Hybrid Model for Evaluation and Selection of R&D Projects, Journal of the Korean Operations Research and Management Science Society, 24(4), 1-12.
  40. Science and Technology Policy Institute (STEPI) (2010), Standard Performance Indices of Information and Communication Promotion Fund R&D Program, STEPI, Korea.
  41. Seiford, L. M. and Thrall, R. M. (1990), Recent Development in DEA : The Mathematical Programming Approach to Frontier Analysis, Journal of Econometrics, 46(1-2), 7-38. https://doi.org/10.1016/0304-4076(90)90045-U
  42. Sherman, H. D. and Gold, F. (1985), Bank Branch Operating Efficiency : Evaluation With Data Envelopment Analysis, Journal of Banking and Finance, 9(2), 297-315. https://doi.org/10.1016/0378-4266(85)90025-1
  43. Sohn, S. and Joo, Y. (2004), Data Envelopment Analysis and Logistic Model for BRAIN KOREA 21, IE Interfaces, 17(3), 249-260.
  44. Thompson, R. G., Langemeier, L. N., Lee, C. T., Lee, E., and Thrall, R. M. (1990), The Role of Multiplier Bounds in Efficiency Analysis With Application to Kansas Farming, Journal of Econometrics, 46(1-2), 93-108. https://doi.org/10.1016/0304-4076(90)90049-Y
  45. Zhu, J. (2003), Quantitative Models for Performance Evaluation and Benchmarking : Data Envelopment Analysis With Spreadsheets and DEA Excel Solver, Springer, Boston.