Journal of KIISE (정보과학회 논문지)

Korean Institute of Information Scientists and Engineers (한국정보과학회)
권호 표지
DOI QR코드

DOI QR Code

A Comparison of the Search Based Testing Algorithm with Metrics

메트릭에 따른 탐색 기반 테스팅 알고리즘 비교

  • 최현재 (부산대학교 전기전자컴퓨터공학과) ;
  • 채흥석 (부산대학교 전기전자컴퓨터공학과)
  • Received : 2015.12.11
  • Accepted : 2016.02.10
  • Published : 2016.04.15
Download PDF
⟨ Previous Next ⟩

Abstract

Search-Based Software Testing (SBST) is an effective technique for test data generation on large domain size. Although the performance of SBST seems to be affected by the structural characteristics of Software Under Test (SUT), studies for the comparison of SBST techniques considering structural characteristics are rare. In addition to the comparison study for SBST, we analyzed the best algorithm with different structural characteristics of SUT. For the generalization of experimental results, we automatically generated 19,800 SUTs by combining four metrics, which are expected to affect the performance of SBST. According to the experiment results, Genetic algorithm showed the best performance for SUTs with high complexity and test data evaluation with count ${\leq}20,000$. On the other hand, the genetic simulated annealing and the simulated annealing showed relatively better performance for SUTs with high complexity and test data evaluation with count ${\geq}50,000$. Genetic simulated annealing, simulated annealing and hill climbing showed better performance for SUTs with low complexity.

탐색 기반 테스팅은 넓은 탐색 범위에서 효과적으로 테스트 데이터를 생성하는 대표적인 기술중 하나이다. 탐색 기반 테스팅의 성능이 프로그램의 구조적 특성에 영향을 받는 것이 알려져 있음에도 구조적 특성을 고려한 탐색 기반 테스팅 비교 연구는 제한적으로 수행되었다. 본 연구는 탐색 기반 테스팅 비교를 통해 테스트 대상의 구조적 특성 차이에 따른 최적 알고리즘을 분석하고자 한다. 실험 결과의 일반화를 위해 탐색 성능에 영향을 주는 4가지 메트릭 값을 조합하여 19,800개의 테스트 대상 프로그램을 자동 생성하였다. 실험 결과 복잡도가 높은 프로그램을 20,000번 이하의 횟수로 분석하였을 경우에는 유전 알고리즘이 가장 우수한 성능을 보였으나 50,000번 이상 분석하였을 경우에는 유전 담금질 기법과 담금질 기법이 다른 알고리즘들 보다 우수한 성능을 보였다. 복잡도가 낮은 프로그램에서는 유전 담금질 기법, 담금질 기법, 언덕 오르기 방법이 다른 알고리즘들 보다 우수한 성능을 보였다.

Keywords

References

  1. Ali, Shaukat, et aI., "A systematic review of the application and empirical investigation of search-487 based test case generation," Software Engineering, IEEE Transactions on, Vol. 36, No. 6, pp.742-762, 2010. https://doi.org/10.1109/TSE.2009.52
  2. Xiao, Man, et aI., "Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques," Empirical Software Engineering, Vol. 12, No. 2. pp. 183-239, 2007. https://doi.org/10.1007/s10664-006-9026-0
  3. Harman, Mark, et aI., "Testability transformation ," Software Engineering, IEEE Transactions on, Vol. 30, No. 1, pp. 3-16, 2004. https://doi.org/10.1109/TSE.2004.1265732
  4. The Motor Industry Software Reliability Association, Report 5: Software Metrics, MIRA Ltd., Feb. 1995.
  5. Kernighan, Brian, "JPL Institutional Coding Standard for the C Programming Language," California Insititute of Technology, Mar. 2009.
  6. Martin, Lockheed, "Joint strike fighter air vehicle c++ coding standards for the system development and demonstration program," 2005.
  7. Michael, Christoph C., Gary McGraw, and Michael Schatz, "Generating software test data by evolution," Software Engineering, IEEE Transactions on, Vol. 27, No. 12, pp. 1085-1110, 2001. https://doi.org/10.1109/32.988709
  8. Tlili, Marouane, Stefan Wappler, and Harmen Sthamer, "Improving evolutionary real-time testing," Proc of the 8th annual conference on Genetic and evolutionary computation, ACM, pp. 1917-1924, 2006.
  9. Harman, Mark, et aI., "Optimizing for the number of tests generated in search based test data generation with an application to the oracle cost problem," Software Testing, Verification, and Validation Workshops (ICSTW), 2010 Third International Conference on IEEE, pp. 182-191, 2010.
  10. Wegener, Joachim, Andre Baresel, and Harmen Sthamer, "Evolutionary test environment for automatic structural testing," Information and Software Technology, Vol. 43, No. 14, pp. 841-854, 2001. https://doi.org/10.1016/S0950-5849(01)00190-2
  11. McMinn, Phil, "Search-based software test data generation: a survey," Software testing, Verification and reliability, Vol. 14, No. 2, pp. 105-156, 2004. https://doi.org/10.1002/stvr.294
  12. Harman, Mark, and Phil McMinn, "A theoretical and empirical study of search-based testing: Local, global, and hybrid search," Software Engineering, IEEE Transactions on, Vol. 36, No. 2, pp. 226-247, 2010. https://doi.org/10.1109/TSE.2009.71
  13. Affenzeller, Michael, et aI., Genetic algorithms and genetic programming: modern concepts and practical applications, Crc Press, 2009.
  14. McMinn, Phil, et aI., "Input domain reduction through irrelevant variable removal and its effect on local, global, and hybrid search-based structural test data generation," Software Engineering, IEEE Transactions on, Vol. 38, No. 2, pp. 453-477, 2012. https://doi.org/10.1109/TSE.2011.18
  15. Lakhotia, Kiran, Mark Harman, and Hamilton Gross, "AUSTIN: An open source tool for search based software testing of C programs," Information and Software Technology, Vol. 55, No. 1, pp. 112-125, 2013. https://doi.org/10.1016/j.infsof.2012.03.009
  16. Lundy, Miranda, and Alistair Mees, "Convergence of an annealing algorithm," Mathematical Programming, Vol. 34, No. 1, pp. 111-124, Jan. 1986. https://doi.org/10.1007/BF01582166
  17. Arlt, Stephan, and Martin Schaf, "Joogie: Infeasible code detection for java," Computer Aided Verification, Springer Berlin Heidelberg, pp.767-773, 2012.