A Comprehensive Review on Regression Test Case Prioritization Techniques for Web Services

Abstract

Test Case Prioritization (TCP) involves the rearrangement of test cases on a prioritized basis for various services. This research work focuses on TCP in web services, as it has been a growing challenge for researchers. Web services continuously evolve and hence require reforming and re-execution of test cases to ensure the accurate working of web services. This study aims to investigate gaps, issues, and existing solutions related to test case prioritization. This study examines research publications within popular selected databases. We perform a meticulous screening of research publications and selected 65 papers through which to answer the proposed research questions. The results show that criteria-based test case prioritization techniques are reported mainly in 41 primary studies. Test case prioritization models, frameworks, and related algorithms are also reported in primary studies. In addition, there are eight issues related to TCP techniques. Among these eight issues, optimization and high effectiveness are most discussed within primary studies. This systematic review has identified that a significant proportion of primary studies are not involved in the use of statistical methods in measuring or comparing the effectiveness of TCP techniques. However, a large number of primary studies use 'Average Percentage of Faults Detected' (APFD) or extended APFD metrics to compute the performance of techniques for web services.

1. Introduction

Regression Testing helps to determine the correctness of a web service, following functional and non-functional changes. A system integrated with the web services undergoes evolution and maintenance. It is due to the addition of a new function or development in the existing ones. Any error in a web service could impact the entire system. Regression testing ensures that changes do not have a significant adverse effect on the correctness of web services [1].

Regression testing is costly in terms of time and money because half of the cost accounts for software maintenance. Rerunning previously executed test case induces a high cost [2]. Regression testing can be performed in a systematic way to substitute the rerunning of all test cases by selecting test cases that cover modifications made in a system. In other words, a set of test cases are chosen, which helps to detect maximum faults within a web service. In addition to cost, the time factor also needs to be considered when executing test cases. These factors are significantly considered for the selection and reordering of test cases, as the size of the regression test suit is substantial usually [3]. Traditional industrial applications are written in the orchestration languages like ‘Web Services-Business Process Execution Language' (WS-BPEL) and link with the web services via 'Web Services Descriptive Language' (WSDL) and XPath. Any error in these artifacts results in the extraction of wrong information from messages. Retesting of all services to reveal the incorrect information is constrained due to a shortage of resources [4]. To increase the efficiency of regression testing, ‘regression test selection’ (RTS) or ‘test case prioritization’ (TCP) is performed [5].

In the literature, there are several definitions of TCP. We have found the following two definitions to be more comprehensive because they both consider the importance of the scheduling (time) of test cases for effectiveness, cost, and performance goals.

“Test case prioritization, i.e. scheduling test case executions in an order that attempts to increase their effectiveness at meeting some desirable properties, is a powerful approach to maximise the value of a test suite and to minimise the testing cost” [6].

“One approach to selecting test cases is to schedule the test cases according to some criterion to satisfy a performance goal; scheduling test cases in this manner is known as test case prioritization”[7].

These two definitions of TCP discuss test cases regarding TCP execution and performance goals. The commonality between these two is the execution of test cases. The first definition focuses more on the sequence of executing test cases and their advantages. The second one focuses on the criteria for running test cases for the satisfaction of performance goals. Time and cost factors determine the effectiveness of a TCP technique, as revealed in the literature [8][9]. Walcott et al. mentioned that an industrial product with approximately 20k lines of code required seven weeks to test in its entirety. Seven weeks for regression testing is relatively expensive in terms of time [10]. To overcome this long processing time, Garg and Datta claimed that the parallel execution of the entire test suit required less than three hours [11]. A similar claim was verified by Liu et al. [12]. They found that it took 39.2 minutes to finish the parallel execution of all test cases.

Although TCP is time-costly in some cases, it is still advantageous because it reorders test cases within a limited time to ensure the earlier detection of faults [13]. TCP is used to automate the selection of a minimum number of test cases from the total numbers of test cases. The intention behind this selection is to uncover the maximum number of faults within the components of web services. Prioritized test cases are executed to prove that the selected test cases provide sufficient satisfaction for software functionalities in reduced time [14].

Catal and Mishra [15] examined regression testing techniques, which were widely used in primary studies published between 2001 and 2011. However, they did not include a comprehensive comparison of TCP techniques. Moreover, they did not include a discussion on TCP approaches regarding web services. Until recently, there is no systematic literature review (SLR) that identifies current issues regarding TCP techniques for web services. There is a growing call for a comprehensive SLR in order to provide an analysis of TCP techniques, models, frameworks, and algorithms as a solution to the emerged issues and challenges related to web services TCP techniques. Our efforts to provide a comprehensive SRL will fill this void, including all papers published up until 2017.

2. Background

To present this SLR’s context, an overview of web services and TCP is provided in the subsequent subsections.

2.1 Web Services

A large number of web applications have procedures for WSDL [16], specifying messages and functional interface parameters. WSDL can be applied to XML documents for the representation of messages. A formal XML notation describes a web service. A WSDL document augments the interface information. Furthermore, external services observe XML documents. Farrag et al. have stated that WSDL has been mostly used in combination with XML schema and ‘Simple Object Access Protocol’ (SOAP) [17]. The combined services and schema have ensured communication over the internet.

In addition to the above-discussed procedures, web services have some standards. In literature, several definitions for the web standards of web services have been presented. Two of these definitions are given as follows.

“The WS-standards define how middleware aspects (security, reliability, transactions, etc.) can be realized through web services” [18].

“The most commonly used standards and protocols include, but are not necessarily limited to, the Extensible Markup Language (XML), Simple Object Access Protocol (SOAP), the Web Services Definition Language (WSDL) and Universal Discovery Description and Integration (UDDI)” [19].

The first definition of web standards, as presented by Simon et al. [18], focuses on various aspects of the middleware of web services, while the second definition of web standards only enumerates on many web standards that are applied to communicate transactions over the internet [19].

A service description contains all information required to interact with the web services, including location, transport protocol, and message format [20]. A web service is modular, self-describing, and self-contained. Above, the authors have described web services and web standards, mainly used for communication over the internet. Communication over the internet is increasingly being used, resulting in the evolution of web services. Alone, the WSDL specification of web services cannot verify the retesting of web services because web service providers have source codes regarding dynamic web services [21].

2.2 Test Case Prioritization (TCP)

TCP problem was defined in the earlier research works by Elbaum et al. [22][23], where they stated T as a test suit; PT as permutations of T and f represented a function from PT to real numbers. In this definition, PT is taken as a set of possible orders, and f is applied to any such order to achieve an award value for the order.

Problem definition stated in the works, as mentioned earlier, is similar to other works [24][25]. TCP problem in the proposed approaches is the same to date because researchers have attempted to resolve this problem in terms of time, and the cost of ordering test cases. A hybrid TCP technique is not necessarily composed of two techniques of the same criteria-based group of techniques. For example, a hybrid TCP technique may involve a combination of users’ requirements priority scores and users’ session information. On many occasions, researchers have incorporated one TCP technique into another TCP technique to create a hybrid TCP technique. Marchetto et al. [26] proposed a hybrid TCP technique by including low code coverage and high requirement coverage information. The proposed hybrid TCP technique has been generalized to all size projects [27].

3. The Systematic Literature Review Method

Researchers use SLRs as an instrument for identifying, evaluating, and interpreting existing research on specific research questions and topic areas [28]. Contributing studies in SLR are considered primary studies. A well-defined and explicit process is required to conduct an SLR. For the proposal and execution of this SLR, this paper's authors have leveraged the recommended guidelines by Kitchenham and Charters [28], and the following adjustment recommendation made by Kitchenham et al. [29].

Fig. 1 depicts three phases, and the activities performed in each phase. This three-phase model of SLRs is used by Brereton et al. [30] and Kazmi et al. [31] in their studies.

Fig. 1. The Phases of a Systematic Literature Review

3.1 Research Questions

It is essential, in all types of systematic reviews, that research questions should be specified [32]. Research questions are used to build the search strings and determine the accurate extraction of data from selected primary studies. Research questions become part of the protocol and do not be changed after the protocol is accepted [30]. Questions to be addressed in this research include the following: 

RQ 1. What approaches are proposed as a means to prioritize web services test cases?

RQ 2: How have the proposed web services TCP techniques been validated?

RQ 3: Are there statistical methods used in web services TCP techniques?

RQ 4. What issues related to TCP exist in regression testing, with respect to web services?

Next, a set of keywords are established to find relevant primary studies on the research topic. Finding suitable primary studies has supported the authors in seeking out answers to the above-given research questions.

3.2 Identification of Appropriate Search Keywords

Searching the string with the use of keywords is an essential part of SLR because there is a need to find relevant studies. Thus, the author constructed simple search strings from the topic of interest and research questions [33]. Alternative and synonymous keywords were also used. Both Boolean ‘AND’ and ‘OR’ operators were used for joining and using alternative keywords, respectively. Simple search strings need less effort on refinement. The search strings have been used in different repositories, as detailed in the following Table 1.

Table 1. Search strings/keywords

3.3 Primary Studies’ Inclusion and Exclusion Criteria

Primary studies within regression testing, test case prioritization, and web validation have been examined, depending on the proposed criteria and empirical evidence. Primary studies with regression testing, test case prioritization techniques, and web services validation contained in their titles, keywords, abstracts, and full texts, have been examined.

3.3.1 Inclusion Criteria

The inclusion criteria of research papers have been defined based on abstracts and titles:

- Studies which discuss regression testing for web services

- Studies which discuss test case prioritization for web services

- Studies which discuss web services validation

- Studies which discuss any type of regression testing

- Studies which discuss algorithms for the test case prioritization technique

If the titles and abstracts of a study contained any of the above points, serving as inclusion criteria, the study was subsequently included in this SLR.

3.3.2 Exclusion Criteria

In addition to the inclusion criteria detailed above, this paper’s authors also developed a series of exclusion criteria for the current SLR.

- Studies which did not focus on regression testing for web services were excluded

- Studies which did not focus on test case prioritization for web services were excluded

- Technical reports, Ph.D. dissertations, and studies, of less than five pages, were also excluded. Sufficient evidence was required from empirical studies regarding regression testing for web services.

- Duplicate studies were excluded. It was noticed that some of the researchers had published journal papers, which are versions of their previous conference or workshop papers. A journal paper has more detailed experiments, results, and discussion details as compared to conference or workshop papers.

- Studies that do not provide technical details on this paper’s research topic were also excluded.

The authors have followed the grey literature exclusion criteria given by Kitchenham et al. [33] as they assumed that grey literature was not published due to publishing bias [34]. The quality of primary studies was assessed through quality assessment criteria, as detailed in the following section.

3.4 Quality Assessment Criteria (QAC)

Kitchenham et al. [29] have reported that quality assessment is mandatory for ensuring aggregation results, which are taken from the best available databases. To verify this research finding, Zhou et al. have suggested that a large number of SLRs, such as 110 out of 127, have used the explicit quality assessment criteria [35]. It means that the findings of Zhou et al. are in line with those of Kitchenham et al. regarding QAC’s importance. Therefore, we have used the quality assessment criteria for evaluating chosen primary studies, as proposed in Kitchenham et al. [32], through the use of quality assessment questions.

This paper’s authors have used four quality assessment questions to evaluate primary studies, which are the following:

QA 1. Is the study focused on the relevant research problem domain?

QA 2. Does the study explicitly involve the web services test case prioritization technique?

QA 3. Is the study about regression testing for web services?

QA 4. Is the study about the validation of web services?

The quality assessment questions have been scored, as detailed in the following table: Each question has been scored on a scale from Yes (1), Partly (0.5), to No (0 “Scores were summed up accordingly. In order to increase the reliability of this SLR, the authors involved other independent researchers in assessing the primary studies. Any confusion or disagreement was settled before reaching a final agreement.

3.5 Search Process

A traditional review is restricted to literature, which is already known by authors. The same studies are cited more often than cursory searches [36]. Like other SLRs, this research study adopts a broad research strategy that is based on predefined search strings and keywords. The following digital libraries (DLs) were used to find relevant literature on the research topic:

- ACM

- IEEE Xplore

- ScienceDirect (The Journal of System and Software, Information and software Technology

- Springer Link

- Wiley Online Library

The above-listed repositories are the most popular because many SLRs in the area of software engineering use them for journal and conference papers. Other repositories were excluded because the search was restricted to journal papers and conference papers within the field of regression testing, and particularly regarding TCP for web services. Table 2 presents the number of studies found in each of these databases after each search. 

Table 2. Selection Procedure

The researchers applied a broad search to find these research papers. The researchers used keywords given in Table 1 and identified many research papers that focus on regression testing and web services TCP. They applied this search process by choosing an advanced research option for the above-given repositories. They selected the titles, abstracts, and keywords for the first search, and then refined the search process by only selecting abstracts with searched keywords. However, the final search was based on studying articles in full text to ensure that the articles discussed regression testing and TCP for web services. Research articles written in languages other than English were not included [32].

3.6 Data Extraction

Data extraction is a critical part of an SLR. Data extraction processes are employed in an SLR, to extract findings from the chosen primary studies, through a consistent means. We used the data extraction procedure proposed by Kitchenham [34]. They extracted information regarding the focused aspects of regression testing and mainly web services TCP from each primary study.

Table 3. Summary of the Research Focuses of Primary Studies

Table 3 presents a summary of each primary study selected in this SLR. The summary includes the central idea of each proposed TCP approach, along with the venue. As mentioned earlier, the authors have focused more on journals and less on conference papers. As given in Table 3, 50 articles (or 77%) are journal papers, and 15 papers (or 23%) are conference papers. The publishing trend on the web services TCP techniques since 2011 has evidence that primary studies were either published in conferences or Journal papers. In summary, the topic web services TCP is in the middle phase as many journal and conferences studies have been already addressed web services TCP issues.

3.7 A Conceptual Framework of SLR on TCP for Web Services

The following Fig. 2 presents a framework for the currently-presented SLR. The proposed conceptual framework is an abstraction of the relevant TCP approaches, the issues of TCP techniques, and the validation methods used in regards to the existing web services TCP approaches in primary studies. The proposed framework is built on the data we have gathered during the search process. Moreover, the proposed framework is validated by using the information from retrieved papers.

Fig. 2. A Conceptual Framework of TCP Techniques

The proposed framework in Fig. 2, in its composition, contains three main components. The first is the challenges to the TCP techniques, the second is the criteria-based classification of TCP techniques, and the third is the evaluation of web services TCP techniques. This framework has been proposed to group the TCP techniques along the lines of similar features and to find answers to the SLR's research questions.

4. Results & Discussion

4.1 (RQ1) What Approaches are proposed, as a Means to Prioritize Test Cases?

RQ1 revisits web services TCP approaches, including proposed TCP models, frameworks, and algorithms.

Rothermel et al. [24] and Tsai et al. [37] were among the earliest academic researchers who presented techniques for web service testing. In the latter study, researchers proposed an XML-based Coyote framework that converted the WSDL specifications into scenarios of testing in its first part and information tracing in the second part. That time distributed services required integration testing, and the Coyote framework was used for integration testing instead of module testing.

Four SLRs were identified, along with several non-SLRs in literature. Both SLRs and numerous non-SLRs attempted to address the TCP problem. A comprehensive comparison of SLRs, from the perspective of TCP techniques, has been contained in the following content.

In the first SLR, Qiu et al. [2] sought to explore regression testing techniques used within the context of five diverse stakeholders. In the same study, the roles of all five stakeholders within regression testing were studied, as well. In addition to the position of web services stakeholders, a large proportion of the SLR results have highlighted the TCP techniques, and have provided a limited classification of TCP techniques. In the second SLR, Kazmi et al. adopted a different approach to TCP techniques and stated that 'Regression Testing Selection' (RTS) involved fault, coverage, and cost-based adequacy criteria [31]. Kazmi et al.’s content did not include any other than the three factors listed above. However, they also investigated a comprehensive set of standard metrics applied in regression testing. While the first SLR by Qiu et al. identified evaluation as a validation method for TCP techniques, Kazmi et al. used 11 metrics as a means of evaluating techniques in their SLR. The third SLR by Yoo and Harman detailed state of the art regarding session-based and coverage-based techniques, from available research papers, published up to 2010 [54]. In both of these criteria-based techniques, Yoo and Harman more emphasized on an increasing fault detection rate [54]. In the same SLR, various types of TCP techniques were studied under the coverage-based criteria of the TCP techniques, such as branch total, statement total, and the fault-exposing potential information included in other criteria-based TCP techniques. The second SLR by Kazmi et al. [31] is in line with the third SLR, in detailing the types of regression testing. However, Yoo and Harman overemphasized the oracle problem of regression testing and did not focus on other TCP challenges [54]. In the fourth SLR, Catal and Mishra followed the pattern of a mapping study and included research papers published between 2001 and 2011 [15]. They did not describe the TCP techniques for researchers and practitioners in detail.

From the comparison of four SLRs, the authors have found that these SLRs have missed, including the newly-proposed TCP techniques for web services. Therefore, we have provided a sufficient literature review of the criteria-based existing and new-emerged web services TCP techniques. The TCP techniques are revisited, discussed in four SLRs conducted by Yoo and Harman [54], Qiu et al. [2], Kazmi et al. [31], and Catal and Mishra [15]. Based on data extraction, as shown in Table 3, we provide the classification of TCP approaches by following criteria, algorithms, frameworks, and models proposed in the selected studies.

Fig. 3 shows that 63% of the total 65 research studies discussed the proposal of criteria-based web services techniques. Algorithm-based web services TCP techniques were found in 17% of the total 65 primary studies. Model-based web services techniques were examined in 11% of total 65 primary studies, and framework based TCP approaches were reported in 9% of total 65 primary studies. It showed that web services TCP is no more examined in the context of classification algorithms and clustering approaches.

Fig. 3. TCP Approaches

Before this study, no SLR or mapping study investigated the models, frameworks, and algorithms separately in order to find a solution to the TCP problem. In summary, researchers have focused on proposing TCP techniques, models, frameworks, and algorithms as a means of increasing the effectiveness and optimization of TCP techniques for web services.

Fig. 4 shows a map of selected primary studies over the TCP techniques and statistical methods. Note that the information includes not only data extracted from chosen primary studies [96]. Detailed information about TCP techniques has been provided within the above section of this SLR. Additionally, abbreviations have been used for each identified TCP technique. For example, the abbreviations coverage based (CB), session-based (SB), similarity-based (SiB), requirement based (RB), flow graph-based (FgB), risk-based (RiB) and location-based (LB) techniques, are all presented in Fig. 4.

Fig. 4. A Bubble Chart of the Distribution of TCP Technique and Statistical Methods

Fig. 4 shows that CB approaches have been widely proposed to address the TCP issue. Moreover, the bubble chart, as Fig. 4 indicates that several papers introduced CB techniques, and evaluation of them was done by statistical methods including p-value, Tukey test, ANOVA, and descriptive statistics. Besides, CB techniques, only SiB techniques, have been validated by Anova Analysis statistics. Due to the availability of open-source and industrial interfaces of web services, researchers evaluated their proposed approaches [1][5][18][19][26][35]. Hence, they found it convenient to propose CB approaches of TCP. Other than CB techniques, RB approaches are popular techniques because many researchers proposed TCP approaches using functional specifications of web services. SiB and LB, and RiB based TCP approaches were least examined in the literature because the latter mentioned approaches remained less effective to detect the faults. To overcome the frequent changes in web services, code coverage based approaches performed better for patched web services. Cost-saving is another achievement from coverage based TCP approaches because multiple test cases can be executed at once [88]. Since regression testing and particularly TCP approaches’ proposal and evolution have spawned in the last many years, coverage based approaches such as code based and requirement based TCP approaches have become popular due to their long history in comparison to other TCP approaches.

4.2 (RQ2) How have the proposed web services TCP Techniques been validated?

To quantify the results of primary studies in regression testing for web services TCP approaches (combining TCP techniques and algorithms), some useful metrics have been applied to compare the effectiveness of different web services TCP techniques. Web services TCP techniques differ in terms of their designs and objectives. Owing to the different goals and designs of TCP approaches, it becomes hard to apply a common estimation mechanism for evaluating the results of TCP approaches. The most commonly-applied evaluation metrics are provided in Table 4. Mei et al. mentioned three web service metrics, namely APFD, 'Harmonic Mean of the rate of Fault Detection’ HMFD, and average ‘Relative Position’ RP [76]. They found that each of the three web service metrics measured different aspects of web services TCP approaches.

Table 4. The Distribution of Studies over Evaluation Metrics

Table 4 shows the distribution of identified evaluation metrics in primary studies. The top evaluation metric APFD has been reported in 22 primary studies. Wang and Zeng identified the APFD and FDR metrics, indicating that APFD is also discussed with other metrics [85]. In Table 4, some primary studies other than those reported in Table 3 were also used to discuss the APFD and NAPFD metrics. This SLR’s findings regarding the metrics proposed for validation of web services TCP have not been mentioned in the previous mapping study by Qiu et al. [2]. As shown in Table 4, a total number of 17 metrics have been identified that fill the research gap on the current state of the art on validation of web services techniques. To compare the findings on RQ2, the authors mention 17 metrics, while Catal and Mishra [15] reported only seven metrics in their SLR. Kazmi et al. [31] mentioned 12 metrics, and include those metrics which were stated for test case selection. The lateral mentioned SLR includes precision, recall, efficiency, and F-measure metrics. These metrics are known as inclusiveness metrics. It is noticed that precision, recall, efficiency, and F-measure metrics are best examined in a combination of classifiers [101]. Four lateral mentioned metrics are best explained for fault prediction in traditional programs. Code metric for interface application of web services is used to predict changes produced in web services. Code metrics are widely used for traditional programs. Kumar et al., in their proposed web services TCP, used code metrics on eBay web services, and gained better results than those using a set of other metrics [100]. Askarunisa et al. used APFD, and Harmonic Means (HM) metrics to validate their web services TCP on benchmark weather monitoring services (WMS), and Bible web services [99]. Mei et al. applied the APFD metric to measure the effectiveness of the TCP technique on eight WS-BPEL programs [16].

4.3 (RQ3): Are there Statistical Methods used in web services TCP Techniques?

Since 2016, the trend of using statistical methods in regression testing has increased. The increasing trend of using statistical methods is evident in many primary studies. In a primary study, Jiang et al. integrated statistical methods with the TCP for fault localization [51]. Moreover, they used ANOVA analysis to determine differences between TCP techniques in a group of TCP techniques.

Tahat et al. combined the APFD metric with the HMFD metric to determine a fault detection rate by assuming the multiple faults [91]. To investigate the effectiveness of TCP techniques, they determined that detecting defects in regression testing was difficult due to restrictive measures when using the two metrics HMFD and APFD in a combined form. Thus, they established the relative position of test cases from the application of varying TCP techniques. Ledru et al. assessed various conditions of their evaluated experiment through the use of the t-test ANOVA analysis [55]. The t-test ANOVA analysis measured differences between the random and distance-based prioritization of test cases. In some experiments, ANOVA analysis shows its limitations in regards to the size of test cases. For those test experiments, Tukey's HSD test, as an option of ANOVA analysis, is performed to determine differences in multiple mean values.

Table 5 shows that 11 primary studies applied statistical methods regarding TCP techniques for web services. ANOVA analysis is the most-cited statistical method, examined in 15.38% (10/65) x100 of primary studies. Statistical tests, including p-value, t-test, and Tukey's HSD test, also belong to the ANOVA family of analysis tests. Apart from ANOVA analysis, no family or class of statistical method has been observed, which is used for comparing, evaluating, or verifying TCP techniques for web services. Only Marchetto et al. [26] utilized a descriptive analysis. Therefore, a low percentage (1.53%) referred to a statistical analysis other than ANOVA analysis.

Table 5. Primary Studies Using the Statistical Method​​​​​​​

4.4 (RQ4): What Issues Related to TCP Exist in Regression Testing, with Respect to Web Services?

Cloud computing services for ‘regression testing’ are discussed in various primary studies, within the context of using third party services for testing modified web services. The interface information of web services is merely exposed, and its implementation is also kept in-accessible. In these circumstances, the testing of web services by a third party becomes a challenge [16][97]. Various TCP approaches have been proposed to address the TCP problem, despite mentioning or discussing the challenges related to using third party services. Eight types of potential issues have been identified, that prevail in terms of TCP techniques for web services, and these issues are given as the following.

4.4.1 Scalability

The scalability challenge is identified during the testing of web services on clouds. The 'Software as a Service' (SaaS) infrastructure of MIDAS is only accessible for the functional testing of web services. The MIDAS platform provides software testing in a controlled manner. To make cloud testing more efficient, large-scale software and hardware distributed systems are planned as a means of troubleshooting and testing complex web systems [6]. In recent research work, Wang et al. revealed that the scalability and stability issues of newly-proposed TCP techniques are not as severe when compared to most previous TCP techniques [103]. From these two primary studies, it can be inferred that scalability remains an issue because limited research efforts have been made in regards to scalability, in terms of regression testing.

4.4.2 Overhead

A high overhead issue has been reported to introduce performance regression testing. The above factors involve spotting performance regression because several commits have merged from the last testing of web services. The extra efforts and development time has been spent on finding solutions to commits. The overhead issue of TCP techniques has been discussed in the literature. Different researchers have discussed the overhead TCP issue in various ways. Within primary studies, the overhead issue has been identified as a rapid pace of software evolution, which requires regression testing [6][46][70][104][105].

4.4.3 Overfitting Patch

During test case generation, overfitting becomes an issue when a repair technique produces bad fixes. A bad fix is the human-made modification of source code, used for repairing software. In this regard, repairing the system requires retesting and prioritizing test cases. TCP techniques may suffer from this overfitting issue, while systems are being corrected. Dobolyi et al. have highlighted the issue of over-fitting, which is commonly seen during the validation of approaches [70][106]. To overcome this issue of overfitting patch, Xin and Reiss proposed a technique called DiffTGen for testing patches and for avoiding the overfitting issue during test case generation [107][108]. The DiffTGen technique identified almost 50% of overfitting patches within a few minutes. However, this DiffTGen technique required more sophistication to make it an optimized technique.

4.4.4 Effectiveness

The effectiveness of regression testing is profoundly affected by many TCP factors for web services. These factors include test case size, fault coverage, fault proneness, and data flow. Additionally, the fault severity and cost of correcting web services determine the effectiveness of a TCP technique. Some of the TCP techniques do not account for fault severity and cost, and both factors are considered equally effective [51][62][64]. Therefore, TCP techniques that ignore these two factors cannot produce an appropriate order of test cases [71][91][109].

4.4.5 Optimization

In regression testing, optimization of TCP approaches is a severe concern of researchers as they require the ordering of test cases, specifically by executing those with a high priority, before those with a low priority [26][75][111]. The optimization of TCP techniques is a valid reason for seeing the practical implementation of techniques. Many recent attempts have been made to highlight the optimality issue of TCP techniques [82][63][65][105][110]. In these primary studies, the optimization of test case reduction and test case prioritization were attributed as being future research areas for regression testing. Neither of these studies presented a solution for the optimization of TCP approaches. However, in the first study, Anwar et al. [9] concluded that the optimization of TCP techniques was crucial for the cost-effective execution of test suits. The risk of unsafe optimization increased, because some of the vital test cases were skipped and because faults within these skipped test cases remained unaddressed.

4.4.6 Network Fault

The positioning of faults among networks is a familiar concept used in software testing. A node position is represented as a fault in a system. In a primary study, Kayes et al. presented the proposal of a network approach to resolving the concerns of researchers, regarding network faults [73]. Dependency among network faults occurs because other faults create some fault consequences. Leading failures need to be addressed before consequent faults are removed.

4.4.7 Robustness

Web services are utilized in complex and critical business systems. The robustness of web services is essential for running a business through these complicated and critical systems [58]. A robustness assessment of critical systems is in itself an issue, which is corroborated by a limited number of primary studies. A comprehensive, robust solution to the TCP technique is mandatory because the security vulnerabilities produced as a result of the behavior and design of web services are only addressed through the use of a robustness approach.

4.4.8 Service Integration

The integration of web services is itself a big challenge for web services providers. It is because a vendor who uses the web services of others does not know the internal behavior of these services. The integration of web services results in a lack of control regarding test case execution and software artifacts, without having complete knowledge about the internal behavior of web services [53].

Fig. 5 shows the number of primary studies that tried to highlight or conquer each issue. Therefore, optimization remains a particular concern to researchers in ten primary studies (15% of the total 65 primary studies), and high effectiveness is a concern in nine primary studies (14% of the total 65 primary studies “Overhead and over-fitting are prominent in 8% and 5% of publications, respectively. Scalability issue is now a focused area of research in web services TCP as an increased number of web services use the same interface for services. The scalability issue remained less addressed in the existing literature, and two primary studies (3% of total 65 primary studies) revealed this issue for web services TCP. The remaining matters identified, namely robustness, integration, and network fault, all each appear in 2% of the primary studies.

Fig. 5. Identified Issues Impacting TCP techniques​​​​​​​

5. Threats to Validity

This paper’s authors are conscious of threats that may impact the validity of the findings and outcomes of this SLR. First, selection bias could be possible. Although recognized data sources including IEEE Xplore DL, ScienceDirect, ACM DL, Springer, and Wiley online were all used as primary studies, other sources have not been considered due to their scope and time constraints. Thus, it is possible that some relevant studies are missing.

Another possible concern is the need to find adequate search strings. The search strings and repositories used within the research method section have been presented in the method section. The search strings have been refined many times, to determine the maximum number of relevant research studies, and to access only relevant data sources.

Quality assessment may present a problem, leading to inaccurate data results. It was challenging to answer RQ2, RQ3, and RQ4 due to a lack of appropriate measures required for these questions. Therefore, based on the framework proposed for this SLR, the researchers were able to assess primary studies that ensured quality assessment for chosen primary studies.

6. Conclusion and Future Work

In this SLR, researchers have classified TCP approaches for criteria-based TCP techniques, models, frameworks, and algorithms. Among the seven criteria-based classes of TCP techniques, coverage-based TCP techniques have been greatly covered in primary studies. Most primary studies have mentioned that APFD is a primarily-applied metric, used to compute or compare the effectiveness of TCP approaches. Three primary studies discussed two tools apart from APFD or extended APFD metrics, which were implemented in the MATLAB and C programming languages. In addition, identified statistical methods were employed in primary studies, including ANOVA analysis (p-value, t-test, and Tukey's HSD test) and descriptive statistics. ANOVA analysis is mainly used to find variances among TCP techniques. The researchers have identified cost-overhead, optimization, and high effectiveness as essential issues related to web services TCP techniques. Besides these issues, overfitting patch, scalability, robustness, network faults, and services integration have also been mentioned, needing additional attention from researchers, within future research work. The current research on TCP techniques for web services contains a few concrete works. However, more issues to TCP techniques for web services have been identified for consideration in future research studies.