A Study of Resource Utilization Improvement on Cloud Testing Platform

Abstract

This paper developed the software testing factory-cloud testing platform (STF-CTP) to address the software compatible issues in various smart devices. Software developers who only require uploading the application under test (AUT) and test script can test plenty of smart devices in STF-CTP. The challenge for the cloud test platform is how to optimize the resource and increase the performance in the limited resource. This paper proposed a new scheduling mechanism and a new process of the system operation which is based on the OpenStack platform. We decrease about 40% memory usage of OpenStack server, increase 3% to 10% Android device usage of STF-CTP, enhance about 80% test job throughput and reduces about 40% test job average waiting time.

1. Introduction

With the growing popularity of smart mobile devices, people are more and more dependent on the App developed on the device. Therefore, users experience has been getting more and more important in recent years [1] [2]. Users concern about the software compatibility on the various smart devices rather than the hardware specification. Developers are aware of the importance to the testing the App. However, testing on a single device is not enough. In recent years, the specifications of mobile devices developed by different vendors are not the same. Not only different OSs but also different sizes of devices may cause applications to make errors on different devices. Therefore, compatibility testing is very important before the software is shipped. The concerns of software compatibility are a major topic in smart device manufacturers and thousands of hundreds of software developers as well.

The software factory laboratory at National Taipei University of Technology has developed an integrated cloud testing platform, STF-CTP [3]. STF-CTP mainly focuses on the unit testing and automated acceptance testing with Android devices. It can solve the compatible issue on Android platform, which saves developers’ time and money. STF-CTP enables App developers or testers to upload apps and test scripts for testing. After STF-CTP receives the test, it will dispatch the work to integrated station (IS) located in OpenStack virtual machine. [4] [5] [6] The IS connect to the specified device through Wi-Fi. After finish the test, Android device will be released and return the test result. At last, developers can access the test report and performance data via STF-CTP.

As the need of STF-CTP increases day by day, testers may want to upload a large number of test jobs to execute tests. In order to execute many test jobs at one time, STF-CTP needs to launch a lot of virtual machine to dispatch jobs to different IS. Besides, each testing device can only execute one test at a time; so, only when IS finish converting testing video record, it will release the device to let device get another testing job, which cost a lot of waiting time. After whole jobs are finished, IS will not close it selves, which will waste resource of OpenStack server. Therefore, how to use and allocate resources of devices and OpenStack server become an important issue.

Ala'a Al-Shaikh et al. [7] propose a utilization schedule algorithm to filter all requests that do not meet the optimal resource through layer-by-layer of sorting and puts the matching requests into the schedule, which eventually gets the best use of the utilization schedule. Bo Xu et al. [8] group virtual machines that are related to a certain service into a cluster and propose a segmentation method that can reduce the overall bandwidth requirement after deployment. How to use and allocate resources becomes an important issue. This study will explore how to maximize the utilization of resources, thereby improving the consumption of resources to achieve the same resources. The proposed architecture of this paper includes how to maximize the utilization of resources, how to improve the consumption of resources, how to improve the quality of STF-CTP production, and how to reduce average waiting time to the overall test. The remainder of this paper is organized as follows. Section 2 presents related work and method for finding the solution. In Section 3, we present an effective architecture to change the quantity of Integration Station (IS) in each OpenStack virtual machine and manage IS dynamical which can improve the resource utilization of STF-CTP. In addition, we propose a method to improve the utilization of testing devices that can reduce the execution time and average waiting time on the overall test. In Section 4, we conduct three experiments including the utilization of OpenStack Server, the utilization of Android device, and the performance of proposed STF-CTP architecture to confirm the effectiveness of the proposed solution. In Section 5, we summarize the conclusions and the contributions of this paper.

2. Related Work

This section introduces the background knowledge and research content related to this paper, including the choice of cloud-computing platform, the introduction and application of STF-CTP, how to make maximum use of resource utilization in cloud service, the algorithm of scheduling and deploy virtual machine and how to minimize the makespan of tasks.

2.1 OpenStack

Rakesh Kumar et al. [9] conducted a comparative study of architecture, operating system support, virtual machine migration and image management for several cloud-computing platforms such as Eucalyptus, OpenStack, CloudStack, and OpenNebula. They conclude that OpenStack is the most suitable solution for the cloud computing.

OpenStack is a free and open-source software platform for cloud computing and easy to build and manage public and private clouds. It allowed users deploy virtual machines and other instances that handle different tasks for managing a cloud environment. The STF-CTP is developed under OpenStack.

2.2 STF-CTP

The STF-CTP, software testing factory cloud testing platform, is a cloud testing service platform, developed by the National Taipei University of Technology software testing factory, which is based on OpenStack cloud computing services and Web UI. It mainly focuses on the unit testing and automated acceptance testing with Android devices. Acceptance testing supports different Android automation tools, including Monkey [10], MonkeyTalk [11], Robotium [12], UiAutomator [13] and Robot Framework [14]. In the STF-CTP environment, test jobs will be dispatched by Rabbit Message Queue [15]. Integration station (IS) will receive the job in rabbit message queue. The STF-CTP is required to execute many test jobs. It was developed on OpenStack to build virtual machines, and manage IS. IS connects Android device to execute test via Android Debug Bridge (ADB) [16]. After finish the test, Android device will be released and return the test result. At last, developers can access the test report and performance data via STF-CTP.

2.3 A Hierarchical Scheduling Strategy

A hierarchical scheduling strategy [17] addresses the problem of composition service scheduling and resource allocation in the cloud, especially the data-intensive and compute intensive service. The implementation of each step regarded as the smallest unit and resource allocation depends on the job completion rate which is based on the work as a weight to reduce the work time to complete. Observing STF-CTP, we took the similar approach, but the scenario of test application test is hard to implement like the data-intensive and compute-intensive service. The software must be entirely executed on the target device to ensure the compatibility of the operating version and the hardware difference.

2.4 Resource Utilization in Cloud Computing as an Optimization Problem

Ala'a Al-Shaikh et al. [7] combined 0/1 Knapsack Problem [18] and Activity-Selection Problem [19] to propose a way to maximize resource scheduling. The paper mentioned that in the case of a given resource, hundreds of requests accumulate over time to use that resource. Each request has a profit index. For a given resource regarding profit obtained by utilizing that resource and the number of time slices during which the resource will be utilized, they design an algorithm. The algorithm filters all requests that do not meet the optimal resource through layer-by-layer of sorting and puts the matching requests into the schedule, which eventually gets the best use of the utilization schedule. Then obtain the maximum profit of that resource.

The shortcomings of the algorithm are that the use of 0/1 Knapsack Problem's picks up or not pick up the request problem. The resource does not accept all requests and the requests that are not put into the schedule will be released and will not be used.

2.5 Deployment Method of VM Cluster Based on Graph Theory

The deployment of the virtual machine is the core problem in the cloud computing. The efficiency of the single virtual machine is too low; the overall bandwidth after deployment is high. Therefore, it will be more efficient to group virtual machines that are related to a certain service into a cluster and configure physical hosts in the form of a cluster.

Bo Xu et al. [8] use the energy minimization model to segment virtual machines into different clusters. Then, Bo Xu et al. change the deployment of virtual machine cluster by using max-flow min-cut problem that is based on graph cut theory. After that, the structure assumes the shape of a network. When modeling an energy minimization network, add a source point and a sink point to the network. This virtual machine segmentation method can reduce the overall bandwidth requirement after deployment.

2.6 An Optimized Task Scheduling Algorithm in Cloud Computing

Scheduling of job and maintaining load is a main issue in cloud environment by considering parameters such as throughput, resource utilization, cost, computational time, priority, performance, bandwidth, resource availability. In order to provide better quality of service (QoS), Shubham Mittal et al. [20] , Sharma et al. [21] , and Cui et al.[22] implement an optimized task schedule algorithm to reduce the makespan by dispatching tasks into different resources. This algorithm adapts the advantage of Min-Min, Max-Min and RASA algorithms to perform various check for finding an optimized task which can lead to minimum makespan. Their optimized task schedule algorithm first compute total execution time for task on each resources and compare each resources execution time for tasks to decide which task will be dispatched to which resource. Because this algorithm aims to reduce the makespan of all task, all the task may will be dispatched to the fastest resource when other resources handle the execute time of one task is greater than the total execute time of task in the fastest resource. This situation may did not make good use of resource.

2.7 Self-Organizing Map

Scheduling jobs to minimize the completion time of tasks is important, as it can increase the utilization, productivity, or profit of a cloud. In order to minimize the Job Completion Time (JCT), Li et al. [23], determine task placement plan and resource allocation plan for jobs and then formulate the problem of scheduling a single job as a Non-linear Mixed Integer Programming problem. They focus on the problem of scheduling embarrassingly parallel jobs composed of a set of independent tasks and consider energy consumption during scheduling. Because each task should be placed on only one server, they define resource availability constraint and energy consumption constraint to limit the total amount of computing resource that can be allocated to that job on a server and to limit the total amount of energy that can be consumed by a job.

2.8 Comparisons of Related Approach

In this section, three approaches, Manikandan et al. [24], Pan et al. [25], and Veerendra et al. [26], are compared with our proposed approach are shown in Table 1.

Table 1. The comparison experiment architecture

In the study [24], the research is through the use of independent tasks in cloud computing to allocate resources through the summary of the improved canopy fuzzy c-means algorithm (MCFCMA). In order to assign tasks to their corresponding resources, a particle swarm-based optimization algorithm (PSO) is used. The proposed method can independent task selected based on load feed-back cluster the requested task using MCFCMA and schedule a task to each virtual machine. The proposed system overcomes issues in load balancing and load scheduling; this can be proved by its precision and privacy calculation.

In the study [25], the researches address the performance-variation-aware workflow scheduling problem by leveraging a time-series-based prediction model and a Critical-Path-Duration-Estimation based (CPDE) VM Selection strategy. The proposed method can exploit real-time trends of performance changes of cloud infrastructures and generate dynamic workflow scheduling plans. This study performs extensive experimental case analysis over real-world third-party commercial clouds and show the effectiveness of the proposed method. In the work [26], the research is based on Adaptive Fruit Fly Optimization (AFO) technique called the metaphorical approach to optimize resources in cloud computing. Balance workload and enhance results for all mentioned tasks using the certain Time period of execution, Response Time of Resources (RTR), Energy Utilization Criterion Adaptive Fruit Fly Optimization technique improves comprehensive process flow, Resource Response time and optimizes adequate time. The proposed method can improve speed and resources by adopting hybrid technology. The implemented model works entirely by an increase in utilization thereby lower costs. Energy-intensive AFO-CS algorithms improve overall process flow and reduce the total processing time and RTR.

By comparing with other cloud computing methods shown in Table 1, it can be seen that our proposed approach is better than the three approaches for improving cloud computing resource utilization.

3. The Optimization of STF-CTP Resource Management

This section describes the system architecture and workflow of STP-CTP, and analyzes the utilization of CPU resource. Based on the analysis of the current process, we present a method to optimize the resource of STF-CTP.

3.1 STF-CTP Operation Procedure and Architecture

STF-CTP bases on OpenStack to manage virtual machines and the rabbit message queue uses as a communication bridge between the web server and virtual machines. Fig. 1 shows STF-CTP operation model and procedure. First, developers will upload an application and test scripts. When the web server receives test job, it will dispatch the job to rabbit message queue. Integration station (IS) receives the job in rabbit message queue. IS will connects devices to execute test via Android Debug Bridge (ADB). When finishing the test, IS will release devices. Test result will transfer to rabbit message queue. Developers can review the test result through the web server.

Fig. 1. STF-CTP operation model

Job scheduler of STF-CTP was followed Kuo et al. [27][28]. The monitoring and dynamic resource scheduling method which is based on FIFO, except the current required resource was occupied by another job. The next job will be executed earlier than previous one. Also with reference to Kuo et al. [28], proposed the STF-CTP resource monitoring and management service designed and applied enabled direct OpenStack operation and management. In addition, the automated VM monitoring mechanism was incorporated in the STF-CTP, which provided the STF-CTP with the operation statuses of all VM-related resources.

STF-CTP consists of the web server, rabbit message queue, OpenStack, and smart devices. Web server is responsible for the test job scheduling. Rabbit message queue is used for inter process communication between the Web server and IS. OpenStack manages virtual machines. Every virtual machine has its own IS. The smart devices execute the test and report the result to IS. Each component has its elements, as shown in Fig. 2. The details are described below.

Fig. 2. STF-CTP architecture

1. Web UI: It’s a user interface for developers.

2. Device Queue: The job is assigned to the device queue by resource scheduler depends on the usage of the physical smart device.

3. Resource Scheduler: The scheduler updates the latest usage of devices, scheduling the job in FIFO method.

4. Job Queue: The job will be dispatched to the device queue when the smart device is available.

5. Job Result Queue: This queue stores the test result for web server access.

6. IS: IS checks the job queue, connecting the devices to execute the job through the ADB. The result will be reported to job result queue by IS as well.

7. ADB Daemon: It is a program installed on the smart device. IS will send the ADB commands to ADB Daemon and ADB Daemon will execute the commands.

8. Device Agent: It is a monitor program that propose by Kuo et al. [28] that responsible for the preparation of test jobs and monitors the network flow of the Apps during the test.

3.2 Analysis of STF-CTP CPU Utilization

The CPU is the major resource in the STF-CTP. The specification of the virtual machine show in Table 2. In Fig. 3, X-axis is execution time, and Y-axis is the utilization of CPU. We observed that the peak of CPU usage is about 50% in six different tests, and it happens on the final stage of the entire test. Table 3 shows the purpose of six different test jobs. The application we use to run the test is OnMyWay that was developed by STF-CTP team. We design four different scripts except Installer test and Monkey test. It is because Installer test is to test whether application can be successfully install or not and Monkey test is to simulate the user touch the screen, pressing keyboard and other operations, mainly aimed at the robustness and stability of the application. The rest of the CPU usage remains low. The reason is that the test is executed on the device and CPU resource is not required in this phase. After that, the test details will be converted to the test video that consumes the CPU resource at the end of the test.

Table 2. Virtual machine specification

Fig. 3. CPU utilization in different test job

Table 3. The purpose of test job

Fig. 4 shows three steps for executing the test job on the smart device. The details are described below.

Fig. 4. Testing procedure on device

1. Set up: Connect and wake up the smart device through Wi-Fi. Install and launch the Apps according to the test type. Launching the recording program records the test video while executing the test.

2. Execute test: Execute the test according to the test type. Some tests are executed by the smart device, others executed by PC terminal.

3. Tear down: Terminate the recorded program, and get the test video from the smart device to IS. After converting the test video, it will uninstall the Apps and release the smart device.

3.3 Proposed Method on STF-CTP

This section describes the procedures of our proposed method on STF-CTP.

3.3.1 Test Environment Setup

A virtual machine has more than one IS. In other words, a virtual machine can execute multiple tests at the same time. It also means that each type of test has its method. Unfortunately, one virtual machine only can execute one Robot Framework test. Robot Framework is not supported parallel execution, but it can execute with another test type at same time. Fig. 5 illustrates how the scheduler dispatches the Robot Framework job.

Fig. 5. Scheduler decision tree of Robot Framework job

In order to make all the test types can be executed in parallel; each test type will be modified as shown in Table 4. Robot Framework test type is modified because it does not support parallel execution. Therefore, the modification method of Robot Framework makes it execute with tests of other test types.

Table 4. Modified method for parallel execution

3.3.2 Effect of Quantity of IS

Regarding STF-CTP architecture, the quantity of IS will have different test result. This paper takes monkey test for example because it consumes the highest CPU average utilization among all tests. We assumed that there are 20 money jobs in the queue, waiting for executing with different quantity of IS. Because of the limitation of our testing environment, it supported to create eight IS at the same time.

It can be observed that the execution time cannot have obvious improvement when the quantity of IS more than four (see Fig. 6). As shown in Fig. 7, the average waiting time increased with an increase of the quantity of IS. Under this circumstance, we can conclude the quantity of IS will be an important factor of the execution time and average time.

Fig. 6. Execution time under different quantity of IS

Fig. 7. Average waiting time under different of quantity IS

3.3.3 Improve the Utilization of Device

Originally, the device resource will be released after converting the video which makes the device occupied and cannot be released until finish converting video. If the device is occupied, it cannot receive the new job, which will make the test job’s average waiting time longer. Hence, we proposed a new tear down the procedure to release the device resource earlier before converting videos. The released device can be assigned or executed by other IS job regardless of converting video. Fig. 8 depicts the revised tear down procedure.

Fig. 8. Revised tear down procedure

3.3.4 Dynamic IS management

To optimize the quantity of IS in OpenStack servers, we propose a model to achieve IS dynamic, as shown in Fig. 9. Each component is described below:

Fig. 9. IS management model

1. IS Manager: Get the status of OpenStack server, virtual machine and IS by Server Manager and IS Monitor. And, manage the quantity of IS and a virtual machine as well.

2. IS Monitor: Monitor the status of IS and communicate with IS.

3. Server Manager: Monitor the status of the server and create or delete virtual machine by OpenStack API.

While IS is dynamic, the status of IS become three types from two types. Each component is described below:

1. Idle: IS is available to execute the new job.

2. Busy: IS is busy and cannot execute a new job until job finish.

3. Shutdown virtual machine: When the closed IS is the only IS on the virtual machine, it needs to be closed with the virtual machine as well. Because it takes a short time to close the virtual machine, this state is added to let STF-CTP know the state of this IS which will avoid assigning the job to this IS.

3.3.5 Mechanism of Dynamic IS

The IS dynamic mechanism having two-part, dynamic start IS and dynamic shutdown IS. The details are described below.

1. Dynamic start IS: When STF-CTP has a job to do, and the total IS quantity (IS_total) is less than the maximum quantity of IS(IS_max), and the quantity of job (Job) is more than the quantity of idle IS (IS_idle), STF-CTP will start the quantity of IS_start , as shown in equation (1). And the CPU and memory utilization of OpenStack server decide the IS_max. If the utilization is over 90%, STF-CTP will not start new IS.

\(I S_{\text {start }}=\left\{\begin{array}{cc} \text { Job }-I S_{\text {idle }}, & \text { If } J o b>I S_{\text {idle }} \text { and } I S_{\text {total }}       (1)

2. Dynamic shutdown IS: When IS is idl0e (IS_{dile_time}) for a while (timeout 𝛼), STF-CTP will shutdown the quantity of IS_shutdown. Timeout 𝛼 is set by STF-CTP manager, as shown in formula (2). The manager can adjust Timeout 𝛼 according to the test interval and the number of test jobs. If the Timeout value is very big, it may cause unnecessarily memory usage of the server.

\(I S_{\text {shutdown }}=I S_{\text {idle_time }}>\alpha\)       (2)

3.3.6 Proposed System Architecture

Adding and modifying five components on original STF-CTP architecture, details have shown in Fig. 10. Each component is described below:

Fig. 10. Proposed system architecture

1. Resource Scheduler: Add Robot Framework decision tree into scheduler which is still first in first out. If Robot Framework job cannot be executed, STF-CTP will let next job execute first.

2. IS: Receive web server notification and execute the job from Rabbit Message Queue. Create or terminate IS on the virtual machine by receiving the command from IS Manager. When IS itself is idle and over timeout 𝛼, it will be shut down and notify Server Manager to shut down the virtual machine as well if the IS is the last one on the virtual machine.

3. Server Manager: Monitor OpenStack server and deal with the problem about the virtual machine by OpenStack API.

4. IS Monitor: Monitor IS and communicate with IS.

5. IS Manager: Receive the status of OpenStack server, virtual machine and IS from Server Manager and IS Monitor. Manage the quantity of IS and virtual machine and decide the new IS create at which virtual machine and which IS execute the job.

By using IS Manager to dynamically add or delete IS and virtual machine, it can save a lot of waiting time and make good utilize the resource. IS Manager monitors IS status through IS Monitor, knowing if IS ready for work or a type of work test in progress. Assignment work is also transmitted through IS Monitor to the IS. When IS received instructions, it would add a new IS it their own virtual machine. When the entire virtual machine mounts the maximum load capacity IS, it is necessary to learn from the Server Manager whether the OpenStack server can add a new virtual machine. Server Manager will complete the addition or deletion of virtual machine through calling OpenStack API. We can add a new condition that only when OpenStack server uses no more than 90% of the CPU and Memory then OpenStack can add a new virtual machine.

3.3.7 Proposed System Workflow

Fig. 11 illustrates the new workflow on STF-CTP, and details are described as below. The following procedure starts with the star mark is the new procedure.

Fig. 11. Proposed system workflow

1. An application under test and test scripts are uploaded by developers.

2. *IS Manager gets information by IS Monitor and Server Monitor and create new IS.

3. The job is assigned to Rabbit Message Queue by the web server.

4. *IS Manager notifying IS to get the test job from Rabbit Message Queue.

5. Integration Station (IS) receives job in Rabbit Message Queue.

6. IS connects the smart device to execute test via Android Debug Bridge (ADB). The smart device will be released when the test is finished.

7. The test result will be transferred to Rabbit Message Queue.

8. Developers can review the test result through the web server.

9. *IS will shut down itself when it’s idle time over timeout.

4. Experimental Results

The three experiments were conducted to verify the different objectives of the system.

4.1 Experimental Environment

The experiments were conducted in a 100-Mbit network environment, in which the combination of 802.11-n and 802.11-ac wireless networks was used. A physical machine was employed as the CTP server, and five VMs that had been turned on were used as the test running units. Ten smart devices with Android platform were used as the hardware devices for running tests. The environments are shown in Table 5 and Table 6. The mobile phone brands which shown in Table 7 included HTC, Sony, LG, Samsung, and OPPO, and the Android operating system version is from 4.1 to 4.4. Table 8 lists the test application what we use to develop the test script and run test.

Table 5. CTP web server specifications

Table 6. OpenStack server specifications

Table 7. The smart device specifications

Table 8. Test application

4.2 Utilization of OpenStack Server

To compare the utilization of OpenStack server, Table 9 shows the test scenario between the original architecture and proposed architecture. The first fifty tests will be executed and the coming fifty tests will be executed after five minutes on 10 Android devices. The original architecture uses 10 IS and proposed architecture IS are dynamic according to the usage and the test job amount.

Table 9. Test application

According to the experimental test result in Fig. 12 and Fig. 13, proposed architecture has higher CPU usage and save about 40% usage of memory capacity than original architecture in average. The main reason is each VM supports multiple IS in proposed architecture result in less VM needed in execution. In addition, the drop curve during 601s to 701s in proposed architecture is the idle VMs were closed, which also can save unnecessary memory usage.

Fig. 12. CPU utilization of OpenStack server

Fig. 13. Memory utilization of OpenStack server

4.3 Utilization of Android Device

This section will present the proposed architecture improvement about utilization of Android device. There two kinds of video on STF-CTP. One is video only and the other is a video with the subtitle, so the experiment executes two different tests. The test pattern is also shown in Table 10. The Android device specification is shown in Table 11.

Table 10. Utilization of Android device experiment specifications

Table 11. Android device specifications

The test result is shown in Fig. 14. It can be observed that the usage of Android device in original architecture has a short idle time because Android device was not released until converting the video finish. Oppositely, there is no idle Android device in switching test based on our proposed architecture. The main change is the Android device will be released before converting the video; therefore, the Android device will immediately be assigned to execute other test job so that proposed architecture’s device will be occupied more than the original architecture’s device which increases the utilization of Android device and reduces the execution time. To sum up, the utilization of Android device in proposed architecture is better than the original one.

Fig. 14. Utilization of Android device

4.4 Performance of Proposed STF-CTP

The experiment shows the improvement of the proposed architecture by executing 60, 120, 240 tests. Detail is shown in Table 12.

Table 12. The performance of STF-CTP experiment specifications

The average waiting time of STF-CTP test is shown in Fig. 15. It can be observed that the proposed one is less than original one because the quantity of IS in proposed architecture is more than original architectures. Although the overhead in proposed architecture is to create new IS, it still takes less time than the original one.

Fig. 15. Average Waiting Time of STF-CTP Test

The throughput test result of STF-CTP test is shown in Fig. 16. Multiple IS in one VM can execute more tests in parallel without waiting for other VM available. The two results show that the proposed architecture can reduce the average waiting time and enhance the throughput with the same OpenStack server, Android device and hardware resource.

Fig. 16. Throughput of STF-CTP Test

After 2015, the scheduler technology has been transferred to the organization called Taiwan Testing and Certification Center (ETC) [29] where the technology is used and extended for commercial/business operations. The extension of the technology mainly focuses on the enhancement of system functionality to support security testing and the change of system architecture to improve the overall performance through parallelism by using additional servers. The core technology of the scheduler used in the framework of ETC has not been optimized or further improved. Table 13 lists the comparison of the two frameworks.

Table 13. Comparison of two architectures

As can be seen from Table 13, this study has expanded the FIFO-based monitoring, dynamic resource scheduling methods and dynamic IS management. The Taiwan Testing and Certification Center was transformed into a commercial operation after the transfer of the technology bank. The work of focus is on how to use this technology to guide companies to efficiently test their mobile software application, and to create test cases based on test methods to lower the cost for commercial/business operations.

5. Conclusion

This paper proposes an effective architecture to change the quantity of IS in each OpenStack virtual machine and dynamically manage IS which can improve the resource utilization of STF-CTP. In addition, we propose a scheduling method to improve the utilization of testing devices that can reduce the execution time and average waiting time on overall test. We decrease about 40% memory usage of OpenStack server, increase 3% to 10% Android device usage of STF-CTP, enhance about 80% test job throughput and reduces about 40% test job average waiting time.

In future, we hope that Robot Framework test job can be supported in parallel execution, and improve the test job average waiting time by predicting the execution time of each test job. It can reduce the test job average waiting time by scheduling the test job according to the completion of test job also let IS dynamic can support load balance on the virtual machine.