Reducing the Flow Completion Time for Multipath TCP

Abstract

The modern mobile devices are typically equipped with multiple network interfaces, e.g., 4G LTE, Wi-Fi, Bluetooth, but the current implementation of TCP can support only a single path at the same time. The Multipath TCP (MPTCP) leverages the multipath feature and provides (i) robust connection by utilizing another interface if the current connection is lost and (ii) higher throughput than single path TCP by simultaneously leveraging multiple network paths. However, if the performance between the multiple paths are significantly diverse, the receiver may have to wait for packets from the slower path, causing reordering and buffering problems. To solve this problem, previous MPTCP schedulers mainly focused on predicting the latency of the path beforehand. Recent studies, however, have shown that the path latency varies by a large margin over time, thus the MPTCP scheduler may wrongly predict the path latency, causing performance degradation. In this paper, we propose a new MPTCP scheduler called, choose fastest subflow (CFS) scheduler to solve this problem. Rather than predicting the path latency, CFS utilizes the characteristics of these paths to reduce the overall flow completion time by redundantly sending the last part of the flow to both paths. We compare the performance through real testbed experiments that implements CFS. The experimental results on both synthetic packet generation and actual Web page requests, show that CFS consistently outperforms the previous proposals in all cases.

1. Introduction

In recent years, the production and demand of mobile devices e.g., smart phones, tablet PCs, that have multiple network interfaces, such as 4G LTE, Wi-Fi, and Bluetooth, have increased dramatically. This behavior renders an opportunity to utilize the multiple network interfaces to provide better robustness and efficiency. For example, Apple's iOS 7 supports multipath TCP(MPTCP), as defined in RFC 6824 [1]. Additionally, KT, South Korea’s largesttelecommunications company, has provisioned an MPTCP proxy server to allow users toleverage LTE and Wi-Fi simultaneously [2].

MPTCP is an extension of TCP and uses two or more paths, each as subflows, by extending the single TCP connection. MPTCP is similar to TCP in many ways, but it provides additional features to utilize the multiple network interfaces. For example, if one connection is lost, datacommunication can still continue via the remaining path, providing robustness to link failures. MPTCP can also provide higher throughput, because it can simultaneously leverage multiplepaths. Furthermore, MPTCP provides backward compatibility; if one node does not support MPTCP, the other node can revert to using the legacy TCP connection.

Unlike the traditional TCP, the MPTCP packets are sent to multiple paths, which may havediverse latency, causing out-of-order delivery at the MPTCP receiver. Here, the receivershould store the unordered packets in its receive buffer, and reorder them for orderly reception at the application layer. If the paths have diverse delay performance, which is evidenced in recent measurement studies [3], the receiver buffering can be extreme, causing performancedegradation. To overcome this issue, some MPTCP schedulers tried to predict the path performance beforehand, and redesign the MPTCP scheduler accordingly [4, 5, 6]. Theseapproaches generally show better performance, but if the path latency fluctuates over time, they may suffer from the inaccurate predictions. Recent research has found that two thirds of the Internet latency samples exceeded 100ms of variation, supporting our argument [7].

In this paper, we propose a new MPTCP scheduler called choose fastest subflow (CFS) scheduler. CFS reduces the overall flow completion time by cleverly sending the last part of the flow to both paths. We give some simple numerical analysis to show the benefits of CFS[20-26]. We compare the performance through two types of real testbed experiments, namelysynthetic packet generation and actual Web page requests, and show the CFS performance compared to the previous work.

This paper is organized as follows. In Section 2, we introduce the basic operation principles of MPTCP and it scheduler. In Section 3, we show related work on the MPTCP scheduler. In Section 4, we present our proposal, CFS, and provide algorithms and numericalanalysis. Section 5 describes the experimental testbed setup and Section 6 describes the results of the experiments from the testbed. Finally, in Section 7, we conclude this paper.

2. Background

In this section, we discuss the basic operating principles and problems of MPTCP. First, we describe the basic concept and structure of MPTCP and its default scheduler. Due to the multipath transmission, the out-of-order problem can occur at the receiver’s buffer.

2.1 MPTCP

MPTCP is a relatively new standard that enables nodes to simultaneously utilize various network interfaces. Unlike TCP, which can only transmit data over a single path, MPTCP uses multipath. Therefore, even if one connection is lost, it can still transmit data. It can also provide higher throughput, by simultaneous utilizing multipaths [8]. Fig. 1 shows the basic structure of MPTCP [9]. It is recognized as TCP in the application layer, but when data comesto the transport layer through a socket, it is allocated to multiple paths on different interfaces. Setting up the MPTCP connection is similar to the original TCP three-way handshake: SYN, SYN/ACK, and ACK. The only difference is the MP_CAPABLE option in each packet, which contains a key value to add a new subflow [10]. After the first flow is established, anotherthree-way handshake with the MP_JOIN option is used to add another MPTCP subflow.

Fig. 1. MPTCP Structure

2.2 MPTCP Scheduler

Unlike the legacy TCP, MPTCP transmits data using multiple subflows, thus it needs todetermine which path to use when transmitting each packet. The MPTCP scheduler is responsible for this decision [11]. The default MPTCP scheduler uses the minimal round-triptime (RTT) scheduler that finds a subflow with the lowest RTT. First, it allocates packets to the lowest RTT subflow until the congestion window (cwnd) becomes full, then finds the next fastest subflow, and allocates the remaining packets, and so on. Fig. 2 shows the MPTCP scheduler as defined in the current implementation [12]. In this example, packets 1 to 3 arefirst scheduled to the first subflow with the lowest RTT, then the remaining ones are sent to thenext subflow.

Fig. 2. MPTCP Scheduler [11]

2.3 “Out-of-Order” Problem

Since the packets sent in MPTCP may travel through multiple paths with diverse delay, there is a high probability that the packets will not arrive in order at the receiver. Theout-of-order packets at the receiver have to be buffered at the receiver’s buffer space, since they will not be transferred to the application layer until the delayed packets arrive.

Let us give a simple example. We assume the sender transmits packets 1 to 30 to thereceiver, as shown in Fig. 3, using the MPTCP default scheduler. Per the given minimal RTT(min RTT) scheduler, packets are transmitted first by the lowest RTT subflow (subflowf) untilthe cwnd is full. For example, the first 10 packets are sent to subflowf. Next, the min RTT scheduler transmits the remaining packets (11 to 20) to the next-fastest subflow (subflows). Since subflowf has a much smaller RTT (10ms vs. 100ms), it can transmit the remaining packets (21 to 30) using subflowf, and they arrive at the receiver before packet 11 to 20. Insummary, packets 1 to 10 and 21 to 30 arrive at the receiver, whereas packets 1 to 10 arealready sent to the application layer. Because packets 11 to 20 have not arrived until 50ms, i.e., half of subflows RTT 100ms, packets 21 to 30 continuously occupy the buffer space of thereceiving side. In a severe situation where the two paths’ RTT are diverse, there is a chance that the receiver buffer becomes full, and the fast subflow is no longer utilized until the delayed packets from the slow subflow arrive. This situation is called head-of-line blocking.

Fig. 3. Out-of-Order Problem

To solve this problem, there is an opportunistic method of retransmitting packets from the slow subflow to the faster subflow, called opportunistic retransmission [13]. When the receive window becomes 0 due to the out-of-order packets, the sender perceives that the receiver has caused head-of-line blocking and retransmits the packets that were in the slow subflow at the fast subflow, provided that the fast subflow’s cwnd is not full. However, the opportunisticretransmission is performed only when the buffer on the receiving side is full. When the flowsize is short, the problem persists since the buffer does not become full. Thus, opportunisticretransmission is not activated.

Next, there is a penalization (PR) method that reduces the cwnd of the slow subflow by half to prevent it from being further utilized [13]. However, today's routers and APs have expanded their buffer sizes to prevent packets from being lost during transmission. Thus, small buffered data, including ACKs, may be buffered, causing buffer bloat [11]. Therefore, in the case of PR, it is difficult to restore the reduced cwnd to its original size.

If the performance differences (e.g., RTT, bandwidth) between two paths is small, and the MPTCP is used more than a single TCP, higher throughput can be expected. Thus, large-flow(i.e., large files) can benefit, but short-flow (i.e., smaller files, message services) are moreaffected by delay than bandwidth. Therefore, MPTCP is more sensitive to delays than highband width, in the case of the small files. Therefore, by overcoming the performancedifferences between the two paths, it is possible to improve performance.

3. Related Work

In this section, we discuss some recently proposed MPTCP schedulers that attempt to solvethe aforementioned out-of-order problem.

Delay-Aware Packet Scheduler (DAPS) [4] is a scheduler for solving head-of-line blocking and out-of-order problems when using multiple paths. The next packet to be sent is assigned to a subflow based on the previous cwnd and RTT, so that the packets sent will arrive in order. As the RTT increases, less packets are allocated, and, as the RTT decreases, morepackets are allocated. However, because RTT varies with time [7], if its variation is large overtime, the DAPS scheduler may incorrectly distribute packets.

Fig. 4. BLEST Scheduling [5]

Blocking Estimation-Based MPTCP Scheduler (BLEST) [5] is a scheduler that enablespackets to arrive in order based on the size of the previous transmission’s send window. Themin RTT scheduler sends its cwnd, in order, from the faster subflow to the slower subflow. However, if the send widow is filled by the window of a slower subflow, a faster subflow will not be able to use it. Therefore, BLEST reduces the chance of sending to the slower subflow; itsends the data by allocating as much as the cwnd through the fast subflow’s ACK. For example, in Fig. 4, subflow 2 has a higher RTT than subflow 1. In the case of the minRTT scheduler, the packets are transmitted first to subflow 1’s as much as the cwnd size and then to the slowersubflow 2. If the minRTT scheduler is used as-is, the slower subflow 2 occupies the currentsend window, and the faster subflow 1 cannot transmit data. To prevent this, BLEST'sscheduler allocates as much as the slow subflow’s cwnd. However, when the fast subflow can receive and send ACK, it transmits data only using fast subflow.

The ECF scheduler [6] aims to reduce the completion time of short flows. If there is a large performance difference between the two paths, but the slow subflow is still utilized, then the fast subflow must transmit all of its packets and then wait for the transmission of the packets sent by the slow subflow to be completed. The key idea here is as follows; each subflow’scwnd, the RTT, and the k (i.e., byte unit) remaining in the connection level’s send buffer prior to packet allocation determines whether or not to use the slow subflow. If using a slowersubflow will complete sooner than using a faster subflow, it uses the slower subflow. If using a slower subflow takes more time than the fast subflow, rather than using the slow subflow, it waits for the fast subflow to finish its current transmission, and then send the rest of the trafficto the fast subflow. Like BLEST, the ECF determines whether to use a slower subflow or not. However, the ECF focuses on the packets’ arrival time, not order. For BLEST, the choice is based on the send window. For ECF, the choice is based upon the connection level send buffer, the cwnd of each subflow, and the RTT.

Fig. 5. ECF Scheduling [6]

Fig. 5 shows the basic mechanism of ECF. If the slow subflow takes more time than the fast subflow, ECF waits to use the fast subflow. Thus, ECF is the MPTCP scheduler that shows the best performance when the difference between two paths is large. This method canimprove MPTCP performance, because it can correctly predict network fluctuations in RTT and cwnd, under stable conditions. The work in [6] show that ECF outperforms both DAPS and BLEST. However, when the performance fluctuates over time, the MPTCP scheduler may wrongly predict the path performance in advance. Hence, the performance enhancement diminishes.

In Freeze MTCP [14], the scheduler freezes the slow path if the RTT difference is above acertain threshold, e.g., 100ms. Freeze MPTCP improves the default MPTCP by a large margin. The key idea is simple, but it did not show any comparisons with the previous approaches.

3. Choosing the Fastest Subflow Scheduler (CFS)

The schedulers studied in the previous section generally show better performance than the default MPTCP scheduler by predicting the path performance beforehand. However, due to the dynamic Internet traffic and the recent increase of mobile and wireless traffic, the path performance tends to fluctuate over time [7]. In result, the prediction can become inaccurate, thus resulting in performance degradation. In this section, we present our proposed MPTCP scheduler called CFS that overcomes this problem.

3.1 CFS Algorithm

When sending data via MPTCP, data is first stored in the connection-level send buffer, then after being assigned to each subflow, will be assigned to the subflow-level send buffer. Therefore, if the size of the connection-level send buffer is larger than the sum of the send buffers at each subflow-level, there are remaining data packets that are not yet assigned to thesend-level buffers.

The above algorithm describes the CFS scheduler in detail. The main principle of the CFS scheduler is similar with the existing minRTT scheduler. However, if a fast subflow is available and there is no more data in the current connection-level send buffer, the CFS scheduler will duplicate the data previously given to the slower subflow, and send it to the fastsubflow. This redundant transmission reduces the overall flow completion time, which canimprove performance for delay-sensitive applications. Furthermore, this scheme is immune to the link quality fluctuation since it does not require any prediction on each subflow path. Ingeneral, if the number of flows is N, the above algorithm gives a computational complexity of O(N).

Note that CFS does not require computation or prediction as in ECF [6]. The main reasons are as follows. First, RTT indicates the time between the sent packet and the received ACK. Therefore, the RTT of a slow subflow must be larger than that of the faster subflow. Next, Eq. (1) compares the time it takes to transmit the send buffer’s remaining data of the connection level by the cwnd of each subflow. The cwnd increases with each received ACK, and, in the case of a slow subflow, the RTT is larger than the RTT of the fast subflow. Thus it is difficult to exceed its cwnd. Therefore, when the faster subflow is used, per Eq. 2, the entire flow completion time can be reduced by repetitively transmitting the packet given to the slowersubflow.

3.2 Numerical Analysis

We give some numerical analysis to show the effectiveness of our proposal. CFS gives benefit only when the RTT of the fast subflow (RTTf) is smaller than that of the slow subflow(RTTs) as shown in the previous subsection. We give the RTTs of the slow and fast subflows as random variables following the exponential distribution: Xs and Xf, with mean values of 1/μsand 1/μf, respectively. We compute the probability when the RTTs may become faster that RTT fas follows:

Pr < = ∫ Pr( < )

∞ ( ) (1)

where ff(y) represents the probability density function (pdf) of Xf. The above equation (1) can be solved as follows:

Pr < = ∫ (1 − − ) & minus;∞=+ (2)

We plot the results in Fig. 6. We set the 1/μf =20ms with varying 1/μs, from 30ms to 300ms. This is consistent with the experimental settings in Section 4. The results show that the probability of the slow subflow overtaking the fast subflow is relatively low, especially in practical conditions where the LTE’s RTT is much larger than that of Wi-Fi [3]. When the LTE’s RTT is 200ms, then the probability becomes less than 10%. In summary, there is a very high probability that the duplicate transmission of the fast path will complete before the slow path, showing the benefit of CFS.

Fig. 6. Probability of RTTs < RTTf as a function of 1/μs.

3.3 Discussions

The potential drawback of our approach is when there is no advantage in duplicating and transmitting the last packets to the fast path. Specifically, when the completion time of the slow path is shorter than the duplicate transmission of the fast path, then the CFS mechanism would have no benefit. The drawback is that the several final packets have been transmitted twice with no edge.

First, according to the analytical results of Section 3.2, there is a very high probability that the duplicate transmission of the fast path will complete before the slow path. So CFS will beadvantageous in most cases. Second, even though CFS may transmit duplicate packets in vain, albeit with small probability, this causes minimum damage since it is only done for the final few packets of the flow. This argument is also verified in the experimental results shown in Section 5.

4. Experimental Setup

In this section, we present the experimental setup. We used the Hansung U54X notebookas the MPTCP client and installed Ubuntu Linux 14.04, which built the MPTCP version 0.89 on the Linux kernel, for the evaluations [12]. The server is also built with MPTCP kernelversion 0.89 on Ubuntu Linux 14.04 on a general PC. The experimental network topology is shown in in Fig. 7.

The MPTCP client is connected to two interfaces, namely Wireless LAN (WLAN) and LTE. WLAN uses the IEEE 802.11ac [15] interface, and the LTE is emulated by using NETEM (i.e., network emulation) [16] over Ethernet. We have also used NETEM to adjust therate and RTT of both WLAN and LTE as shown in Table 1. Similar to our settings, the RTT variation can be very large in real life over time [7]. We have implemented ECF [6] on each client and server [17], as well as our own proposal, CFS for comparison purposes.

The experimental method is as follows. In the first experiment, we use synthetic packet generation; flows of various sizes, 8 types, were randomly generated by the client using socket programming, and flow completion time at the server was measured and compared using each scheduler. We further classify the flow types into 4 types of small flows and 4 types of long flows. In the second experiment, we stored the actual Web page on the MPTCP server and measured and compared the time taken by the client to request it after using Linux’s w getapplication [18]. We measured and compared the flow completion time taken of the client torequest the web page. The final experiment is similar to that of wget. We compared the actual performance with the curl application [19] used to parse the web page.

Fig. 7. MPTCP experimental setup

Table 1. Rate and RTT controlled by NETEM

5. Performance Evaluation

5.1 Synthetic Packet Generation Experiment

Clients synthetically generated flows of different sizes and sent them to the server. Then, itmeasured the overall flow response time. Fig. 8 shows the short-flow of flow sizes of 64 KB, 128 KB, 256 KB, 512 KB, and 1 MB, repeating 30 times for each flow size.

Fig. 8. Completion time for short flows - various traffic size (s)

In Fig. 8, the horizontal axis represents each MPTCP scheduler and the vertical axis represents the flow completion time. The lower the flow completion time, the faster theresponse time for the application user. Overall, CFS has a low completion time compared to the default MPTCP scheduler [12] and ECF [6]. If the latency difference between the twopaths is large and the network performance changes frequently over time, the predicted valuemay be inaccurate for ECF. By utilizing the characteristic of these paths, the CFS lowers the completion time by duplicating the slow subflows’ data onto the fast subflows.

Table 2. Average completion time for short flows (sec)

Table 3. Reduced percentage of average completion time over the default scheduler for short flows

Table 2 shows the average value of each experiment shown in Fig. 8. Furthermore, Table 3 shows the reduced percentage of completion time for ECF and CFS over the min RTT scheduler. As shown in Table 3, the performance is similar or slightly higher than that of ECF. In particular, when the flow size is 256KB.

We note that the main focus of our work is to reduce the completion time of the flows. Thus, the completion time is meaningful for relatively short flows. However, we next conductexperiments on the large flows (i.e., 4 MB, 8 MB, 16 MB, 32 MB), similar to the work in [3]. In the case of large flows, as the traffic increases, it may cause redundant transmission reducing the overall performance. However, at the end of the flow, there is no impact on performance, because it is redundant only if there is no more data to send to the send buffer at the connection level. Our argument is shown in Fig. 9. There is not much difference between performance of the existing scheduler and the ECF. The average values for these experiments are shown in Table 4.

Fig. 9. Completion time for large flows- various traffic size

Table 4. Average completion time for large flows (sec)

5.2 Actual Web Page Request Experiment

In this experiment, we use real web applications for performance comparisons. We measured the total flow completion time when the client requests the Web pages from the MPTCP enabled web server. The MPTCP server’s actual web page data set is shown in Table 5.

Table 5. Web file size

We then measured the overall flow completion time using two applications (i.e., w get, curl), used when retrieving web pages or data from a Linux server.

Fig. 10. Objection download average completion time - various file size (sec) Table 6. Object download average completion time (sec)

First, we measured flow completion time of each web page stored on the web server using wget [18]. Wget is a free software package for retrieving files using widely used Internet protocols such as HTTP and HTTPS. It shows a traffic pattern similar with the general web-surfing via a web browser. The experiment was repeated 30 times for each web page. Fig. 10 shows the flow completion time (i.e., time between the web page request and receipt of data) using each MPTCP scheduler. The horizontal axis represents each MPTCP scheduler, and thevertical axis represents the average value of the flow completion time 30 tests. Overall, the CFS scheduler has a lower flow completion time than the minRTT scheduler and ECF. Table 6 summarizes the mean values for the figures. The table also shows that CFS has slightly betterperformance than the minRTT or ECF.

Fig. 11. Web-parsing average completion time - various file size

Table 7. Web parsing average completion time (sec)

Next, we measured the flow completion time using curl [19]. Curl is similar to wget, that it downloads data, but curl supports libraries and various protocols. The experiment was repeated 30 times for each web page. Fig. 11 shows the overall flow completion time of parsing the web page. The horizontal axis represents each scheduler and the vertical axis represents the average flow completion time of each scheduler. The minRTT scheduler and ECF have a slightly higher flow completion time than CFS. Table 7 summarizes their mean values.

In summary, CFS outperforms the previous schedulers for real web traffic as well assynthetic traffic, due to its ability to improve the completion time without the delay prediction.

6. Conclusion

In this paper, we proposed a new MPTCP scheduler, called choose fastest subflow (CFS) scheduler, that copes well in general environments where the two paths have diverse and dynamic performances. CFS utilizes the characteristics of these paths to reduce the overall flow completion time by redundantly sending the last part of the flow to both paths. We present some numerical analysis to show that, there is a very high probability that the duplicatetransmission of the fast path will complete before the slow path, showing the benefit of CFS.

We conducted evaluation through two types of real experiments, synthetic packet generation and actual Web page requests. We compared CFS with the current MPTCP implementation on the Linux kernel, and also with the state-of-art scheduler, ECF. The experimental results showed that CFS consistently outperforms the previous proposals in all cases.