Optimal Opportunistic Spectrum Access with Unknown and Heterogeneous Channel Dynamics in Cognitive Radio Networks

Abstract

We study the problem of optimal opportunistic spectrum access with unknown and heterogeneous channel dynamics in cognitive radio networks. There is neither statistic information about the licensed channels nor information exchange among secondary users in the respective systems. We formulate the problem of maximizing network throughput. To achieve the desired optimization, we propose a win-shift lose-stay algorithm based only on rewards. The key point of the algorithm is to make secondary users tend to shift to another channel after receiving rewards from the current channel. The optimality and the convergence of the proposed algorithm are proved. The simulation results show that for both heterogeneous and homogenous systems the proposed win-shift lose-stay algorithm has better performance in terms of throughput and fairness than an existing algorithm.

1. Introduction

Cognitive radio (CR) [1] [2] technologies which can solve the problem of spectrum scarcity have drawn great attention. Recently, the opportunistic spectrum access (OSA) [3-4] has been studied in detail in literature because of the good compatibility between the traditional communication systems and the cognitive radio networks. Although considerable innovative research work has been done on distributed OSA systems, there are still some problems that need more effort. First, most of the methods for global optimization in a distributed OSA system require information exchange between secondary users. In these works, the authors always assume and use a cooperation mechanism to achieve their solutions. The secondary users acquire some intelligence through the information exchange, such as other users’ strategies. Yet such extra information exchange brings extra costs in terms of energy and resources. Secondly, the authors also assume that the secondary users have full knowledge about the environment. However, due to hardware limitations, secondary users cannot sense all the channels simultaneously; sometimes even, they choose only one channel to sense. Another important constraint is the time-varying environment. Typically, the statistical information is hard to estimate. In this paper, we focus on distributed OSA systems without information exchange or a priori statistical knowledge about the channels.

There has been one work [9] about a distributed OSA system in the above environment. However, the results in the cited reference show that the throughput performance in heterogeneous environments is not as good as that obtained in quasi homogeneous ones. And yet the heterogeneous environment is more common in cognitive radio networks due to the diversity of primary users. Thus, more research on the distributed OSA in a heterogeneous environment is necessary.

In this paper, we focus on the problem of global throughput optimization in an unknown heterogeneous distributed OSA system. The goal is a challenge, because there is no statistical information available to secondary users about the channel and no information exchange can be used for coordination. The only available information is the history of each user’s channel selection decisions and the rewards received from the channel. Moreover, the environment is time-varying during the entire process, making it hard to obtain the optimized solution.

To solve the problem, we propose a new channel selection algorithm for secondary users. All the users work in a slotted fashion. The key idea is that after each channel access, the winner of the contention will move on to another channel but the losers will continue to stay in the current channel next slot. We name the algorithm with the secondary users’ strategies: the “win-shift lose-stay” (WSLS) algorithm. The algorithm represents an implicit cooperation scheme using the information contained in the rewards instead of the explicit information received from other users directly. The system finally achieves the maximization of throughput.

To summarize, the main contributions of this paper are as follows. We propose a win-shift lose-stay algorithm to achieve optimal opportunistic spectrum access with unknown and heterogeneous channel dynamics. We analyze the convergence process and prove the optimality of the algorithm. We simulate the proposed approach both in heterogeneous and homogeneous systems. The results show its rapid convergence, as well as its improvement on throughput and fairness performance compared with an existing algorithm.

The rest of this paper is organized as follows. In Section II we review the related work. In Section III we present the system model as well as the problem formulation. In Section IV the WSLS algorithm is proposed and proved to be convergent. In Section V simulation and discussion of results are presented. Finally, we present our conclusions in Section VI.

 

2. Proposed Denoising Method

The subject of opportunistic spectrum access has been researched for quite a long time. The authors in [4] and [13] use the Markov decision process to study the OSA system. But their works do not consider the multiple secondary users situation. There are other theoretical methods employed in the study of the distributed OSA systems [19], such as game theory in [5-6] [14-15], optimal stopping rule in [16-17], and self-organization paradigms in [20]. However, these approaches always assume that there is information exchange, or that the channel statistical knowledge is known for secondary users.

The OSA system in unknown environment is studied in [7] [8] and [18] with the multi-armed bandit theory. These works focus on the minimizing the regret instead of maximizing the system throughput. Recently, the authors in [9] proposed a game-theoretic stochastic learning algorithm to solve the problem of an OSA system in an unknown dynamic environment. But their work focuses on the convergence and finally achieves a suboptimal throughput maximization of the whole network. Compared with [9], the proposed approach achieves a higher throughput in a heterogeneous environment. Besides throughput, fairness is another important index for multiple secondary users systems [10]. In this paper, the algorithm achieves an almost near-fair result at the same time with a high throughput.

 

3. System Model and Problem Formulation

3.1 System Model

We assume a heterogeneous OSA system with M licensed channels and N secondary users, N ≥ M ≥ 1 . Each channel transmission rate is Rm , where 1 ≤ m ≤ M . We assume that each channel provides the same transmission rate for all the users. In this way, we ignore the difference of bandwidth or other characteristics when different secondary users access the same channel. We can find the same assumption in some practical systems, e.g., IEEE 802.16d [11]. Moreover, we consider that the primary users utilize the licensed channel in a slotted fashion and the probability of occupying a channel is independent from channel to channel and from slot to slot. In other words, we can define the independent channel idle probabilities with θm , 1 ≤ m ≤ M . To make our approach more realistic, we assume that the system also has the following characteristics [9]:'

1. The channel idle probabilities θm are fixed but unknown for the secondary users. θm is distributed in a large range to make the system heterogeneous.

2. The number of secondary users N is unknown.

3. There is no information exchange between the secondary users, and they are non-cooperative.

4. The system is distributed. In other words, there is no centralized controller.

The transmission structure of secondary users is shown in Fig. 1. As a result of the hardware limitation [4], each secondary user can sense only one channel in each slot. We assume the sensing results are perfect to simplify the problem. When the sensing results are imperfect, we can find the throughput decline, as a consequence of the mis-detection probability and the false alarm probability. In this situation, the throughput can be improved by a more exact sensing result, such as collaborative sensing, but this is a little beyond our topic in this paper. As described in Fig. 1, at the beginning of each slot, every secondary user selects one channel to sense. Then, after sensing, if the channel is idle, secondary users begin to access the channel. We use the carrier sense multiple access (CSMA) mechanism to decide which secondary user can use the channel when there are two or more users selecting the same channel. The winner has the right to transmit the data during the next period. At the end of the slot, each secondary user updates its channel selection strategy.

Fig. 1.Transmission structure of secondary users.

The whole contention is divided into mini-slots. These mini-slots have same length. Users contend for the channel access with same probability in each mini-slot. The winner of the contention is the user which no other user contends with during the same mini-slot. The others are the loser. According to the CSMA mechanism, the length of contention period should vary in different situations. The more users in one channel, the more time it costs to access the channel successfully. But for simplicity, we assume the length of contention time to be a fixed one. Under this condition, every successful secondary user has the same time to transmit data.

3.2 Problem Formulation

Let an denote a channel selection of secondary users n and a-n denote the channel selection of other users. Let cm denote the set of secondary users who select channel m to sense and transmit, i.e., cm = { n ∈ {1,…, N} : an = m} . Let sm denote the number of the set, i.e., sm = | cm | . Let dm (j) denote the state of channel m in j th slot. That is,

According to the assumption of the same transmission time, we define the transmission time as t . The throughput of user n in the j th slot is

where e(n) = 1 stands for secondary user n accessing channel and transmitting data successfully; otherwise, e(n) = 0 . The total throughput of the network is

where a = (a1,a2,…,aN) is a channel selection profile and D = {d1,d2,d3,… dM} is the channel state set. We know that D may have 2M different states. D is unknown before secondary users sense. The channel state set D also keeps changing with time. We rewrite the equation (3) from the point of view of channels as follows:

where δ(sm ) denotes whether there is a secondary user in channel m or not. That is,

From (4), to maximize throughput is to make sure that there is always at least one secondary user on the idle channel. Formally, for m ∈ {1, 2, …, M}, if dm = 1, then δ(sm ) = 1 . The objective is to find the optimal channel selection aopt to maximize Us (a) . That is,

It is easy to see that making the most use of the idle channels in each slot is a challenge because (i) N is unknown in a distributed system and the other secondary user selections are also unknown, and (ii) the D may be different from channel to channel and slot to slot. A distributed algorithm with no information exchange is desirable in this work.

 

4. Win-shift Lose-Stay Algorithm for achieving Global Optimization of Throughput

To solve the problem of global optimization in a distributed OSA system with unknown heterogeneous environment, we have to design a new mechanism. In this situation, the most helpful information is the history of sensing and access. Based on this knowledge, we propose a "win-shift lose-stay" algorithm which converges to a global optimal solution.

4.1 Description of the Algorithm

The proposed algorithm is described as Algorithm 1. The main idea of the algorithm is that secondary users have different channel selections based on the different sensing and accessing results. The updating rules specified by (7) show the different strategies. As described in (7), the winner of contention will quit the channel and shift to another one in the next slot; the losers continue to stay in the current channel next slot. Through such a mechanism, the secondary users in one channel may be divided into two groups: the shifting users and the staying users. In this way, the secondary users will disperse to all the channels. Therefore, no matter what the channel state D will be, we make sure that the idle channels will be selected by secondary users. Thus the algorithm avoids the waste of available channels and achieves the maximization of throughput. Let I0 denote the number of channels which are not selected by secondary users, e.g., I0 = |{m ∈ {1,…, M} : sm = 0}|.

From the description in Algorithm 1, it is easy to find the obvious features of WSLS algorithm: (i) the algorithm just needs the rewards without any other extra information and it is completely distributed, and (ii) there is no an explicit stopping criterion in the algorithm. However, at last, the algorithm goes into a state which may change with slots but always can achieve the maximal possible throughput in each slot, which just satisfies the maximization in objective (6). We will explain the special meaning of convergence in the WSLS algorithm in the next subsection.

4.2 Convergence of the Algorithm

The meaning of the convergence in the WSLS algorithm is different with regard the ordinary acceptance of the term. It is similar with the absorbing state in a Markov process. The final channel selection an is not a fixed one and may vary with slots. We would like to use an example in Fig. 2 to illustrate the special "convergence" of the WSLS algorithm. From the figure, in the first slot, we can see that there are two channels not selected and one of them is idle. This wastes the resource. According to the selection rules described in (7), three users in Channel 2 will shift to Channel 1, and User 5 shifts to Channel 2, while User 2 still stays in Channel 3 in the third slot. We show a random state of the channels from Slot 2 to 6. Based on these states, we show a possible evolution of channel selection. In Slots 5 and 6, we find that all the channels are selected by secondary users. The system can achieve the maximization of throughput. Although the channel selections are totally different in the Slots 5 and 6, they can still be regarded as convergence states, because they make the most use of the idle channel no matter what the channel state D will be. Moreover, there may be some more channel selections that satisfy (6) and they are all the "convergence" states of the WSLS algorithm.

Fig. 2.An example to illustrate the convergence (Slot 1-5 are the convergence process and Slot 5-6 are both the "convergence" state).

We use a turntable model to illustrate the algorithm in Fig. 3. The numbers inside the circle are from 1 to M, which stands for channels. The outside number represents the secondary users selecting the corresponding channel. Each secondary user will make a decision on the channel state dm and the result of contention. From slot to slot, the system varies as the turntable turns step by step with the outside numbers changing. We focus on the number of secondary users in channels and ignore the relationship between channels and secondary users in the proof of convergence.

Fig. 3.Description of the WSLS algorithm using a turntable(numbers inside stand for channels and the ones outside for the secondary users in corresponding channels and the arrow is the SUs moving direction).

Theorem 1: If all the channels of the system are selected by secondary users in some slot, then all the channels will be selected after this slot. Formally,

Proof: Based on the meaning of I0 , when I0 (T ') = 0, it means that all the channels are selected by secondary users. In other words, there is at least one secondary user in each channel. According to the updating rules in (7), we know that whatever the channel state dm is, there is at least one secondary user that will select channel m - 1 in the T ' + 1 slot. Therefore, each channel will be selected in the next slot. The algorithm is still convergent.

Formally, I0 (T ' + 1) = 0. Concluding, I0 (j) = 0, j ≥ T ' .

We have proved the convergence stability of the proposedWSLS algorithm in Theorem 1. Now, we will prove that the system will converge from any random initialization. We classify the number of secondary users (SUs) in one channel sm into four kinds. That is,

The reason to identify number of SUs in each channel is because they play a different part in convergence. N+ contributes considerably to the convergence process. The z0 and z1 show the different trend in convergence process which z0 becomes 0 and z1 becomes 1 or N+ . We use an example to describe (9). We consider a system with ten channels and ten secondary users. One situation of sm may be [1, 4, 0, 1, 0, 1, 2, 1, 0, 0]. Then, using (9), it is expressed as [1, N+ , z0 , 1, z1 , 1, N+ , 1, z0 , z1].

Theorem 2: Consider a system with N secondary users and M channels, N ≤ M . The channel selected by no secondary users will be the one finally selected. Formally, z0 , z1 will turn into 1 or N+ .

Proof: According to (7), if dm (j) = 1 and sm ≥ 1, the secondary users in channel m will be divided into two groups in the next slot. For the z1 in the sequence of z1 , 1, … , 1, N+ , where the number of 1 is k , we define θmin = min{θm , m ∈ {1, …, M}}. We use θmin to represent all θm . We define P(i) meaning z1 turns into 1 after i slots in the worst situation. P(i) is given as follows:

Therefore,

For a fixed θmin , there is always a T 'making q < 1. When i → ∞, P(i) → 0. Therefore, the total probability . In other words, z1 will turn into 1 or N+ at last. In fact, the algorithm does not need a large number of iterations i to turn z1 into 1. This just indicates the inevitability of convergence in the proof. Furthermore, we will find in simulation that the algorithm converges rapidly in simulation.

At the same time when z1 turns into 1, a z0 turns into z1 or 1 which makes I0 decline. The whole convergence process is the process of z1 turning into 1 and z0 turning into z1 or 1 continually. With the assumption of N ≤ M in the system model, z1 and z0 will turn into 1 or N+ at last, and I0 will finally decrease to zero. Based on Proposition 1, I0 will be zero forever.

We have finally proven that theWSLS algorithm converges from a random access situation to a maximization of throughput.

 

5. Simulation and Results Discussion

In this section, we mainly perform a two-part simulation of the WSLS algorithm. In the first part, we illustrate the convergence of the algorithm. We also consider the convergence speed with different secondary users. In the second part, we compare the throughput and fairness of three algorithms: (i) the WSLS algorithm, (ii) the distributed stochastic learning solution proposed in [9], and (iii) an exhaustive approach both in heterogeneous and homogeneous systems. The stochastic learning solution has for its objective to maximize user’s own throughput and finally achieves a NE (Nash Equilibrium) point. The exhaustive approach is a exhaustive search with a central controller that has all the information; this would be the perfect strategy for each user in order to maximize system throughput.

5.1 Convergence of the Proposed WSLS Algorithm

First, we show a given process from its initialization to the convergence state in Fig. 4. We consider a distributed OSA system including ten licensed channels. For different numbers of secondary users, we compare their convergence process. We record the average I0 in each slot of 10000 independent trials. From the figure, we can note that the more secondary users there are, the more rapidly it converges. Even in the slowest situation with ten secondary users, it only takes less than 30 iterations to converge.

Fig. 4.Convergence evolution with 10,15,20 secondary users for 10 channels(θm is randomly generated in the region [0.1-0.9])

Moreover, we consider a distributed OSA system with 10 licensed channels. The number of secondary users increases from 10 to 30. We plot the relationship between the convergence time and the number of secondary users in Fig. 5. The channel idle probabilities θm are distributed in different ranges as shown in the figure. We use some specified data to illustrate the simulation in Table 1. For ten secondary users with θm ∈ {0.1 - 0.9}, the algorithm needs 11.95 iterations to converge. But for 20, it only needs 2.80 iterations. This is because of the algorithm’s random access mechanism. The secondary users have same the probability of accessing each channel. Based on the probability theory, the increase in users will make the distribution of users a uniform distribution, which is close to the convergence state of the algorithm. Moreover, for the same number of secondary users, a higher θm means a higher convergence speed. Therefore, we can conclude that the convergence time decreases quickly with the number of secondary users and with the channel idle probabilities increasing.

Fig. 5.Convergence time with different number of SUs(The system has 10 channels and the SUs increase from 10 to 30. θm is randomly generated).

Table 1.Convergence time for different numbers of SUs.

5.2 Throughput and Fairness Performance of the Proposed Approach

A) The Heterogeneous Systems

In this subsection, we consider OSA systems where the channel idle probabilities vary in a large range. The system contains 10 licensed channels and the channel idle probabilities θm are set as [0.1,0.2,0.3,0.4,0.5,0.5,0.6,0.7,0.8,0.9] to show the heterogeneous characteristics. The transmission rate are set as Rm = 1, m ∈ {1, 2,…, M ). The number of secondary users varies from 10 to 20. For each number, we do 10000 trials independently to obtain the average throughput and fairness performance. We compare the proposed WSLS algorithm, the stochastic learning solution in [9], and the exhaustive solution.

The average throughput with different numbers of secondary users is plotted in Fig. 6. From the figure, we can see that (i) the throughput of WSLS algorithm is higher than the stochastic learning solution [9] and (ii) the proposed WSLS algorithm has the same performance with the exhaustive solution. There is an obvious advantage when 10 secondary users are in the system. Then, the two performance curves both decline but get close. There are two reasons for this. On the one hand, the throughput is an average over all every secondary users. However, the total resource of whole network is fixed, thus the average throughput declines with the number of users increasing. On the other hand, the throughput of the stochastic learning solution gets close to the global optimization with the number of secondary users increasing. However, the throughput of the proposed WSLS is always better than that of the stochastic learning solution.

Fig. 6.Comparison of the achievable system throughput in the heterogeneous OSA system( θm = [0.1,0.2,0.3,0.4,0.5,0.5,0.6,0.7,0.8,0.9]).

We also get the fairness performance in Fig. 7. We use the Jain’s Fairness Index (JFI) [12] to describe the fairness. It is clear that the JFI of WSLS is always staying above that of the other two algorithms. According to the JFI characteristics, if a JFI is greater than 0.9, we can say that it achieves a good fairness. The JFI of the proposed WSLS is greater than 0.99 from the figure which achieves a near perfect fairness.

Fig. 7.Comparison results of the JFI of 10 channel selection schemes in the heterogeneous OSA system( θm = [0.1,0.2,0.3,0.4,0.5,0.5,0.6,0.7,0.8,0.9])

B) The Homogeneous Systems

We perform the above simulation for a homogeneous environment again where the channel idle probabilities are θm = 0.6 m ∈ {1, …, M}. The throughput performance is showed in Fig. 8. From the figure, it is easy to find that the proposed WSLS algorithm has almost the same performance as the exhaustive algorithm and the stochastic learning solution. The fairness results are showed in Fig. 9. The proposed WSLS is almost a straight line. The curves of other two algorithms decrease at first until the number of secondary users is beyond 15. It is easy to explain why there is a depression. For example, the fairness is perfect when there are two channels and two secondary users in the system. When there are three users, two users have to share one channel while the third still owns one channel in two algorithms. However, if there is one more user, it will be fair again where each channel has two secondary users. The number of users increasing may cause the difference of rewards among the users some times.

Fig. 8.Comparison results of the throughput of ten channel selection schemes in the homogeneous OSA system( θm = 0.6 ).

Fig. 9.Comparison results of the JFI of ten channel selection schemes in the homogeneous OSA system( θm = 0.6 ).

It is noted from the above figures that (i) the proposed WSLS converges fast especially with large number of secondary users or high channel idle probabilities, (ii) the proposed WSLS algorithm has the same throughput with the exhaustive solution, and they are both better than stochastic learning solution [9] in heterogeneous systems, (iii) the throughput values of the three algorithms in homogeneous systems are practically the same, and (iv) the WSLS achieves near fair results both in heterogeneous and homogeneous systems.

 

6. Conclusion

In this paper, we study the problem of optimal opportunistic spectrum access with unknown and heterogeneous channel dynamics in cognitive radio networks. The channel idle probabilities and number of secondary users are unknown for secondary users. There is no information exchange among users, either. We formulate the problem of maximizing throughput and propose a win-shift lose-stay (WSLS) algorithm. We prove the convergence and optimality of theWSLS algorithm. It is shown that the WSLS algorithm converges rapidly. Simulation results validate that the proposed approach achieves a better performance of throughput and fairness both in heterogeneous and homogeneous systems.

However, it is also seen that we do not consider the channel switching cost in this paper. In fact, channel switching costs energy and resources. When the channel switching cost is considered, the users would stay in the current channel rather than switch frequently. We will investigate the issue of behavior cost to further improve the system performance in the future.