Reinforcement Learning-Based Intelligent Decision-Making for Communication Parameters

Abstract

The core of cognitive radio is the problem concerning intelligent decision-making for communication parameters, the objective of which is to find the most appropriate parameter configuration to optimize transmission performance. The current algorithms have the disadvantages of high dependence on prior knowledge, large amount of calculation, and high complexity. We propose a new decision-making model by making full use of the interactivity of reinforcement learning (RL) and applying the Q-learning algorithm. By simplifying the decision-making process, we avoid large-scale RL, reduce complexity and improve timeliness. The proposed model is able to find the optimal waveform parameter configuration for the communication system in complex channels without prior knowledge. Moreover, this model is more flexible than previous decision-making models. The simulation results demonstrate the effectiveness of our model. The model not only exhibits better decision-making performance in the AWGN channels than the traditional method, but also make reasonable decisions in the fading channels.

1. Introduction

In recent years, the rapid development of Internet of Things (IoT) and Vehicle Networks have made the requirements of communication quality even higher. At the same time, the exploration of extreme communication environments, such as polar and space, also raises higher demands on the environment adaptability of communication systems. As the current research trend of wireless communication, the concept of cognitive radio (CR) was first proposed by Joseph Mitola in 1999 [1], which is a further development based on software radio (SR) [2]. Currently, the research on CR primarily focuses on the spectrum, including spectrum sensing, dynamic spectrum allocation, and spectrum management [3,4]. However, CR should not remain in the "dynamic spectrum using radio" [5]. In our opinion, the next generation intelligent radio systems should be able to sense and extract effective information from the complex channels, then it would be able to make decisions based on the perceived results, and intelligently generate the waveform parameter configuration that is most suitable for the current channel [6]. Finally, the reliable and efficient transmission of information can be achieved. In this paper, we aim to study the intelligent decision-making of communication parameters which is the core component of the above process.

The decision-making of the waveform parameter can be considered as a further expansion and evolution of traditional adaptive modulation and coding (AMC) [7]. However, intelligent decision-making methods do not require the complex analysis and calculation of signal to noise ratio (SNR) thresholds in AMC, and are not restricted by the limited number of thresholds. RL-based decision-making can interact with the environment and does not rely on background knowledge and prior information. It has strong learning ability and flexibility. so that it can be applied to any channel condition. This paper proposes a new intelligent decision-making model by introducing the idea of RL. The complete decision-making framework is shown in Fig. 1 [8].

Fig. 1. The proposed decision-making model framework based on RL.

The previous decision-making models pay too much attention to the complicated analysis and modeling for channels. However, in the complex channel, for example, the bit error rate (BER) of different modulation and coding is difficult to be calculate. In this paper, we want to solve the problem from a new perspective. We do not care about the specific characteristics of the channel, but directly obtain the real feedback of the channel through the on-line interaction between reinforcement learning (RL) and the environment. Compared with previous decision-making models, our model has the following advantages:

•The concise state-action pairs and the modeling method avoid the large-scale problem and can converge faster.

•The model has low complexity and high efficiency, which is suitable for practical applications.

•The model is not limited by conditions and is applicable in any channel environment.

We can find the optimal parameter configuration for the communication system in any channel, and the model does not depend on any prior knowledge. In practical applications, the RL-based method may exhibit poor performance in the initial exploration stage. In the future, we will continue to complete the model to solve this problem. We will design a classifier to store and utilize the existing decision-making results reasonably to guide future decisions.

The remainder of this paper is organized as follows: The related work about communication decision-making is presented in Section 2. Section 3 gives a brief overview of RL theory. Then we describe the communication system used in the proposed model in Section 4. Based on the above, we discuss the problems of existing models and our improvement idea, then propose a new RL-based decision-making model in Section 5. The simulation results are shown and discussed in Section 6. Finally, the paper is concluded in Section 7

2. Related Work

We will classify and introduce the decision-making methods according to their dependence on prior knowledge form high to low. The earliest expert system model is enhanced by continuous off-line simulation, and ultimately provides decision rules that can be used on-line to adapt the radio equipment to various environments [9,10]. However, the complex and varied channels in practice bring about great challenges to the design of expert systems. Since the system relies entirely on the prior knowledge, it can be difficult for it to work beyond its knowledge reserve.

The idea of optimization-based decision-making is to transform the decision-making problem into a multi-optimization problem. Based on the designed objective function that characterizes the communication performance, genetic algorithm (GA) performs an optimal search within a configurable range, and finally obtains the optimal parameters. Rieser proposed a biologically inspired cognitive radio framework based on GA, and then carried out simulation and hardware testing [11]. The related research, such as the cognitive decision engines based on ant colony optimization (ACO) [12], bacterial foraging optimization [13], etc., have mainly focused on improving the optimization performance of the algorithm. In addition, there are cognitive decision-making based on binary chaotic particle swarms (BPSO) [14], hill climbing genetic algorithm [15], and other algorithms engine. However, most of the researches in this area concentrate on the ideal additive white gaussian noise (AWGN) channel. When the real channel is complex, it is difficult for us to make accurate theoretical formulas and apply the decision engine. Moreover, the process of the optimization search requires a lot of time.

The ways of off-line learning solve the problem can be roughly divided into two categories. One method is collecting a large amount of data under the guidance of theoretical analysis and previous experience to train classifiers that can obtain output parameter configuration according to the input channel condition. The algorithms include case-based reasoning (CBR) [16], neural network (NN) [17], support vector machine (SVM) [18,19], etc. This raises a concern: when the deviation between the input channel and the historical training set is large, the output result will not be reliable. The other method is to introduce the idea of regression prediction into decision-making. Dong used neural networks to predict performance BER [20]. Dou proposed a NN-based predictive model by randomly measuring the performance of part of the parameter values [21]. Thus, the off-line learning methods require many historical cases and are not suitable for guiding decision-making individually. The learning universality and real-time performance are both poor.

On-line learning decision-making is mainly achieved by RL [22]. RL interacts with the environment, obtains environmental information and reinforcement signals, and learns based on practical experience [23]. RL is very suited to solve decision-making problems because of its inherent characteristics. RL was originally used for the spectrum to solve the problems of spectrum congestion and insufficient spectrum utilization [24,25]. Later, RL was used for the parameter decision-making problem [26,27]. However, there are still some problems in these existing applications: they are still following the idea of optimization based decision-making and do not make full use of the interaction between the RL algorithm and the environment. We will analyze these RL-based models and propose our improved decision-making model in Section 5.

3. Background On Reinforcement Learning

In this paper, we just introduce some necessary conceptions to assist readers to easily understand the proposed model. The clearly detailed description and mathematical derivations are provided in [28].

We describe RL model as shown in Fig. 2.

Fig. 2. A typical reinforcement learning interaction model.

At each step s_m =1,2,3,4,5,6 , the agent executes action A_t ,and receives observation S_t and scalar reward R_t .The environment receives action A_t , then emits observation S_t+1 and scalar reward R_t+1 . The agent's job is to maximize the cumulative reward. RL algorithms can be categorized into model-based and model-free according to whether the agent attempts to build a model. In model-free algorithms, the agent interacts directly with the environment to estimate and optimize the value function. The algorithms can be categorized into Monte-Carlo (MC) learning, Temporal-Difference (TD) learning, and TD( λ ) learning.

Model-free algorithms learn directly from episodes of experience under the condition of having no knowledge of MDP transitions /rewards. MC methods must learn from complete episodes, which means that MC learning is not bootstrapping. The main idea of MC is to replace the value with the mean return. However, TD learning can learn from incomplete episodes, which means that it is bootstrapping. In MC learning, we update the value V (S_t) towards the actual return G_t :

V (S_t) ← V(S_t) + α (G_t - V(S_t))       (1)

Where α is the updated factor. In TD learning, we update the value V (S_t) towards the estimated return R_t+1 + γV (S_t+1) :

V (S_t) ←V (S_t) + α(R_t+1 + γV (S_t+1) - V (S_t))       (2)

R_t+1 + γV (S_t+1) is called the TD target, and (δ_t = R_t+1 + γV (S_t+1) - V (S_t)) is called the TD error.

By comparing MC learning and TD learning, we find that TD can learn before knowing the final outcome and learn online after every step, but MC must wait until the end of the episode. TD has low variance and is usually more efficient than MC. TD is also more efficient in Markov environments because it can exploit the Markov property.

Sarsa and Q-learning are two classic TD algorithms. Sarsa usually chooses a more conservative strategy compared with Q-learning. For example, in a trapped environment, Sarsa usually chooses the safest route away from traps. Q-learning, however, is bolder and more efficient, it generally chooses the fastest route. In real-time decision-making. problems, efficiency is a very important element. We choose the efficient and practical Q-learning algorithm to build the proposed model. The convergence of Q-learning can be proven.

Q-learning is a kind of off-policy learning algorithm. The core idea of off-policy learning is learning about policy π from experience samples from another policy µ :

\(\begin{aligned}V\left(S_{t}\right) \leftarrow V\left(S_{t}\right)+\alpha\left(\frac{\pi\left(A_{t} \mid S_{t}\right)}{\mu\left(A_{t} \mid S_{t}\right)} \cdot\left(R_{t+1}+\gamma V\left(S_{t+1}\right)\right)-V\left(S_{t}\right)\right)\end{aligned}\)       (3)

These methods can also re-use experience generated from old policies and learn about the optimal policy following the exploratory policies. Q-learning is one of the best algorithms to apply the off-policy idea. The key component of Q-learning is that when the agent updates the value of Q( S_t, A_t ), it uses the next state-action value of the target policy instead of the value of the current behavioral policy. The updated equation is given as (4):

Q( S_t, A_t ) ← Q( S_t, A_t ) + α[R_t+1 + γQ( S_t+1, A_t ) - Q( S_t, A_t)]       (4)

In (4), the behavioral policy α is the ε − greedy policy and the target policy γ is the greedy policy. ε − greedy is the simplest idea for ensuring continual exploration. It makes all m actions tried with none-zero probability in any state. Under ε − greedy exploration, the agent chooses the greedy action with probability 1− ε and chooses an action at random with probability ε .

\(\begin{aligned}\pi(a \mid s)=\left\{\begin{array}{ll}\varepsilon / m+1-\varepsilon & \text { if } \mathrm{a}^{*}=\arg \max _{a \in A} Q(s, a) \\ \varepsilon / m & \text { otherwise }\end{array}\right.\end{aligned}\)       (5)

In this way, the agent can converge to the optimal state-action value function while ensuring that it sufficiently experiences reaching new states.

4. Background on Reinforcement Learning

We consider an orthogonal frequency division multiplexing (OFDM) system in our model design because of its unique efficiency and flexibility. OFDM is a special multicarrier transmission scheme that can be viewed as a modulation technique, and it is also a diversity technology. Because of the orthogonality of OFDM subcarriers, OFDM can overcome the shortcoming of low spectrum utilization in general multi-carrier modulation. Meanwhile, multi-carrier transmission can resist frequency selective fading and narrowband interference to some extent. OFDM technology has been extensively used in various communication scenarios.

Another feature of OFDM is the ease of the control of the modulation and coding scheme. We can arbitrarily configure any M-PSK or M-QAM modulation mode in the transmission process. Because the subcarriers are easily allocated and the modulation mode is easy to be changed, the OFDM system is suitable for building a parameter configurable decision-making model. In the proposed model, we presently stipulate that each subcarrier is modulated with the same modulation mode. In future research, we will make more in depth decisions on the parameter configuration of each subcarrier according to the channel and spectrum environment. The convolutional code is also selected as the way of channel coding in our system. Additionally, a cyclic prefix of the appropriate length is inserted as the guard interval in each OFDM symbol so that the inter-symbol interference (ISI) can be eliminated. More specific details of the parameter configuration will be provided in the simulation section.

In this paper, we discuss not only the additive white Gaussian noise (AWGN) channels, but also the fading channels. We assume that the channel does not vary within one OFDM symbol. At the receiver, we do ideal estimation and equalization in the fading channels to avoid the impact of different performance of different channel estimation algorithms. In the decision-making model, the channel estimation algorithm is not the focus of research. In practical applications, we can choose the appropriate algorithm according to different communication environments and needs.

5. RL-Based Decision-Making Model

In this section, we propose a waveform parameter decision-making model with a new idea based on the analysis of the relationship between digital communication system performance and communication waveform parameters.

The decision-making problem of communication waveform parameters can be simplified as choosing a best performing parameter configuration to adapt to the current channel. Therefore, the problem can be divided in to three key parts, the relationship among them is shown in Fig. 3:

Fig. 3. The disassembly of the waveform parameter decision-making problem.

The parameters to be decided: Our research is based on the digital communication system. By analyzing a typical digital communication link and its modules, we find that The modulation scheme and the coding rate of channel coding have the most influence on the communication performance.

The environmental factors affecting decision-making：Various adverse factors in the Channel impact on the communication performance, and mostly include noise, interference, and fading. We can estimate these factors by channel estimation and spectrum detection.

Theoretical basis for decision-making：The basis for the decision-making problem is the way to measure the communication performance. The performance of a digital communication system can be described in terms of both effectiveness and reliability. BER is the most intuitive parameter to indicate reliability, and the transmission rate is usually used to describe effectiveness.

Based on the above analysis, we select the modulation order m and the coding rater as the parameters to be decided in the OFDM system. Meanwhile, the OFDM Transmission rate and BER are the indicators for decision-making. However, effectiveness and reliability are two opposite performance indicators．In the same channel environment, when m and rate increased, the values of the BER and transmission rate both increase. This also denotes that when the effectiveness of the system is getting better, the reliability becomes worse. Therefore, the communication performance cannot be absolutely optimal. If the effectiveness is supposed to be improved, it will inevitably lead to a decline in the reliability. In the decision-making model, we can merely make trade-offs based on the actual demands.

At present, the research on RL-based waveform parameter decision-making still follows the idea of optimization-based decision-making, such as paper[29],[30], etc. The decision-making process of this model type is described in Fig. 4.

Fig. 4. Previously typical RL-based decision-making model.

In their model, the state is defined as the current channel information and the action refers to selecting one modulation and coding scheme from the optional range. The reward function still follows the multi-objective function in the optimization-based decision-making:

f_{decision max} (x) = ω₁f_max (R) + ω₂f_min (BER) + ω₃f_constant (W)       (6)

As described in (6), the goals for their system are: maximizing rate R , minimizing BER and keeping the bandwidth W constant. In [29], they also mentioned the problem that attempting to achieve multiple goals at the same time can cause the competition. Their approach to minimize this dispute is using weight. In every episode, they send training sequences by using the current parameter configuration， and then they can obtain the optimization function value as the reward. Applying RL to a decision-making problem in this way is a single-step problem, and it simplifies the episode to only one step. Thus, they modify the updated (4) as:

Q( S_t, A_t ) ← Q( S_t, A_t ) + α[R_t+1 - Q(S_t, A_t)]       (7)

However, RL essentially concerns a sequential decision-making problem, timing, which is a key factor in RL. Obviously, this type of application does not make full use of the interaction between the RL agent and the environment. It may need to try all the optional parameter configurations once to make a decision. When there are many optional parameter values, this kind of model will be difficult to converge and may suffer from limitations in a large-scale search.

Inspired by the classical application of RL in the maze games, we design a new RL-based decision-making model, which is represented in Fig. 5. We want to solve the decision-making problem in a new way by making full use of the interactivity of RL.

Fig. 5. A new application mode of RL in waveform parameter decision-making.

We desire to simulate the direct process of human making decision. Alternatively, when trying to make decision for waveform parameters, we first select one parameter configuration arbitrarily. We then try to increase or decrease the parameter values and determine the direction of next action according to the communication performance after changing parameter value. The decision-making problem is analogous to the problem of finding the shortest path to the optimal parameters. It is very similar to the classical maze problem in RL. Therefore, in our model, no matter where the parameters locate initially, they will eventually arrive at the optimal position. In the context of this idea, RL model can be mapped as follows.

In our new RL-based decision-making model, we set the state S to the current waveform parameter configuration, including the modulation order and the coding rate. The states of the modulation mode consist of BPSK, QPSK, 8QAM, 16QAM, 32QAM, and 64QAM, which means the modulation order m is from 1 to 6, and we number them as s_m = 1,2,3,4,5,6 . The state of the coding rate c include 1/3, 1/2, 2/3, and 1, which are numbered as s_c = 1,2,3,4 .

The action A in the proposed model is the changing direction of the parameter, which means whether we increase or decrease the modulation order or the coding rate. In order to build the model, we summarize the three optional actions as adding or subtracting s_m and s_c by 1 or keeping the value unchanged. This modeling way avoids too much complexity and too long exploration process caused by the excessive actions.!!!

The most important part of the model is the reward R, which determines the goal of the decision-making. We set the reward as the communication performance value at the next time after the agent takes an action. As mentioned above, the BER and transmission rate are considered as the digital communication performance in this model. They are contradictory and cannot be improved simultaneously. Similar to (6), in the optimization-based decision-making, they want to solve this problem by introducing a set of weighting factors:

\(\begin{aligned}f(x)=\sum_{i=1}^{k} \omega_{i} f_{i}(x)\end{aligned}\)       (8)

where f_i(x) is the optimization sub-objective. The weighting factors can take different values for different application scenarios. However, there is no reference telling us how to choose the weight. This is difficult to measure for the multi-objective optimization. In this case, we might as well simplify the optimization objective as obtaining the peak balance between BER and transmission rate. In this paper, we establish the performance equation with reference to the general throughput formula in AMC:

η = (1-P_e) ⋅ ρ_i ⋅ log₂ M_i       (9)

where ρ_i is the data rate and M_i means the number of modulation constellations. On this basis, we modify (9) to adapt to our actual modeling situation. Specifically, the reward equation suitable for our model combining the BER and transmission rate is shown in (10)-(12).

C = C_ber ⋅ C_rate       (10)

\(\begin{aligned}C_{\mathrm{rate}}=\sqrt{m \cdot\left(r_{c} \cdot 6\right)}\end{aligned}\)       (11)

C_ber = -log₁₀ ( ber )       (12)

C_ber and C_rate respectively represent the reliability and the effectiveness. ber is the actual bit-error-rate value, which is measured by the power of 10. m is the modulation order which has the same meaning as log₂ M_i in (9) and r_c is the coding rate.

As we are only concerned about the order of the magnitude of ber, we take the logarithm of ber and adopt the opposite value as the indicator of reliability in (12). The greater the value of C_ber is, the greater reliability the system exhibits. In addition, we make two constraints to ber due to the limited precision:

•When ber > 0.1, we consider that the BER is beyond the scope of tolerance and record it as 1.

•When ber < 10⁻⁶ , we consider that the BER reaches the ideal state and record it as 10^-6.

The value of the modulation order m varies from 1 to 6, and the coding rate r_c ranges from 1/3 to 1. In order to balance the proportion of these two factors in the transmission rate, we expand r_c proportionally. We then multiply them together and obtain the square root of this product as C_rate to maintain an equilibrium with C_ber. The system performance C is jointly represented by numerically balanced C_ber and C_rate, as (10). In the practical application, (10) can be adjusted according to the actual situation. Therefore, the reward R can be expressed as R = C_t+1 .

Following the analysis of RL algorithms in this section, we use the Q-learning algorithm to implement the proposed model. Algorithm 1 demonstrates the procedures to build a decision-making model based on Q-learning, indicating the entire decision-making process. In the following algorithm, a_m and a_c are the optional actions in the decision-making.

Algorithm. 1 Decision-making model based on Q-learning

Initialize Q( s_m,a_m ) and Q( s_c,a_c ) arbitrarily, for all ∀s ∈ S, ∀( s_m,s_c ) ∈ S, ∀a_m ∈ A( s_m ) , ∀a_c ∈ A( s_c ) .

Repeat

(for each episode) Initialize S_m, S_c .

Repeat

(for each step of episode) ( n_m and n_c are alternately decided, A = A_m or A_c , S = S_m or S_c .

Choose A from S using policy derived from Q(eg., ε − greedy) .

Take action A ,observe R , S′ .

\(\begin{aligned}Q(S, A) \leftarrow Q(S, A)+\alpha\left[R+\gamma \max _{a^{\prime}} Q\left(S^{\prime}, a^{\prime}\right)-Q(S, A)\right]\end{aligned}\)

S ← S'

Until episode is end.

Until S are terminal

6. Simulation Results and Discussion

In this section, a set of simulation results are exhibited to analyze the performance of the proposed decision-making model for various variable configurations. We then choose the optimal variable configuration for Q-learning, and demonstrate its performance compared to that of the traditional model. The main parameter configuration as shown in Table 1.

Table 1. The main parameter configuration for simulation channels and the OFDM system.

Fig. 6 shows that the final decision-making result for the AWGN ( E_b / N₀ = 0dB ) channel. The left graph of Fig. 6 presents the decision curve for the modulation order. The simulation result demonstrates the modulation order gradually converges at s_m = 2 , where the modulation mode is QPSK. Similarly, the right graph illustrates that the coding rate ultimately converges at s_c = 1 The small fluctuations in the decision curve are due to the exploratory character of the ε − greedy policy and the closest suboptimal parameters. In order to show the decision-making process more clearly, we also take AWGN ( E_b / N₀ = 0dB ) channel as an example to present the learned Q-value tables for the decisions.

Fig. 6. The decision curves for the AWGN ( E_b / N₀ = 0dB ) channel based on Q-learning.

In Table 2 and Table 3, Each state S corresponds to three optional actions A. The bold data in the table represents the most valuable one of the three actions which indicates the final decision-making result. We get to know that no matter which initial state s_m and s_c are, they will eventually reach the optimal states, s_m = 2 and s_c = 1 , and this is consistent with the decision result shown in Fig. 6. The position with Q-value of "0" in the table indicates that this state-action pair does not appear in the exploration process, and the Q-value of other states not shown in the table is also "0". It illustrates that the RL-based model can avoid the exploration of some invalid state-action pairs and improve the decision-making efficiency. Therefore, the final decision-making result for AWGN ( E_b / N₀ = 0dB ) is (QPSK, r_c = 1/3 ).

Table 2. Q-value table for the modulation-order decision in the Q-learning model

Table 3. Q-value table for the coding-rate decision in the Q-learning model

Fig. 7 shows that the final decision-making result for the AWGN ( E_b / N₀ =10dB ) channel is s_m = 4 , s_c = 3 , which denotes (16QAM, r_c = 2 / 3 ). Similarly, the decision-making result for the AWGN ( E_b / N₀ = 20dB ) channel is (64QAM, r_c = 2 / 3 ), as shown in Fig. 8. By analyzing the decision-making results, we can find that the modulation order and the coding rate increase with the increase of the E_b / N₀ value. This demonstrates the rationality of the results.

Fig. 7. The decision curves for the AWGN ( E_b / N₀ = 0dB ) channel based on Q-learning.

Fig. 8. The decision curves for the AWGN ( E_b / N₀ = 20dB ) channel based on Q-learning.​​​​​​​

Fig. 9. Convergence performance comparison between the two decision-making models​​​​​​​

Firstly, taking the decision-making in AWGN ( E_b / N₀ = 20dB ) channel as an example, we compare the convergence performance of BPSO、the existing RL-based decision-making model mentioned in Section 5 and the new RL-based model proposed in this paper. The specific parameters of the communication system are also configured according to Table 1. The simulation results are the average of 500 Monte-Carlo experiments.

Comparing these three convergence curves in Fig. 9, the BPSO-based decision engine optimization process is relatively gentle and slow, and it converges after about 150 iterations. the existing RL-based decision-making model converges at about 75 times. The RL-based decision model in this paper gradually converges after exploring about 40 times. The existing RL-based decision model has a large number of actions, which leads to a large exploration space and a long time to complete the exploration. However, the state-action pairs of the new RL-based decision model is simple, avoiding the large- scale exploration problem. It makes the decision process more effective and efficient. The initial mean square error (MSE) of the convergence curve of the new RL-based decision model in Fig. 9 is large because the model is set to start with s_m = 1 and s_c = 1 for each decision-making process.

We continue to simulate our RL-based decision-making model in the AWGN channels with various E_b / N₀ . values, and compare the decision-making results with the MCS method

The decision-making results we obtained in the ideal AWGN channels with different E_b / N₀ , values using the RL-based model and MCS mapping respectively are shown in Fig. 10. From Fig. 10, it is evident that the performance of the RL-based model is not inferior to the MCS method. This means the proposed model can provide reasonable decision-making results. The difference in performance between the two methods is primarily due to the limited thresholds of the MCS mapping. The coarse discretization of the channel state may lead to performance loss in specific regions. Nevertheless, the RL-based model does not need prior knowledge or extensive simulations over different scenarios to obtain the E_b / N₀ thresholds.

Fig. 10. Convergence performance comparison between the two decision-making models​​​​​​​

In this subsection, we further compare the performance of the RL -based model and the MCS mapping in typical fading channels. We simulate three fading environments defined by 3GPP TS36.104 protocol in LTE standard for different application scenarios, including extended pedestrian A (EPA) channel, extended vehicular A (EVA) channel and extended typical urban (ETU) channel by using the Jakes model based Rayleigh-fading channel. In Table 4, we list the specific parameter values of each channel model.

Table 4. Channel parameter values including excess tap and relative power for each path.​​​​​​​

Fig. 11 respectively present the decision-making performance curves of the RL-based model and MCS in three kind of Rayleigh- fading channels with different values. Compared to Fig. 10, there are apparent inconsistencies between the two decision-making methods. Moreover, from the EPA channel to the ETU and EVA channels, the fading becomes severe, and the performance of MCS is even less able to keep up with the RL-based model. The problem of the MCS mapping is clearly exposed when the Gaussian assumption does not exist. In the fading channels, the switching thresholds set on the ideal channel will no longer apply. At this time, the proposed RL-based model is still able to learn on-line from the actual environment, and exhibits excellent decision-making performance.

Fig. 11. Performance comparison between RL-based model and MCS.​​​​​​​

In order to improve decision-making efficiency and make the model more suitable for waveform parameter decision-making problems, this subsection will optimize the exploration process of RL-based decision model.

In [30], based on the RL-based waveform parameter decision model shown in Fig. 4, it is proposed to classify actions into "good actions" and "bad actions" according to the set performance threshold. The threshold is set to a certain percentage of the maximum performance value. At the same time, the "action rejection probability" is used to control the exploration of "good actions" and "bad actions", and the decision-making efficiency of the model can be improved by filtering out "bad actions" to a certain extent. Inspired by this idea, this section will also optimize the online decision-making process of the RL-based decision model by adding judgments of "good" and "bad" action performance.

According to the communication performance reward function constructed in the RL-based decision model, while the reward is set to R = C_t+1 , the communication performance function change ∆C = C_t+1 - C_t obtained after the action is used to determine the “good” and “bad” of the current action. "Make judgments. If ∆C ≥ −0.1, the action is considered to be a "good action" for the current state, then keep the action and continue the exploration process; if ∆C < −0.1, then the action is considered a "bad action" for the current state. Remove the action from the range of optional actions in the current state, and re-select from the remaining actions to explore.

Comparing the performance of the RL-based decision model after adding the "action judgment mechanism" and the decision model before the improvement, the communication system and channel in the improved decision model still adopt the parameter configuration in Table 1.

Fig. 12-Fig. 14 compare the decision curves of the improved model and the original RL-based decision model under three different AWGN channels with different E_b / N₀ values. It can be seen from Fig. 12-Fig. 14 that the trend of the decision curve before and after the improvement is basically similar, and the final decision result is also the same, but the exploration process of the model after adding the "action judgment mechanism" is obviously shortened, and the maximum can be shortened by 9 times, which shows that the improved decision model no longer needs to spend time on "poor performance" actions and states, can quickly find the position of the optimal waveform parameters and converge, and the exploration process is more efficient.

Fig. 12. Performance comparison in the AWGN( E_b / N₀ = 0dB ) channel

Fig. 13. Performance comparison in the AWGN( E_b / N₀ = 10dB ) channel

Fig. 14. Performance comparison in the AWGN( ^{E_b / N₀ = 20dB} ) channel

7. Conclusion

In this paper, we have proposed a new RL-based decision-making model for the communication waveform parameters. Simulation results have demonstrated that the configured model has the ability to search for optimal parameters in any channel and is more efficient than the existing RL-based decision-making model. Additionally, we have also compared the decision-making performance of the proposed model and the MCS mapping method in both ideal AWGN channels and fading channels. The proposed model made reasonable decisions in the ideal channels, and performed better in several typical fading channels. Moreover, we introduced an "action judgment mechanism" to optimize the decision-making model. Simulations show that the improved model can reduce the exploration process of online learning by up to 9 times while ensuring the same decision-making performance. There are still some problems need to be solved, such as the adaptability of the model to the actual communication channels. In the future research, we will perform simulation experiments and further improvements on the practical application of the model. Meanwhile, we will further apply machine learning to this system model.