Time-Series Forecasting Based on Multi-Layer Attention Architecture

Abstract

Time-series forecasting is extensively used in the actual world. Recent research has shown that Transformers with a self-attention mechanism at their core exhibit better performance when dealing with such problems. However, most of the existing Transformer models used for time series prediction use the traditional encoder-decoder architecture, which is complex and leads to low model processing efficiency, thus limiting the ability to mine deep time dependencies by increasing model depth. Secondly, the secondary computational complexity of the self-attention mechanism also increases computational overhead and reduces processing efficiency. To address these issues, the paper designs an efficient multi-layer attention-based time-series forecasting model. This model has the following characteristics: (i) It abandons the traditional encoder-decoder based Transformer architecture and constructs a time series prediction model based on multi-layer attention mechanism, improving the model's ability to mine deep time dependencies. (ii) A cross attention module based on cross attention mechanism was designed to enhance information exchange between historical and predictive sequences. (iii) Applying a recently proposed sparse attention mechanism to our model reduces computational overhead and improves processing efficiency. Experiments on multiple datasets have shown that our model can significantly increase the performance of current advanced Transformer methods in time series forecasting, including LogTrans, Reformer, and Informer.

1. Introduction

Time-series forecasting has been extensively used in stock forecasting, transportation, et al. In these real-world uses, an urgent problem to be solved is to increase the forecast time, which is conducive to long-term planning and decision-making. With the arrival of big data era, the time-series forecasting model begins to face the scenario of predicting a longer time span. Therefore, for each rolling window, the model should also be able to process more past information. Because the specific structure of classical time-series forecasting algorithms such as ARIMA [1,2] and SSM [3] needs to be manually selected after considering various relevant factors, its prediction ability is insufficient and cannot meet the needs of long-term prediction.

The model based on deep learning is a good candidate model to solve the above problems, especially transformer [4-7] model. Compared with CNN [8-10] or RNN [11-13], the self-attention mechanism of transformer helps the model to treat any length of time series equally, which enables the transformer to better handle long-term time sequence and capture long-term dependencies. However, existing transformer based time series prediction algorithms have the following problems: 1) The encoder-decoder structure of the transformer makes the model more complex and inefficient, which limits the ability to construct deep temporal information dependencies by stacking multiple layers of attention. 2) In existing algorithms, the information interaction between historical and predicted sequences is poor, which affects the predictive performance of the model. 3) The self-attention mechanism of the transformer brings the calculation expense and memory usages of the model to increase twice with the input length, so the issue of computational complexity further limits the algorithm's expansion in model depth.

To address the above problems, a novel time-series forecasting algorithm is designed to simplify traditional models and achieve breakthroughs in constructing deep temporal dependencies, as shown in Fig. 1. The model abandons the traditional encoder-decoder structure of transformer, it is only composed of multi-layer attention modules. The head and tail of the model are composed of sparse self-attention modules, and the middle part is composed of history-prediction cross-attention modules. The main work of this article is introduced as follows.

Fig. 1. Our time-series forecasting network architecture

(1) The complex encoder-decoder structure in traditional Transformer based time series prediction models is abandoned. Using only multi-layer attention modules to construct a time series prediction model makes the model structure more concise and improves the depth of the model, thereby improving the algorithm's ability to mine deep time dependencies.

(2) A cross-attention module is designed to optimize the information of history sequences and prediction sequences separately, and realize the information interaction between history sequences and prediction sequences through the cross-attention mechanism. It improves the information exchange capabilities of history sequences and prediction sequences.

(3) Apply sparse attention mechanism to the algorithm to decrease the computation complexity of the model. Inspired by ProbeSpare attention mechanism proposed in Informer, the self-attention and cross attention modules in our model are replaced with ProbeSpare attention modules, reducing the computational complexity of long sequence self-attention from O(L²) to O(LlogL) and improving the efficiency of the entire model.

The test results indicates that the model raised in this article has obtained good results on main public datasets, which exceeds the traditional time-series forecasting methods and the recently proposed time-series forecasting method based on transformer.

2. Related work

2.1 Traditional time-series forecasting methods

Many time-series forecasting models have been well developed because of its important role, and time-series forecasting algorithms begin with classical tools [14, 15]. ARIMA [1, 2] uses the difference methods to transform non-stationary process into stationary process in time series forecasting. The stationarity of time-series data is an important prerequisite for constructing ARIMA model. In addition, the recursive neural network model (RNN) is used to model the time dependence of time-series [16-19]. RNN is a kind of feedback neural network, which is windy used to deal with sequence data. But it cannot learn too long sequence features. DeepAR [20] integrates autoregressive model and RNN to simulate the probability distribution of the coming sequences. LSTNet [21] brings in convolutional neural networks (CNNs) with recursive skip connections to catch short-term and long-term time dependencies. Refs [22-24] introduces temporal attention to seek the long-term time pattern of forecasting. LSTM [25] is a special form of RNN, which can avoid the gradient disappearance problem and learn long-term information. Relying on the traditional self-attention mechanism, different attention mechanism modules are proposed in the recent years [41-44]. PSP learning framework designs a novel Pyramid Polymerizing Attention (PPA) mechanism, which is able to supplement the body, part and joint level semantic information [41]. SDS-CL designs a new Spatiotemporal-decoupling Intra-Inter Attention (SIIA) that intended to capture spatiotemporal specific information separately [42]. In addition, many studies based on time convolution network (TCN) [26-29] try to use causal convolution to model time causality. TCN is a method that introduces the idea of convolutional neural network into time-series data prediction, that is, convolutional network is used to process time-series forecasting. Compared with LSTM, the main advantage of TCN is that its training and processing speed is much faster than LSTM when the accuracy of timing prediction tasks is similar or even higher than LSTM. This is because TCN is based on the idea of image parallelism and the confluence of convolution neural core, which can integrate a large amount of bottom information in a small processing unit. It can enhance the exactitude and efficiency of TCN in processing a large number of multi-dimensional data and longtime time-series data.

2.2 Transformer model in time-series forecasting

Recently, the Transformer [30, 31] model has shown strong ability to process sequence information and is widely used in tasks such as natural language processing [32, 33], audio signal processing [34]. Some scholars have also adopted this model in the images [35, 36]. Therefore, some researchers try to use transformer-based models to predict time series data. This method uses self-attention mechanism to mine useful information association relationships in time series data, including univariate and multivariate time series information. Transformer contains two parts: encoder and decoder. The former takes the historical sequence as input, and the decoder part takes the splicing of the historical sequence and the sequence to be predicted as input. The latter passes information from the historical sequence to the decoder. In this way, the model can "focus" on the most valuable information in the historical sequence before making a prediction. Decoders use masked self-attention to prevent information leakage. However, the self-attention mechanism in Transformer has a large drawback in dealing with long-term time series prediction because of the computational complexity of O(L²). Some work [37, 38] proposed different attention models to alleviate this problem. Recently, the ProbSparse attention mechanism based on KL dispersion was proposed in Informer [7], which implements the computational complexity of O(LlogL). As can be seen, above algorithms are based on the classical architecture of Transformer, attempt to change the self-attention mechanism to a sparse form. It still adopts an encoder-decoder structure. The reason why this structure is complex is that the encoder and decoder are each composed of self-attention layers, which is equivalent to two sets of parallel attention modules executing calculations simultaneously, resulting in a decrease in computational efficiency. Our algorithm abandons this inherent structure and only uses a more concise multi-layer attention structure to build a deep attention-based model.

3. Network architecture

Our network architecture is composed of head-tail ProbSparse Self-Attention (PSA) layer and History-Prediction Cross-Attention (HPCA) layer. The structure diagram is shown in Fig. 1. In the figure, green represents the historical sequence and orange represents the predicted sequence. Firstly, the model accepts long sequence input, which is composed of history sequences and sequences to be predicted, in which the prediction sequences are filled with zeros. The sequences are sent to the PSA layer to learn the dependencies between sequences, and then the output is sent to the HPCA layer, which optimizes the history sequences and prediction sequences independently, realizes the cross information interaction between the two sequences, and then splices the two sequences. The spliced sequences are sent to the PSA again for further overall optimization. Finally, the ultimate forecasting results are achieved through a fully connected layer. Through the modeling of the deep dependency between sequences, the accurate prediction of future data is realized.

3.1 Head-tail probsparse self-attention layer

The original self-attention is based on tuple input, i.e. Query (Q), Key (K) and Value (V). Given Q, K and V, the attention function is scaled dot-product attention, which is defined in equation (1).

\(\begin{align}\operatorname{Attention}(\boldsymbol{Q}, \boldsymbol{K}, \boldsymbol{V})=\operatorname{softmax}\left(\frac{\boldsymbol{Q} \boldsymbol{K}^{T}}{\sqrt{d_{k}}}\right) \boldsymbol{V}\end{align}\)       (1)

where d_k is the dimension of K.

Expanding the self-attention mechanism to multiple heads for considering different attention distributions, it allows the model to focus on multiple aspects of information. The multi-head attention mechanism is described in equation (2). More comprehensive explanation can be found in Ref. [30].

MultiHead(Q, K, V) = Concat(h₁, ..., H_nh)W^O       (2)

H_i = Attention(QW^Q_i, KW^K_i, VW^V_i)       (3)

where ∈ ℝ^d_mxd_k, W^K_i ∈ ℝ^d_mxd_k, W^V_i ∈ ℝ^d_mxd_v, W^O ∈ ℝ^n_hd_vxd_m is a parameter matrix.

This self-attention mechanism has high computational overhead and high memory consumption, which limits the time-series forecasting ability of transformer model.

Informer breaks the limitation that each multi-headed self-attention in Sparse transformer [39], LogSparse transformer [6] and Longformer [40] is processed with the same strategy, and proposes a ProbSparse self-attention mechanism, which cuts down the computational complexity of self-attention from O(L²) to O(LlogL), thus improving the processing efficiency.

The ProbSparse attention mechanism is applied to all attention layers of the model in this paper. The self-attention layer receives the long sequences spliced by history sequences and prediction sequences, and outputs the long sequences with the same dimension. Specifically, the following vectors are input into the attention layer

X^t_i = Concat(X^t_p, X^t₀) ∈ ℝ^{(L_P+L_y)×d_m}       (4)

where X^t_p ∈ ℝ^L_P×d_m is the history sequence; and X^t₀ ∈ ℝ^L_y×d_m is the prediction sequence, which is filled with 0. Taking the prediction data of 168 time points (the experimental part is 7-day data) as an example, the first 4 days of the known sequences are taken as the history sequences, and the spliced sequence is X^t_i = {X_4d, X₀). X₀ contains the timestamp of the target sequences. Then, this model predicts the output through a forward process, rather than the ‘dynamic decoding’ in the traditional encoder-decoder architecture.

The tail attention layer obtains the terminal output through the fully connected layer, and the scale of the output is determined by making the univariate or multivariate predictions.

The structural diagram of ProbSparse Self-Attention layer (PSA) is shown in Fig. 2. The purple dotted arrow indicates the residual structure. The purple dot indicates the matrix Q, and the orange dot indicates the matrix K and V.

Fig. 2. PSA structure diagram

Under the action of self-attention mechanism, the history sequences and prediction sequences are regarded as a whole to construct the dependence between different time points. At the same time, under the front multi-head attention mechanism, the prediction sequences can also learn some information from the history sequences, which is conducive to the subsequent processing. In this paper, we set the multi-head ProbSparse Self-Attention stack two layers to mine the deeper time dependence.

3.2 History-prediction sequence cross-attention layer

The history-prediction sequence cross-attention layer is located in the middle of the head-tail ProbSparse self-attention layer. The history sequences and prediction sequences are separated and optimized separately, and the two are associated by using the cross-attention module. By highlighting their independence, the information interaction ability of history sequences and prediction sequences is improved, and then the forecasting effects of the algorithm is increased.

Specifically, the output of the ProbSparse self-attention layer in the head of algorithm is used as the input of this layer. At this time, the prediction sequences have learned some information from the history sequences. First, the two parts of the long sequences (history sequences and prediction sequences) is separated, as shown in equation (5):

X^t_p, X^t_f = 𝑆plit(X^t_i) ∈ ℝ^L_P×d_m, ℝ^L_f×d_m       (5)

where X^t_p ∈ ℝ^L_P×d_m is the history sequence and X^t_f ∈ ℝ^L_f×d_m is the prediction sequence. The above two vectors are input into the multi-head ProbSparse self-attention module respectively.

The multi-head ProbSparse self-attention module adaptively integrates the information from different time points of history sequences and prediction sequences in the form of residual and multi-head ProbSparse self-attention. Through the self-attention operation of the history sequences and the prediction sequences respectively, the information association within the two sequences can be established, and the heterogeneity of the information of the two sequences can be highlighted, which is convenient to realize the interaction of high-value information in the history sequences and the prediction sequences in the subsequent cross-attention module. Because the attention mechanism does not have the ability to distinguish the relative position relationship between different time points, position location is introduced to encode the input X^t_p and X^t_f, and a sinusoidal function is adopted to produce time location code. In the end, the mechanism of multi-head self-attention module can be summarized.

X_SA = X + MultiHead(X + P_x, X + P_x, X)       (6)

where P_x ∈ ℝ^L_p×d_m, ℝ^L_f×d_m represents time position code, X_SA ∈ ℝ^L_p×d_m, ℝ^L_f×d_m is the output of the module. The output is fed into the multi-head ProbSparse cross-attention module. The schematic diagram of the self-attention stage of the history-prediction HPCA is shown in Fig. 3. The purple dotted arrow indicates the residual structure. The purple dot indicates the matrix Q, and the orange dot indicates the matrix K and V.

Fig. 3. Schematic diagram of HPCA self-attention stage

The multi-head ProbSparse cross-attention module also adopts the form of residual, and uses the multi-head ProbSparse cross-attention to fuse the feature vectors of two input to realize the information interaction between the history sequences and the prediction sequences. Similar to the multi-head ProbSparse self-attention module, time location code is also used in the multi-head ProbSparse cross-attention module. Furthermore, to improve the fitting performance of the model, an FFN module is used. The module is a fully connected feedforward network, it contains two linear transformations with a relu activation function ReLU in the middle, i.e.

FFN(X) = max(0, XW₁ + b₁)W₂ + b₃       (7)

where W and b indicate the weight matrix and the basis vector respectively. Subscripts indicate different layers. So the action mechanism of multi-head ProbSparse cross-attention module can be summarized.

X_CA = X'_CA + FFN(X'_CA),

X'_CA = X_q + MultiHead(X_q + P_q, X_kv + P_kv, X_kv)       (8)

where X_q is the input and P_q is the corresponding spatial location code of X_q. X_kv is an input from another branch, P_kv is the time coding of X_kv. X_CA is the output of the multi-head ProbSparse cross-attention module. According to equation (8), the cross-attention module computes the attention map based on the multiple scaled point products between X_kv and X_q, then reweights X_kv based on the attention map and puts it into X_q to strengthen the expression performance of time-series information. The schematic diagram of the cross-attention stage of the history-prediction HPCA is shown in Fig. 4. The purple dotted arrow indicate the residual structure. The purple dot indicates the matrix Q, and the orange dot indicates the matrix K and V.

Fig. 4. Schematic diagram of HPCA cross-attention stage

Under the action of the cross-attention layer of history-prediction sequences, the history sequences and prediction sequences are separated, the independence of the two is highlighted, the two are optimized separately, and the correlation of the two sequences is learned through the cross-attention mechanism, it enhances the information interaction ability of history sequences and prediction sequences. In this paper, the history-prediction sequence cross-attention module is stacked two layers to obtain a deep time-series forecasting algorithm based on attention, it greatly improves the ability of information interaction.

4. Experimental results and performance analysis

The same datasets (ETT, ECL and Weather) as Informer [7] is used to do experiments in this paper. A detailed description of the dataset can be found in reference [7]. This paper selects three Transformer-based time-series forecasting methods as comparison, including Reformer, LogTrans and Informer. Compared to various high-performance time-series forecasting algorithms such as ARIMA, Prophet and LSTM, these Transformer-based algorithms have demonstrated their advanced performance through experiments. Therefore, it is meaningful to deeply study the extent to which our algorithm can improve Transformer-based time-series forecasting methods.

4.1 Experimental details

Through several groups of experiments, it is confirmed that our algorithm can improve the prediction effect and efficiency of transformer or like transformer model in time-series forecasting. The loss function is MSE. All methods use Adam optimizer, whose learning rate starts from 1e-4 and decays twice per epoch. The entered history sequence length is set to 96. The total number of epoch is 8, which should be stopped early appropriately. The batchsize is set to 32. Each individual experiment is repeated five times and the mean results are used. All the experiments are conducted on a single NVIDIA GTX 3090 24GB GPU. More experimental descriptions will be presented in the following parts. MSE loss function is selected in the prediction of the target sequence.

4.2 Analysis of experimental results

The performance of these four time series prediction algorithms is evaluated under univariate and multivariate conditions respectively, in order to understand the accuracy improvement of this method over Transformer-based methods. ETTh1, ETTh2, ETTm1, Weather and ECL datasets are used for testing. To explore the performance with various granularities, the length of the forecasting series are set to {24, 48, 168, 336} for ETTh1, ETTh2 and Weather datasets, {24, 48, 96, 288} for ETTm1 dataset and {48, 168, 336, 720} for ECL dataset. The indicators used to compare the accuracy of different algorithms are: MSE = \(\begin{align}\frac{1}{n} \sum_{\mathrm{i}=1}^{\mathrm{n}}\left(\mathrm{y}-\mathrm{y}^{\wedge}\right)^{2}\end{align}\) and MAE = \(\begin{align}\frac{1}{n} \sum_{\mathrm{i}=1}^{\mathrm{n}}\left|\mathrm{y}-\mathrm{y}^{\wedge}\right|\end{align}\), n represents the size of the forecasting window. The experiment in this part is conducted on five datasets, ETTh1, ETTh2, ETTm1, Weather and ECL, and is divided into univariate prediction and multivariate prediction. If necessary, these two indexes will also be used to estimate the prediction effect of subsequent experiments. When our model is tested, the training time on the ETTh1 dataset is 1.5 hours, and the testing time is 8.65 seconds, 9.22 seconds, 9.64 seconds and 9.98 seconds for 24, 48, 168 and 336 prediction length, respectively. The specific experimental results are list in Table 1 ~ 2.

Table 1. Univariate forecasting results on ETTh1, ETTh2, ETTm1, Weather and ECL

Table 2. Multivariate forecasting results on ETTh1, ETTh2, ETTm1, Weather and ECL

It can be shown from Table 1 ~ 2 that the effect of our algorithm is better than Transformer-based algorithms in all the univariate and multivariable cases, which shows that our proposed architecture does improve the prediction ability and performance of like transformer algorithm in time-series forecasting problems. Especially in the case of multivariable prediction, the proposed architecture is significantly improved compared with other algorithms, which shows that the deep attention-based architecture constructed in this paper has strong ability to mine the time dependence between multivariable and has higher prediction accuracy compared with the encoder-decoder based Transformer architecture. It is helpful to meet the actual demand for time-series prediction performance (especially multivariable time-series prediction).

4.3 Ablation Experiment Analysis

To evaluate the effect and contribution of the two important components of the proposed algorithm (head-tail ProbSparse self-attention layer and history-prediction sequences cross-attention layer) in time-series forecasting, two sets of ablation experiments are carried out on the ETTh1 dataset based on MSE and MAE indexes.

In the first group of experiments, the number of head-tail ProbSparse self-attention layers is reduced to one layer, univariate and multivariable prediction are carried out on the ETTh1 dataset, and then the head-tail ProbSparse self-attention layer is completely removed for the same experiment. The role of head-tail ProbSparse self-attention layer in this algorithm is verified by comparing the experimental results. The experimental results are list in Table 3.

Table 3. Ablation experiments of head-tail ProbSparse self-attention layer

In the second group of experiments, the number of head-tail ProbSparse self-attention layers is kept unchanged, the number of cross-attention layers of history-prediction sequences is reduced to one layer, the univariate and multivariable prediction on ETTh1 dataset are carried out, and then the of history-prediction sequences cross-attention layer is completely removed for the same experiment. The comparison experimental results verify the role of history-prediction sequence cross-attention layer in this algorithm. The experimental results are shown in Table 4.

Table 4. Ablation experiments of history-prediction cross-attention layer

It can be seen from Table 3 that in the case of multivariate prediction and short prediction sequences, ProbSparse self-attention layer may bring negative effects. The reason is that the periodic relationship of short sequences is not obvious. For the prediction of short sequences, whether the algorithm can catch the subtle and short-term dependence between sequences is very important. ProbSparse self-attention mechanism tends to focus on the periodic information of the sequence, which will ignore or even destroy the subtle local information to a certain extent, and in the case of multivariable, the periodic dependence between variables becomes worse, so this disadvantage of ProbSparse self-attention mechanism will be further amplified. The ProbSparse self-attention layer destroys the local fine and short-term dependence information beneficial to the model prediction, it will negatively affect the forecasting accuracy of the algorithm. However, in most other cases, the algorithm of stacking two layers of head-tail ProbSparse self-attention layers can achieve good results.

As can be seen from Table 4, the algorithm of stacking two layers of history-prediction sequences cross-attention layer can achieve better results in most cases, and only in a few cases achieve suboptimal results, but the gap is not significant. It can be seen that in addition to in the case of univariate and long series prediction it will bring a small loss of accuracy, in most other cases, the history-prediction sequence cross-attention layer designed in this paper can bring a significant performance improvement. This is because the proposed history-prediction sequence cross-attention layer can enhance the information interaction between history sequences and prediction sequences, simultaneously increase the depth of the algorithm, and make the model more capable of capturing time-dependent relationships.

Through the analysis of the ablation experiment results, it can be seen that each module constructed in this model can increase the effect of time-series forecasting to some extent, and make a certain contribution in time-series forecasting. The integrated algorithm structure architecture is simple and clear. By means of multi-level self-attention layer connection and information interactive fusion layer based on cross attention, multi-layer attention architecture can give full play to its potential in time-series forecasting task. The experiments results confirm that our algorithm can achieve higher prediction effect in most cases, it further demonstrates the robustness and effectiveness of the algorithm and its modules designed in this paper.

5. Conclusion

A novel time-series forecasting algorithm based on multi-layer attention architecture is proposed and tested on multiple public datasets in this paper, it has achieved excellent prediction performance. Our algorithm abandons the classical encoder-decoder architecture of Transformer and only uses multi-layer feedforward structure to construct the deep attention-based model, which improves the ability of mining deep time dependencies. At the same time, the head-tail ProbSparse self-attention layer and history-prediction sequence cross-attention layer are designed to enhance the ability of information interaction and mining deep information relevance. Experiments results confirm that our model can get better forecasting performance on five datasets. In the follow-up research, we will explore the limitations of ProbSparse attention mechanism, explore more effective methods of time-series relationship modeling based on attention, and further increase the effect of the algorithm.