A Bit Allocation Method Based on Proportional-Integral-Derivative Algorithm for 3DTV

Abstract

Three-dimensional (3D) video scenes are complex and difficult to control, especially when scene switching occurs. In this paper, we propose two algorithms based on an incremental proportional-integral-derivative (PID) algorithm and a similarity analysis between views to improve the method of bit allocation for multi-view high efficiency video coding (MV-HEVC). Firstly, an incremental PID algorithm is introduced to control the buffer "liquid level" to reduce the negative impact on the target bit allocation of the view layer and frame layer owing to the fluctuation of the buffer "liquid level". Then, using the image similarity between views is used to establish, a bit allocation calculation model for the multi-view video main viewpoint and non-main viewpoint is established. Then, a bit allocation calculation method based on hierarchical B frames is proposed. Experimental simulation results verify that the algorithm ensures a smooth transition of image quality while increasing the coding efficiency, and the PSNR increases by 0.03 to 0.82dB while not significantly increasing the calculation complexity.

1. Introduction

 In a 3DTV/FTV system, the type of method used for the multiview acquisition part and the 3D stereo display part is inconsequential. Various key technologies of three-dimensional (3D) video systems are receiving extensive attention and research is being conducted on them owing to the continuous development of computer vision theory, video capture technology, and stereo display technology, especially the rapid development of naked-eye display technology in recent years. The development of 3DTV has introduced a new generation of video technology. A strong sense of perspective and immersion can be experienced through 3D video. The field of image processing has always been a hotspot [1-5]. Compared with traditional two-dimensional (2D) video, 3D video contains 2D information from multiple angles for the reconstruction of the depth information of the third dimension to produce a 3D feeling [6-9]. When scene switching occurs in the video, the time-domain redundancy of the video at the switching point disappears. At this time, if the adjacent video frames at the switching point are still used as the same coding unit, a larger error occurs in the rate control of the 3D video coding, thereby considerably reducing the video compression performance. There is a significant amount of scene switching in the actual video; hence, it is necessary to detect scene switching. This paper proposes two algorithms based on the incremental proportional-integral-derivative (PID) algorithm and similarity estimation between viewpoints to improve the bit allocation and rate control algorithm for multiview high-efficiency video coding (MV-HEVC).

 The remainder of this paper is structured as follows. Section 2 presents our review of previous work on rate control. Section 3 introduces the PID rate control model and provides details of bit allocation algorithms for MV-HEVC based on PID. Section 4 describes the extensive experiments conducted to evaluate the performance of the proposed method. Finally, conclusions are drawn in Section 5.

2. Related Work

 The bit allocation method is an important part of the video encoding process. It controls the encoding parameters to ensure that the generated bitstream meets the application requirements and maintains the subjective quality between the frame images in the sequence without significant fluctuations while maintaining high encoding quality. The bit rate fluctuation of the bitstream generated after encoding is very large. A buffer must be added at the output of the encoder to smoothen the bit rate fluctuation, but the buffer still faces the possibility of overflow and underflow. The bit rate control algorithm must verify that the buffer overflow level is used to adjust the encoder output bit stream. Bit rate control also takes into account the high quality of the encoded video while maintaining a constant (smooth change) quality. The average PSNR of the video sequence must be improved. Moreover, the subjective quality between the front and back images in the sequence must not fluctuate significantly. The fluctuation of the decoded image with time has a significant impact on the subjective quality experienced by the audience. The latest bit rate control needs to consider obtaining a higher objective quality and also the human subjective visual effects.

 At present, most domestic and foreign research on rate control is based on H.264, HEVC, MVC, and 3D-HEVC [10-14]. The research on MV-HEVC is not as good as that on the other rate control technologies. However, the prediction characteristics according to the parallax between each viewpoint are allocated by bits. In the rate control algorithm of stereoscopic video coding, Vizzotto BB and others only considered the case of two left and right viewpoints [15], but MV-HEVC generally has more than five viewpoints, and there are many types of image coding. The accuracy of the target bit allocation method based on TM5 will continue to deteriorate, resulting in very high bit allocation errors and difficulty in controlling the bit rate. Seanae Park et al. considered the effect of using layered B frames in the temporal coding layer in multiview video coding [16].

 Since the introduction of HEVC, many scholars at home and abroad have been devoted to researching HEVC based multiview video coding rate control. Shao et al. used a fixed allocation ratio to assign a coding rate to the viewpoint and depth map [17]. This algorithm was simple and effective, but it did not allow for different sequences to obtain the best coding efficiency. Xiao et al. proposed the use of scalable bit rate allocation algorithms for different bandwidths to meet the actual application scenarios [18]. Pan Geng et al. [19] proposed a depth-based 3D-HEVC code rate control algorithm that uses a fixed color and depth code rate ratio of 4:1, but this algorithm could not achieve the best virtual viewpoint rendering quality. Fang et al. [20] estimated a virtual visual distortion model based on the coding information. The model could accurately allocate bits, but the complexity was higher.

 Xiao Jianmin et al. [21] proposed a depth and texture grading rate control algorithm. Reference [22] established an R-D model between viewpoints by combining texture and virtual viewpoint distortion functions; however, this method did not use the correlation between viewpoints, and the rate control error was relatively large, with an average code rate error of 1.56%. Wang et al. [23] proposed a bit allocation rate control algorithm for 3D-HEVC. Because the algorithm directly used the traditional H.264 bit allocation model, the R-D model prediction was not accurate, thereby resulting in an unacceptable rate control accuracy. In 2018, Lei J et al. [24] proposed a bit allocation method based on the R-λ model. The experimental results were ideal, but the correlation among viewpoints was not considered, thereby leading to high computational complexity. Therefore, further research on the bit allocation optimization model for MV-HEVC is needed.

 PID control theory has achieved considerable success in the field of mechanical automation. Many scholars have introduced PID controllers in 2D video coding rate control. Sun et al. [25] proposed a bit allocation method for MPEG-4 based on PID as early as 2004. This method enabled precise control of the buffer and achieved good video quality, buffer control, and quality fluctuation. Xu et al. [26] proposed rate control in the linear domain. A PID controller was used at the macroblock level. Simultaneously, they considered the impact of detecting scene changes and adopted a simple method to detect scene changes. Excessive output code flow may block the transmission channel, resulting in delayed video transmission or frame loss. The compression ratio is high, and the output code flow is too small; thus, the transmission channel resources cannot be fully utilized, and video quality degradation will be caused at the decoding end.

 The main method to solve this problem is adopting rate control technology, that is, the size of the output code stream at the video coding end is adjusted by reasonably distributing the number of video coding bits so that the video coding can reach the best quality under the conditions of full video transmission. Zhou Yimin et al. [27] adopted an exponential code rate control model and adopted a PID controller for frame rate code allocation, but they used a fixed PID in the process of PID control, as in the aforementioned study. The control parameters affect the control accuracy, and the control has no flexibility.

 Considering the limitations of the bit rate allocation of the frame layer and the GOP layer, Shen et al. [28] proposed a frame-level target bit allocation scheme based on incremental PID and frame complexity. First, they introduced incremental PID to control the buffer “liquid level” to reduce the negative impact on the target bit allocation of the frame layer owing to the fluctuation of the buffer “liquid level.” Next, a target bit allocation scheme based on frame complexity was proposed in which the MAD of the previous frame, time-domain activity of the previous frame, and time-domain activity of the current frame were used to predict the current frame complexity of the frame.

 The aforementioned algorithms are all 2D video coding rate control algorithms based on the PID control theory, in which PID theory has not yet been introduced into 3D video coding rate control. This paper proposes a 3D video coding rate control algorithm (MV-HEVC) based on an incremental PID algorithm and similarity estimation between viewpoints. First, we introduce incremental PID to control the “liquid surface” of the buffer, and then, we use image correlation to analyze inter-view correlation and frame complexity to perform bit allocation and coding between the viewpoint layer, frame layer, and basic unit layer rate control. The experimental results verify that the proposed algorithm is consistent with the current popular multiview texture and depth. Compared with the algorithm, under the same coding bit rate, the objective quality of the virtual viewpoint and the basic viewpoint improved by an average of 0.45 dB; hence, it has a higher coding efficiency.

3. Proportional-Integral-Derivative Controller for MV-HEVC

 The PID controller is one of the most popular feedback controllers. PID control is particularly suitable for complex control environments. We cannot predict the next frame in video coding; hence, this practical environment is particularly suitable for control with a PID controller. The PID controller has the characteristics of simple operation and high efficiency. These advantages are precisely why most engineers favor the PID controller. Video code rate control has achieved very successful results [28-29]. PID control uses the ratio of the error signal (proportional), integral, and derivative to generate control signals so that the system obtains the required output. Fig. 1 is a schematic diagram of a typical PID controller. The entire system is controlled by a PID controller. And controlled devices.

Fig. 1. PID control principle

 The goal of PID control is to ensure that the difference between the output and the demand signal of the controlled device is as small as possible. In order to perform PID control, an error signal is required as an input. Generally, the error signal is the demand signal and the feedback signal The difference between . After the error signal is obtained, the integral and derivative of the error signal are calculated, and then the three signals are k _p、 k_i、 k _d -weighted to form the control signal E_PID of the controlled module.

 Where EP, EI, and ED are the proportional, integral, and derivative control parameters, respectively, and they can be computed by

\(E_{I}(t)=k_{I} \cdot \int_{0}^{t} e(t) d t=k_{I} \cdot \sum_{i=0}^{t} e(t)\)       (1)

\(E_{P}(t)=k_{P} \cdot e(t)\)       (2)

\(E_{I}(t)=k_{I} \cdot \int_{0}^{t} e(t) d t=k_{I} \cdot \sum_{i=0}^{t} e(t)\)       (3)

\(E_{D}(t)=k_{D} \cdot \frac{d e(t)}{d t} \approx k_{D} \cdot[e(t)-e(t-1)]\)       (4)

 In the formula, e(t), \(\sum_{i=0}^{t} e(i) \text { and } \frac{d e(t)}{d t}\) are respectively expressed as the error signal itself, as well as the integral and derivative of the error signal. When there is only proportional control, there is a steady-state error in the system output. The value of p_k determines the speed of proportional adjustment. When the p_k factor is increased, the adjustment speed of the regulator will be accelerated. If p_k is too large, then the control system will oscillate or overshoot. Increasing i_k can suppress the impact of accidental factors on the system, reduce oscillations or reduce fluctuations. Increasing d_k can increase the sensitivity to speed of error response, but it should be avoided to cause unnecessary fluctuations.

 The error function e( t ) represents the difference between the demand function and the feedback functions. The output of the proportional controller is proportional to the input error signal, while the output of the integral controller is proportional to the integral of the input error signal. This causes the controller's output to increase and the steady-state error further decreases until it equals zero. The output of the differential controller is directly proportional to the rate of change of the error of the input error signal. The differential term can predict the trend of error changes, thereby avoiding serious overshoot of the controlled variable. This can improve the dynamic characteristics of the control system during the adjustment process.

 In contrast to the above PID, the incremental PID control focuses on the impact of error accumulation on the system. The output of the controller is only the increment \(\Delta E_{P I D}(t)\) of the control amount. The control formula is:

\(\Delta E_{P I D}(t)=k_{P} \cdot \Delta e(t)+k_{I} \cdot e(t)+k_{D} \cdot[\Delta e(t)-\Delta e(t-1)]\)       (5)

 In the formula: k_p、 k_i、 k_d is the proportional, integral and differential coefficients respectively; is the error signal. And can be expressed as follows (6): 

\(\Delta e(t)=e(t)-e(t-1)\)       (6)

 Compared with the PID algorithm, the incremental PID algorithm has the following four advantages: (1) Each output signal of the PID algorithm is related to the entire previous PID controller state. The calculation formula uses the accumulated value \(\sum_{i=0}^{l} e(t)\) of the past deviation, which is prone to generate large cumulative errors. The incremental method only needs to calculate the increment. When there is a calculation error or insufficient accuracy, the influence on the calculation of the control amount is small. Therefore, the reliability of the incremental PID algorithm is relatively high. (2) The incremental PID algorithm is only related to the past three samplings, and the calculation amount is relatively low. (3) When the control conditions change, because the incremental PID algorithm only outputs the increment, it will not have much impact on the system. (4) Incremental PID control is more conducive to eliminating the effect of error accumulation on the system.

 Therefore, as shown in this section, we use incremental PID control. Determination of the control increment \(\Delta E_{P I D}(t)\) is only related to the last three samplings, and it is easier to obtain a better control effect through weighting processing.

4. Rate Control Algorithm for MV-HEVC

4.1 Bit allocation based on incremental PID for the view layer

 According to the previous analysis, it is known that the target bit allocation in the GOP layer rate control of the results obtained in previous research is unreasonable. The problem with MV-HEVC bit rate control is that it is difficult to allocate bits between viewpoints, and it is impossible to assign a reasonable bit to each viewpoint accurately [30]. The weight, w_k , represents the proportion of all viewpoints from viewpoint k . A larger w_k indicates that the viewpoint is the main viewpoint. For different situations, we use similarities between viewpoints to determine a reasonable rate control and bit allocation. The viewpoint with the highest similarity to other viewpoints has a correspondingly higher weight; hence, that viewpoint should be given more bits. As a reference viewpoint for other viewpoints, the benefit of this viewpoint encoding is directly related to the entire video encoding result; therefore, a greater allocation of bits is needed. The total number of bits allocated to the i^th viewpoint in the encoded GGOP picture group is given by (7).

\(T_{G o p}(i)=\left\{\begin{array}{l} \frac{u(i)}{F_{r}} \times N_{\text {gop }}(i) \times N_{\text {view }} * w_{0} \quad i=0, \text { initialization } \\ \min \left(T_{\text {Gop }}(i-1)+\Delta T_{G o p}(i), \frac{u(i)}{F_{r}} \times N_{\text {gop }}(i) \times N_{\text {view }} * w_{i}+\Delta T_{\text {buffer }}\right) \quad i \neq 0 \end{array}\right.\)(7)

 where Nview represents the number of multi-view video viewpoints, F_r represents the channel bit rate, u(i) represents the frame rate, and \(N_{\text {gop }}(i)\) indicates the encoded GOP length. \(\Delta T_{\text {Gop }}(i)\) represents the remaining bits of the last GGOP. ∆Tbuffer represents the remaining bits of the last buffer.

\(w_{k}=\frac{\frac{1}{N_{\text {view }}} \cdot \sum_{j=0, j \neq k}^{N-1} S\left(V_{j}, V_{k}\right)}{\sum_{k=0}^{N_{\text {vew }}-1} \frac{1}{N_{\text {view }}-1} \cdot \sum_{j=0, j \neq k}^{N_{\text {wide }}-1} S\left(V_{j}, V_{k}\right)}\)       (8)

 where \(S\left(V_{j}, V_{k}\right)\) is the similarity between viewpoints V_j and V_k. This paper uses a bilinear similarity measurement algorithm to test the similarity of two viewpoints. This algorithm is superior to the traditional distance measurement to test the similarity of two features, and there are no restrictions such as distance measurement. The algorithm has been successfully used in the field of image retrieval. Among them, \(S_{j}^{d}, S_{k}^{d}\) are the feature vectors of the two images, respectively.

\(S\left(V_{j}, V_{k}\right)=\frac{S_{j}^{k} \cdot S_{k}^{d}}{\left|S_{j}^{d}\right| \cdot\left|S_{k}^{d}\right|}\)       (9)

4.2 Bit allocation and rate control based on incremental PID for frame layer

From the above analysis, it can be observed that the target allocation bits of P frames are T_p allocated by T_p and T_buf weighting. T_buf allocation depends on the current target buffer overflow, frame rate, channel bit rate, and true buffer overflow. When the scene is switched or the scene is moving violently, the predicted QP is not accurate. The actual number of bits generated by the current frame is much higher or lower than the number of allocated bits, which makes the real buffer “liquid level” far away. Moved the target buffer overflow. In order to meet the fullness of the target buffer, this will inevitably affect the allocation of target bits in the next frame or frames, causing fluctuations in PSNR. Therefore, incremental PID is used to control the "liquid level" of the buffer to balance the negative impact of the buffer fluctuation on the target bit allocation of the frame.

 The definition of the error function is the key to the incremental PID controller, and the error function is defined on the basis of the demand function and the feedback function. This paper uses (10) as the error function of the buffer incremental PID control:

\(e(j)=T b l(j)-B_{c}(j)\)       (10)

 In the formula: \(\operatorname{Tbl}(j) \text { and } B_{c}(j)\) are the buffer target overflow degree and the actual buffer occupancy of the first P frame of the current GOP. The error increment is:

∆ e (j )= e (j )− e (j − 1) (11)

 Next, the incremental PID algorithm is used to control the buffer. The output of the controller (increment of \(T_{\text {buf }}\)) \(\Delta T_{b u f}\) is as shown in (12).

\(\Delta T_{\text {buf }, P I D}=k_{P} \cdot \Delta e(t)+k_{I} \cdot e(t)+k_{D} \cdot[\Delta e(t)-\Delta e(t-1)]\)       (12)

 The parameters of the incremental PID controller in the formula are determined using the PID trial and error method; hence, based on experience, a set of controller parameters is temporarily and tentatively determined. Different sequences are tested, and the PSNR and buffer overflow curves are observed. If the control quality is not achieved, then the parameters of the controller are changed according to the influence of each parameter on the control process. Many tests are conducted until this requirement is satisfied.

 According to the number of remaining bits, the number of remaining B frames in each layer, the scaling factor of the current GOP, and the incremental PID controller, determine the number of bits allocated to the current frame as

\(T_{l, i}^{B}=\chi_{1} \cdot \frac{\omega_{l} B_{l}(i)}{\sum_{k=1}^{d-1} \omega_{k} N_{B}^{i}(k)}+\eta_{1} \cdot \Delta T_{b u f, P I D}\)       (13)

\(T^{P / I}=\chi_{2} \cdot \frac{\omega^{P / I} B_{t o t}}{\omega^{P / I}+\sum_{k=1}^{d-1} \omega_{k} N_{B}(k)}+\eta_{2} \cdot \Delta T_{b u f, P I D}\)       (14)

 Among them, \(T^{P / I}\) indicates the total number of bits allocated to the P picture or I picture, \(T_{l, i}^{B}\) indicates the number of bits allocated to the i th B picture of the l th layer of the current GOP, and \(B_{i}(i)\) indicates the i th B of the l th layer of the current GOP when encoding The number of remaining bits in the picture, \(N_{B}^{i}(k)\) N indicates that the remaining bits have not been encoded in the k th layer. \(\Delta T_{b u f, P I D}\) indicates that the remaining bits of the buffer are based on the PID controller. \(B_{\text {tot }}\) P represents the bits allocated by the current GOP, and \(\omega^{P / I}\) and \(\omega_{k}\) represent the scaling factor and k th score of the P picture or I picture, respectively. This is the scaling factor of the layer B image. \(\chi_{1}, \chi_{2}, \eta_{1}, \eta_{2}\) respectively represent the weight coefficients.

5. Experimental Test

The six synthetic sequences in this study, verify the effectiveness of this algorithm. Among them, 6 sequences Balloons-PoznanHall2, PoznanStreet-Balloons, Kendo-Newspaper, BookArrival-PoznanStreet, BookArrival-PoznanStreet, Newspaper-Balloons, PoznanHall2- BookArrival with scene switching, they are respectively stitched by two different sequences (Balloons-PoznanHall2, PoznanStreet-Balloons, Kendo-Newspaper, BookArrival-PoznanStre et, Newspaper-Balloons, PoznanHall2-BookArrival). Balloons-Kendo-Newspaper is a new sequence synthesized by Balloons, Kendo and Newspaper. The synthesis of the above sequence requires resampling and preprocessing. As a result, each scene has a different complexity. The main coding parameters were as follows: coding structure I-P-B-B, GOP length of 12, each composite sequence encoded 480 frames or 720 frames, the entropy coding used CABAC, RDO was set to enable, the reference frame was 2, the search method was Fast Search, and the search range was 64 pixels. The test platform was an Intel(R) Core(TM) i7-5500U Duo CPU with 2.40 GHz (Double) and 8.00 GB of memory.

(a) Experimental PSNR results of PoznanStreet-Balloons sequence

(b) Experimental PSNR results of Kendo-Newspaper sequence

Fig. 2. Objective quality fluctuation of sequences

 In this study, we compare the proposed algorithm based on frame complexity and incremental PID with our previous algorithm. Fig. 2 presents a PSNR fluctuation chart of the PoznanStreet-Balloons and Kendo-Newspaper sequences. It can be observed from Fig. 2 that the PSNR change in the proposed rate control algorithm has small fluctuations, thereby ensuring a smooth transition of image quality, and no obvious image quality changes are visually observed. The complexity of the rate model is relatively high, and the error is relatively large. Scene switching frames consume too many bits, and the “liquid level” of the buffer suddenly increases. For nonswitching gentle viewpoints and frames, fewer bits need to be allocated. The image quality based on the ρ-domain rate control algorithm is low and fluctuates considerably.

Table 1. Simulation results Yan's

Fig. 3. Experimental results of sequences: (a) ″Balloons-PoznanHall2″, (b) ″PoznanStreet-Balloons″, and (c) ″Kendo-Newspaper″

 Table 1 and Fig. 3 present the coding efficiency of the three rate control algorithms. We previously proposed a rate control algorithm based on the ρ domain for MV-HEVC, but its coding efficiency was relatively low. Compared with the fixed ratio algorithm, the average PSNR gain was 0.47 dB. Compared with our previous algorithm, the PSNR of the algorithm proposed this paper is improved, and the average gain of PSNR is 0.27 dB. This is mainly because the scene switching frame consumes too many bits, and the “liquid level” of the buffer is suddenly increased. If a fixed bit allocation rate between the viewpoints is used, then the rate control algorithm of the fixed ratio is used to meet the full overflow of the buffer “liquid level” target. The bits allocated in the following frames must be significantly reduced, resulting in a sudden drop in the encoding quality of the following frames.

 In summary, the rate control algorithm proposed in this paper is more accurate than the algorithm we previously proposed. The rate deviation is smaller, and the PSNR fluctuation is relatively small, thereby improving the visual effect. The scheme adopts incremental PID control for the “liquid level” of the buffer, which can distribute the impact of excessive bit consumption of scene switching frames to subsequent frames, to avoid large fluctuations in video quality. Experimental results verify that the algorithm can accurately control the output bit rate and significantly improve the overall coding efficiency of a video sequence.

6. Conclusions

Excessive output code flow may block transmission channels, resulting in delayed video transmission or frame loss. The compression ratio is large, and the output code flow is too small. Thus, transmission channel resources cannot be fully utilized, and the video quality is degraded at the decoding end. The main method to solve this problem is adopting rate control technology, in which by reasonably distributing the number of video coding bits, the size of the output code stream at the video coding end is adjusted so that the video coding can achieve the best quality under the conditions of full video transmission.

 As the current bit allocation method for MV-HEVC has not been studied in depth, after an extensive review of related research, we found that the existing rate control is insufficient and proposed a rate control algorithm based on the PID control theory. The algorithm performs effective code rate control at the viewpoint layer and the frame layer. Experimental results verified that the introduction of PID theory to 3D video coding bit allocation achieves very good results, but the bit allocation between viewpoints needs to be improved. In the future, the correlation between viewpoints will be studied further to improve the rate control algorithm.