1. Introduction
The essential prerequisite for stable and reliable operation of the electrical power system is to maintain a balance between the generation and load at all time. Conventionally, frequency control for real-time power balancing is implemented by directly handling centralized, integrated utility [1-4]. However, one of major changes driven by deregulation in electric power sector is to treat frequency control as commodities. These are called frequency control reserves (FCRs) as an essential part of ancillary services (AS), consist of primary control reserve, secondary control reserve and tertiary control reserve based on a framework in Union for the Co-ordination of Transmission of Electricity (UCTE), which is the major portion of the European Network of Transmission System Operators for Electricity (ENTSO-E). These reserve services are generally procured through a market mechanism [1, 4-6].
There are lots of literatures dealing with FCRs problems, which generally can be categorized into three types. One of research themes is energy and FCR scheduling problem by the system operator (SO). This problem is about market auctions for trading of energy and FCRs. There are two forms of auctions in energy and FCR scheduling in the deregulated electricity market. The first method adopts a sequential reserve procurement approach which decides the quantity of each reserve in a series of auctions [7-18]. Although this approach has an advantage in the computational aspect, experience has shown that the model may lead to quite undesirable situation due to price reversal problem or the probability of reserve capacity insufficient, illustrated by the failure of the California FCR market at the end of the 1990s [11]. The second method is a simultaneous auction of energy and FCR to overcome this price reversal problem. This approach currently is used at the New York Independent System Operator (ISO) and New England ISO. Another research theme is bidding problems for energy and FCR. From the view of suppliers, energy market and reserve market are interdependent due to each generating resource's capacity limit and technical aspect. This problem is about a decision making problem to achieve total benefit maximization [19-23]. Decision support for bidding has been studied using deterministic and stochastic approaches to deal with unknown market prices [21]. The other research theme is a problem to determine FCR requirements. Most of the current research on this problem assumes that the requirement of reserves is predetermined, which means that there is no need to consider short-term uncertainty in real time. In practice, according to a grid code such as UCTE operation book, FCR requirement is pre-determined as quantities based on some reliability criteria without in the consideration of coupling relationship in economic aspects between preoperation and post-operation phase [11-13, 17, 18].
In this paper, we will focus on the issue of establishment of secondary control reserve requirement in the consideration of coupling relationship between ex-ante planning and real-time balancing through reserves activation. This approach determines the capacity of secondary control reserve that minimizes the total cost of holding and utilizing secondary and tertiary control reserve. The organization of the reminder is as follows. Section 2 describes a process of planning, operation, and settlement for FCR in the competitive market. Section 3 presents the optimization framework that allows deriving an optimal determination method of secondary reserve capacity. Section 4 is a detailed case study where results are reported and analyzed. Some relevant conclusions are drawn in Section 5.
2. Ancillary Service Market
As the definition and classification of FCRs differ from a country to another, it is required to investigate how to define FCR in different product according to various countries and technical roles. There is no standard definition of the AS globally accepted [3, 15, 24]. This section outlines a framework that can accommodate definitions for FCRs without confusion because of difference defined in two different criteria. We classify FCR for frequency control that are currently used in UCTE, and compare the name of reserves defined by North American Electric Reliability Council (NERC). Then we describe the principle for real-time balancing and market clearing process.
2.1 Frequency control reserve definition and classifycation
An appropriate real time balancing between load and generation depends on the available FCRs. Its adequate planning for each reserve therefore is an important task for the ISO. FCRs can be classified into three levels based on the time scale separation principle: primary, secondary and tertiary frequency control reserves as depicted in Fig. 1 [24, 26]. This figure explains three sequential phase in frequency control. Primary control reserve stabilizes frequency deviation within a few short seconds. It acts as a proportional controller that restrains large frequency errors. Due to the proportional structure of primary control, frequency error still remains as quasi-steady states. Secondary control reserve takes over a linear combination of the frequency deviation and tie-line electrical power exchange known as Area Control Error (ACE). The main purpose of secondary control is to eliminate the ACE to zero within some minutes. Tertiary control replaces secondary control, based on an economic generation dispatch that reschedules the set point of generating units. In the event of incessant deviation and in order to free up secondary control reserves, ISO manually activates tertiary control reserve through Economic Dispatch (ED) program running at the control center [3, 7, 16, 24, 26].
Fig. 1.Frequency control mechanism in UCTE region
Fig. 2 explains how FCR types classify and compares the difference between NERC and UCTE policies. A main difference between NERC and UCTE is that NERC separates FCRs into non-event reserve and event reserve, whereas UCTE does not [26]. Non-event means continuous small deviation considered as normal conditions and happened continuously. Events are infrequent things that are more severe than in normal conditions. NERC uses the term contingency reserve to highlight severe situation such as unexpected failure, large error of load forecasting.
Fig. 2.Frequency control reserve categories
Spinning reserve is generally defined as extra amount of generating capacity, synchronized with the grid, and ready to serve additional demand. Non-spinning reserve is generating reserve, not connected but able to be synchronized within a specified time. However NERC uses the term spinning reserve or non-spinning reserve to refer to secondary control reserve under contingency reserve. Supplement reserve is defined as tertiary reserve under contingency. Regulating reserve under non-event in NERC classification only can be translated to secondary reserve, however, both primary and secondary reserve take cover a role of regulating reserve in UCTE area. Load following is the action to follow expected load pattern as a part of optimal dispatch, only separately considered in NERC [7, 26]. However, tertiary control reserve takes a role as load following. In this paper, to avoid confusion, we follow the terms defined by UCTE.
2.2 Process of FCR planning in Pre-operation phase
There are various market structures to procure FCR. However, we only consider a pool market for FCRs through centralized auction. Other market structures such as a bilateral market are not our research scope. Since load changes instantaneously, it is essential that ex-ante markets secure enough reserves associated with real time operation. The important characteristic of FCR in this paper is that capacity is reserved and procured ahead and then incremental or decremental energy is activated in response to real time imbalance between expected energy and load. This means that FCRs are different from other power products in that they are paid for availability, whether or not they are dispatched. Of course, if they dispatch based on auto or manual request in real time, delivered energy is remunerated.
In a short, there are two kinds of prices in FCR markets to be paid to participants with generating resources. One is the price paid for reserved capacity availability and the other is the price for delivered energy that is activated under specific conditions. However, there are different structures of remuneration for FCRs as illustrated Table 1 [2].
Table 1.Remuneration structure in different countries
In this paper, we follow the structure of settlement in PJM and Spain. It means that the provision of primary reserve is a mandatory, non-paid service. Therefore, primary reserve dose not influence the FCRs operation cost. Apart from primary reserve, secondary reserve is remunerated as market products settled by its availability based on the capacity in ex-ante market. Tertiary reserve providers are compensated for both availability and utilization. Moreover, we consider separate auctions for each category of reserve (secondary reserve capacity, tertiary reserve capacity and tertiary reserve delivered energy). The bids for each reserve service are arranged in price order to form an increasing staircase for each operating hour. We assume that capacity prices are determined in accordance with the most expensive bid taken by ISO as a single buyer every 15 minutes. Bid curves for each reserve in ex-ante market are illustrated in Fig. 3. In this paper, we assume that the size of positive and negative capacity is same.
Fig. 3.Reserve capacity bids in day-ahead market
The SO clears the reserve capacity markets based on the submitted bids with a 15 minutes time basis. However, the method of auction for utilized quantities of tertiary control reserve is linked with real-time price, usually settled at expost, i.e. the real time price at the time of energy delivery is uncertain for market participants as depicted in Fig. 4
Fig. 4.Up/down reserve energy bids in real-time market
2.3 Continuous adjustment in real-time operation
As explained in section 2.1, primary control reserve stabilizes frequency almost immediately following the disturbance. Secondary control reserve is deployed through automatic generation control (AGC) to release primary control reserve within some minutes. Tertiary control reserve is supervisory with respect to secondary control, refers to manual changes in the dispatching of generating units and load to free up secondary control reserve. Operation and settlement rules for provision of FCRs are well defined in many literatures including grid codes. In contrast, grid codes like UCTE operation handbook, FERC Order 888 do not describe definitive and unambiguous methods for implementing of tertiary control reserve. As a consequence, different methods have been applied to execute tertiary control reserve without standards. Methods may be a control by heuristic or probabilistic method. According to previous studies and practices, tertiary reserve activation is generally depends on the utilized level of secondary reserve [27, 28]. Therefore, we assume that execution of tertiary control reserve is based on a specific level of secondary control reserve, considering time delay and load deviation pattern. In this assumption, capacity requirement of secondary control reserves is referred to as the bandwidth of secondary control. If the secondary reserve meets its limits of the bandwidth defined as saturation, it jeopardizes the system security. To keep power system stable, ISO must activate tertiary control to relieve secondary control reserve before saturation [29]. When secondary control reserve meet certain level of its capacity, ISO calls tertiary reserve to release amount of required energy in the consideration of time delaying characteristics of tertiary reserve. In this paper, amount of activated tertiary reserve is determined on the value of the integer multiples of certain level of secondary reserve as a following equation (1).
Here, qkSEC,C,contract is contracted quantity to determine the relevant price in the day-ahead market, and qkSEC is a value of the bandwidth as an activation point of tertiary reserve. In this paper, we set the weight value χ to 1 based on the value for a simple optimization problem without considering delay time of tertiary reserve. The delivered energy for tertiary control reserve is determined based on the scheduled quantities for secondary control reserve and load fluctuation. For example, if large amount capacity of secondary control reserve is scheduled in the pre-operation phase, SO may save more during real-time operation with tertiary reserve utilization.
3. Methodology
In this chapter, we make mathematical formulations to describe real time balancing with several reserves. Then, we suggest an optimization framework to determine secondary reserve requirement considering the interaction between the day ahead and the real time operation.
3.1 Real time balancing equations
Demand error, qDk,t, means imbalances of load and generation mainly given by power plant outages, load oscillations, load forecast errors and changes in exchanges schedule [2]. Demand errors are recovered by secondary and tertiary reserve utilization, as follow
subject to
Demand error is eliminated by secondary and tertiary reserves according to amount of error and current level of secondary control reserve. If secondary reserve meets its limit (capacity), tertiary reserve is released by amount of secondary capacity to free up saturation of secondary reserve. Therefore, tertiary reserve is activated based on integer multiples (βk,t) of secondary reserve. The delivered energy for tertiary reserve can be calculated as following equation. βk,t is related to the activated quantity based on demand error, degree of saturation of the bandwidth. Positive value of βk,t is positive adjustment of tertiary reserve, on the other hand, its negative value means negative adjustment of tertiary reserve.
Here, t is the time index based on 2 seconds and T is a settlement period. We set T to 30 minutes (1800 seconds), 2T is 3200 seconds which means 1 hour used for scaling MWh value. We assume that activated tertiary reserve should be continued for at least 15 minutes, terminated in end of next time index k. Fig. 5 shows a simple example to illustrate equation (7).
Fig. 5.Example of tertiary reserve energy adjustment
3.2 Market settlement process in pre and post operation phase
In this paper, we consider coupling between ex-ante and ex-post reserve markets. In detail, we focus on the problem with day-ahead capacity acquisition and real-time adjustment which relate to procuring secondary and tertiary reserves. Capacity prices for secondary, tertiary reserve are formulated as follows,
During settlement time duration k (15 minutes), demand errors are detected every frequency control signal time t (2 seconds). To remove demand errors, appropriate secondary and tertiary reserve is utilized every time index t. Therefore, the settlement price for tertiary delivered energy during time period k is divided into 3 cases according to the values of as following equations.
Beginning of each stage, the number of calling tertiary reserve is initialized to 0 (10). Every time the SO calls tertiary reserve, n+k,0 (positive order) or n-k,0 (negative order) is cumulative according to each activation order. Price for delivered energy of tertiary reserve is determined to compensate a caused imbalance when the SO activates tertiary reserve in real time.
Case 1: βk,t > 0
subject to
Case 2: βk,t < 0
subject to
Case 3: βk,t = 0
subject to
As describe above, settlement price for real-time balancing through tertiary control reserve is formulated as follows:
Cost of each reserve from the SO's view can be calculated from following equations.
Therefore, the objective function during time index (1 hour) is expressed as the following.
3.3 Demand error model
In the frequency control market, uncertain demand fluctuations are the main factor which determines the reserve provision cost through the ex-post market settlement. We define demand error as power imbalance in the real time operation phase. In this paper, we use the term demand errors as real-time imbalance to affect the provision schedule of secondary and tertiary reserves. We describe demand errors as a mean-reverting process known as an Ornstein-Uhlenbeck process as below. This process, although satisfying the Markov property, does not have independent increments. We consider the demand errors as a relevant state variable, and model it as a mean-reverting process with drift [30]-[32]. By changing each parameter based on operational condition, various demand model can be tested.
Here, η is the speed of reversion, is the mean value of qDt, ε is normally distributed with mean zero and σ is instantaneous standard deviation of this process.
3.4 Stochastic programming framework
We use the stochastic programming (SP) based methods for distributed decision marking under uncertainties. SP is used to formulate and solve a problem with uncertain load errors. In particular, the types of decisions of interest are to find optimal scheduling for secondary reserve capacity in ex-ante considering relevant amount of delivered energy of tertiary reserve in ex-post clearance procedure. The overall objective is to optimize expected performance; this is done in an interactive way, since the strategy for markets operation for the delivery of frequency control is adapting as more information is gained in real-time about the uncertainty. In this paper, we adopt the dynamic programming (DP) backward algorithm to calculate the expected total reserve cost at time k, translated as a stage [31, 32]. DP framework is illustrated in Fig. 4. In the DP framework, we define as:
1. State variable, xk, is qkD at time index k.2. Decision variable, uk, is qkSEC,C at time index k.3. Random variable, wk, is qkTER,E during the period k.
Fig. 6.DP framework for discrete finite state problem
Fig. 7.3 States model for SP framework
In this paper, we divide the system state at each stage into 3 states based on the value given from PJM, illustrated in Fig. 5. Then, we generate 10000 Monte Carlo samples of the demand errors models to estimate transition probability at each state in each stage. Variable lk means a position of demand errors at stage k, which describes a discrete state in SP framework as described a following equation. State at stage k, xk is described as lk.
4. Case Study
This section provides the simulation results of applying the proposed technique using stochastic programming as an optimization on the base of the daily demand error data from Korea electricity market to generate demand models. Simulation assumptions and parameters are described in Table 4. Moreover, we assume that each reserve requirements vary throughout a day but remain fixed during 15 minutes period (stage k). In these case studies, we are comparing the dynamic reserve requirement determination method (proposed method) and the static reserve requirement determination method (traditional method). Tables 5-7 show different provision schedules depending on various speed of reversion, η = 0.0, 0.2 and 0.5 to test various demand model scenarios.
Table 4.Parameters used in case study
Table 5.Provision schedule for secondary control reserve capacity (η = 0 )
Table 6.Provision schedule for secondary control reserve capacity (η = 0.2 )
Table 7.Provision schedule for secondary control reserve capacity (η = 0.5 )
Table 8 compares the total reserve cost between the proposed model and the previous method. This illustrates that the determination method for secondary reserves requirement with consideration of interaction between exante and ex-post markets shows lower reserve cost than determination methods through pre-determined reserve capacity without consideration of interaction between dayahead reserve market and real time balancing.
Table 8.Comparison between dynamic and static provision methods
5. Conclusion
In this paper a new decision model for secondary reserve capacity is presented with the following characteristics:
1. Incorporation of capacity requirement of secondary reserve in ex-ante and delivered energy of tertiary reserve in ex-post market settlement2. Comparison between the dynamic and static determination of secondary reserves capacity approaches3. Allowance of testing several demand errors models in order to apply a demand model closed to actual demand4. Provision of detailed results for SO in terms of optimal decision and costs
We show that capacity requirement of secondary in dayahead market and the delivered energy of tertiary control reserve in real time are physically and economically linked. Moreover, this paper supposes a determination method for secondary reserve capacity considering coupling between pre and post-operation. Proposed dynamic determination method shows more economical efficiency than the traditional method. Simulated results in this paper provide some insights about determining reserve capacity in a different view.
Nomenclature
t The time index for frequency control signal, (2 seconds)k The time index for reserves schedule, (15 minutes)T The time index for hourly energy schedule, (1 hour) Price for hourly scheduled energy, [$/MW] Price for secondary reserve capacity, [$/MW] Price for tertiary reserve capacity, [$/MW] Increased price for tertiary reserve energy, [$/MWh] Decreased price for tertiary reserve energy, [$/MWh] Price for tertiary reserve delivered energy, [$/MWh] Quantity of demand errors, [MW] Quantity of secondary reserve capacity, [MW] Quantity of tertiary reserve capacity, [MW] Level of secondary reserve at time index (k, t), [MW] Level of tertiary reserve at time index (k, t), [MW] Quantity of tertiary reserve delivered energy at time index (k, t), [MWh] Cost of secondary reserve capacity at time k, [$] Cost of tertiary reserve capacity at time k, [$] Cost of tertiary reserve delivered energy at time k, [$] Total cost of frequency control at time k, [$] The cumulated number of tertiary reserve positive activation at time index (k, t) The cumulated number of tertiary reserve negative activation at time index (k, t) Slope for secondary reserve capacity bid curve, [$/MW] Slope for tertiary reserve capacity bid curve, [$/MW] Slope for tertiary reserve energy bid curve (positive adjustment part), [$/MWh] Slope for tertiary reserve energy bid curve (negative adjustment part), [$/MWh] Y-intercept for secondary reserve capacity bid curve, [$] Y-intercept for tertiary reserve capacity bid curve, [$] Y-intercept for tertiary reserve energy bid curve (positive adjustment part), [$] Y-intercept for tertiary reserve energy bid curve (negative adjustment part), [$]σ Standard deviation value for load errors , [MW]μ Mean value for demand errors, [MW]𝜂T Speed of reversion for demand error model [0~1]xk State variable in DP frameworkuk Decision variable in DP frameworkwk Random variable in DP frameworkfk (xk, uk, 𝜔k) Cost function in DP framework𝜏k (xk, uk, 𝜔k) State transition function in DP framework Transition probability distribution in DP frameworklk The state at the stage k in DR framework
References
- M. D. Ilic and X. S. Liu, "A modeling and control framework for operating large-scale electric power system under present and newly evolving competitive industry structure," Mathematical Problems in Eng, vol.1, pp. 317-340, Dec. 1995. https://doi.org/10.1155/S1024123X95000196
- Yann G. Rebours, Daniel S. Kirschen, Marc Trotignon and Sebastien Rossignol, "A Survey of Frequency and Voltage Control Ancillary Services - Part I: Technical Features," IEEE Trans. Power Systems, vol. 22, no. 1, pp. 350-357, Feb. 2007.
- Yann G. Rebours, Daniel S. Kirschen, Marc Trotignon and Sebastien Rossignol, "A Survey of Frequency and Voltage Control Ancillary Services - Part II: Economic Features," IEEE Trans. Power Systems, vol. 22, no.1, pp.358-366, Feb. 2007. https://doi.org/10.1109/TPWRS.2006.888965
- Joskow, Paul L. "Remedying Undue Discrimination through Open Access Transmission Service and Standard Electricity Market Design," Federal Energy Regulatory Commission , Docket No. RM01-12-000, July. 2002.
- Galiana, F.D., Bouffard, F., Arroyo, J.M. and Restrepo, J.F. "Scheduling and Pricing of Coupled Energy and Primary, Secondary, and Tertiary Reserves," Proceedings of the IEEE, vol.93, no.11, Nov. 2005.
- Mohammad Shahidehpour, Hatim Yamin and Zuyi Li, "Market Operations in Electric Power Systems: Forecasting, Scheduling, and Risk Management," Wiley-IEEE Press, March. 2002.
- AA Khatir, R Cherkaoui and M Zima, "Literature Survey on Fundamental Issues of Frequency Control Reserve (FCR) Provision," Swiss Electric Research, Nov. 2010. [Online]. Available:http://mars.ethz.ch/.
- P. Griffes, "A Report to the California Power Exchange: The benefits of a simultaneous vs. sequential PX auction for energy and ancillary services," March. 1999.
- H. Singh and A. Papalexopoulos, "Competitive procurement of ancillary services by an independent system operator," IEEE Transactions on Power Systems, Vol. 14, No. 2, May. 1999.
- Marcelino Madrigal and Victor H. Quintana, "A security-constrained energy and spinning reserve markets clearing system using an interior-point method," IEEE Transactions on Power Systems, Vol. 15, No. 4, Nov. 2000.
- Eric Hirst and Brendan Kirby, "Costs for electric-power ancillary services," The Electricity Journal, Vol.9, Issue 10, pp.26-30, Dec. 1996.
- Tong Wu, Mark Rothleder, Ziad Alaywan and Alex D. Papalexopoulos, "Pricing energy and ancillary services in integrated market systems by an optimal power flow," IEEE Trans. Power Systems, vol.19, no.1, Feb. 2004.
- Jose M. Arroyo and Francisco D. Galiana, "Energy and reserve pricing in security and network-constrained electricity markets," IEEE Trans. Power Systems, vol.20, no.2, May. 2005.
- Shmuel S. Oren, "Design of Ancillary Service Markets," System Sciences, Proceedings of the 34th Annual Hawaii International Conference on, Jan 2001.
- Allen J. Wood and Bruce F. Wollenberg, Power Generation, Operation, and Control: John Wiley & Sons, 2006.
- Yann Rebours, "A comprehensive assessment of markets for frequency and voltage control ancillary services," A thesis submitted to the university of Manchester for the degree of philosophy in the faculty of Engineering and Physical sciences, 2008.
- Rajnish Kamat and Shmuel S. Oren, "Rational Buyer Meets Rational Seller: Reserves Market Equilibria under Alternative Auction Designs," Journal of Regulatory Economics, vol.21, Issue.3, pp.247-288, May. 2002. https://doi.org/10.1023/A:1015378914995
- Zuyi Li and Mohammad Shahidehpour, "Security-Constrained Unit Commitment for Simultaneous Clearing of Energy and Ancillary Services Markets," IEEE Trans. Power Systems, vol.20, NO.2, May. 2005.
- Elaheh Mashhour and Seyed Masoud Moghaddas-Tafreshi, "Bidding strategy of virtual power plant for participating in energy and spinning reserve markets - Part I: Problem formulation," IEEE Trans. Power Systems, vol.26, no.2, pp.949-956, May. 2011. https://doi.org/10.1109/TPWRS.2010.2070884
- Matthew J.C. Walley and David R. Fortin, "Behavioral outcomes from online auctions: reserve price, reserve disclosure, and initial bidding influences in the decision process," Journal of Business Research, vol.58, 2005.
- Derk J. Swider and Christoph Weber, "Bidding under price uncertainty in multi-unit pay-as-bid procurement auctions for power systems reserve," European Journal of Operational Research, vol. 181, Issue 3, 1297-1308, 2007. https://doi.org/10.1016/j.ejor.2005.11.046
- Ernan Ni, Peter B. Luh and Stephen Rourke, "Optimal integrated generation bidding and scheduling with risk management under a deregulated power market," IEEE Trans. Power Systems, vol.19, no.1, pp.600-609, Feb. 2004. https://doi.org/10.1109/TPWRS.2003.818695
- Pathom Attaviriyanupap, Hiroyuki Kita, Eiichi Tanaka and Jun Hasegawa, "New bidding strategy formulation for day-ahead energy and reserve markets based on evolutionary programming," Electrical Power & Energy Systems, vol.27, no.3, pp.157-167, 2005. https://doi.org/10.1016/j.ijepes.2004.09.005
- UCTE Operation Handbook, "Load-frequency control and performance," Policy 1, 2004. [Online]. Available: https://www.entsoe.eu/publications/system-operations-reports/operation-handbook/.
- Gregor Verbic and Ferdinand Gubina, "Cost-based models for the power-reserve pricing of frequency control" IEEE Trans. Power Systems, vol.19, Issue.4, Nov. 2004.
- Erik Ela, Michael Milligan and Brendan Kirby, "Operating Reserves and Variable Generation," National Renewable Energy Laboratory Technical Report, Aug. 2011. [Online]. Available: http://www.nrel.gov/ docs/fy11osti/51978.pdf/.
- Christoph Maurer, Simon Krahl and Holger Weber, "Dimensioning of secondary and tertiary control reserve by probabilistic methods," Euro. Trans. Electr. Power, vol.19, pp.544-552, Jan. 2009. https://doi.org/10.1002/etep.326
- Farzaneh Abbaspourtorbati, Marc Scherer, Andreas Ulbig and Goran Andersson, "Towards an Optimal Activation Pattern of Tertiary Control Reserves in the Power System of Switzerland," American Control Conference (ACC), pp.3629-3636, June. 2012.
- Jan Machowski, W. Bialek Janusz and R. Bumby James, Power System Dynamics: John Wiley & Sons, 1997.
- Avinash K. Dixit and Robert S. Pindyck, Investment under Uncertainty: Princeton University Press, 1994.
- Martin L. Puterman, Markov Decision Processes: Discrete Stochastic Dynamic Programming: John Wiley & Sons, 1994.
- Dimitri P. Bertsekas, Dynamic Programming and Optimal Control: Athena Scientific, Edition.3, July. 2001.
Cited by
- On the Use of Electrical Humidifiers in Office Buildings as a Demand Side Resource vol.32, 2014, https://doi.org/10.1016/j.procs.2014.05.482