(303h) Learning-Based Economic Model Predictive Control for Energy Storage Systems Under Imperfect Forecast Information

Energy storage systems have to be managed in the presence of different forms of uncertainty: time-varying and stochastic weather conditions, spot electricity prices, and energy demand. An important feature in many of these problems is the availability of rolling forecasts with varying levels of accuracy (Powell, 2014; Anderson et al., 2011). A standard approach to solve these problems is to use a deterministic economic model predictive control formulation, where the available forecast is used to compute control actions under the assumption that the predictions are correct. However, this certainty equivalence approach can result in arbitrarily poor performance depending on the level of accuracy of the forecast. In this work, we explore a way of learning the optimal control policy in the presence of imperfect rolling forecasts by combining economic model predictive control (EMPC) with reinforcement learning (RL) techniques. We illustrate this approach in an energy storage case study, where the time-varying energy demand from a building must be satisfied by a wind farm (stochastic, but free supply), a battery, and/or the power grid (unlimited supply, but stochastic prices). We assume that inaccurate rolling forecasts of the wind, demand, and electricity prices are available and can be used for control purposes. The objective is to design a controller to operate the system such that the energy demand from the building is satisfied at minimum cost.

It has recently been shown that an EMPC scheme can be tuned to deliver the optimal policy of the real system even when using the wrong model (Gros and Zanon, 2019). This suggests that EMPC can be used as a function approximator within RL. In this work, we use this idea to design an EMPC scheme where the stage cost, terminal cost, and constraints are carefully parametrized. These parameters are then adjusted online to improve performance under imperfect forecast information using reinforcement learning techniques such as Q-learning. We show in closed-loop simulations how the RL-tuned MPC is able to improve performance under imperfect forecast information.

References:

Anderson, R. N., Boulanger, A., Powell, W. B., and Scott, W. (2011). Adaptive stochastic control for the smart grid. Proceedings of the IEEE, 99(6):1098–1115.

Gros, S. and Zanon, M. (2019). Data-driven economic NMPC using reinforcement learning. IEEE Transactions on Automatic Control, 65(2):636–648.

Powell, W. (2014). Energy and uncertainty: Models and algorithms for complex energy systems. AI Magazine, 35(3):8–21.