(362g) Soft Actor-Critic Deep Reinforcement Learning with Hybrid Actions for Scheduling of Energy Systems Under Demand Response

Recent developments in the field of Deep Reinforcement Learning (DRL) have enabled more complex applications in terms of both environment (dynamic system) and control policy complexity. One such development was the Soft Actor-Critic (SAC) algorithm (Haarnoja et al., 2022), an actor-critic, off-policy method that generates stochastic policies by maximizing the policy’s entropy while simultaneously maximizing the expected return. This creates a natural way of handling the exploration vs. exploitation tradeoff and has been shown to work well for continuous actions following a spherical gaussian distribution. In this work, we outline how hybrid (combined discrete-continuous) actions, commonly found in the operation of energy systems, may be integrated into the SAC framework, and apply the designed hybrid-SAC algorithm to the operation of District Cooling plants under demand forecast uncertainty (Campos et al., 2022). Real plant data is used for building cooling demand and day-ahead electricity prices. Several alternatives are identified based on the recent literature and compared in terms of performance for the selected case study. The best performing method is then further tested under different scenarios. Results show that the hybrid-SAC method presents a good performance, quickly avoiding constraint violations and improving steadily toward the optimum policy. We further show that the hybrid-SAC is robust against demand forecast uncertainty and can learn the optimal policy even under incomplete or partial state information. ­

References

Campos, G., El-Farra, N. H., Palazoglu A. (in press). Soft Actor-Critic Deep Reinforcement Learning with Hybrid Mixed-Integer Actions for Demand Responsive Scheduling of Energy Systems. Industrial & Engineering Chemistry Research, 2022. DOI: 10.1021/acs.iecr.1c04984.

Haarnoja, T., Zhou, A., Abbeel, P., Levine, S. Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor. arXiv:1801.01290 [cs.LG], 2018. Available: https://arxiv.org/abs/1801.01290.