(339g) Applying Reinforcement Learning to Control Batch Processes

Reinforcement Learning (RL) is one of the three basic machine learning paradigms, alongside supervised and unsupervised learning. RL focuses on how a smart agent can learn the optimal policy from the maximization of cumulative rewards from its environment. The recent development of model-free RL has achieved remarkable success in various control tasks, where multiple applications have been reported in literature including parameter tuning for existent single PID control loops [1], supply chain management [2] and robotic control [3].

In the process control domain, there is an extensive state of art that applies RL algorithms to different types of chemical processes [4,5]. Nevertheless, most of these studies focus on the implementation and application of a specific RL method. These studies usually do not consider comparison with other RL algorithms or with traditional methods in the process control area. Additionally, there is no guideline regarding the RL algorithm selection.

In this work, we will provide a comprehensive investigation of RL applications in the process control area. The above practical issues would be addressed through the case study of a batch bio-reactor. Different types of RL methods including value-based, policy-based, and actor-critic algorithms will be compared in this case study. Specifically, their training stability, convergence, reproducibility and sample efficiency would be evaluated. Beyond the algorithm level evaluation, RL performance on common control scenarios, including control for optimization, control to set point, and control with state constraints would also be performed. Furthermore, the performance of the RL algorithms would be compared with a model predictive controller. Based on these comparisons, we will summarize some guidelines and suggestions on current model-free RL approaches in the control area, as well as some future perspectives regarding RL development and its application in the chemical industry.

Reference

[1] Badgwell, Thomas A., et al. "Adaptive PID controller tuning via deep reinforcement learning." U.S. Patent Application No. 16/218,650.

[2] Oroojlooyjadid, Afshin, et al. "A Deep Q-Network for the Beer Game: A Deep Reinforcement Learning algorithm to Solve Inventory Optimization Problems." arXiv preprint arXiv:1708.05924 (2017).

[3] Peng, Xue Bin, et al. "Sim-to-real transfer of robotic control with dynamics randomization." 2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018.

[4] Spielberg, Steven, et al. "Toward self‐driving processes: A deep reinforcement learning approach to control." AIChE Journal 65.10 (2019): e16689.

[5] Petsagkourakis, Panagiotis, et al. "Reinforcement learning for batch bioprocess optimization." Computers & Chemical Engineering 133 (2020): 106649.