(161a) Systematic Process Design Via Reinforcement Learning: An Hydrodealkylation Example

Computer-aided process design plays a critical role in chemical process industry which can quantitatively evaluate process alternatives and identify ways of improvements. However, it is a challenging task to decide on the optimal design among plethora of plausible process configurations. Process synthesis [1-2] provides an instrumental tool striving to determine the most promising equipment and stream combinations together with optimal design and operating conditions driven by mathematical optimization. These strategies typically require a postulation of a superstructure network encapsulating all flowsheets of interest [3-4], which restrict the optimality and creativity of process design by expert knowledge. More recently, reinforcement learning (RL)-assisted process design has received increasing interest [5-7] which offers the potential to identify optimal process solutions without pre-postulating process structures. However, a key open research question lies in how to enable the RL agent to effectively search the resulting combinatorial design space, in which the number of infeasible designs can be larger than the feasible ones in orders of magnitude.

In this work, we present an extension of the RL approach for systematic process design recently developed under a collaborative project [8]. Therein, the workflow starts with the users selecting a pool of candidate process units (e.g., flash column, reactor), the rigorous models of which are supported by the Institute for the Design of Advanced Energy System (IDAES) Integrated Platform [9]. Initial process designs are generated via random initializations as stream inlet-outlet matrices and optimized using the IDAES platform, the objective function value of which is the reward to RL agent. Deep Q-Network [10] is employed as the RL agent including a series of convolutional neural network layers and fully connected layers to compute the actions of adding or removing any stream connections, thus creating a new process design. A new posterior recycle screening procedure is incorporated which can systematically identify the redundant recycle loops due to the existence of mixer(s) and splitter(s). The process design will be informed back to the RL agent to refine its learning. The iteration continues until the maximum number of steps is reached with a number of feasible process designs generated. To further expedite the RL search of the design space which can comprise the selection of any candidate unit(s) with arbitrary stream connections, we investigate the different definitions of RL reward function and their impacts in driving RL agent to explore more complicated versus intensified process configurations. A sub-space search strategy is also developed to branch the feasible design space with parallel computing to accelerate the discovery of feasible process design solutions and ensure the robust search particularly when a large pool of candidate process units is selected by the user. The potential and efficiency of the enhanced RL-assisted process design strategy will be demonstrated via a hydrodealkylation example.

References

[1] Mencarelli, L., Chen, Q., Pagot, A., & Grossmann, I. E. (2020). A review on superstructure optimization approaches in process system engineering. Computers & Chemical Engineering, 136, 106808.

[2] Pistikopoulos, E. N., Barbosa-Povoa, A., Lee, J. H., Misener, R., Mitsos, A., Reklaitis, G. V., Venkatasubramanian, V., You, F, & Gani, R. (2021). Process systems engineering–the generation next?. Computers & Chemical Engineering, 147, 107252.

[3] Siirola, J. J. (1996). Strategic process synthesis: Advances in the hierarchical approach. Computers & chemical engineering, 20, S1637-S1643.

[4] Ryu, J., Kong, L., de Lima, A. E. P., & Maravelias, C. T. (2020). A generalized superstructure-based framework for process synthesis. Computers & Chemical Engineering, 133, 106653.

[5] Stops, L., Leenhouts, R., Gao, Q., & Schweidtmann, A. M. (2022). Flowsheet synthesis through hierarchical reinforcement learning and graph neural networks. arXiv preprint arXiv:2207.12051.

[6] Midgley, L. I. (2020). Deep reinforcement learning for process synthesis. arXiv preprint arXiv:2009.13265.

[7] Göttl, Q., Tönges, Y., Grimm, D. G., & Burger, J. (2021). Automated flowsheet synthesis using hierarchical reinforcement learning: proof of concept. Chemie Ingenieur Technik, 93(12), 2010-2018.

[8] Wang, D., Bao, J., Zamarripa-Perez, M., Paul, B., Chen, Y., Gao, P., Ma, T., Noring, A., Iyengar, A., Schwartz, D., Eggleton, E., He, Q., Liu, A., Marina, O., Koeppel, B. & Xu, Z. (2022). Reinforcement learning for automated conceptual design of advanced energy and chemical systems. https://doi.org/10.21203/rs.3.rs-2248780/v1.

[9] Lee, A., Ghouse, J. H., Eslick, J. C., Laird, C. D., Siirola, J. D., Zamarripa, M. A., Gunter, D., Shinn, J., Dowling, A., Bhattacharyya, D., Biegler, L. T., Burgard, A. P. & Miller, D. C. (2021). The IDAES process modeling framework and model library – Flexibility for process simulation and optimization. Journal of Advanced Manufacturing and Processing, 3(3), e10095.

[10] Li, Y. (2017). Deep reinforcement learning: An overview. arXiv preprint arXiv:1701.07274.