(61i) Uncertainty Quantification of Physics Informed Neural Networks Using Bayesian-Last-Layer Approach, and Its Application to Real-World Bioprocess

Physics informed neural networks (PINNs) is an emerging hybrid modeling technique that combine the adaptability of deep learning with the rigor of mechanistic laws ¹. Compared to traditional hybrid modeling techniques where data-driven models (DDMs) and first principle models (FPMs) are arranged in a structural manner, PINN completely integrates the FPM component into a neural network by enforcing the mechanistic laws (usually in the form of ordinary or partial differential equations) during the model training process. This ensures model predictions fall within realistic bounds. The PINN hybrid modeling framework has shown itself to be useful in describing a growing number of diverse and complex systems while remaining computationally scalable ².

For highly regulated applications such as pharmaceutical manufacturing, quantifying the confidence associated with model predictions is critical for process transparency and robustness ³. The predictive accuracy of PINN is inherently limited by the presence of uncertainty, stemming from sources such as unmeasured drivers of variability and measurement error (aleatoric) or modeling assumption and data availability (epistemic). Therefore, it is imperative to quantify the uncertainty associated with PINN prediction to be aware of the intrinsic system noise and “knowledge-poor” regions of the search space ⁴.

Bayesian-last-layer (BLL) is a scalable statistical approach for uncertainty quantification of deep learning models, which is particularly suited for real-time applications ⁵. By introducing the Bayesian framework to only the last layer of a deep neural network, BLL can estimate both the epistemic as well as aleatoric uncertainty of model output while remaining computationally tractable.

This study discusses a novel Bayesian formulation of PINN that is suited for real-time application. This strategy uses BLL formulation to quantify the model uncertainty of PINN. By introducing mechanistic knowledge as the prior for latent variables representing the derivatives ⁵, the proposed probabilistic model will be able to generate confidence-aware predictions while respecting the mechanistic laws.

The proposed modeling approach will be illustrated using both simulation case studies as well as real-world data from Pfizer’s bioreactors across different scales. Predictive performance, as well as uncertainty quantification of the proposed BLL-PINN model, are compared with traditional PINNs and Gaussian Processes.

References:

(1) Raissi, M.; Perdikaris, P.; Karniadakis, G. E. Physics-Informed Neural Networks: A Deep Learning Framework for Solving Forward and Inverse Problems Involving Nonlinear Partial Differential Equations. Journal of Computational Physics 2019, 378, 686–707. https://doi.org/10.1016/j.jcp.2018.10.045.

(2) Bradley, W.; Kim, J.; Kilwein, Z.; Blakely, L.; Eydenberg, M.; Jalvin, J.; Laird, C.; Boukouvala, F. Perspectives on the Integration between First-Principles and Data-Driven Modeling. Computers & Chemical Engineering 2022, 166, 107898. https://doi.org/10.1016/j.compchemeng.2022.107898.

(3) Makrygiorgos, G.; Berliner, A. J.; Shi, F.; Clark, D. S.; Arkin, A. P.; Mesbah, A. Data-Driven Flow-Map Models for Data-Efficient Discovery of Dynamics and Fast Uncertainty Quantification of Biological and Biochemical Systems. Biotechnology and Bioengineering 2023, 120 (3), 803–818. https://doi.org/10.1002/bit.28295.

(4) Yang, L.; Meng, X.; Karniadakis, G. E. B-PINNs: Bayesian Physics-Informed Neural Networks for Forward and Inverse PDE Problems with Noisy Data. Journal of Computational Physics 2021, 425, 109913. https://doi.org/10.1016/j.jcp.2020.109913.

(5) Watson, J.; Lin, J. A.; Klink, P.; Pajarinen, J.; Peters, J. Latent Derivative Bayesian Last Layer Networks. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics; PMLR, 2021; pp 1198–1206.