(110e) Evaluation of Metamodeling Techniques for Discrete-Time Approximations

In many applications of multi-scale modeling, the simulation requires a large computational time. Examples of expensive simulations in science and engineering are detailed atomic calculations of metal organic frameworks for membrane separations, crash simulations of aircrafts and automobiles for safety, and structural analysis and protein structure predictions using molecular dynamics. As a result, these simulations are not well suited for engineering applications like design, control or optimization, because of the large number of function evaluations that would be required.

Many techniques have been developed within mathematics, computer science and engineering [1,2] to build a ?model of the model? or metamodel [3] from simulation input-output data. The goal of metamodeling is to create a surrogate model that accurately describes the nonlinear relationships among key variables in the original simulation, but with reduced computation. Despite the number of regression-based metamodeling strategies and their wide range of applications, few of them have been used as surrogates for expensive dynamic simulations.

In this work, we compare different metamodeling approaches as surrogates for stochastic dynamic simulations. The metamodels tested include equation-free prediction, in-situ adaptive tabulation, Gaussian process models, response surface methodology and radial basis functions. These metamodels are used to approximate the discrete-time evolution of a multidimensional state under the assumption of a time-invariant Markov process. The metamodels are compared by fixing the total number of simulations used to build the metamodel. This total number of simulation evaluations includes the initial evaluations for preliminary parameter estimation and additional evaluations designed for further improvements in the predictions. Thus, we allow flexibility in the construction of the sampling scheme, tailored toward the particular metamodel.

The aim of our work is to explore the strengths and weaknesses of commonly used metamodeling strategies in a dynamic application. Performance indicators in our study include root mean square error and computational cost. We also compare the response of the metamodel under different signal-to-noise ratios in the stochastic simulations. Results show how a Gaussian process model uses repeated functions evaluations to separately identify locally correlated errors and uncorrelated simulation noise. A Gaussian process model balances the repeated evaluations with a space filling sampling strategy to minimize the error prediction across the entire state space. As a result, a 2% relative error in mean prediction is achieved with 90% reduction of computational cost, compared with the original stochastic simulations. Finally, we conclude our study providing recommendations about the use of metamodels in large-scale dynamic problems and research directions towards uncertainty estimation in metamodeling.

References

[1] D. Givon, R. Kupferman, and A. Stuart, ?Extracting macroscopic dynamics: Model problems and algorithms,? Nonlinearity, vol. 17, pp. R55-R127, 2004.

[2] G. G. Wang and S. Shan, ?Review of metamodeling techniques in support of engineering design optimization,? Journal of Mechanical Design, vol. 129, pp. 370-380, 2007.

[3] T. W. Simpson, J. D. Peplinski, P. N. Koch, and J. K. Allen, ?Metamodels for computer-based engineering design: Survey and recommendations,? Engineering with Computers, vol. 17, pp. 129-150, 2001.