(106c) Inverse Backward Analysis of Neural Approximants of Ordinary Differential Equations

When extracting neural network approximants of differential equations (DEs) from data, various factors can combine to introduce error into the learned DEs. However, even assuming the best-case convergence of the optimizer and cleanliness of the data, the structure of the network used to fit the data can itself introduce a bias into the equations extracted.

We extend previous work [1] in which we defined this inverse modified differential equation, and show that the known results of of forward analysis (where we produce approximate trajectories from a true DE), such as order of convergence, have natural extensions to this field of inverse backward analysis (where we produce an approximate DE from true trajectories).

We establish a theoretical basis for hyperparameter selection when training neural ordinary differential equations [2], by implying a lower bound on the accuracy identification possible with a particular set of integrator hyperparameters. We focus particularly on learning with neural networks templated on implicit numerical integration.

We use a fixed point (FP) iteration in the forward pass of our neural network, and propose an adaptive algorithm to adjust the number of FP iterations during the training process to accelerate training while preserving accuracy. Several numerical experiments are performed to demonstrate the superiority of the proposed algorithm and verify the theoretical analysis.

[1]: Aiqing Zhu, Pengzhan Jin, and Yifa Tang. "Inverse modified differential equations for discovery of dynamics."

[2]: Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, David Duvenaud. "Neural Ordinary Differential Equations." NeurIPS 2018.