(402g) Connections between Residual Networks and Explicit Numerical Integrators, with Applications to Identification of Noninvertible Dynamical Systems

Noninvertibility, an intrinsic property of certain discrete-time dynamical systems, is defined as non-uniqueness of the backward-time dynamics. Moreover, this noninvertibility concept could be also be adapted to neural network predictions that approximate the ground-truth fpr some unknown system [1]. In many cases, the governing equations for discrete-time systems could be treated as a discretized form of a continuous time, ordinary differential equation, which could be solved numerically by, say, explicit or implicit Euler integration methods. These methods, to some extent, are similar in architecture to residual neural networks (ResNets), which have recently seen wide use in machine learning [2].

We begin our work with numerical simulations of chemical oscillatory dynamics. Extending previous work [1], we implemented an algorithm to determine the range of noninvertible regions for the Euler discretization map. Then, training a neural network to discrete time data, we show how noninvertibility arises naturally in neural networks, and how to quantify its occurrence.
In addition to this study of the noninvertibility itself, we investigate its dynamical consequences: dynamic instabilities and pathological phenomena that occur when this feature of the discrete map interacts with the dynamics and bifurcations of the underlying continuous-time system. This study yields useful insights into the long-term prediction power of network architectures. Additionally, our work establishes connections between a network’s invertibility and the estimation of its Lipschitz constants, a connection which has value beyond dynamical systems applications [3].

[1] R. Rico-Martinez, I. G. Kevrekidis and R. A. Adomaitis, "Noninvertibility in neural networks," IEEE International Conference on Neural Networks, San Francisco, CA, USA, 1993, pp. 382-386 vol.1.

[2] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015.

[3] J. Behrmann, W. Grathwohl, R.T.Q. Chen, D. Duvenaud, and J. Jacobsen. Invertible Residual Networks. arXiv preprint arXiv: 1811.00995, 2019.