(406d) MFIX-Exa: A CFD-DEM Code for Exascale Computer Architectures

The combination of computational fluid dynamics and discrete element method (CFD-DEM) offers an accurate way to model gas-solids reactors. However, CFD-DEM simulations require much computational effort, which currently limits their application to small reactors, typically lab-scale reactors that at most contain tens of millions of particles. To apply CFD-DEM for accelerating fossil energy technology development, the modeling of pilot-scale reactors (~1 MWe) would be required. The modeling of such reactors, which contain hundreds of billions of particles, poses both a computational and data management challenge.

The development of a code called MFIX-Exa, which enables such simulations on exascale computers, began in November 2016. This development project takes place under the Exascale Computing Project (ECP), a collaborative effort of two US Department of Energy organizations – the Office of Science and the National Nuclear Security Administration. ECP was established to accelerate delivery of a capable exascale computing system that integrates hardware and software capability to deliver approximately 50 times more performance than the most powerful supercomputers in use today in the US. ECP’s work encompasses applications, system software, hardware technologies and architectures.

The first version of MFIX-Exa has been developed by migrating the core hydrodynamic models from the widely used, open-source code MFIX into AMReX, a new ECP software framework that uses block-structured adaptive mesh refinement algorithms to solve systems of partial differential equations with complex boundary conditions on exascale architectures. The correctness of MFIX-Exa was established by comparing its solutions with MFIX (version 2016-1) solutions for a set of benchmark cases that represent flow regimes encountered in the project’s challenge problem – a 1 MWe chemical looping reactor. The parallel performance of the code has been improved by implementing logical tiling, enabling OpenMP threading over tiles, and employing a dual grid approach for fluid and particle load balancing. To handle complex geometries in the simulations, AMReX embedded boundary (EB) data structures and iterators were incorporated and an algorithm to address particle collisions with EB walls was implemented. The first major performance milestone for MFIX Exa has been reached with the 0.025 s simulation of 1 billion particles undergoing homogeneous cooling in 12.9 hours of wall time using 6912 processors of NREL’s Peregrine supercomputer.