(151c) Parallelization of Three-Dimensional Discrete Particle Model for Dense Gas-Solid Flows On Unstructured Mesh

        A parallel code for three-dimensional discrete
particle/element model (DPM or DEM) coupled with the computational fluid
dynamics (CFD) model has been developed and used to simulate dense gas-solid
flows in fluidized bed reactors. The gas-phase hydrodynamics is described in
the Eulerian framework while the dispersed phase is in
the Lagrangian one. The finite volume method is
applied to discretize the continuity and momentum
equations of the gas phase on 3D unstructured mesh. The fractional time-marching
scheme is designed for the two-phase momentum coupling. The particle motion is
tracked by directly solving the dynamical equations of each individual
particle. Particle-particle and particle-wall interactions are taken into
account by a soft-sphere model. The numerical solution procedure in a time step
begins by calculating the particle contact forces and integrating the particle
momentum equations over fractional DPM time steps, followed by solving the
gas-phase hydrodynamics with SIMPLE algorithm. During integration of the
particle momentum equations, the hydrodynamic forces on particles are
calculated and the corresponding source terms of the gas momentum are
integrated.

        The coupled
numerical solution of the DPM-CFD problem, described above, poses new
challenges in parallelization. The total load on the DPM part of the code is
directly related to the number of particles, which in our design is targeted to
be in the order of millions of particles. The computational load in the CFD
part can be distributed evenly on many processors using domain decomposition
within the framework of MPI. However, if we use the same structure for DPM, the
heterogeneous spatial distribution of particles will result in poor load
balancing among the many processors.

        The
hybrid parallel computing technique is used in handling the coupled CFD-DPM
model to partly overcome the difficulties identified above and achieve
efficient load balancing. Firstly at the coarse-grain level, the solution
domain is decomposed into partitions using bisection algorithm to minimize the
number of faces at the partition boundaries while keeping almost equal number
of cells in each partition. The solution of the gas-phase governing equations
is performed on these partitions. Particles and the solution of their dynamics
are associated with partitions according to their hosting cells. This makes no
data exchange between processors for calculating the hydrodynamic forces on
particles. By introducing proper data mapping between partitions, the cell void
fraction is calculated accurately even if a particle is shared by several
decomposed partitions. Neighboring partitions are grouped by a gross evaluation
before simulation, with each group having similar number of particles. The
computation task of a group of partitions is assigned to a compute node, which
has multi-cores or multi-processors with a shared memory. Each core or
processor in a node takes the computation of the gas governing equations in one
partition. Processors communicate and exchange data through Message Passing
Interface (MPI) at the coarse-grain parallelism.

        Secondly,
the multithreading technique is used to parallelize the computation of the
dynamics of the particles in each partition. The number of compute threads is
determined according to the number of particles in partitions and the number of
cores or processors in compute node. Thread pooling is
employed to assign computing threads to cores or processors. In such a way there
is almost no waiting of the processors in a compute node. Since the particle
numbers in all compute nodes are almost the same, the above strategy yields an
efficient load balancing among compute nodes. Test numerical experiments on HPC
cluster show that the developed model is scalable up to 128 CPUs and can be
used to simulate dense gas-solid flows in fluidized bed with more than 10 millions
of particles.