How to decompose a mesh of huge size effectively?

General Discussion
1

Hi FreeFem Community,

I am working on a code to decompose a 3D mesh by calling FFDDM. The mesh was generated with Gmsh, and has about 46 million elements. The code is running on the HPC of our Lab, and each node has 1T memeory and 56 cores. The key codes read:

include “ffddm.idp”
include “macro_ddm.idp”
load “gmsh”
load “msh3”
mesh3 ThGlobal = gmshload3(“xxx.msh”);
ffddmbuildDmesh(M3d, ThGlobal, mpiCommWorld);

The related options in the command line are: -ffddm_overlap 10 -ffddm_partitioner 1.

I tried to run the code on 1 or 2 nodes to partition the mesh into 56 or 112 subdomains, it terminates with the error: Failed to allocate memory for adjncy.

I also launched the task by allocating 4 nodes, but the log stays at the building stage for over 4 hours: Building decomposition from mesh of 45826143 elements.

Is there more effective method to decompose such a big mesh?

2

Use macro_ddm.idp with ParMETIS instead of METIS (-Dpartitioner=parmetis).

3

Thanks to your quick reply.

Is there a complete example to illustrate the usage of ParMETIS?

4

There is nothing to illustrate beside setting the macro partitioner to parmetis as shown above.

1 Like
5

Hi prj, I’m sorry to bother you again. I have tested the method of inserting a macro in the beginning of the code like:

macro dimension 3 // EOM, 2D or 3D
macro partitioner() parmetis // EOM

include “ffddm.idp”
include “macro_ddm.idp”
load “gmsh”
load “msh3”
load “iovtk”
load “hpddm”

The code works for the mesh of moderate size (about 7 million), but still fails for the mesh having 56 million by allocating 2 nodes (2T memory). When allocating 4 nodes, the log stays at the building stage for over 2 hours: Building decomposition from mesh of 45826143 elements.

Could you give me some suggestions, or are there more options for improving the performance?

6

How are you partitioning the mesh? You have to call DmeshCreate(), as of today, ffddmbuildDmesh() cannot use ParMETIS.

1 Like
7

I have tested this method, and it really works.

But I have some more problems. Suppose ThGlobal is the global mesh to be partitioned. By calling

macro partitioner() parmetis // EOM
…….
DmeshCreate(ThGlobal);

how can I get the local mesh in each subdomain? In addition, is there a convenient way to incorporate DmeshCreate into ffddm or hpddm?

8

ThGlobal will be replaced by just the local mesh (it is your duty to save ThGlobal into a backup mesh if you need the global mesh).

1 Like