# Stall in some multi-node parallel calculations

**URL:** <https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143>\
**Category:** General Discussion\
**Created:** [August 6, 2021, 11:36am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143 "2021-08-06T11:36:47Z")\
**Posts on this page:** 20\
**Page:** 4

<div class="post-metadata">

**Author:** ![etj](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@etj](https://community.freefem.org/u/etj)\
**Post date:** [August 13, 2021, 7:01am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/61 "2021-08-13T07:01:44Z")

</div>

No, I applied them yesterday and didn’t change anything after that. And `stokes-2d-PETSc.edp` contains the line `include "macro_ddm.idp"`. Could it be that `stokes-2d-PETSc.edp` is such that the for loop or the if statement in the part with `true neighbor` and `false neighbor` is not visited?

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [August 13, 2021, 7:07am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/62 "2021-08-13T07:07:45Z")

</div>

Did you edit the `.edp` as I told you (`createMat` instead of `buildMat`)? There is a `cout` for both the `if` and the `else`, if nothing is printed, it means there are no intersection between subdomains, but then there is output from the PETSc plugin saying there are intersections, so that’s not possible.

---

<div class="post-metadata">

**Author:** ![etj](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@etj](https://community.freefem.org/u/etj)\
**Post date:** [August 13, 2021, 7:18am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/63 "2021-08-13T07:18:53Z")

</div>

Yes, the modified `.edp` is the following:

```auto
// run with MPI: ff-mpirun -np 4 script.edp
// NBPROC 4

load "PETSc" // PETSc plugin
macro dimension()2// EOM // 2D or 3D
include "macro_ddm.idp" // additional DDM functions

macro def(i)[i, i#B, i#C]// EOM // vector field definition
macro init(i)[i, i, i]// EOM // vector field initialization
macro grad(u)[dx(u), dy(u)]// EOM // two-dimensional gradient
real Sqrt = sqrt(2.);
macro epsilon(u)[dx(u), dy(u#B), (dy(u) + dx(u#B)) / Sqrt]// EOM
macro div(u)(dx(u) + dy(u#B))// EOM
func Pk = [P2, P2, P1]; // finite element space

mesh Th;
{
    mesh ThGlobal = square(getARGV("-global", 40), getARGV("-global", 40), [x, y]); // global mesh
    ThGlobal = trunc(ThGlobal, (x < 0.5) || (y < 0.5), label = 5);
    Th = movemesh(ThGlobal, [-x, y]);
    Th = ThGlobal + Th;
}
Mat A;
//buildMat(Th, getARGV("-split", 1), A, Pk, mpiCommWorld)
  createMat(Th, A, Pk);

fespace Wh(Th, Pk); // local finite element space
varf vPb([u, uB, p], [v, vB, q]) = int2d(Th)(grad(u)' * grad(v) + grad(uB)' * grad(vB) - div(u) * q - div(v) * p + 1e-10 * p * q) + on(1, 3, 5, u = 0, uB = 0) + on(2, u = y*(0.5-y), uB = 0);
real[int] rhs = vPb(0, Wh);

set(A, sparams = "-pc_type lu -pc_factor_mat_solver_type mumps");
Wh<real> def(u);

A = vPb(Wh, Wh);
u[] = A^-1 * rhs;

macro def2(u)[u, u#B]// EOM
macro def1(u)u// EOM
plotMPI(Th, def2(u), [P2, P2], def2, real, cmm = "Global velocity")
plotMPI(Th, uC, P1, def1, real, cmm = "Global pressure")

```

Let me retry interactively, just to make sure.  
By the way, it doesn’t matter if one puts a final semicolon or not after `createMat(Th, A, Pk)`?

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [August 13, 2021, 7:22am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/64 "2021-08-13T07:22:19Z")

</div>

No, it’s a macro so that’s not big deal. You can put a random `cout` in `macro_ddm.idp` near the `mpiWaitAny` that you changed, my guts tell me it will never be displayed, you probably have another `macro_ddm.idp` file in your system that is being used instead.

---

<div class="post-metadata">

**Author:** ![etj](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@etj](https://community.freefem.org/u/etj)\
**Post date:** [August 13, 2021, 7:30am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/65 "2021-08-13T07:30:54Z")

</div>

Ok, for some reason line 251 is `for(...) mpiWaitAll(rq);`, but it should be `for(...) mpiWaitAny(rq);`, is that correct?

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [August 13, 2021, 7:31am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/66 "2021-08-13T07:31:53Z")

</div>

Yes, that is correct.

---

<div class="post-metadata">

**Author:** ![etj](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@etj](https://community.freefem.org/u/etj)\
**Post date:** [August 13, 2021, 7:32am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/67 "2021-08-13T07:32:51Z")

</div>

Ok, my mistake, I misread this part:

> [@prj](#):
>
> Excellent! Now it should be rather straightforward to debug. Line 251, could you please replace  
> `mpiWaitAll(rq);` by `for(int debugI = 0; debugI < rq.n; ++debugI) mpiWaitAny(rq);`. Does the code still hang?

---

<div class="post-metadata">

**Author:** ![etj](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@etj](https://community.freefem.org/u/etj)\
**Post date:** [August 13, 2021, 7:56am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/68 "2021-08-13T07:56:20Z")

</div>

So, coming back to where we were yesterday evening, here is the output for my code on 2x2 processes (runs to completion, but hangs on 2x8 processes):

```auto
-- FreeFem++ v4.9 (Fri Jun 18 14:45:02 CEST 2021 - git v4.9)
 Load: lg_fem lg_mesh lg_mesh3 eigenvalue parallelempi 
 load: init metis (v 5 )
 (already loaded: msh3) sizestack + 1024 =9128 ( 8104 )

0 true neighbor is 1 (1)
0 true neighbor is 2 (2)
0 false neighbor bis is 3 (2), eps = 0.502167, epsTab = 0
0: 
2	
	  1 2	
2747	2047	
rank 0 sending/receiving 2747 to 1
1 true neighbor is 0 (1)
1 true neighbor is 2 (2)
1: 
2	
	  0 2	
2747	2995	
rank 1 sending/receiving 2747 to 0
rank 0 sending/receiving 2047 to 2
3 false neighbor bis is 0 (0), eps = 0.445833, epsTab = 0
3 true neighbor is 2 (1)
3: 
1	
	  2	
2950	
rank 3 sending/receiving 2950 to 2
rank 1 sending/receiving 2995 to 2
rank 0 received from 1 (1) with tag 0 and count 2747
rank 1 received from 0 (0) with tag 0 and count 2747
rank 0 received from 2 (2) with tag 0 and count 2047
rank 1 received from 2 (2) with tag 0 and count 2995
Done
Done
2 true neighbor is 0 (1)
2 true neighbor is 1 (2)
2 true neighbor is 3 (3)
2: 
3	
	  0 1 3	
2047	2995	2950	
rank 2 sending/receiving 2047 to 0
rank 2 sending/receiving 2995 to 1
rank 2 sending/receiving 2950 to 3
rank 3 received from 2 (2) with tag 0 and count 2950
rank 2 received from 3 (3) with tag 0 and count 2950
rank 2 received from 0 (0) with tag 0 and count 2047
rank 2 received from 1 (1) with tag 0 and count 2995
Done
Done

```

And here is the output for `stokes-2d-PETSc.edp` (modified with `createMat` instead of `buildMat`) on 2x2 processes (hangs):

```auto
-- FreeFem++ v4.9 (Fri Jun 18 14:45:02 CEST 2021 - git v4.9)
 Load: lg_fem lg_mesh lg_mesh3 eigenvalue parallelempi 
 load: init metis (v 5 )
 sizestack + 1024 =10800 ( 9776 )

  -- Square mesh : nb vertices =1681 , nb triangles = 3200 , nb boundary edges 160
  -- Square mesh : nb vertices =1681 , nb triangles = 3200 , nb boundary edges 160
  -- Square mesh : nb vertices =1681 , nb triangles = 3200 , nb boundary edges 160
  -- Square mesh : nb vertices =1681 , nb triangles = 3200 , nb boundary edges 160
rank 2 sending/receiving 426 to 3
rank 1 sending/receiving 480 to 0
rank 1 sending/receiving 964 to 3
 --- global mesh of 4800 elements (prior to refinement) partitioned with metis --metisA: 4-way Edge-Cut: 3, Balance: 1.01 Nodal=0/Dual 1
 (in 4.355907e-03)
 --- partition of unity built (in 2.466662e-01)
rank 0 sending/receiving 480 to 1
rank 1 received from 0 (0) with tag 0 and count 480
rank 0 received from 1 (1) with tag 0 and count 480
rank 1 received from 3 (3) with tag 0 and count 964
rank 3 sending/receiving 964 to 1
rank 3 sending/receiving 426 to 2
rank 2 received from 3 (3) with tag 0 and count 426
rank 3 received from 2 (2) with tag 0 and count 426
rank 3 received from 1 (1) with tag 0 and count 964
 --- global numbering created (in 5.788803e-03)
 --- global CSR created (in 7.145405e-04)
 Warning: -- Your set of boundary condition is incompatible with the mesh label.
 Warning: -- Your set of boundary condition is incompatible with the mesh label.
 Warning: -- Your set of boundary condition is incompatible with the mesh label.
 Warning: -- Your set of boundary condition is incompatible with the mesh label.

```

Putting a few `cout` in `macro_ddm.idp` suggests that my code calls the correct `macro_ddm.idp` but that `stokes-2d-PETSc.edp` does not… What can we do?

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [August 13, 2021, 7:57am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/69 "2021-08-13T07:57:56Z")

</div>

As I told you already, you have another `macro_ddm.idp` somewhere on your machine that you need to locate and modify as well. Probably in `../../idp/macro_ddm.idp` (relatively to `stokes-2d-PETSc.edp`).

---

<div class="post-metadata">

**Author:** ![etj](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@etj](https://community.freefem.org/u/etj)\
**Post date:** [August 13, 2021, 8:00am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/70 "2021-08-13T08:00:04Z")

</div>

Output of `find ./ -name "macro_ddm.idp"`:

```auto
./lib/ff++/4.9/idp/macro_ddm.idp
./FreeFem-sources/idp/macro_ddm.idp

```

---

<div class="post-metadata">

**Author:** ![etj](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@etj](https://community.freefem.org/u/etj)\
**Post date:** [August 13, 2021, 8:32am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/71 "2021-08-13T08:32:41Z")

</div>

In `FreeFem-sources/idp/`, after applying successively the 2 patches, and modifying line 251 of `macro_ddm.idp` as `for(int debugI = 0; debugI < 2*intersection[0].n; ++debugI) mpiWaitAny(rq);`, the output for `stokes-2d-PETSc.edp` is now:

```auto
-- FreeFem++ v4.9 (Fri Jun 18 14:45:02 CEST 2021 - git v4.9)
 Load: lg_fem lg_mesh lg_mesh3 eigenvalue parallelempi 
 load: init metis (v 5 )
 The Identifier intersection does not exist 

 Error line number 207, in file ../../idp/macro_ddm.idp, before token intersection

  current line = 207 mpirank 0 / 4
Compile error : 
	line number :207, intersection
  current line = 207 mpirank 2 / 4
error Compile error : 
	line number :207, intersection
 code = 1 mpirank: 0
  current line = 207 mpirank 1 / 4
  current line = 207 mpirank 3 / 4

```

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [August 13, 2021, 9:12am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/72 "2021-08-13T09:12:21Z")

</div>

You’ve messed up the file, it runs fine on my machine.

---

<div class="post-metadata">

**Author:** ![etj](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@etj](https://community.freefem.org/u/etj)\
**Post date:** [August 13, 2021, 10:59am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/73 "2021-08-13T10:59:37Z")

</div>

Thank you for this constructive comment. So what can I do? Should the file in `FreeFem-sources/idp/` be the same as in `lib/ff++/4.9/idp/`?

---

<div class="post-metadata">

**Author:** ![etj](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@etj](https://community.freefem.org/u/etj)\
**Post date:** [August 13, 2021, 11:04am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/74 "2021-08-13T11:04:12Z")

</div>

Note that `diff lib/ff++/4.9/idp/macro_ddm.idp FreeFem-sources/idp/macro_ddm.idp` returns:

```auto
1d0
< cout << "toto" << endl;
252d250
< cout << "test 1" << endl;
254d251
< cout << "test 2" << endl;
256d252
< cout << "test 3" << endl;
260d255
< cout << "test 4" << endl;
268c263
< } else cout << mpirank << " false neighbor bis is " << intersection[0][i] << " (" << numberIntersection << "), eps = " << eps << ", epsTab = " << epsTab[i] << "\n";
---
> } else cout << mpirank << " false neighbor bis is " << intersection[0][i] << " (" << numberIntersection << ")\n";

```

So if is true that I forgot to modify one `cout` statement, I don’t think I messed up the file.

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [August 13, 2021, 11:16am UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/75 "2021-08-13T11:16:30Z")

</div>

> So what can I do?

You can remove the `-ns` flag and figure out why the variable `intersection` is not defined.

---

<div class="post-metadata">

**Author:** ![etj](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@etj](https://community.freefem.org/u/etj)\
**Post date:** [August 13, 2021, 12:14pm UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/76 "2021-08-13T12:14:54Z")

</div>

Now, although I have not modified `macro_ddm.idp` further, I can’t reproduce the error about `intersection`… The output is the following with the `-ns` flag:

```auto
-- FreeFem++ v4.9 (Fri Jun 18 14:45:02 CEST 2021 - git v4.9)
 Load: lg_fem lg_mesh lg_mesh3 eigenvalue parallelempi 
 load: init metis (v 5 )
 sizestack + 1024 =10824 ( 9800 )

  -- Square mesh : nb vertices =1681 , nb triangles = 3200 , nb boundary edges 160
  -- Square mesh : nb vertices =1681 , nb triangles = 3200 , nb boundary edges 160
  -- Square mesh : nb vertices =1681 , nb triangles = 3200 , nb boundary edges 160
  -- Square mesh : nb vertices =1681 , nb triangles = 3200 , nb boundary edges 160
2 true neighbor is 1 (1)
2 true neighbor is 3 (2)
2: 
2	
	  1 3	
190	426	
rank 2 sending/receiving 190 to 1
 --- global mesh of 4800 elements (prior to refinement) partitioned with metis --metisA: 4-way Edge-Cut: 3, Balance: 1.01 Nodal=0/Dual 1
 (in 4.319191e-03)
0 true neighbor is 1 (1)
 --- partition of unity built (in 2.465439e-01)
0: 
1	
	  1	
480	
rank 0 sending/receiving 480 to 1
1 true neighbor is 0 (1)
1 false neighbor bis is 2 (1)
1 true neighbor is 3 (2)
1: 
2	
	  0 3	
480	964	
rank 1 sending/receiving 480 to 0
rank 1 sending/receiving 964 to 3
3 true neighbor is 1 (1)
3 true neighbor is 2 (2)
3: 
2	
	  1 2	
964	426	
rank 3 sending/receiving 964 to 1
rank 3 sending/receiving 426 to 2
rank 0 received from 1 (1) with tag 0 and count 480
rank 2 sending/receiving 426 to 3
rank 1 received from 0 (0) with tag 0 and count 480
rank 2 received from 3 (3) with tag 0 and count 426
rank 1 received from 3 (3) with tag 0 and count 964
rank 3 received from 2 (2) with tag 0 and count 426
rank 3 received from 1 (1) with tag 0 and count 964

```

and without the `-ns` flag:

```auto
(...)
   41 : sizestack + 1024 =10824 ( 9800 )

  -- Square mesh : nb vertices =1681 , nb triangles = 3200 , nb boundary edges 160
  -- Square mesh : nb vertices =1681 , nb triangles = 3200 , nb boundary edges 160
 --- global mesh of 4800 elements (prior to refinement) partitioned with metis --metisA: 4-way Edge-Cut: 3, Balance: 1.01 Nodal=0/Dual 1
 (in 4.315138e-03)
0 true neighbor is 1 (1)
 --- partition of unity built (in 2.509263e-01)
0: 
1	
	  1	
480	
rank 0 sending/receiving 480 to 1
1 true neighbor is 0 (1)
1 false neighbor bis is 2 (1)
1 true neighbor is 3 (2)
1: 
2	
	  0 3	
480	964	
rank 1 sending/receiving 480 to 0
rank 1 sending/receiving 964 to 3
rank 0 received from 1 (1) with tag 0 and count 480
2 true neighbor is 1 (1)
2 true neighbor is 3 (2)
2: 
2	
	  1 3	
190	426	
rank 2 sending/receiving 190 to 1
rank 2 sending/receiving 426 to 3
rank 1 received from 0 (0) with tag 0 and count 480
rank 1 received from 3 (3) with tag 0 and count 964
3 true neighbor is 1 (1)
3 true neighbor is 2 (2)
3: 
2	
	  1 2	
964	426	
rank 3 sending/receiving 964 to 1
rank 3 sending/receiving 426 to 2
rank 2 received from 3 (3) with tag 0 and count 426
rank 3 received from 2 (2) with tag 0 and count 426
rank 3 received from 1 (1) with tag 0 and count 964

```

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [August 13, 2021, 2:35pm UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/77 "2021-08-13T14:35:47Z")

</div>

Let me get this straight. In [this](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/71) post, FreeFEM returned a syntaxical error `The Identifier intersection does not exist `. Then, in [that](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/76) post, without changing anything, the error magically disappeared? That’s not possible. The FreeFEM parser is 100% reproducible, nothing random involved, nothing in parallel. By the way, I can reproduce the same error you had, i.e.:

```auto
The Identifier intersection does not exist

 Error line number 207, in file ../../idp/macro_ddm.idp, before token intersection

  current line = 207 mpirank 0 / 4
Compile error :
	line number :207, intersection

```

by messing up `macro_ddm.idp` in the following way:

```auto
diff --git a/idp/macro_ddm.idp b/idp/macro_ddm.idp
index a29dacf0..2a930fb3 100644
--- a/idp/macro_ddm.idp
+++ b/idp/macro_ddm.idp
@@ -250,3 +250,4 @@ ENDIFMACRO
         n2oNeighbor.resize(0);
- mpiWaitAll(rq);
+ // mpiWaitAll(rq);
+ for(int debugI = 0; debugI < 2*intersection[0].n; ++debugI) mpiWaitAny(rq);
         for(int i = 0; i < intersection[0].n; ++i) {

```

Notice the commented code, that’s not OK to put `//` comments in macro definition.

---

<div class="post-metadata">

**Author:** ![etj](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@etj](https://community.freefem.org/u/etj)\
**Post date:** [August 13, 2021, 2:43pm UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/78 "2021-08-13T14:43:07Z")

</div>

Ok, so perhaps instead of modifying a line I had commented it out, and then I guess I deleted it before doing `diff ...`.

Now, what about the output of `stokes-2d-PETSc.edp`?

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [August 13, 2021, 2:50pm UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/79 "2021-08-13T14:50:52Z")

</div>

> Ok, so perhaps instead of modifying a line I had commented it out, and then I guess I deleted it before doing `diff ...` .

Thank you for this informative comment.

> Now, what about the output of stokes-2d-PETSc.edp?

Are you using `mpiWaitAll` or a loop of `mpiWaitAny` in `macro_ddm.idp`? If you are using a loop of `mpiWaitAny` and it is still not working, try also the following.

```auto
diff --git a/idp/macro_ddm.idp b/idp/macro_ddm.idp
index a29dacf0..dd01df6b 100644
--- a/idp/macro_ddm.idp
+++ b/idp/macro_ddm.idp
@@ -250,3 +250,4 @@ ENDIFMACRO
         n2oNeighbor.resize(0);
- mpiWaitAll(rq);
+ for(int debugI = 0; debugI < 2*intersection[0].n; ++debugI) mpiWaitAny(rq);
+ for(int debugI = 0; debugI < intersection[0].n; ++debugI) epsTab[debugI] = 1.0;
         for(int i = 0; i < intersection[0].n; ++i) {

```

---

<div class="post-metadata">

**Author:** ![etj](https://avatars.discourse-cdn.com/v4/letter/e/a698b9/32.png) [@etj](https://community.freefem.org/u/etj)\
**Post date:** [August 13, 2021, 3:03pm UTC](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/80 "2021-08-13T15:03:46Z")

</div>

> [@prj](#):
>
> Thank you for this informative comment.

🙂

I was using a loop of `mpiWaitAny`. I modified the file, and now the output is:

```auto
-- FreeFem++ v4.9 (Fri Jun 18 14:45:02 CEST 2021 - git v4.9)
 Load: lg_fem lg_mesh lg_mesh3 eigenvalue parallelempi 
 load: init metis (v 5 )
 sizestack + 1024 =10832 ( 9808 )

Entering FreeFem-sources/idp/macro_ddm.idp
  -- Square mesh : nb vertices =1681 , nb triangles = 3200 , nb boundary edges 160
Entering FreeFem-sources/idp/macro_ddm.idp
  -- Square mesh : nb vertices =1681 , nb triangles = 3200 , nb boundary edges 160
Entering FreeFem-sources/idp/macro_ddm.idp
  -- Square mesh : nb vertices =1681 , nb triangles = 3200 , nb boundary edges 160
Entering FreeFem-sources/idp/macro_ddm.idp
  -- Square mesh : nb vertices =1681 , nb triangles = 3200 , nb boundary edges 160
 --- global mesh of 4800 elements (prior to refinement) partitioned with metis --metisA: 4-way Edge-Cut: 3, Balance: 1.01 Nodal=0/Dual 1
 (in 4.330397e-03)
0 true neighbor is 1 (1)
 --- partition of unity built (in 2.576466e-01)
0: 
1	
	  1	
480	
rank 0 sending/receiving 480 to 1
1 true neighbor is 0 (1)
1 true neighbor is 2 (2)
1 true neighbor is 3 (3)
1: 
3	
	  0 2 3	
480	190	964	
rank 1 sending/receiving 480 to 0
rank 1 sending/receiving 190 to 2
2 true neighbor is 1 (1)
2 true neighbor is 3 (2)
2: 
2	
	  1 3	
190	426	
rank 2 sending/receiving 190 to 1
rank 0 received from 1 (1) with tag 0 and count 480
3 true neighbor is 1 (1)
3 true neighbor is 2 (2)
3: 
2	
	  1 2	
964	426	
rank 3 sending/receiving 964 to 1
rank 3 sending/receiving 426 to 2
rank 1 sending/receiving 964 to 3
rank 2 sending/receiving 426 to 3
rank 1 received from 0 (0) with tag 0 and count 480
rank 1 received from 2 (2) with tag 0 and count 190
rank 1 received from 3 (3) with tag 0 and count 964
rank 2 received from 3 (3) with tag 0 and count 426
 --- global numbering created (in 4.596710e-04)
rank 2 received from 1 (1) with tag 0 and count 190
 --- global CSR created (in 6.911755e-04)
rank 3 received from 2 (2) with tag 0 and count 426
rank 3 received from 1 (1) with tag 0 and count 964
 Warning: -- Your set of boundary condition is incompatible with the mesh label.
 Warning: -- Your set of boundary condition is incompatible with the mesh label.
 Warning: -- Your set of boundary condition is incompatible with the mesh label.
 Warning: -- Your set of boundary condition is incompatible with the mesh label.
 --- system solved with PETSc (in 4.188831e-01)
times: compile 6.200000e-01s, execution 8.000000e-01s, mpirank:0
 ######## We forget of deleting 0 Nb pointer, 0Bytes , mpirank 0, memory leak =2456640
 CodeAlloc : nb ptr 7304, size :654304 mpirank: 0
Ok: Normal End
times: compile 0.61s, execution 0.8s, mpirank:1
 ######## We forget of deleting 0 Nb pointer, 0Bytes , mpirank 1, memory leak =2457456
times: compile 0.6s, execution 0.79s, mpirank:2
 CodeAlloc : nb ptr 7304, size :654304 mpirank: 1
 ######## We forget of deleting 0 Nb pointer, 0Bytes , mpirank 2, memory leak =2458448
 CodeAlloc : nb ptr 7304, size :654304 mpirank: 2
times: compile 0.61s, execution 0.8s, mpirank:3
 ######## We forget of deleting 0 Nb pointer, 0Bytes , mpirank 3, memory leak =2457952
 CodeAlloc : nb ptr 7304, size :654304 mpirank: 3

```

[Previous page](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143.md?page=3)

[Next page](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143.md?page=5)