# Memory allocation problem in MPI computation using PETSc

**URL:** <https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071>\
**Category:** General Discussion\
**Created:** [October 18, 2022, 11:27am UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071 "2022-10-18T11:27:55Z")\
**Posts on this page:** 20\
**Page:** 2

<div class="post-metadata">

**Author:** ![marcodeba](https://avatars.discourse-cdn.com/v4/letter/m/977dab/32.png) [@marcodeba](https://community.freefem.org/u/marcodeba)\
**Post date:** [October 19, 2022, 9:53am UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/21 "2022-10-19T09:53:51Z")

</div>

What do you mean with the command `part[] = A.D.`? What should I insert instead of A.D.? This is not clear to me, I’m sorry

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [October 19, 2022, 9:55am UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/22 "2022-10-19T09:55:22Z")

</div>

If `foobar` is the name of your PETSc Mat, then insert `foobar.D`, see, e.g., [FreeFem-sources/navier-stokes-2d-PETSc.edp at master · FreeFem/FreeFem-sources · GitHub](https://github.com/FreeFem/FreeFem-sources/blob/master/examples/hpddm/navier-stokes-2d-PETSc.edp#L88).

---

<div class="post-metadata">

**Author:** ![marcodeba](https://avatars.discourse-cdn.com/v4/letter/m/977dab/32.png) [@marcodeba](https://community.freefem.org/u/marcodeba)\
**Post date:** [October 19, 2022, 10:00am UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/23 "2022-10-19T10:00:08Z")

</div>

Oh sorry, that’s the point… I was supposing that A.D was something link an achronim, while A is nothing but the Mat name!

I’m trying to implement this strategy and I’ll let you know if I’ll need sme more help (which is actually very realistic 🙂 )

---

<div class="post-metadata">

**Author:** ![marcodeba](https://avatars.discourse-cdn.com/v4/letter/m/977dab/32.png) [@marcodeba](https://community.freefem.org/u/marcodeba)\
**Post date:** [October 19, 2022, 11:51am UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/24 "2022-10-19T11:51:26Z")

</div>

I was trying to understand how the solution you proposed is working and I have developed this very stupid script:

```auto
//prova soluzione PRJ in caso semplicissimo

load "PETSc"
load "msh3"
macro dimension()3//
include "macro_ddm.idp"

macro def(i)i// EOM
macro init(i)i// EOM
func Pk = P2;
int nn = 70;

mesh3 ThNo, Th = cube(nn,nn,nn);
real Volume = int3d(Th)(1.0);
if (mpirank == 0)
{
  cout << "Volume = " << Volume << endl;
}
fespace Vh(Th,Pk);

Mat AA;

buildMat(Th, getARGV("-split",1), AA, Pk, mpiCommWorld);

Vh part = AA.D;
ThNo = trunc(Th, abs(part - 1.0) < 1e-1);

func real integral(real[int]& X) {
  Vh uu;
  ChangeNumbering(AA, uu[], X, inverse = true, exchange = true);
  real glob, loc = int3d(ThNo)(uu);
  mpiAllReduce(loc,glob, mpiCommWorld, mpiSUM);
  return glob;
}

real tin = clock();

Vh valore = 2.0;
real[int] vPETSc;
ChangeNumbering(AA, valore[], vPETSc);

real volx2 = integral(vPETSc);

if (mpirank == 0)
{
  cout << "volx2 = " << volx2 << endl;
}

real duration = clock() - tin;

if (mpirank == 0)
{
  cout << "Duration = " << duration << " [s]" << endl;
}

```

This script should compute in parallel the volume of a unit cube multiplied by a certain factor (2 in this case). I have noticed that with one core the result is properly evaluated, while increasing the number of cores the result is under estimated. I am pretty sure that is something missing in my script, but I’m not able to understand where. (see attached table for error description)

 ![ErroreLegion_function](https://canada1.discourse-cdn.com/flex030/uploads/freefem/original/2X/3/3ce1c09c8e025792bc200d0237c6b7e159fbc122.png)

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [October 19, 2022, 12:18pm UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/25 "2022-10-19T12:18:20Z")

</div>

My bad, I thought this would work, but the multiplicity scaling is wrong with such a `fespace`. Here is how to bypass the issue. Replace:

```auto
buildMat(Th, getARGV("-split",1), AA, Pk, mpiCommWorld);

Vh part = AA.D;
ThNo = trunc(Th, abs(part - 1.0) < 1e-1);

```

By:

```auto
fespace Ph(Th, P0);
Ph part;
partitionerSeq(part[], Th, mpisize);
partitionerPar(part[], Th, mpiCommWorld, mpisize);

buildMatWithPartitioning(Th, part[], getARGV("-split",1), AA, Pk, mpiCommWorld);
part = part;

ThNo = trunc(Th, abs(part - mpirank) < 1e-1);

```

Is this an acceptable solution for you?

---

<div class="post-metadata">

**Author:** ![marcodeba](https://avatars.discourse-cdn.com/v4/letter/m/977dab/32.png) [@marcodeba](https://community.freefem.org/u/marcodeba)\
**Post date:** [October 19, 2022, 1:34pm UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/26 "2022-10-19T13:34:46Z")

</div>

Great! This is working very well in this simple test case! I’m trying to translate it in the real problem application!

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [October 19, 2022, 1:35pm UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/27 "2022-10-19T13:35:41Z")

</div>

Sounds good, you know where to find me if this does not work or need something else 🙂

---

<div class="post-metadata">

**Author:** ![marcodeba](https://avatars.discourse-cdn.com/v4/letter/m/977dab/32.png) [@marcodeba](https://community.freefem.org/u/marcodeba)\
**Post date:** [October 19, 2022, 6:38pm UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/28 "2022-10-19T18:38:15Z")

</div>

Here I am again! I’m trying to get step by step closer to the final result, but I have encountered a strange issue which I am not able to understand… everything seems to work smoothly but at a certain point the execution stops (ExitCode 59) showing the error code at the end of the attached .log file (…possibly illegal memory access).  
The adopted script is attached too.

Do you have any guess on the reason for this crush?  
[NewImplTEST.log](https://community.freefem.org/uploads/short-url/4z1bgPLWFMfGcVbH6EGHKP4DabK.log) (751.0 KB)  
[NewImplementation\_onlyPower.edp](https://community.freefem.org/uploads/short-url/ePEqK82wYqI68jnzshFmt1PRzOT.edp) (6.9 KB)

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [October 19, 2022, 8:26pm UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/29 "2022-10-19T20:26:04Z")

</div>

You need to initialise `real[int] bbb(vh.ndof);` _after_ the refinement, or alternatively, call `bb.resize(vh.ndof);` after `buildMatWithPartitioning`.

---

<div class="post-metadata">

**Author:** ![marcodeba](https://avatars.discourse-cdn.com/v4/letter/m/977dab/32.png) [@marcodeba](https://community.freefem.org/u/marcodeba)\
**Post date:** [October 20, 2022, 11:39am UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/30 "2022-10-20T11:39:36Z")

</div>

Thanks again for making me noticing this!  
In the very simple case I have used as a test I was able to make properly work examples with very heavy meshes!  
By the way I’m still facing some issues in the real case application, and the problem arises once I try to assemble the matrix (AA = vAmp(vh,vh)). PETSc is showing an error stating that “Argument is out of range” as in the attached sneepshot.

 ![image](https://canada1.discourse-cdn.com/flex030/uploads/freefem/original/2X/a/a488a2b3a1d5a64e3a69af4d479601112436d04e.png)

Do you know which could be the reason of this issue? Should I define the matrix on the base of other fespaces (maybe defined on ThNo?).

For completeness the updated version of the “final script” is here attached.

[NewImplementation\_onlyPower.edp](https://community.freefem.org/uploads/short-url/7pUfGvlw3QKNWJr6tDP2xI1zib4.edp) (8.0 KB)

Thanks again for the support!

---

<div class="post-metadata">

**Author:** ![marcodeba](https://avatars.discourse-cdn.com/v4/letter/m/977dab/32.png) [@marcodeba](https://community.freefem.org/u/marcodeba)\
**Post date:** [October 20, 2022, 4:45pm UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/31 "2022-10-20T16:45:25Z")

</div>

I’m here again to add some new information. Similar, but not same errors appears (Segmentation Violation, probably memory access out of range) whenever I try to solve a simple heat diffusion problem within the usual cube as happen running the attached script (notice that the mesh is light so the computation can be easily afforded by a simple laptop!)

Any suggestion for the crash reason @prj ?

Sorry again for disturbing you a lot, and thank you very much for the support!

[ProvaSoluzionPRJ\_heatconduction.edp](https://community.freefem.org/uploads/short-url/9o6yt8kYwGTE3I1UlyPM1hSztes.edp) (3.1 KB)  
[cubotestL.mesh](https://community.freefem.org/uploads/short-url/j0lhWokaKSRRd39zgX67O2D4GRk.mesh) (2.9 MB)

EDIT: This is running on another machine but the computed average temperature is 4000 [K], which is impossible given the side temperatures of 500 [K] and 300 [K]

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [October 21, 2022, 5:36am UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/32 "2022-10-21T05:36:14Z")

</div>

For the first script, if you use edge elements, you need to define a special macro `PkPart`, cf. [FreeFem-sources/maxwell-3d.edp at master · FreeFem/FreeFem-sources · GitHub](https://github.com/FreeFem/FreeFem-sources/blob/master/examples/hpddm/maxwell-3d.edp#L18). Then, you’ll have to call `buildMatEdgeRecursiveWithPartitioning`. For the second script, if you use a FreeFEM `matrix` and run the code in sequential, does it provide the correct results?

---

<div class="post-metadata">

**Author:** ![marcodeba](https://avatars.discourse-cdn.com/v4/letter/m/977dab/32.png) [@marcodeba](https://community.freefem.org/u/marcodeba)\
**Post date:** [October 21, 2022, 8:06am UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/33 "2022-10-21T08:06:39Z")

</div>

Thanks again @prj for the suggestion, just to be sure:

- I should define the macro `macro ThPkPart()Edge03ds0// EOM` at the beginning of the script (i.e. just after the def and init macro)
- I should substitute `buildMatWithPartitioning(Th, part[], getARGV("-split",1), AA, Pk, mpiCommWorld);` with `buildMatEdgeRecursiveWithPartitioning(Th, part[], getARGV("-split",1), AA, Pk, mpiCommWorld);` so basically using the same input argument, is it? Or are there some different arguments to be provided to the new command?

**(EDIT: I suppose I need to load/include some external “packages” / .idp since running with only these modification the job fails stating that the identifier buildMatEdgeRecursiveWithPartitioning does not exist)**

**(EDIT2: Looking within macro\_ddm.idp I have noticed that exists two different macros which are macro buildMatEdgeRecursive and macro buildMatEdgeWithPartitioning. I have tryed to use the second one and no errors are appearing when assembling the matrix, but I’m still waiting to understand if it is actually computing or not The new version of the buildMat is the following:)**

```auto
buildMatEdgeWithPartitioning(Th, part[], getARGV("-split",1), AA, Pk, mpiCommWorld, Edge03ds0, defPart, initPart);

```

For what concerns the second script, my fault… the problem is solved properly! By the way I’m not still able to store the global result on a .vtu file. I was trying to do something like this:

```auto
load "iovtk"

real[int] Tglob;

mpiAllReduce(TT[],Tglob, mpiCommWorld, mpiSUM);

if (mpirank == 0)
{
  savevtk("TemperaturaBlocco.vtu",Th,Tglob);
}

```

but this was generating an error telling that it is not possible to store such kind of data (but if I define Tglob as Vh it is not working too…)

Thanks again!

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [October 21, 2022, 2:33pm UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/34 "2022-10-21T14:33:12Z")

</div>

1. You need to use `buildMatEdgeWithPartitioning`, there is nothing else to include but `macro_ddm.idp`.
2. You should not `load "iovtk"` to use `savevtk` in parallel, it’s automatically shipped with `load "PETSc"`. See [FreeFem-sources/diffusion-mg-3d-PETSc.edp at master · FreeFem/FreeFem-sources · GitHub](https://github.com/FreeFem/FreeFem-sources/blob/master/examples/hpddm/diffusion-mg-3d-PETSc.edp#L41).

---

<div class="post-metadata">

**Author:** ![marcodeba](https://avatars.discourse-cdn.com/v4/letter/m/977dab/32.png) [@marcodeba](https://community.freefem.org/u/marcodeba)\
**Post date:** [October 23, 2022, 8:02am UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/35 "2022-10-23T08:02:53Z")

</div>

Dear @prj ,  
I hope to disturb you for the last time, at least for this topic! 🙂

I have introduced the modifications you suggested me, but in the real case application I’m still having problems. Indeed the error “Caught signal number 7 BUS: Bus Error, possibly illegal memory access” is appearing now when calling partitionerseq and partitionerpar.

The relevant part of the adopted script is attached below.

Thanks again for the support! Have a nice sunday!

```auto
load "msh3"
load "PETSc"
load "mat_edgeP1"
load "Element_Mixte3d"
include "getARGV.idp"

int iresta = getARGV("-iresta",0);
int cm = 15; //getARGV("-cm",0);

string meshname = "FixedTFVV"+cm+".mesh";

lockOrientation = false;
mesh3 Th = readmesh3(meshname);
lockOrientation = true;

mesh3 ThNo;

// region definition

int pt = Th(0.0,9.0,0.0).region;

int pp = Th(0.0,10.0,0.0).region;

int cassa = Th(0.0,3.3,0.0).region;

int VV = Th(0.0,13.7,0.0).region;

macro dimension()3// EOM // 2D or 3D
include "macro_ddm.idp"

macro defPart(u)u// EOM // partition of unity definition
macro initPart(u)u// EOM // partition of unity initialization

macro def(i)[i, i#B, i#C] //EOM
macro init(i)[i, i, i] //EOM
macro ThPkPart()Edge03ds0// EOM special FE for the domain partitioning

real mu = 4*pi*1.0e-7;
real nu=1.0/mu;

func ppp = P0;
func pj = [P0, P0, P0];
func Pk = Edge03d;
/*
fespace vhp(Th,ppp);
fespace vhj(Th,pj);
fespace vh(Th,Pk);
*/
real rhocase = 80e-8; // [Ohm*m] ref ENEA
real sigmaCase = 1.0/rhocase;

real rhoVV = 54.4e-8; // [Ohm*m] ref ENEA
real sigmaVV = 1.0/rhoVV;

//vhp Sigma = 1.0 + (sigmaCase-1.0)*(region == cassa) + (sigmaVV-1.0)*(region == VV);

// macros
macro Curl(Ax,Ay,Az) [dy(Az)-dz(Ay), dz(Ax)-dx(Az), dx(Ay)-dy(Ax)]//EOM

real tempo;

real Ip = 3.6e6; //[A]
real rmajor = 9.0;

func Xc = rmajor*(x/sqrt(x^2+y^2));
func Yc = rmajor*(y/sqrt(x^2+y^2));

func rr = sqrt(x^2+y^2);

func oriz = (rr-rmajor)*(region == pp);

func cosalpha = oriz/sqrt(z^2+oriz^2);

func sinalpha = z/sqrt(z^2+oriz^2)*(region == pp);

real rout = 1.4;
real rin = 0.9;

func sezione = (rout - rin)*2.0*pi*(rmajor+((rout+rin)/2.0)*cosalpha);

func J0p = Ip/sezione;

func Jmodtp = 0.0*(tempo<0.5) + J0p*((tempo-0.5)/7.0)*(tempo >= 0.5 && tempo < 7.5) + J0p*(tempo >= 7.5 && tempo < 20.0) + J0p*(1-(tempo-20.0)/0.057)*(tempo>=20.0 && tempo < 20.057) + 0.0;

func cosbeta = Xc/rmajor;
func sinbeta = Yc/rmajor;

real It = 18.0e6;
real rt = 0.75;
real sezt = pi*rt^2;

real J0t = It/sezt;

func Jmodtt = 0.0*(tempo<0.5) + J0t*((tempo-0.5)/7.0)*(tempo >= 0.5 && tempo < 7.5) + J0t*(tempo >= 7.5 && tempo < 20.0) + J0t*(1-(tempo-20.0)/0.057)*(tempo>=20.0 && tempo < 20.057) + 0.0;

real tend = 25.0;
real dt;

if (mpirank == 0)
{
  cout << "Variational formulation defined" << endl;
}

Mat AA;

if (mpirank == 0)
{
  cout << "PETSc matrix defined" << endl;
}
//macro ThPkPart()Edge03ds0// EOM

fespace Ph(Th, P0);
Ph part;
partitionerSeq(part[], Th, mpisize);
partitionerPar(part[], Th, mpiCommWorld, mpisize);

if (mpirank == 0)
{
  cout << "partitioner done" << endl;
}

//buildMatWithPartitioning(Th, part[], getARGV("-split",1), AA, Pk, mpiCommWorld);
buildMatEdgeWithPartitioning(Th, part[], getARGV("-split",1), AA, Pk, mpiCommWorld, Edge03ds0, defPart, initPart);
if (mpirank == 0)
{
  cout << "buildMatEdgeWithPartitioning done" << endl;
}

part = part;

```

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [October 23, 2022, 8:10am UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/36 "2022-10-23T08:10:07Z")

</div>

Without the mesh: impossible to debug.

---

<div class="post-metadata">

**Author:** ![marcodeba](https://avatars.discourse-cdn.com/v4/letter/m/977dab/32.png) [@marcodeba](https://community.freefem.org/u/marcodeba)\
**Post date:** [October 23, 2022, 9:05am UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/37 "2022-10-23T09:05:17Z")

</div>

You are right, I’m sorry. I was hoping that thanks to your expertise you were able to easily catch the mistake! 🤗

The mesh is saved at this GDRIVE link. Not such a small mesh, but to execute the first few lines this should not be an issue.

> **[FixedTFVV15.mesh](https://drive.google.com/file/d/1rHMOS5VHRF-9YiTdqdK8xP7zKDM-B6Lt/view?usp=sharing)**
>
> Google Drive file.

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [October 23, 2022, 9:09am UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/38 "2022-10-23T09:09:42Z")

</div>

This is a 1GB mesh. The exact goal of this script is not to have to deal with such a large mesh and instead use the `-split` parameter. If it was not working before with the 1GB mesh, it still won’t work here.

---

<div class="post-metadata">

**Author:** ![marcodeba](https://avatars.discourse-cdn.com/v4/letter/m/977dab/32.png) [@marcodeba](https://community.freefem.org/u/marcodeba)\
**Post date:** [October 23, 2022, 9:15am UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/39 "2022-10-23T09:15:29Z")

</div>

No, this mesh was actually working with the “old style” approach… let me say that this was the biggest mesh that can be used!

By the way I am checking if with lighter meshes it is instead working!

---

<div class="post-metadata">

**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\
**Post date:** [October 23, 2022, 9:29am UTC](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071/40 "2022-10-23T09:29:38Z")

</div>

The problem remains. Dealing with large duplicated meshes is far from ideal. Alternatively, you could try to launch your script with the additional command line argument `-Dpartitioner=parmetis`, and see if it’s any better.

[Previous page](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071.md?page=1)

[Next page](https://community.freefem.org/t/memory-allocation-problem-in-mpi-computation-using-petsc/2071.md?page=3)