# Error with Nested matrix using PETSc with Local Communicator

**URL:** https://community.freefem.org/t/error-with-nested-matrix-using-petsc-with-local-communicator/1848
**Category:** General Discussion
**Created:** [June 24, 2022, 2:57pm UTC](https://community.freefem.org/t/error-with-nested-matrix-using-petsc-with-local-communicator/1848 "2022-06-24T14:57:09Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![Loic](https://avatars.discourse-cdn.com/v4/letter/l/e9c0ed/32.png) [@Loic](https://community.freefem.org/u/Loic)
#### Post date: [June 24, 2022, 2:57pm UTC](https://community.freefem.org/t/error-with-nested-matrix-using-petsc-with-local-communicator/1848/1 "2022-06-24T14:57:09Z")

</div>

Hello all,

I am using PETSc with a Local Communicator. Below I give you an example of code (random problem just to show the problem)

```auto
verbosity=0;
mpiComm commGlobal(mpiCommWorld,0,0);

load "PETSc"
load "gmsh";

macro dimension() 2 // EOM 2 for 2d 3 for 3d
macro ThKComm()commLocal // Local communicator or the mesh ThK
include "macro_ddm.idp"

for (int k=0; k<4; k++) //loop on the coarse element
{
bool elemtest =(mpirank==k%mpisize);
 if (elemtest)
 {
  mpiComm commLocal(mpiCommWorld,mpirank,0);// MPI_Comm_split //definition of the local communicator
  mesh ThK=square(10+k,10+k);

 // ********************************************************************************
 func PkX=P2;
 func PkM=P1;
 func PkX2 = [PkX, PkX];
 // **********************************************************************************
 fespace VhK(ThK, PkX2); //FE space to solve the local problem
 fespace PhK(ThK, PkM);

 //Variational formulation //random problem
 varf vA([u,v],[uu, vv])=int2d(ThK)((dx(u)*dx(uu)+dy(u)*dy(uu)+dx(v)*dx(vv)+dy(v)*dy(vv))) + on(1,2,3,u=0,v=0) + on(4, u=1, v=0);
 varf vB([p], [uu,vv])= int2d(ThK)(-p*(dx(uu)+dy(vv)));

 //Creation of the corresponding FreeFEM matrices
 matrix A=vA(VhK,VhK);
 matrix B=vB(PhK,VhK);

//Creation of the corresponding matrices in PETSc
Mat Ap, Cp;
{
  buildDmesh(ThK)
  {
    macro def(u)[u, u #B] // dont touch
    macro init(u)[u, u] // dont touch
    createMat(ThK, Ap, PkX2)
  }
  createMat(ThK, Cp, PkM)
}
Mat Bp(Ap, Cp, B); // Bp = B
Ap = A; //Ap=A

Mat N=[[Ap, Bp],
   [Bp',0]]; // Creation of a Nested Matrix

// RHS vector
real[int] vel(VhK.ndof) ;
vel= 1;

real[int] press(PhK.ndof);
press=0;

real[int] rhsPETSc;
real[int] velPETSc;
real[int] pressPETSc;

ChangeNumbering(Ap, vel, velPETSc);
ChangeNumbering(Cp, press, pressPETSc);
rhsPETSc = [velPETSc, pressPETSc];

//real[int] xx1= Ap^-1 *velPETSc; // this operation work
real[int] xx3= N^-1 *rhsPETSc; //This operation does not work

cout << "element " << k << " treated " << " with processor " << mpirank << endl;

}
}

```

I run this code with

```auto
mpiexec -np 4 FreeFem++-mpi mycode.edp

```

The operation `real[int] xx1= Ap^-1 *velPETSc;` works perfectly.

However when I do the same with the nested matrix N, `real[int] xx3= N^-1 *rhsPETSc;` a problem of dimension occurs. Indeed, the error for one processor is for example example :

`Submatrix dimension (144,1058) incompatible with nest block (630,1058)`

I don’t know where the size 630 comes from. It seems that something is replicate 4 times.

Could someone help me to build correctly the nested matrix N ?

Thank you in advance,

Best regards,

Loïc,

---

<div class="post-metadata">

### Author: ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)
#### Post date: [June 25, 2022, 8:04am UTC](https://community.freefem.org/t/error-with-nested-matrix-using-petsc-with-local-communicator/1848/2 "2022-06-25T08:04:48Z")

</div>

Your script is correct, but there was no support for `MatNest` living on a local communicator. I have fixed this in [Adds support for MatNest on subcommunicators. · FreeFem/FreeFem-sources@63cb3d3 · GitHub](https://github.com/FreeFem/FreeFem-sources/commit/63cb3d30e18797708bb096d5ab85fcabae311c55). Thank you for reporting this.

---

<div class="post-metadata">

### Author: ![Loic](https://avatars.discourse-cdn.com/v4/letter/l/e9c0ed/32.png) [@Loic](https://community.freefem.org/u/Loic)
#### Post date: [June 27, 2022, 6:37am UTC](https://community.freefem.org/t/error-with-nested-matrix-using-petsc-with-local-communicator/1848/3 "2022-06-27T06:37:52Z")

</div>

Hello @prj

Thank you for your help,  
I will try this and let you know,

Best regards,

Loïc,

---

<div class="post-metadata">

### Author: ![Loic](https://avatars.discourse-cdn.com/v4/letter/l/e9c0ed/32.png) [@Loic](https://community.freefem.org/u/Loic)
#### Post date: [July 6, 2022, 3:11pm UTC](https://community.freefem.org/t/error-with-nested-matrix-using-petsc-with-local-communicator/1848/4 "2022-07-06T15:11:09Z")

</div>

Thanks @prj ,

My code now is working,

Best regards,

Loïc,