# Request for Reproduction Setup of Cobra Cavity Example on 6144 Cores (198 Million DoFs)

**URL:** <https://community.freefem.org/t/request-for-reproduction-setup-of-cobra-cavity-example-on-6144-cores-198-million-dofs/3882>\
**Category:** General Discussion\
**Created:** [April 25, 2025, 8:07pm UTC](https://community.freefem.org/t/request-for-reproduction-setup-of-cobra-cavity-example-on-6144-cores-198-million-dofs/3882 "2025-04-25T20:07:27Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dhl2011dj](https://avatars.discourse-cdn.com/v4/letter/d/3ab097/32.png) [@dhl2011dj](https://community.freefem.org/u/dhl2011dj)\
**Post date:** [April 25, 2025, 8:07pm UTC](https://community.freefem.org/t/request-for-reproduction-setup-of-cobra-cavity-example-on-6144-cores-198-million-dofs/3882/1 "2025-04-25T20:07:27Z")

</div>

Hello everyone,

I am currently trying to reproduce the large-scale electromagnetic scattering simulation from the Cobra Cavity example shown on the FreeFEM gallery ([https://freefem.org/gallery/cobra](https://freefem.org/gallery/cobra)), where the problem reportedly reaches **198 million degrees of freedom (DoFs)** using **6144 cores**.

In my attempt to reproduce this result, I used the following command:

```auto
ff-mpirun -np 3200 Maxwell_Cobracavity.edp -wg -raspart -ffddm_schwarz_method oras -ffddm_medit -frequency 1.6e+10

```

To increase the number of DoFs, I adjusted the `distx` parameter in `Maxwell_Cobracavity.edp` to **0.12** , which refined the mesh. However, when the DoF reaches several tens of millions, the computation runs into **out-of-memory errors**.

I also found that the `-noGlob` option in `ffddm_parameters.idp` helps avoid global mesh generation, which reduces the memory usage. But even with this option, as the DoFs grow further, the memory issue still persists.

Here is my current cluster setup:

- **100 compute nodes**
- **32 CPU cores per node** (total 3200 cores)
- **124 GB memory per node**

Given this, I would like to ask:

- Is it possible to share the exact `.edp` script and command-line options used to achieve the large-scale 198 million DoFs computation on 6144 cores?
- Are there any additional parameters or best practices needed to successfully scale to this problem size?

Thank you very much for your help and suggestions!

Best regards,  
Dylan

---

<div class="post-metadata">

**Author:** ![phtournier](https://avatars.discourse-cdn.com/v4/letter/p/3da27b/32.png) [@phtournier](https://community.freefem.org/u/phtournier)\
**Post date:** [April 30, 2025, 10:25am UTC](https://community.freefem.org/t/request-for-reproduction-setup-of-cobra-cavity-example-on-6144-cores-198-million-dofs/3882/2 "2025-04-30T10:25:30Z")

</div>

The exact setting of the test case is detailed in the [paper](https://arxiv.org/pdf/1711.03789), section 6.5. It is said that

- The problems are discretized with order 2 edge elements using 10 points per  
wavelength, so you need to set

```auto
func Pk = Edge13d;
func PkP0 = Edge13ds0;

```

for order 2 (degree 1) edge elements, and

```auto
int nloc = 10./mysplit*sec3/lambda;

```

to use 10 points per wavelength instead of 20.
- The mesh for the coarse problem corresponds to a discretization with 3.33 points per wavelength, so there is a refinement factor of 3 between the coarse and the fine mesh. Thus, you need to set the splitting factor to 3:

```auto
int mysplit = 3;

```

- Finally, it is said that the sides of the box are placed 4 wavelengths away from the cavity in each direction, so you need to set

```auto
real distx = 4*lambda;
real disty = distx;
real distz = distx;

```

As you pointed out, you can reduce memory usage by using the `-noGlob` flag. You can also reduce the maximum size of the Krylov subspace in GMRES by setting a lower maximum number of iterations

```auto
u[] = MfGMRES(x0, rhs, 1.e-6, 50, "right");

```

or by using restarted GMRES with a smaller restart, e.g. with `-ffddm_gmres_restart 50`.

I just reproduced the results of the paper on the Irene supercomputer, using

```auto
ff-mpirun -np 6144 Maxwell_Cobracavity.edp -ns -nw -raspart -ffddm_schwarz_method oras -noGlob -frequency 1.6e+10

```

With your cluster setup I am afraid that the 16Ghz case may be difficult to pass, but you should be able to at least reproduce the 10Ghz case with no problem.