# FreeFEM compilation: issues on HPC
**URL:** https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788
**Category:** FreeFEM installation
**Created:** [February 28, 2025, 10:25pm UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788 "2025-02-28T22:25:16Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@aszaboa](https://community.freefem.org/u/aszaboa)
#### Post date: [February 28, 2025, 10:25pm UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/1 "2025-02-28T22:25:16Z")
Hello everyone,
I am having trouble installing FreeFEM + PETSc on a HPC cluster. Precisely, the installation seems OK at first glance, and everything runs smoothly when I run the codes on one node; however, when I try to run on multiple nodes, the code just stops randomly, similarly to a previously [problem report](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/84).
First, I tried the installation with the following modules:
```auto
module load git/2.39.1
module load intel/2022.1.2
module load openmpi/5.0.3
```
With this, the installation is successful, but the program stops without a message on multiple nodes.
Then, [as suggested in a previous post](https://community.freefem.org/t/stall-in-some-multi-node-parallel-calculations/1143/84), I tried to switch to the only intel MPI version which was the following: `module load impi/2021.5.1 `.
Then, the PETSc compilation was successful, while the compilation FreeFEM failed with the following message:
```auto
mv -f $depbase.Tpo $depbase.Po
depbase=`echo ../lglib/lg.tab.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
mpicxx -DHAVE_CONFIG_H -I. -I../.. -DPARALLELE -I./../fflib -I./../Graphics -I./../femlib -I./../bamglib/ -I"/sw/pkgs/arc/intel/2022.1.2/mpi/2021.5.1/include" -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker -Xlinker -rpath -Xlinker -Xlinker --enable-new-dtags -I/home/andrasz/FFinstall/petsc/arch-FreeFem/include/suitesparse -I/home/andrasz/FFinstall/petsc/arch-FreeFem/include -I./../../3rdparty/include/BemTool/ -I./../../3rdparty/boost/include -I../../plugin/mpi -DPARALLELE -DHAVE_ZLIB -g -DNDEBUG -O3 -mmmx -mavx2 -mavx -msse4.2 -msse2 -msse -std=gnu++14 -DBAMG_LONG_LONG -DNCHECKPTR -fPIC -MT ../lglib/lg.tab.o -MD -MP -MF $depbase.Tpo -c -o ../lglib/lg.tab.o ../lglib/lg.tab.cpp &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo compositeFESpace.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
mpicxx -DHAVE_CONFIG_H -I. -I../.. -DPARALLELE -I./../fflib -I./../Graphics -I./../femlib -I./../bamglib/ -I"/sw/pkgs/arc/intel/2022.1.2/mpi/2021.5.1/include" -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker -Xlinker -rpath -Xlinker -Xlinker --enable-new-dtags -I/home/andrasz/FFinstall/petsc/arch-FreeFem/include/suitesparse -I/home/andrasz/FFinstall/petsc/arch-FreeFem/include -I./../../3rdparty/include/BemTool/ -I./../../3rdparty/boost/include -I../../plugin/mpi -DPARALLELE -DHAVE_ZLIB -g -DNDEBUG -O3 -mmmx -mavx2 -mavx -msse4.2 -msse2 -msse -std=gnu++14 -DBAMG_LONG_LONG -DNCHECKPTR -fPIC -MT compositeFESpace.o -MD -MP -MF $depbase.Tpo -c -o compositeFESpace.o compositeFESpace.cpp &&\
mv -f $depbase.Tpo $depbase.Po
depbase=`echo parallelempi.o | sed 's|[^/]*$|.deps/&|;s|\.o$||'`;\
mpicxx -DHAVE_CONFIG_H -I. -I../.. -DPARALLELE -I./../fflib -I./../Graphics -I./../femlib -I./../bamglib/ -I"/sw/pkgs/arc/intel/2022.1.2/mpi/2021.5.1/include" -Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker -Xlinker -rpath -Xlinker -Xlinker --enable-new-dtags -I/home/andrasz/FFinstall/petsc/arch-FreeFem/include/suitesparse -I/home/andrasz/FFinstall/petsc/arch-FreeFem/include -I./../../3rdparty/include/BemTool/ -I./../../3rdparty/boost/include -I../../plugin/mpi -DPARALLELE -DHAVE_ZLIB -g -DNDEBUG -O3 -mmmx -mavx2 -mavx -msse4.2 -msse2 -msse -std=gnu++14 -DBAMG_LONG_LONG -DNCHECKPTR -fPIC -MT parallelempi.o -MD -MP -MF $depbase.Tpo -c -o parallelempi.o parallelempi.cpp &&\
mv -f $depbase.Tpo $depbase.Po
g++: error: unrecognized command line option ‘-rpath’
make[3]: *** [Makefile:710: ffapi.o] Error 1
make[3]: *** Waiting for unfinished jobs....
g++: error: unrecognized command line option ‘-rpath’
g++: error: unrecognized command line option ‘-rpath’
g++: error: unrecognized command line option ‘-rpath’
g++: error: unrecognized command line option ‘-rpath’
make[3]: *** [Makefile:710: ../lglib/mymain.o] Error 1
make[3]: *** [Makefile:710: ../lglib/lg.tab.o] Error 1
make[3]: *** [Makefile:710: parallelempi.o] Error 1
make[3]: *** [Makefile:710: compositeFESpace.o] Error 1
make[3]: Leaving directory '/home/andrasz/FFinstall/FreeFem-sources/src/mpi'
make[2]: *** [Makefile:554: all-recursive] Error 1
make[2]: Leaving directory '/home/andrasz/FFinstall/FreeFem-sources/src'
make[1]: *** [Makefile:896: all-recursive] Error 1
make[1]: Leaving directory '/home/andrasz/FFinstall/FreeFem-sources'
```
I attach the config.logs of both cases. Any help with finding the issue is much appreciated.
[config\_intel\_openmpi.zip](https://community.freefem.org/uploads/short-url/mTisYGUiUMmGLpJYHizg8qy7txW.zip) (643.0 KB)
[config\_intel\_intelmpi.zip](https://community.freefem.org/uploads/short-url/mUjeBWo9NsOgqjoH9f3wBI5uudW.zip) (631.5 KB)
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [March 1, 2025, 6:48am UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/2 "2025-03-01T06:48:10Z")
I don’t know of a better fix than to manually remove `-Xlinker --enable-new-dtags -Xlinker -rpath -Xlinker -Xlinker -rpath -Xlinker -Xlinker --enable-new-dtags` from the various `Makefile` in the arborescence.
---
### Author:  [@aszaboa](https://community.freefem.org/u/aszaboa)
#### Post date: [March 2, 2025, 4:24am UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/3 "2025-03-02T04:24:52Z")
Thanks for the tip. Removing the corresponding string is not enough. There is a occurence of it in the file plugin/seq/WHERE\_LIBRARY-config which is created by the ./configure command. If I remove the command there completely, the compilation seems fine. However, the original issue remains: the code runs fine on a single node, but stalls when I run it on two nodes. Do you have any ideas how to figure out the solution?
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [March 2, 2025, 8:17am UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/4 "2025-03-02T08:17:25Z")
Are you sure you are not mixing MPI implementations at compile- and run-time?
---
### Author:  [@aszaboa](https://community.freefem.org/u/aszaboa)
#### Post date: [March 3, 2025, 7:12pm UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/5 "2025-03-03T19:12:15Z")
I do not think so. I load either the module `impi/2021.5.1` or `openmpi/5.0.3`, and set the compiler flags/environmental variables myself. Still, the issue persists in both cases.
I will try to contact the HPC admins about this. In the meantime, if you have any ideas where I should investigate further, I would appreciate it.
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [March 3, 2025, 7:40pm UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/6 "2025-03-03T19:40:27Z")
How are you launching the FreeFEM code?
---
### Author:  [@aszaboa](https://community.freefem.org/u/aszaboa)
#### Post date: [March 3, 2025, 7:45pm UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/7 "2025-03-03T19:45:32Z")
I tried the following:
`ff-mpirun -n 384 Nonlinear-solver.edp -Re 50 -v 0 `
`srun --mpi=pmi2 -n 384 FreeFem++-mpi Nonlinear-solver.edp -Re ${Re} -v 0`
Neither did work.
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [March 3, 2025, 7:55pm UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/8 "2025-03-03T19:55:21Z")
Can you do `ldd FreeFem++-mpi` and send the output, please? Does a simple `cout << mpirank << endl;` run?
---
### Author:  [@aszaboa](https://community.freefem.org/u/aszaboa)
#### Post date: [March 3, 2025, 9:01pm UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/9 "2025-03-03T21:01:46Z")
Sure, here is the log.
Note that in the meantime, I switched back to openmpi as its compilation was easier.
[ldd.log](https://community.freefem.org/uploads/short-url/9prqz8IiObRwBYHWFeQxwUHdZbs.log) (4.6 KB)
Yes, the program displays all the mpiranks.
What is really odd (in the case of multiple nodes ofc) that with the current openmpi installation, when I use `ff-mpirun -n NPROC`, the code stalls, but when I launch with `srun -n NPROC FreeFem++-mpi ` (without `--mpi=pmi2`), the code seems to work. However, it displays the following error message: `[lh1659:3771795] Error: coll_hcoll_module.c:241 - mca_coll_hcoll_module_enable() coll_hcol: mca_coll_hcoll_save_coll_handlers failed`
I suppose this happens everytime when communication occurs between the nodes.
I am not sure how serious of an issue this is, if it is one at all. Upon googling, I think this may cause suboptimal performance.
Anyways, I would appreciate if you could possibly provide some insight into this behavior which would help me (and the community) with sorting out similar issues.
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [March 4, 2025, 7:06am UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/10 "2025-03-04T07:06:28Z")
This error comes from your machine, not the installation.
---
### Author:  [@aszaboa](https://community.freefem.org/u/aszaboa)
#### Post date: [March 4, 2025, 2:06pm UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/11 "2025-03-04T14:06:02Z")
Ok, thank you so much for your help!!
---
### Author:  [@aszaboa](https://community.freefem.org/u/aszaboa)
#### Post date: [March 16, 2025, 5:05am UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/12 "2025-03-16T05:05:32Z")
In the meantime, I got the idea that I should compile PETSc and FreeFEM with MPICH. The installation is successful, and when I run an example on a login node using ff-mpirun, it seems to work fine. However, when I run the code with slurm + srun, it does not seem to work as intended: there are only one process (rank 0, mpisize 1). It seems like the connection/link between mpich and slurm needs to be specified somehow. Do you have any idea how that issue may be resolved?
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [March 16, 2025, 4:58pm UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/13 "2025-03-16T16:58:51Z")
`srun` probably uses the system MPI, meanwhile you want to use your own MPICH. But you should avoid compiling MPICH yourself and instead rely on a system module, precisely to avoid such issues. If your system module is bugged (cf. previous error message `[lh1659:3771795] Error: coll_hcoll_module.c:241 - mca_coll_hcoll_module_enable() coll_hcol: mca_coll_hcoll_save_coll_handlers failed`), you should ask how to get this fixed.
---
### Author:  [@aszaboa](https://community.freefem.org/u/aszaboa)
#### Post date: [March 17, 2025, 2:48pm UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/14 "2025-03-17T14:48:52Z")
Thanks for the information. I thought about MPICH because it has worked in the past, and I have trouble with both openMPI/intel MPI which are the available MPIs on this HPC.
I have managed to run FreeFEM with openMPI, and run it with `export OMPI_MCA_coll="^hcoll"` plus `srun --mpi=pmix`, but I there seem to be some issues. When I tried to examine the scaling of the program on 60 nodes/11k processes, real PETSc seem to work, but the complex version stops with the following message:
`Internal error 2 with small buffers`. Furthermore, I now once again recompiled the code, and my code what runs fine on 5 nodes crashes on 10 nodes (more precisely, one code using real PETSc works, while the complex PETSc crashes with `Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range`). One note that previously, `srun --mpi=pmix`worked, but now I recompiled FreeFEM + PETSc and only `srun --mpi=pmix_v4` works (only one 5 nodes).
Update: now to the code seems to be running on 10 nodes, everything being exactly the same … something is definitely not right.
Regarding intel MPI, I think I will have to clean out manually the make files, but I need to try that. I have been contact with the HPC support about this. Anyway, if based on the above behavior you have any tips how to progress, I would appreciate it.
---
### Author:  [@aszaboa](https://community.freefem.org/u/aszaboa)
#### Post date: [March 17, 2025, 9:00pm UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/15 "2025-03-17T21:00:01Z")
OK, I managed to make FreeFEM work with intel compiler + MPI. The catch is, that once again it does not work on multiple nodes.
Interestingly, I observed the following. When I run a test on one node with an sbatch script with srun within, if I redirect the stdout/stderr output to separate files, the text is written as binary. If I do not redirect it, it appears correctly.
I think there is a mismatch between software(FreeFEm+PETSc) and hardware, both of them were tested extensively, but separately. Can you suggest any tips how I can possibly resolve the issue?
---
### Author:  [@aszaboa](https://community.freefem.org/u/aszaboa)
#### Post date: [March 18, 2025, 7:59pm UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/16 "2025-03-18T19:59:18Z")
I continued to investigate intel MPI.
I the meantime, tried the [https://petsc.org/release/src/ksp/ksp/tutorials/ex15.c.html](https://petsc.org/release/src/ksp/ksp/tutorials/ex15.c.html) example out. I ran it with the following command: `srun -n 384 ex15 -m 200 -n 200 -ksp_type tsirm -pc_type ksp -ksp_monitor_short -ksp_ksp_type fgmres -ksp_ksp_rtol 1e-10 -ksp_pc_type mg -ksp_ksp_max_it 30 `, and it seems to work thus on two nodes.
Another simple test I conducted is [FreeFem-sources/examples/hpddm/stokes-block-2d-PETSc.edp at master · FreeFem/FreeFem-sources · GitHub](https://github.com/FreeFem/FreeFem-sources/blob/master/examples/hpddm/stokes-block-2d-PETSc.edp). It runs on 1 node/192 processes, but stalls on two nodes/382 processes.
Then, I use the matrices exported by the block-stokes example in [FreeFem-sources/examples/hpddm/MatLoad-PETSc.edp at master · FreeFem/FreeFem-sources · GitHub](https://github.com/FreeFem/FreeFem-sources/blob/master/examples/hpddm/MatLoad-PETSc.edp). Then, once again the example works on 1 node, but does not on two nodes.
These experiments indicate for me that something goes sideways in the FreeFEM/PETSc interface, or at least its compatibility with the HPS machine I am using. I attach the full output of the MatLoad examples with options `export I_MPI_DEBUG=30` and `export I_MPI_HYDRA_DEBUG=1`. I also attach an output with two nodes plus the `-start_in_debugger noxterm` option. I do not find these outputs very informative: the KSP solve stage just stops.
If you have any tips how to proceed with the investigation, I would appreciate it.
[MatLoad-384.log](https://community.freefem.org/uploads/short-url/kFNqstbUShD7NEU8klTT2uR5kiO.log) (64.6 KB)
[MatLoad-384-debugger.log](https://community.freefem.org/uploads/short-url/3DFq0rgbsxquz2sz0wYN4j0zF1j.log) (1.0 MB)
[MatLoad-192.log](https://community.freefem.org/uploads/short-url/q0edIveRnrlIQGYxRrqhDg8RdLF.log) (40.8 KB)
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [March 19, 2025, 7:41am UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/17 "2025-03-19T07:41:50Z")
In the `debugger.log` file, I read:
```auto
Mat Object: 384 MPI processes
type: mpiaij
rows=543003, cols=543003
total: nonzeros=15631208, allocated nonzeros=15631208
total number of mallocs used during MatSetValues calls=0
using I-node (on process 0) routines: found 708 nodes, limit used is 5
```
So it looks to me like the code is working?
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [March 19, 2025, 7:43am UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/18 "2025-03-19T07:43:23Z")
Ah, my bad, then there should be a `KSPSolve()`? Could you just stop one of the faulty process (no need to use GDB everywhere, can just run with `-debugger_ranks 0`) and get the back trace of where the process is hanging?
---
### Author:  [@aszaboa](https://community.freefem.org/u/aszaboa)
#### Post date: [March 19, 2025, 3:59pm UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/19 "2025-03-19T15:59:13Z")
Thanks for taking a look. Yes, the `KSPSolve` is where the process should continue. Currently, all the machine is mostly occupied, so I just queued the run without specifying `-N 2`. The weird thing is that now, the `MatLoad()` example works 9/10 times. However, when I run the [FreeFem-sources/examples/hpddm/navier-stokes-2d-PETSc.edp at master · FreeFem/FreeFem-sources · GitHub](https://github.com/FreeFem/FreeFem-sources/blob/master/examples/hpddm/navier-stokes-2d-PETSc.edp) example, does code consistently does not work on two nodes. Once again, it works on one node.
I attach outputs of runs with the option `-debugger-ranks 0` option; the Matload example when it was working/not working, and the NAvier-stokes output (the OK outputs are printed by me; I think it stops just ater the Jacobian is set, before KSPSolve is called internally). Any help is appreciated. I am quite confused by this behavior, especially the inconsistency.
[NavierStokes\_debugger\_notworking.log](https://community.freefem.org/uploads/short-url/hoJGoL7wVreN82PKHZBlABHQEhH.log) (67.1 KB)
[MatLoad-384-debugger-working.log](https://community.freefem.org/uploads/short-url/uFeuPWJnUg3iOij3Cpis7F6gssC.log) (72.6 KB)
[MatLoad-384-debugger-notworking.log](https://community.freefem.org/uploads/short-url/aWWg33VzEHH4diAlRS40kjxsCva.log) (67.2 KB)
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [March 19, 2025, 4:01pm UTC](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788/20 "2025-03-19T16:01:42Z")
When a process hangs, I need you to show the back trace in GDB.
[Next page](https://community.freefem.org/t/freefem-compilation-issues-on-hpc/3788.md?page=2)