# FreeFem++ Not Using GPU with PETSc
**URL:** https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594
**Category:** FreeFEM installation
**Created:** [November 22, 2024, 3:44pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594 "2024-11-22T15:44:21Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@ELfeniz](https://community.freefem.org/u/ELfeniz)
#### Post date: [November 22, 2024, 3:44pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/1 "2024-11-22T15:44:21Z")
Description:
I am encountering issues with FreeFem++ not utilizing the GPU when solving problems with PETSc. Despite following all the necessary steps to enable GPU support, it seems that FreeFem++ does not leverage CUDA during execution. Below, I outline everything I have done so far.
1. PETSc Configuration
I configured PETSc with CUDA support using the following command:
./configure --with-mpi=1
–with-cuda=1
–with-cudac=nvcc
–download-fblaslapack
–with-debugging=0
–prefix=/usr/local/petsc
–with-shared-libraries=1
–download-hpddm
–download-metis
–download-ptscotch
–download-parmetis
–download-superlu
–download-mmg
–download-parmmg
–download-scalapack
–download-mumps
I also set the necessary environment variables:
export PETSC\_DIR=/usr/local/petsc
export PETSC\_ARCH=“”
export PATH=$PETSC\_DIR/lib/petsc/bin:$PATH
export LD\_LIBRARY\_PATH=$PETSC\_DIR/lib:$LD\_LIBRARY\_PATH
PETSc appears to work correctly with CUDA, as verified by running examples such as ex19 and ex2. CUDA was successfully detected and utilized by PETSc during these tests.
2. FreeFem++ Installation
I installed FreeFem++ from the develop branch with support for PETSc and CUDA using the following commands:
./configure
CPPFLAGS=“-I$PETSC\_DIR/include”
LDFLAGS=“-L$PETSC\_DIR/lib”
–enable-download
–enable-optim
–with-mpi=/usr/local/openmpi
–with-petsc=$PETSC\_DIR
–enable-summary
–without-mumps
Additional steps taken to address potential issues:
```
Used ./3rdparty/getall -a to download missing dependencies.
Manually edited the mpic.c file in the MUMPS directory to resolve compilation errors.
Built FreeFem++ with make -j$(nproc) and passed all tests using make check.
```
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [November 22, 2024, 4:26pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/2 "2024-11-22T16:26:14Z")
What preconditioner are you using? What `MatType` are you using?
---
### Author:  [@ELfeniz](https://community.freefem.org/u/ELfeniz)
#### Post date: [November 22, 2024, 4:33pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/3 "2024-11-22T16:33:21Z")
Thank you for your response. Here are the details about the preconditioner and matrix type I am using in the example:
```
Preconditioner: Jacobi (-pc_type jacobi)
Matrix type: aijcusparse (GPU-optimized format)
```
The script I am using is as follows:
load “PETSc”
macro defKSP\_OPTIONS() “-ksp\_type cg -pc\_type jacobi -vec\_type cuda -mat\_type aijcusparse -ksp\_monitor -log\_view” //
int nx = 5000, ny = 5000;
real lx = 1.0, ly = 1.0;
mesh Th = square(nx, ny, [x \* lx, y \* ly]);
fespace Vh(Th, P1);
Vh u, v;
func f = 1.0;
solve Poisson(u, v, solver = “petsc”) =
int2d(Th)(dx(u) \* dx(v) + dy(u) \* dy(v)) -
int2d(Th)(f \* v);
cout \<\< “System solved using GPU with PETSc.” \<\< endl;
I am running this script with the following command:
mpiexec -n 1 FreeFem++ poisson\_gpu.edp -log\_view
The problem is solved without errors, but it seems that the GPU is not being utilized during the execution, even though PETSc was configured with CUDA support and examples such as ex19 and ex2 work correctly using the GPU.
Is there something I might be missing to ensure FreeFem++ leverages the GPU? Any additional guidance would be greatly appreciated.
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [November 22, 2024, 4:35pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/4 "2024-11-22T16:35:30Z")
Could you please send the output of `-ksp_view`?
---
### Author:  [@ELfeniz](https://community.freefem.org/u/ELfeniz)
#### Post date: [November 22, 2024, 4:44pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/5 "2024-11-22T16:44:42Z")
jp@jp:~/Descargas$ mpiexec -n 1 FreeFem++ poisson\_gpu.edp -log\_view
– FreeFem++ v4.14 (mié 20 nov 2024 17:13:44 -05 - git v4.14-95-g6fcdacf31)
file : poisson\_gpu.edp
Load: lg\_fem lg\_mesh lg\_mesh3 init\_mesh3\_array
eigenvalue
1 : load “PETSc”
2 :
3 : macro defKSP\_OPTIONS() “-ksp\_type cg -pc\_type jacobi -vec\_type cuda -mat\_type aijcusparse -ksp\_monitor -log\_view -ksp\_view” ////
4 :
5 : int nx = 1000, ny = 500;
6 : real lx = 1.0, ly = 1.0;
7 : mesh Th = square(nx, ny, [x \* lx, y \* ly]);
8 :
9 : fespace Vh(Th, P1);
10 :
11 : Vh u, v;
12 : func f = 1.0;
13 :
14 : solve Poisson(u, v, solver = “petsc”) =
15 : int2d(Th)(dx(u) \* dx(v) + dy(u) \* dy(v)) -
16 : int2d(Th)(f \* v);
17 :
18 : cout \<\< “System solved using GPU with PETSc.” \<\< endl;
19 :
20 : sizestack + 1024 =1768 ( 744 )
– Square mesh : nb vertices =501501 , nb triangles = 1000000 , nb boundary edges 3000 rmdup= 0
– Solve :
min -1.55651e+10 max -1.55651e+10
System solved using GPU with PETSc.
times: compile 0.074077s, execution 7.02759s, mpirank:0
######## unfreed pointers 23 Nb pointer, 0Bytes , mpirank 0, memory leak =17228288
CodeAlloc : nb ptr 4165, size :560176 mpirank: 0
Ok: Normal End
* * *
\*\*\* WIDEN YOUR WINDOW TO 160 CHARACTERS. Use ‘enscript -r -fCourier9’ to print this document \*\*\*
* * *
------------------------------------------------------------------ PETSc Performance Summary: ------------------------------------------------------------------
FreeFem++ on a named jp with 1 process, by jp on Fri Nov 22 11:43:32 2024
Using Petsc Development GIT revision: v3.22.1-208-g2ad7182b109 GIT Date: 2024-11-20 12:46:14 -0600
```
Max Max/Min Avg Total
```
Time (sec): 7.083e+00 1.000 7.083e+00
Objects: 0.000e+00 0.000 0.000e+00
Flops: 0.000e+00 0.000 0.000e+00 0.000e+00
Flops/sec: 0.000e+00 0.000 0.000e+00 0.000e+00
MPI Msg Count: 0.000e+00 0.000 0.000e+00 0.000e+00
MPI Msg Len (bytes): 0.000e+00 0.000 0.000e+00 0.000e+00
MPI Reductions: 0.000e+00 0.000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N → 2N flops
and VecAXPY() for complex vectors of length N → 8N flops
Summary of Stages: ----- Time ------ ----- Flop ------ — Messages — – Message Lengths – – Reductions –
Avg %Total Avg %Total Count %Total Avg %Total Count %Total
0: Main Stage: 7.0831e+00 100.0% 0.0000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0%
* * *
## See the ‘Profiling’ chapter of the users’ manual for details on interpreting output. Phase summary info: Count: number of times phase was executed Time and Flop: Max - maximum over all processors Ratio - ratio of maximum to minimum over all processors Mess: number of messages sent AvgLen: average message length (bytes) Reduct: number of global reductions Global: entire computation Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop(). %T - percent time in this phase %F - percent flop in this phase %M - percent messages in this phase %L - percent message lengths in this phase %R - percent reductions in this phase Total Mflop/s: 10e-6 \* (sum of flop over all processors)/(max time over all processors) GPU Mflop/s: 10e-6 \* (sum of flop on GPU over all processors)/(max GPU time over all processors) CpuToGpu Count: total number of CPU to GPU copies per processor CpuToGpu Size (Mbytes): 10e-6 \* (total size of CPU to GPU copies per processor) GpuToCpu Count: total number of GPU to CPU copies per processor GpuToCpu Size (Mbytes): 10e-6 \* (total size of GPU to CPU copies per processor) GPU %F: percent flops on GPU in this event
## Event Count Time (sec) Flop — Global — — Stage ---- Total GPU - CpuToGpu - - GpuToCpu - GPU Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s Mflop/s Count Size Count Size %F
— Event Stage 0: Main Stage
* * *
Object Type Creations Destructions. Reports information only for process 0.
— Event Stage 0: Main Stage
## ======================================================================================================================== Average time to get PetscTime(): 2.37e-08 #PETSc Option Table entries: -log\_view # (source: command line) #End of PETSc Option Table entries Compiled without FORTRAN kernels Compiled with full precision matrices (default) sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void\*) 8 sizeof(PetscScalar) 8 sizeof(PetscInt) 4 Configure options: --with-mpi=1 --with-cuda=1 --with-cudac=nvcc --download-fblaslapack --with-debugging=0 --prefix=/usr/local/petsc --with-shared-libraries=1 --download-hpddm --download-metis --download-ptscotch --download-parmetis --download-superlu --download-mmg --download-parmmg --download-scalapack --download-mumps
## Libraries compiled on 2024-11-20 21:56:11 on jp Machine characteristics: Linux-6.8.0-49-generic-x86\_64-with-glibc2.35 Using PETSc directory: /usr/local/petsc Using PETSc arch:
## Using C compiler: mpicc -fPIC -Wall -Wwrite-strings -Wno-unknown-pragmas -Wno-lto-type-mismatch -Wno-stringop-overflow -fstack-protector -fvisibility=hidden -g -O Using Fortran compiler: mpif90 -fPIC -Wall -ffree-line-length-none -ffree-line-length-0 -Wno-lto-type-mismatch -Wno-unused-dummy-argument -g -O
## Using include paths: -I/usr/local/petsc/include -I/usr/local/cuda-12.2/include
## Using C linker: mpicc Using Fortran linker: mpif90 Using libraries: -Wl,-rpath,/usr/local/petsc/lib -L/usr/local/petsc/lib -lpetsc -Wl,-rpath,/usr/local/petsc/lib -L/usr/local/petsc/lib -Wl,-rpath,/usr/local/cuda-12.2/lib64 -L/usr/local/cuda-12.2/lib64 -L/usr/local/cuda-12.2/lib64/stubs -Wl,-rpath,/usr/local/openmpi/lib -L/usr/local/openmpi/lib -Wl,-rpath,/usr/lib/gcc/x86\_64-linux-gnu/11 -L/usr/lib/gcc/x86\_64-linux-gnu/11 -ldmumps -lmumps\_common -lpord -lpthread -lscalapack -lsuperlu -lflapack -lfblas -lparmmg -lmmg -lmmg3d -lptesmumps -lptscotchparmetisv3 -lptscotch -lptscotcherr -lesmumps -lscotch -lscotcherr -lparmetis -lmetis -lm -lcudart -lnvToolsExt -lcufft -lcublas -lcusparse -lcusolver -lcurand -lcuda -lX11 -lmpi\_usempif08 -lmpi\_usempi\_ignore\_tkr -lmpi\_mpifh -lmpi -lgfortran -lm -lgfortran -lm -lgcc\_s -lquadmath -lstdc++ -lrt -lquadmath
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [November 22, 2024, 4:49pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/6 "2024-11-22T16:49:48Z")
You are not using PETSc for doing the linear solve, thus why the GPU is not being utilized. You need to use a `varf`, switch to a `Mat`, and so on. BTW, do you have a good preconditioner in mind for using GPU? Plain Jacobi is terrible, unless you are working on a simple problem.
---
### Author:  [@ELfeniz](https://community.freefem.org/u/ELfeniz)
#### Post date: [November 22, 2024, 4:55pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/7 "2024-11-22T16:55:58Z")
Thanks for your observation. It is clear to me that my current implementation is not set up for PETSc to handle the linear solution directly, and I understand that is necessary to take advantage of the GPU. I am also considering your preconditioner suggestion and am willing to try something more efficient than Jacobi for more complex problems.
Could you provide me with a working example where the GPU is properly utilized with PETSc in FreeFem++? This would be very useful for me to compare and make sure my setup is working correctly.
I really appreciate your help. 😊
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [November 23, 2024, 7:11am UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/8 "2024-11-23T07:11:24Z")
I don’t think there is any example using GPU, yet. Why do you want to use GPU? Are you sure this is appropriate for your application?
---
### Author:  [@ELfeniz](https://community.freefem.org/u/ELfeniz)
#### Post date: [November 23, 2024, 1:20pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/9 "2024-11-23T13:20:21Z")
Thanks for your response. I am working on a simulation of the electrical behavior of the atrium using finite elements and the Courtemanche model. My main goal is to speed up the solution of the finite element system using GPU and additionally optimize the voltages and currents calculations also on GPU.
Do you think this would be feasible using FreeFem++ with GPU, or would you recommend another tool or approach to achieve this acceleration? I really appreciate your advice on how to approach this kind of problem, especially if FreeFem++ is not the best option to take advantage of GPU in this context.
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [November 23, 2024, 5:45pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/10 "2024-11-23T17:45:56Z")
> My main goal is to speed up the solution of the finite element system using GPU and additionally optimize the voltages and currents calculations also on GPU
Again, what is your motivation for using GPU? Why does plain distributed-memory parallelism not fit your needs?
---
### Author:  [@ELfeniz](https://community.freefem.org/u/ELfeniz)
#### Post date: [November 23, 2024, 6:27pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/11 "2024-11-23T18:27:01Z")
My motivation for using GPU is to leverage its massive parallelism for node-specific computations like diffusion, voltages, and currents, which are highly parallelizable.
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [November 23, 2024, 7:41pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/12 "2024-11-23T19:41:58Z")
> highly parallelizable
How? What is the algorithm? Have you implemented it in CUDA?
---
### Author:  [@ELfeniz](https://community.freefem.org/u/ELfeniz)
#### Post date: [November 23, 2024, 7:46pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/13 "2024-11-23T19:46:37Z")
No, I have not implemented the algorithm in CUDA yet. My initial intention was to explore whether it is feasible to use FreeFEM with PETSc to leverage GPU acceleration for my simulations. I was curious to see if FreeFEM could natively utilize PETSc’s GPU capabilities. While I have not developed a CUDA implementation myself, I have observed that other tools leveraging GPU acceleration for similar finite element problems have successfully implemented such features. This prompted my interest in exploring the possibilities with FreeFEM.
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [November 23, 2024, 8:09pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/14 "2024-11-23T20:09:38Z")
> I have observed that other tools leveraging GPU acceleration for similar finite element problems have successfully implemented such features
Using PETSc? Please share a reference, I will let you know how easy it would be to try in FreeFEM.
---
### Author:  [@ELfeniz](https://community.freefem.org/u/ELfeniz)
#### Post date: [November 23, 2024, 8:44pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/15 "2024-11-23T20:44:55Z")
PETSc (Portable, Extensible Toolkit for Scientific Computation) offers support for finite element solvers that leverage GPU acceleration. The `PetscFE` class in PETSc encapsulates finite element discretizations, facilitating the implementation of finite element methods within the PETSc framework.
[Petsc](https://petsc.org/release/manualpages/FE/?utm_source=chatgpt.com)
To utilize GPU capabilities in PETSc, you can configure the library with GPU support and select appropriate backends such as CUDA, HIP, or Kokkos. This enables the execution of algebraic solvers on GPU systems from NVIDIA, AMD, and Intel.
[Petsc](https://petsc.org/release/overview/gpu_roadmap/?utm_source=chatgpt.com)
For practical examples, PETSc provides several tutorials and examples that demonstrate the use of finite element solvers with GPU acceleration. Notably, examples like `ex12`, `ex17`, and `ex62` showcase the application of `PetscFE` in solving finite element problems.
[Petsc](https://petsc.org/release/manualpages/FE/?utm_source=chatgpt.com)
Additionally, the paper “Toward Performance-Portable PETSc for GPU-based Exascale Systems” discusses the integration of GPU support in PETSc, highlighting its performance and portability across different GPU architectures.
[ArXiv](https://arxiv.org/abs/2011.00715?utm_source=chatgpt.com)
These resources provide comprehensive guidance on implementing finite element solvers with GPU acceleration using PETSc.
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [November 23, 2024, 8:54pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/16 "2024-11-23T20:54:12Z")
I am not looking for a ChatGPT answer. You said: `I have observed that other tools leveraging GPU acceleration for similar finite element problems have successfully implemented such features`. What other tools? And how do they relate to `node-specific computations like diffusion, voltages, and currents`?
---
### Author:  [@prj](https://community.freefem.org/u/prj)
#### Post date: [November 23, 2024, 8:57pm UTC](https://community.freefem.org/t/freefem-not-using-gpu-with-petsc/3594/17 "2024-11-23T20:57:39Z")
GPUs are difficult to use. If you don’t know what you are looking for or what you need, chances are that you will get abysmal performance in the end. I would suggest that you try out first plain distributed-memory parallelism. GPUs can work seamlessly in PETSc (and thus in FreeFEM), but again, if you don’t know what you are doing, there is no real point in using them (because you’ll get bad performance).