Dear all,
I am learning the parallel scalability, and I want to start with a 3D heat equation with hypre as its preconditioner. My program is as following which is run on hpc Aire with 1, 2, 4, 8, 16 processes. To make my workload as equal as possible for each process, I distribute around 48,000 tetrahedrons for each processor(not count overlapping elements). I use uniform meshes generated from cube(n,n,n); n = 20, 25, 32, 40, 50 respectively.
load "PETSc"
macro dimension()3// EOM
include "macro_ddm.idp"
int[int] fforder = [1];
int n = getARGV("-n",20);
mesh3 Th = cube(n,n,n);
// if(mpirank == 0){
// cout << "Number of elements: " << Th.nt << endl;
// cout << "Number of elements per processor: " << Th.nt/mpisize << endl;
// }
DmeshCreate(Th);
Mat A;
MatCreate(Th, A, P2);
fespace Vh(Th,P2);
Vh u, uold=0;
macro grad(u)[dx(u),dy(u),dz(u)]// EOM
real T=1, t=0, dt=0.01;
int Nt = T/dt;
varf vHeat(u,v) =
int3d(Th,qforder=3)( u*v + dt*grad(u)'*grad(v) )
+ int3d(Th,qforder=3)( dt*v + uold*v)
+ on(6,u=1);
for(int i = 0; i <= Nt; i++){
t = i*dt;
if(mpirank == 0)
cout << "Time: " << t << endl;
A = vHeat(Vh,Vh,tgv=-1);
real[int] rhsV = vHeat(0, Vh, tgv=-1);
set(A, sparams=" -ksp_type gmres -pc_type hypre -ksp_rtol 1e-12 -ksp_monitor");
u[] = A^-1 * rhsV;
// savevtk("TestScalability/Out", bin=1, Th, u, dataname=" u ", order=fforder, append = i == 0 ? false : true);
uold = 0;
uold[] = u[];
}
The result of my 1, 2, 4, 8, 16 processors are as the following
P1
------------------------------------------------------------------ PETSc Performance Summary: ------------------------------------------------------------------
/usr/freefem/bin/FreeFem++-mpi on a named node017.aire.lee.alces.network with 1 process, by tsww6159 on Sun Sep 6 22:51:06 2026
Using PETSc Release Version 3.24.4, Jan 30, 2026
Max Max/Min Avg Total
Time (sec): 9.070e+01 1.000 9.070e+01
Objects: 0.000e+00 0.000 0.000e+00
Flops: 4.323e+09 1.000 4.323e+09 4.323e+09
Flops/sec: 4.767e+07 1.000 4.767e+07 4.767e+07
MPI Msg Count: 0.000e+00 0.000 0.000e+00 0.000e+00
MPI Msg Len (bytes): 0.000e+00 0.000 0.000e+00 0.000e+00
MPI Reductions: 0.000e+00 0.000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flops
and VecAXPY() for complex vectors of length N --> 8N flops
Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total Count %Total Avg %Total Count %Total
0: Main Stage: 9.0701e+01 100.0% 4.3234e+09 100.0% 0.000e+00 0.0% 0.000e+00 0.0% 0.000e+00 0.0%
------------------------------------------------------------------------------------------------------------------------
Count: number of times phase was executed
Time and Flop: Max - maximum over all processes
Ratio - ratio of maximum to minimum over all processes
Mess: number of messages sent
AvgLen: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flop in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 1e-6 * (sum of flop over all processes)/(max time over all processes)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flop --- Global --- --- Stage ---- Total
Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
MatMult 808 1.0 8.3392e-01 1.0 3.01e+09 1.0 0.0e+00 0.0e+00 0.0e+00 1 70 0 0 0 1 70 0 0 0 3607
MatConvert 101 1.0 2.0924e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatAssemblyBegin 203 1.0 5.0791e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatAssemblyEnd 203 1.0 5.7787e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatGetRowIJ 2 1.0 6.8000e-07 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatZeroEntries 100 1.0 4.7695e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatSetPreallCOO 1 1.0 9.8442e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatSetValuesCOO 101 1.0 1.8672e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSetUp 101 1.0 1.6843e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSolve 101 1.0 2.2894e+01 1.0 4.32e+09 1.0 0.0e+00 0.0e+00 0.0e+00 25 100 0 0 0 25 100 0 0 0 189
KSPGMRESOrthog 808 1.0 1.0366e-01 1.0 1.00e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 23 0 0 0 0 23 0 0 0 9670
PCSetUp 101 1.0 1.6384e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 18 0 0 0 0 18 0 0 0 0 0
PCApply 909 1.0 2.1916e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 24 0 0 0 0 24 0 0 0 0 0
VecMDot 808 1.0 6.8242e-02 1.0 5.01e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 12 0 0 0 0 12 0 0 0 7344
VecNorm 909 1.0 8.0309e-03 1.0 1.25e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 15602
VecScale 909 1.0 6.8538e-03 1.0 6.26e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 9141
VecCopy 101 1.0 3.2084e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 1010 1.0 1.7126e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 101 1.0 2.1916e-03 1.0 1.39e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 6353
VecMAXPY 909 1.0 4.0451e-02 1.0 6.13e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 14 0 0 0 0 14 0 0 0 15143
VecNormalize 909 1.0 1.5921e-02 1.0 1.88e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 0 4 0 0 0 11805
------------------------------------------------------------------------------------------------------------------------
P2
------------------------------------------------------------------ PETSc Performance Summary: ------------------------------------------------------------------
/usr/freefem/bin/FreeFem++-mpi on a named node020.aire.lee.alces.network with 2 processes, by tsww6159 on Sun Sep 6 22:51:38 2026
Using PETSc Release Version 3.24.4, Jan 30, 2026
Max Max/Min Avg Total
Time (sec): 1.220e+02 1.000 1.220e+02
Objects: 0.000e+00 0.000 0.000e+00
Flops: 4.946e+09 1.054 4.820e+09 9.640e+09
Flops/sec: 4.054e+07 1.054 3.951e+07 7.901e+07
MPI Msg Count: 1.013e+03 1.000 1.013e+03 2.026e+03
MPI Msg Len (bytes): 3.935e+07 1.000 3.885e+04 7.870e+07
MPI Reductions: 2.500e+01 1.000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flops
and VecAXPY() for complex vectors of length N --> 8N flops
Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total Count %Total Avg %Total Count %Total
0: Main Stage: 1.2200e+02 100.0% 9.6396e+09 100.0% 2.026e+03 100.0% 3.885e+04 100.0% 8.000e+00 32.0%
------------------------------------------------------------------------------------------------------------------------
Count: number of times phase was executed
Time and Flop: Max - maximum over all processes
Ratio - ratio of maximum to minimum over all processes
Mess: number of messages sent
AvgLen: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flop in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 1e-6 * (sum of flop over all processes)/(max time over all processes)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flop --- Global --- --- Stage ---- Total
Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
BuildTwoSided 1 1.0 5.5871e-05 6.6 0.00e+00 0.0 2.0e+00 4.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatMult 908 1.0 9.4557e-01 1.0 3.37e+09 1.1 1.8e+03 3.1e+04 0.0e+00 1 68 90 72 0 1 68 90 72 0 6941
MatConvert 101 1.0 2.0579e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 8 0 0 0 0 25 0
MatAssemblyBegin 304 1.0 7.3149e-05 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatAssemblyEnd 304 1.0 4.5302e-02 4.2 0.00e+00 0.0 4.0e+00 7.8e+03 1.0e+00 0 0 0 0 4 0 0 0 0 12 0
MatGetRowIJ 4 1.0 1.4700e-06 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatZeroEntries 100 1.0 5.2980e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatSetPreallCOO 2 1.0 1.0183e-02 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatSetValuesCOO 202 1.0 1.8354e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecMDot 908 1.0 1.9926e-01 2.1 6.16e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 12 0 0 0 0 12 0 0 0 6039
VecNorm 1009 1.0 2.5536e-02 2.8 1.37e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 10483
VecScale 1009 1.0 6.0641e-03 1.1 6.85e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 22072
VecCopy 101 1.0 3.0557e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 1110 1.0 1.7826e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 101 1.0 2.2214e-03 1.0 1.37e+07 1.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 12062
VecMAXPY 1009 1.0 5.2391e-02 1.1 7.39e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 15 0 0 0 0 15 0 0 0 27568
VecScatterBegin 908 1.0 1.4058e-02 1.6 0.00e+00 0.0 1.8e+03 3.1e+04 0.0e+00 0 0 90 72 0 0 0 90 72 0 0
VecScatterEnd 908 1.0 6.7309e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecNormalize 1009 1.0 3.2835e-02 2.0 2.05e+08 1.0 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 0 4 0 0 0 12229
SFSetGraph 1 1.0 4.5760e-05 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFSetUp 1 1.0 8.5093e-04 1.0 0.00e+00 0.0 4.0e+00 7.8e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFPack 908 1.0 2.8526e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFUnpack 908 1.0 1.1066e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSetUp 101 1.0 1.7717e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSolve 101 1.0 2.9126e+01 1.0 4.95e+09 1.1 1.8e+03 3.1e+04 0.0e+00 24 100 90 72 0 24 100 90 72 0 331
KSPGMRESOrthog 908 1.0 2.4069e-01 1.7 1.23e+09 1.0 0.0e+00 0.0e+00 0.0e+00 0 25 0 0 0 0 25 0 0 0 9999
PCSetUp 101 1.0 3.6195e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 29 0 0 0 8 29 0 0 0 25 0
PCApply 1009 1.0 2.7997e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 23 0 0 0 0 23 0 0 0 0 0
------------------------------------------------------------------------------------------------------------------------
P4
------------------------------------------------------------------ PETSc Performance Summary: ------------------------------------------------------------------
/usr/freefem/bin/FreeFem++-mpi on a named node024.aire.lee.alces.network with 4 processes, by tsww6159 on Sun Sep 6 22:53:08 2026
Using PETSc Release Version 3.24.4, Jan 30, 2026
Max Max/Min Avg Total
Time (sec): 2.127e+02 1.000 2.127e+02
Objects: 0.000e+00 0.000 0.000e+00
Flops: 5.340e+09 1.116 5.022e+09 2.009e+10
Flops/sec: 2.511e+07 1.116 2.361e+07 9.446e+07
MPI Msg Count: 3.042e+03 1.500 2.535e+03 1.014e+04
MPI Msg Len (bytes): 7.751e+07 1.319 2.642e+04 2.679e+08
MPI Reductions: 2.500e+01 1.000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flops
and VecAXPY() for complex vectors of length N --> 8N flops
Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total Count %Total Avg %Total Count %Total
0: Main Stage: 2.1269e+02 100.0% 2.0090e+10 100.0% 1.014e+04 100.0% 2.642e+04 100.0% 8.000e+00 32.0%
------------------------------------------------------------------------------------------------------------------------
Count: number of times phase was executed
Time and Flop: Max - maximum over all processes
Ratio - ratio of maximum to minimum over all processes
Mess: number of messages sent
AvgLen: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flop in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 1e-6 * (sum of flop over all processes)/(max time over all processes)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flop --- Global --- --- Stage ---- Total
Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
BuildTwoSided 1 1.0 1.4816e-04 7.8 0.00e+00 0.0 1.0e+01 4.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatMult 909 1.0 1.0424e+00 1.1 3.66e+09 1.1 9.1e+03 2.1e+04 0.0e+00 0 68 90 70 0 0 68 90 70 0 13153
MatConvert 101 1.0 2.2623e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 8 0 0 0 0 25 0
MatAssemblyBegin 304 1.0 8.1729e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatAssemblyEnd 304 1.0 5.4024e-02 3.8 0.00e+00 0.0 2.0e+01 5.1e+03 1.0e+00 0 0 0 0 4 0 0 0 0 12 0
MatGetRowIJ 4 1.0 1.2600e-06 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatZeroEntries 100 1.0 5.8797e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatSetPreallCOO 2 1.0 9.2522e-03 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatSetValuesCOO 202 1.0 2.0363e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecMDot 909 1.0 3.6997e-01 3.4 6.59e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 12 0 0 0 0 12 0 0 0 6747
VecNorm 1010 1.0 4.8697e-02 3.9 1.46e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 11392
VecScale 1010 1.0 8.0204e-03 1.3 7.32e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 34583
VecCopy 101 1.0 3.6115e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 1111 1.0 1.8947e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 101 1.0 2.4527e-03 1.2 1.46e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 22618
VecMAXPY 1010 1.0 5.7325e-02 1.2 7.91e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 15 0 0 0 0 15 0 0 0 52256
VecScatterBegin 909 1.0 1.8815e-02 1.5 0.00e+00 0.0 9.1e+03 2.1e+04 0.0e+00 0 0 90 70 0 0 0 90 70 0 0
VecScatterEnd 909 1.0 9.7846e-03 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecNormalize 1010 1.0 5.6056e-02 2.6 2.20e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 0 4 0 0 0 14844
SFSetGraph 1 1.0 8.1250e-05 2.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFSetUp 1 1.0 9.8632e-04 1.1 0.00e+00 0.0 2.0e+01 5.1e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFPack 909 1.0 5.4073e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFUnpack 909 1.0 1.0243e-04 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSetUp 101 1.0 1.9113e-04 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSolve 101 1.0 5.5339e+01 1.0 5.34e+09 1.1 9.1e+03 2.1e+04 0.0e+00 26 100 90 70 0 26 100 90 70 0 363
KSPGMRESOrthog 909 1.0 4.1378e-01 2.6 1.32e+09 1.1 0.0e+00 0.0e+00 0.0e+00 0 25 0 0 0 0 25 0 0 0 12066
PCSetUp 101 1.0 9.6662e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 44 0 0 0 8 44 0 0 0 25 0
PCApply 1010 1.0 5.4088e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 25 0 0 0 0 25 0 0 0 0 0
------------------------------------------------------------------------------------------------------------------------
P8
------------------------------------------------------------------ PETSc Performance Summary: ------------------------------------------------------------------
/usr/freefem/bin/FreeFem++-mpi on a named node025.aire.lee.alces.network with 8 processes, by tsww6159 on Sun Sep 6 22:54:25 2026
Using PETSc Release Version 3.24.4, Jan 30, 2026
Max Max/Min Avg Total
Time (sec): 2.894e+02 1.000 2.894e+02
Objects: 0.000e+00 0.000 0.000e+00
Flops: 5.166e+09 1.145 4.878e+09 3.903e+10
Flops/sec: 1.785e+07 1.145 1.686e+07 1.348e+08
MPI Msg Count: 6.084e+03 2.000 4.310e+03 3.448e+04
MPI Msg Len (bytes): 1.310e+08 2.052 1.968e+04 6.785e+08
MPI Reductions: 2.500e+01 1.000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flops
and VecAXPY() for complex vectors of length N --> 8N flops
Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total Count %Total Avg %Total Count %Total
0: Main Stage: 2.8942e+02 100.0% 3.9026e+10 100.0% 3.448e+04 100.0% 1.968e+04 100.0% 8.000e+00 32.0%
------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on interpreting output.
Phase summary info:
Count: number of times phase was executed
Time and Flop: Max - maximum over all processes
Ratio - ratio of maximum to minimum over all processes
Mess: number of messages sent
AvgLen: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flop in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 1e-6 * (sum of flop over all processes)/(max time over all processes)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flop --- Global --- --- Stage ---- Total
Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
BuildTwoSided 1 1.0 1.9679e-04 7.2 0.00e+00 0.0 3.4e+01 4.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatMult 909 1.0 1.0816e+00 1.2 3.55e+09 1.2 3.1e+04 1.5e+04 0.0e+00 0 68 90 68 0 0 68 90 68 0 24668
MatConvert 101 1.0 2.2781e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 8 0 0 0 0 25 0
MatAssemblyBegin 304 1.0 7.7876e-05 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatAssemblyEnd 304 1.0 1.4015e-01 6.5 0.00e+00 0.0 6.8e+01 3.8e+03 1.0e+00 0 0 0 0 4 0 0 0 0 12 0
MatGetRowIJ 4 1.0 1.7200e-06 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatZeroEntries 100 1.0 5.7262e-02 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatSetPreallCOO 2 1.0 9.8441e-03 2.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatSetValuesCOO 202 1.0 1.9637e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecMDot 909 1.0 5.2162e-01 5.1 6.34e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 12 0 0 0 0 12 0 0 0 9261
VecNorm 1010 1.0 6.4972e-02 3.8 1.41e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 16523
VecScale 1010 1.0 8.0806e-03 1.5 7.04e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 66425
VecCopy 101 1.0 3.6162e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 1111 1.0 1.8687e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 101 1.0 2.3572e-03 1.1 1.41e+07 1.1 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 45542
VecMAXPY 1010 1.0 5.2793e-02 1.2 7.60e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 15 0 0 0 0 15 0 0 0 109805
VecScatterBegin 909 1.0 3.6968e-02 2.9 0.00e+00 0.0 3.1e+04 1.5e+04 0.0e+00 0 0 90 68 0 0 0 90 68 0 0
VecScatterEnd 909 1.0 1.4852e-02 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecNormalize 1010 1.0 7.1851e-02 3.0 2.11e+08 1.1 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 0 4 0 0 0 22411
SFSetGraph 1 1.0 1.1856e-04 2.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFSetUp 1 1.0 1.0634e-03 1.1 0.00e+00 0.0 6.8e+01 3.8e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFPack 909 1.0 8.9674e-03 2.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFUnpack 909 1.0 1.2299e-04 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSetUp 101 1.0 1.7876e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSolve 101 1.0 5.8800e+01 1.0 5.17e+09 1.1 3.1e+04 1.5e+04 0.0e+00 20 100 90 68 0 20 100 90 68 0 664
KSPGMRESOrthog 909 1.0 5.6545e-01 3.8 1.27e+09 1.1 0.0e+00 0.0e+00 0.0e+00 0 25 0 0 0 0 25 0 0 0 17086
PCSetUp 101 1.0 1.6975e+02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 57 0 0 0 8 57 0 0 0 25 0
PCApply 1010 1.0 5.7518e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 20 0 0 0 0 20 0 0 0 0 0
------------------------------------------------------------------------------------------------------------------------
P16:
------------------------------------------------------------------ PETSc Performance Summary: ------------------------------------------------------------------
/usr/freefem/bin/FreeFem++-mpi on a named node017.aire.lee.alces.network with 16 processes, by tsww6159 on Sun Sep 6 23:04:09 2026
Using PETSc Release Version 3.24.4, Jan 30, 2026
Max Max/Min Avg Total
Time (sec): 3.645e+02 1.000 3.645e+02
Objects: 0.000e+00 0.000 0.000e+00
Flops: 5.422e+09 1.168 4.937e+09 7.900e+10
Flops/sec: 1.488e+07 1.168 1.355e+07 2.168e+08
MPI Msg Count: 9.396e+03 2.250 6.656e+03 1.065e+05
MPI Msg Len (bytes): 1.496e+08 2.142 1.541e+04 1.641e+09
MPI Reductions: 2.500e+01 1.000
Flop counting convention: 1 flop = 1 real number operation of type (multiply/divide/add/subtract)
e.g., VecAXPY() for real vectors of length N --> 2N flops
and VecAXPY() for complex vectors of length N --> 8N flops
Summary of Stages: ----- Time ------ ----- Flop ------ --- Messages --- -- Message Lengths -- -- Reductions --
Avg %Total Avg %Total Count %Total Avg %Total Count %Total
0: Main Stage: 3.6447e+02 100.0% 7.9000e+10 100.0% 1.065e+05 100.0% 1.541e+04 100.0% 8.000e+00 32.0%
------------------------------------------------------------------------------------------------------------------------
Count: number of times phase was executed
Time and Flop: Max - maximum over all processes
Ratio - ratio of maximum to minimum over all processes
Mess: number of messages sent
AvgLen: average message length (bytes)
Reduct: number of global reductions
Global: entire computation
Stage: stages of a computation. Set stages with PetscLogStagePush() and PetscLogStagePop().
%T - percent time in this phase %F - percent flop in this phase
%M - percent messages in this phase %L - percent message lengths in this phase
%R - percent reductions in this phase
Total Mflop/s: 1e-6 * (sum of flop over all processes)/(max time over all processes)
------------------------------------------------------------------------------------------------------------------------
Event Count Time (sec) Flop --- Global --- --- Stage ---- Total
Max Ratio Max Ratio Max Ratio Mess AvgLen Reduct %T %F %M %L %R %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------
--- Event Stage 0: Main Stage
BuildTwoSided 1 1.0 2.0869e-04 6.8 0.00e+00 0.0 1.0e+02 4.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatMult 939 1.0 1.3317e+00 1.2 3.71e+09 1.2 9.6e+04 1.2e+04 0.0e+00 0 68 90 68 0 0 68 90 68 0 40307
MatConvert 101 1.0 2.2763e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 0 0 0 0 8 0 0 0 0 25 0
MatAssemblyBegin 304 1.0 1.0364e-04 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatAssemblyEnd 304 1.0 1.4659e-01 6.3 0.00e+00 0.0 2.0e+02 2.9e+03 1.0e+00 0 0 0 0 4 0 0 0 0 12 0
MatGetRowIJ 4 1.0 1.9890e-06 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatZeroEntries 100 1.0 5.6144e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatSetPreallCOO 2 1.0 1.0524e-02 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
MatSetValuesCOO 202 1.0 1.9799e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecMDot 939 1.0 1.0726e+00 9.1 6.76e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 13 0 0 0 0 13 0 0 0 9308
VecNorm 1040 1.0 1.3986e-01 5.1 1.45e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 3 0 0 0 0 3 0 0 0 15323
VecScale 1040 1.0 8.3549e-03 1.6 7.25e+07 1.2 0.0e+00 0.0e+00 0.0e+00 0 1 0 0 0 0 1 0 0 0 128249
VecCopy 101 1.0 5.3719e-03 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecSet 1141 1.0 2.6839e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecAXPY 101 1.0 2.4679e-03 1.2 1.41e+07 1.2 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 84330
VecMAXPY 1040 1.0 8.4348e-02 1.4 8.07e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 15 0 0 0 0 15 0 0 0 141302
VecScatterBegin 939 1.0 2.5078e-02 1.7 0.00e+00 0.0 9.6e+04 1.2e+04 0.0e+00 0 0 90 68 0 0 0 90 68 0 0
VecScatterEnd 939 1.0 2.1006e-02 2.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
VecNormalize 1040 1.0 1.4729e-01 4.1 2.18e+08 1.2 0.0e+00 0.0e+00 0.0e+00 0 4 0 0 0 0 4 0 0 0 21825
SFSetGraph 1 1.0 1.3316e-04 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFSetUp 1 1.0 1.8612e-03 1.0 0.00e+00 0.0 2.0e+02 2.9e+03 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFPack 939 1.0 8.9331e-03 1.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
SFUnpack 939 1.0 1.1136e-04 1.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSetUp 101 1.0 2.1342e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 0 0 0 0 0 0 0 0 0 0 0
KSPSolve 101 1.0 7.4457e+01 1.0 5.42e+09 1.2 9.6e+04 1.2e+04 0.0e+00 20 100 90 68 0 20 100 90 68 0 1061
KSPGMRESOrthog 939 1.0 1.1330e+00 5.9 1.35e+09 1.2 0.0e+00 0.0e+00 0.0e+00 0 25 0 0 0 0 25 0 0 0 17624
PCSetUp 101 1.0 2.2467e+02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00 60 0 0 0 8 60 0 0 0 25 0
PCApply 1040 1.0 7.2866e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 20 0 0 0 0 20 0 0 0 0 0
------------------------------------------------------------------------------------------------------------------------
I find the total time increase from 90.701s for 1 process to 364.47s for 16 processes. I want to ask does my experiment data make sense?