# Fatal error with PMPI\_WaitAny **URL:** \ **Category:** General Discussion\ **Created:** [December 16, 2021, 3:15pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395 "2021-12-16T15:15:31Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![zcwang6](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.freefem.org/zcwang6/32/371_2.png) [@zcwang6](https://community.freefem.org/u/zcwang6)\ **Post date:** [December 16, 2021, 3:15pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/1 "2021-12-16T15:15:31Z")
Dear Prj, currently I saw a strange error with my FF code, when I run it on my school cluster; " _Fatal error in PMPI\_Waitall: Other MPI error, error stack:_ " It happened randomly kind of. I believe this problem only happened when the code is running cross multiple machines [nodes]. I am not sure if this problem has anything related with my freefem code, but I still hope you can provide me some cues about it. Thank you very much ---
**Author:** ![zcwang6](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.freefem.org/zcwang6/32/371_2.png) [@zcwang6](https://community.freefem.org/u/zcwang6)\ **Post date:** [December 16, 2021, 6:30pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/2 "2021-12-16T18:30:37Z")
“Abort(807022991) on node 24 (rank 24 in comm 0): Fatal error in PMPI\_Waitany: Other MPI error, error stack: PMPI\_Waitany(295)…: MPI\_Waitany(count=122, req\_array=0x2d7320d0, index=0x7fffffff972c, status=0x7fffffff9860) failed PMPI\_Waitany(263)…: MPIDI\_Progress\_test\_impl(195).: MPIDI\_OFI\_handle\_cq\_error(991): OFI poll failed (ofi\_events.h:991:MPIDI\_OFI\_handle\_cq\_error:Input/output error)” more information: the cluster is Cray cs400 & using infiniband for comunication; I am using intel/mpi ---
**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\ **Post date:** [December 16, 2021, 7:31pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/3 "2021-12-16T19:31:06Z")
Could you please share a reproducer? ---
**Author:** ![zcwang6](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.freefem.org/zcwang6/32/371_2.png) [@zcwang6](https://community.freefem.org/u/zcwang6)\ **Post date:** [December 16, 2021, 8:03pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/4 "2021-12-16T20:03:03Z")
> [@prj](#): > > reproducer which reproducer you want, the code or the error generated file? since I didn’t run “export I\_MPI\_DEBUG=6” the error message is like above. I am currently trying to run my code again and with the debug flag ---
**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\ **Post date:** [December 16, 2021, 8:08pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/5 "2021-12-16T20:08:05Z")
A code that I could use to try to reproduce this, to see whether the problem comes from your installation or from within FreeFEM. ---
**Author:** ![zcwang6](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.freefem.org/zcwang6/32/371_2.png) [@zcwang6](https://community.freefem.org/u/zcwang6)\ **Post date:** [December 16, 2021, 8:13pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/6 "2021-12-16T20:13:08Z")
sorry Prj, according to the policy I can not distribute the code without permission. Let me talk with my professor to see if I can simply the code and send it to you. ---
**Author:** ![zcwang6](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.freefem.org/zcwang6/32/371_2.png) [@zcwang6](https://community.freefem.org/u/zcwang6)\ **Post date:** [December 16, 2021, 8:14pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/7 "2021-12-16T20:14:48Z")
One more question, do you think if I change the intel/mpi to open mpi this issue maybe go away? ---
**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\ **Post date:** [December 16, 2021, 8:26pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/8 "2021-12-16T20:26:04Z")
Maybe, can’t tell you for sure. ---
**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\ **Post date:** [December 17, 2021, 8:28am UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/9 "2021-12-17T08:28:04Z")
Forgot to mention that I actually don’t want the full code, just the piece that exhibit the error. So if it’s OK for you to just send this piece, that’s all I need. ---
**Author:** ![zcwang6](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.freefem.org/zcwang6/32/371_2.png) [@zcwang6](https://community.freefem.org/u/zcwang6)\ **Post date:** [December 22, 2021, 3:50pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/10 "2021-12-22T15:50:28Z")
sorry for waiting so long I will send you that portion ---
**Author:** ![zcwang6](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.freefem.org/zcwang6/32/371_2.png) [@zcwang6](https://community.freefem.org/u/zcwang6)\ **Post date:** [December 22, 2021, 3:56pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/11 "2021-12-22T15:56:15Z")
[FREE\_FATAL\_ERROR\_DEBUG.edp](https://community.freefem.org/uploads/short-url/bXyzn7DKdpqKMjlVzVHjhD7NgZD.edp) (36.3 KB) here is the code, the error comes from the Adjoint equation ---
**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\ **Post date:** [December 22, 2021, 4:06pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/12 "2021-12-22T16:06:56Z")
Does the error occur on smaller meshes too? Debugging on such a large problem will probably be tricky… ---
**Author:** ![zcwang6](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.freefem.org/zcwang6/32/371_2.png) [@zcwang6](https://community.freefem.org/u/zcwang6)\ **Post date:** [December 22, 2021, 4:23pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/13 "2021-12-22T16:23:10Z")
I am not sure, I never saw this problem with small meshes; maybe it is because I just run it with single machine. This problem is happened kind of randomly. I doubt it is because of the intel mpi or the net work of my school’s cluster. ---
**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\ **Post date:** [December 22, 2021, 8:21pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/14 "2021-12-22T20:21:23Z")
I ran your code on 256 processes. It was OK up until line 763 ```auto [...] +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Max of meshsize: is 1.820027e-01 Min of meshsize: is 4.936972e-02 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ R1 = 1.250000e+00, R2 = 5.000000e-01, tref = 2.000000e-01, dext = 1.500000e+00, 2*hmin = 9.873944e-02 betamax = 7.216878e+00, beta = 1.000000e-01 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1/0 : d d d current line = 763 mpirank 24 / 256 [...] ``` Are you at least reaching this point of the script, or do you have an error before that? ---
**Author:** ![zcwang6](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.freefem.org/zcwang6/32/371_2.png) [@zcwang6](https://community.freefem.org/u/zcwang6)\ **Post date:** [December 22, 2021, 10:38pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/15 "2021-12-22T22:38:57Z")
No, but I think I know what is happening here, I should solve elasticity equation first, and then run this adjoint, otherwise the inpu [FREE\_FATAL\_ERROR\_DEBUG\_correct.edp](https://community.freefem.org/uploads/short-url/nmBgbYrPAHgE1pIKJtgk43qjVZH.edp) (45.0 KB) t is wrong ,sorry for this. let me send you another one; and at line 67 “hmincsd = .1” you can change it to “hmincsd = .2 or 0.25 or 0.5” to make the mesh smaller. ---
**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\ **Post date:** [December 23, 2021, 7:43am UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/16 "2021-12-23T07:43:25Z")
Here is what I get. ```auto +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Max of meshsize: is 1.825596e-01 Min of meshsize: is 4.936972e-02 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ R1 = 9.000000e-01, R2 = 5.000000e-01, tref = 2.000000e-01, dext = 1.080000e+00, 2*hmin = 9.873944e-02 betamax = 5.196152e+00, beta = 1.000000e-01 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Volume of nonpaded design domain is = 3.243770e+02 KSP final norm of residual 0.0111526 KSP final norm of residual 0.000112754 Solved Adjoint ``` The code ends fine. ---
**Author:** ![zcwang6](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.freefem.org/zcwang6/32/371_2.png) [@zcwang6](https://community.freefem.org/u/zcwang6)\ **Post date:** [December 23, 2021, 1:34pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/17 "2021-12-23T13:34:44Z")
Yes, I do get the same thing; The problem is like this: in my code I have a loop to do this process; when I am running this code in my school’s cluster, sometimes at some iteration for example 738 or 1141; it is terminated with that issue at that adjoint equation. Then I will rerun this code, this issue is happened randomly. But thank you for the help Prj; I am trying to reinstall the FF with openmpi; but the avil module of my cluster is " openmpi/gcc/64/3.1.6 " , seems too old for FF. at least currently, the intel mpi is the only one I can compile FF over there. ---
**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\ **Post date:** [December 23, 2021, 3:17pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/18 "2021-12-23T15:17:45Z")
In that case, what you could do is the following. At each iteration, save the state of your script before partitioning. Each iteration, overwrite what was previously written. Then, if the crash occurs at iteration XYZ, we could just restart from there and troubleshoot the issue. ---
**Author:** ![zcwang6](https://yyz2.discourse-cdn.com/flex030/user_avatar/community.freefem.org/zcwang6/32/371_2.png) [@zcwang6](https://community.freefem.org/u/zcwang6)\ **Post date:** [December 23, 2021, 5:31pm UTC](https://community.freefem.org/t/fatal-error-with-pmpi-waitany/1395/19 "2021-12-23T17:31:55Z")
Yes that’s what I am going to do in the next. Thank you I update this with you.