# Error in a nonlinear simulation in parallel, using domain decomposition **URL:** \ **Category:** General Discussion\ **Created:** [March 3, 2025, 7:02pm UTC](https://community.freefem.org/t/error-in-a-nonlinear-simulation-in-parallel-using-domain-decomposition/3791 "2025-03-03T19:02:08Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![clayrc](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@clayrc](https://community.freefem.org/u/clayrc)\ **Post date:** [March 3, 2025, 7:02pm UTC](https://community.freefem.org/t/error-in-a-nonlinear-simulation-in-parallel-using-domain-decomposition/3791/1 "2025-03-03T19:02:08Z")
I am solving a nonlinear unsteady hygro-thermal model with non-homogenous boundary condition using the “macro\_ddm.idp” framework. The PDE are given in [Exploring Historical Perspectives in Building Hygrothermal Models: A Comprehensive Review](https://www.mdpi.com/2075-5309/14/6/1786) eq (22) and (23) I have no problem when running sequential simulation with “Mums\_seq”, however when I try to run a parallel simulation I have an error message when constructing my matrix after a number of iteration. Depending on the mesh and the number of process the error appears more or less rapidly. I have tried with MUMPs and PETSc, the only difference, is that I have an error message with MUMPs, as with PETSc the simulation is just stuck. The solutions does seem to have diverged, but it 's seem that a some point, it is not possible to build the matrices anymore. I have no error when running the MPI code on one process, so I am assuming that the problem is from the domain decomposition. I understand that it could be harder to solve nonlinear problem with domain decomposition, but is there a way to have a more robust simulation, maybe with a specific preconditioner . I am not a specialist, so I don’t known if the information can be easily found or not. I just took my sequential code and made the changes using the tutorial and examples on the webpage of by Pierre Jolivet. Here is the message error with MUMPs Assertion failed in file src/mpi/comm/contextid.c at line 239: mask[idx] & (1U \<\< bitpos) \*\* ERROR RETURN \*\* FROM DMUMPS INFO(1)= -3 ---
**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\ **Post date:** [March 3, 2025, 7:56pm UTC](https://community.freefem.org/t/error-in-a-nonlinear-simulation-in-parallel-using-domain-decomposition/3791/2 "2025-03-03T19:56:08Z")
Could you share a minimal working example? ---
**Author:** ![clayrc](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@clayrc](https://community.freefem.org/u/clayrc)\ **Post date:** [March 4, 2025, 6:36am UTC](https://community.freefem.org/t/error-in-a-nonlinear-simulation-in-parallel-using-domain-decomposition/3791/3 "2025-03-04T06:36:06Z")
I will try to simplify the test case so that I can share a code. ---
**Author:** ![clayrc](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@clayrc](https://community.freefem.org/u/clayrc)\ **Post date:** [March 4, 2025, 2:01pm UTC](https://community.freefem.org/t/error-in-a-nonlinear-simulation-in-parallel-using-domain-decomposition/3791/4 "2025-03-04T14:01:45Z")
I may have found a solution to my problem when I was trying to simplified the problem. I noticed that for some process the solution were actually “diverging” so instead of computing the biais between two iteration at the local domain, I computed the biais in the global space, and now it seems to be ok. I don’t if there is a way to avoid the reconstruction on the full domain, here in the attached file there is a simplified version of the code (the simulation get stuck after time step n °187) [Simplified\_Test1.edp](https://community.freefem.org/uploads/short-url/6X9nZwrVXgWmFgm0W6QKHMZHXa2.edp) (13.7 KB) ---
**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\ **Post date:** [March 5, 2025, 2:27pm UTC](https://community.freefem.org/t/error-in-a-nonlinear-simulation-in-parallel-using-domain-decomposition/3791/5 "2025-03-05T14:27:14Z")
Your “simplified” code is 300+ lines of code, so I can’t give you the most accurate answer, but you definitely do not need to recompute the full solution. ---
**Author:** ![clayrc](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@clayrc](https://community.freefem.org/u/clayrc)\ **Post date:** [March 5, 2025, 5:17pm UTC](https://community.freefem.org/t/error-in-a-nonlinear-simulation-in-parallel-using-domain-decomposition/3791/6 "2025-03-05T17:17:11Z")
Here a more simplified version of the model. I kept the same form of nonlinear terms and boundary condition. With this simplified version the code does not seems to be diverging. [Simplified\_Testv2.edp](https://community.freefem.org/uploads/short-url/cl34kW9vnudkQe7sCdIQpvFHwg9.edp) (4.9 KB) ---
**Author:** ![clayrc](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@clayrc](https://community.freefem.org/u/clayrc)\ **Post date:** [March 5, 2025, 5:31pm UTC](https://community.freefem.org/t/error-in-a-nonlinear-simulation-in-parallel-using-domain-decomposition/3791/7 "2025-03-05T17:31:15Z")
I removed the reconstruction of the solutions over the all domain since it may not be necessary, and simplified all the physics. I only kept what I thought was necessary ie the nonlinear terms, as well as the non classical boundary condition, but in a way much simpler form. But since the physics has been simplified a lot, I can’t not mimic the bug that I had with the more “complex” exemple. ---
**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\ **Post date:** [March 6, 2025, 2:39pm UTC](https://community.freefem.org/t/error-in-a-nonlinear-simulation-in-parallel-using-domain-decomposition/3791/8 "2025-03-06T14:39:31Z")
Well, the computation of your stopping criterion is wrong, it should be a global value, obtained with, e.g., a `mpiAllReduce()`. ---
**Author:** ![clayrc](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@clayrc](https://community.freefem.org/u/clayrc)\ **Post date:** [March 6, 2025, 3:10pm UTC](https://community.freefem.org/t/error-in-a-nonlinear-simulation-in-parallel-using-domain-decomposition/3791/9 "2025-03-06T15:10:38Z")
So do I have to actually reconstruct the solution over the all domain at each iteration to compute the norm, or can I just compute local norms, and then take the maximal value over all the subdomain as stopping criterion since I am computing a l infinity norm. In this case, is the correct syntax mpiAllReduce(MaxLocal, MaxGlobal, mpiCommWorld, mpiMAX); in order to get the maximal value over all subdomain in “MaxGlobal”, if in MaxLocal I have the local l infinity norm ? ---
**Author:** ![clayrc](https://avatars.discourse-cdn.com/v4/letter/c/f05b48/32.png) [@clayrc](https://community.freefem.org/u/clayrc)\ **Post date:** [March 6, 2025, 3:13pm UTC](https://community.freefem.org/t/error-in-a-nonlinear-simulation-in-parallel-using-domain-decomposition/3791/10 "2025-03-06T15:13:28Z")
Do I have to compute the local norm over all the subdomain (including the overlapping region). I am not sure I really understood what is in the dof within the overlapping region. ---
**Author:** ![prj](https://avatars.discourse-cdn.com/v4/letter/p/ecae2f/32.png) [@prj](https://community.freefem.org/u/prj)\ **Post date:** [March 6, 2025, 7:56pm UTC](https://community.freefem.org/t/error-in-a-nonlinear-simulation-in-parallel-using-domain-decomposition/3791/11 "2025-03-06T19:56:36Z")
You do not need to reconstruct the global solution just to get its norm.