I am conducting a parameter study in which I call PEPSolve about a thousand times inside a double for loop. Oddly, as a computation proceeds, more and more RAM is used by the program, which is surprising as everything is scoped due to the for loop. Since apart from PEPSolve, several additional operations take place inside the loop, the code is structured using multiple macros and funcs. Although my 2D mesh is small (~10-20k triangles), due to the large number of calculations, the memory increase is substantial, and I would like to get rid of this behavior. I managed to reproduce this behavior with the attached code, which mimics the for loop + macro structure and also produces a slight increase in memory as a function of time.
How can I proceed with the investigation of the behavior? Can this happen due to incorrect use of func/macro usage (e.g., not every variable used inside a func/macro is an input)? Should one avoid defining variables inside the loop, and it is better to define them outside and just resize them inside the loop? Or maybe there are some pointers that are not deleted in the PETSc/SLEPc interface? Any help is appreciated. Blasius-stability-1d-PEP-SLEPc-complex-memcheck.edp (6.3 KB)
Thanks for the tip. I checked out Valgrind memcheck and also Valgrind/Massif, but the output does not seem helpful for me since the FreeFem code is not referenced in the output (I assume that’s why there are a lot of ???-s in the file). I attach the outputs from the MWE above I sent you. Do you have a suggestion how to proceed?
The Massif log does not report any substantial memory consumption increase. I’m running the script locally, now for 10 minutes, with no memory increase whatsoever. What tells you that there is such an increase?
Thanks for checking it out. In htop, I see that the memory consumption increases continuously. In my other scripts (which are 2D problems, not 1D, and a larger parameter range is swept, but the code is also long), this is so severe that the program runs out of memory eventually and stops prematurely.
I will check my large problem, run for a longer time, and see whether something is different.
I ran my other script. Massif output does not seem to detect a lot of memory consumption. However, htop tells a different story: as you can see in the screenshot, initially the memory consumption is ~2.5 GB, and it almost triples later. When I want to sweep a larger parameter space with my script, I can even run out of swap ram.
I suppose I can just move one of the for loops from the double for loop into a bash script, and call FreeFem multiple times, but this I find this behavior extremely frustrating. Do you have any further suggestions what I can do to further investigate this issue?
Yes I think it is possible, but then the code would look really messy, as there are actually multiple macros inside the macro loopMacro. Would it be better to use func instead of macro to structure the code, or should I just try putting the raw code inside the loop?
I think it would be worth not using either macro or func. There is no leak from the PETSc side of things, so I’m guessing it’s coming from FreeFEM, but I’m not really sure why.
Thanks for the tips. Unfortunately, neither not running as root nor running the script without the macros solved the memory increase in the case of the problem I am studying.
However, I investigated the MWE I sent above further. I found that with the option -v 1, FreeFem reports the memory leak like:
times: compile 1.031710e-01s, execution 5.105332e+02s, mpirank:0
######## We forget of deleting 15 Nb pointer, 0Bytes , mpirank 1, memory leak =17892208
CodeAlloc : nb ptr 15025, size :914768 mpirank: 1
######## We forget of deleting 15 Nb pointer, 0Bytes , mpirank 0, memory leak =17964320
CodeAlloc : nb ptr 15025, size :914768 mpirank: 0
Ok: Normal End
I ran the MWE for several for loop numbers, and I found the total loss (sum of ranks 0 and 1) is proportional to the number of for loops; see the figure below. This tells me that the example I send is indeed an MWE, and something nasty happens inside the for loop (although I misinterpret the results as my understanding of things is rather shallow). I checked, and the same behavior is observed regardless I use a macro inside the for loop, or I just copy the piece of code inside the for loop.
Do you have any further suggestions the issue can be investigated further?
The FreeFEM memory counter is unreliable. Do not trust it, only trust Valgrind and Massif, and PETSc -log_view (where there must be the same numbers of creations and destructions).
I see. Well, neither Massif, Valgrind, nor PETSc reports any strange behavior. Although not being reliable, the output of the FreeFem memory counter seems to correlate with the memory consumption I see in htop.
Nevertheless, the issue persists: more and more memory is consumed by FreeFem as my calculations proceed. The example I send above also demonstrates this behavior. I checked, and the lines Apep[2] = A2mat(VhC,VhC,tgv=-1); do not alter the memory increase. Is there any way I could investigate the issue further, maybe even try to resolve it with moderate effort? Or is this just the way FreeFem operates at the moment, and it requires cumbersome investigation of the FreeFem source code to find and sort out the unreasonable memory increase? If the case is the latter, I will move the outer for loop into the bash script and call FreeFem multiple times, although I do not like that solution.
Yes, I could reproduce the same behavior on different machines too. In each case, I used openmpi installed with apt-get install libopenmpi-dev . Based on your suggestion, I removed openmpi, and installed mpich also with apt-get, then recompiled both PETSc-SLEPc and FreeFem. This solved the memory leakage. Furthermore, now that I use mpich instead of openmpi, the programs do not start sleeping processes (these I mentioned in this post) . Moreover, I think my programs also run faster!
Although your suggestion above significantly decreased the memory consumption of my code, it still seems to be unreasonably high. I have an additional question regarding this topic. Is it possible that a VecRestoreArray call is missing in SLEPc-code.hpp (considering this code segment)?
Although as I explained before, I did not receive any strangeness using Valgrid/Massif, nor do I understand the precise need for VecRestoreArray, not to mention which pointers need to be deleted or can cause memory leaks, but the if-else structure tells me that something might be off in the code I referenced above.
Together with PEPSolve(), right? Then the code path reaches FreeFem-sources/SLEPc-code.hpp at develop · FreeFem/FreeFem-sources · GitHub and there is no call to the getter. There shouldn’t be a call to the restorer, so that’s indeed a bug in the logic, but I’m not sure this will fix a memory leakage. Could you try this patch, please?
