trixi-framework / trixi-framework/Trixi.jl
Investigate (and fix) excessive memory use for parallel simulations
- Dominant language
- Julia
- Stars
- 731
- Forks
- 166
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 25
Description
At the moment it seems like we have considerable issues with non-parallelized memory usage when running massively parallel simulations. What do I mean by that?
When doing a weak scaling, i.e, a parallel scaling experiment where the problem size is *fixed per rank*, we always seem to reach a point where we run out of memory. For example, I am able to scale with a problem size of 1024 elements/rank, 128 ranks/node, to up to 16 nodes on Hawk. When going beyond that (e.g., 64 nodes), the jobs fail with an OOM error.
Maybe something else is at play here, but my first suspect would be that we somehow allocate memory that is of size O(#ranks) or O(nelements_global), and at some point this becomes just too much for the memory per node.
Contributor guide
Research direction
The issue names no files or tests. Start by reproducing the weak-scaling experiment with 1024 elements per rank on Hawk, compare memory use through 16 and 64 nodes, and trace allocations to determine whether rank- or global-element-sized state causes the OOM; done means the cause is identified and the 64-node run no longer fails from excessive memory use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100