trixi-framework / trixi-framework/Trixi.jl

Investigate (and fix) excessive memory use for parallel simulations

Open
#1,353 2 comments 0 reactions 0 assignees View on GitHub
bug enhancement parallelization
Dominant language
Julia
Stars
731
Forks
166
Avg merge
2d 18h
Merged PRs (30d)
25

Description

At the moment it seems like we have considerable issues with non-parallelized memory usage when running massively parallel simulations. What do I mean by that?

When doing a weak scaling, i.e, a parallel scaling experiment where the problem size is *fixed per rank*, we always seem to reach a point where we run out of memory. For example, I am able to scale with a problem size of 1024 elements/rank, 128 ranks/node, to up to 16 nodes on Hawk. When going beyond that (e.g., 64 nodes), the jobs fail with an OOM error.

Maybe something else is at play here, but my first suspect would be that we somehow allocate memory that is of size O(#ranks) or O(nelements_global), and at some point this becomes just too much for the memory per node.

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests. Start by reproducing the weak-scaling experiment with 1024 elements per rank on Hawk, compare memory use through 16 and 64 nodes, and trace allocations to determine whether rank- or global-element-sized state causes the OOM; done means the cause is identified and the 64-node run no longer fails from excessive memory use.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
distributed-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.