FluidNumerics / FluidNumerics/SELF
EC-DG / ES-Atmo GPU models crash with a GPU memory access fault on multi-rank runs (pre-existing; no MPI test coverage)
- Dominant language
- Fortran
- Stars
- 92
- Forks
- 13
- Avg merge
- 20h 38m
- Merged PRs (30d)
- 7
Description
## Symptom
Running any of the EC-DG / ES-Atmo GPU tests on 2 MPI ranks aborts with a GPU memory access fault shortly after initialization:
```
$ mpirun -np 2 ./test/ec_advection_3d_entropy_conservation
...
SELF_DomainDecomposition_t.f90 : Rank 1 : n_elements = 14
SELF_DomainDecomposition_t.f90 : Rank 2 : n_elements = 13
Memory access fault by GPU node-3 (Agent handle: ...) on address 0x.... Reason: Unknown.
Program received signal SIGABRT
```
Reproduced with `ec_advection_2d_entropy_conservation`, `ec_advection_3d_entropy_conservation`, `esatmo3d_motionless`, and `esatmo3d_hydrostaticbalance` on 8×MI300X (ROCm 6.4.3, OpenMPI 5.0.8 GPU-aware, 2 ranks / 2 GPUs).
## This is pre-existing, not related to #142
Verified by same-node A/B across three images: the crash is **identical** on
1. the pre-#142 baseline (per-side-per-variable exchange, current main),
2. #142 through its third commit, and
3. #142 with the EC/ES-Atmo overlap restructure (fourth commit).
The halo exchange machinery is exercised multi-rank by the 12 passing MPI regression tests (BR gradient, vector DG divergence, advection-diffusion 2-D/3-D), so the fault is in something specific to the EC/ES-Atmo model classes under domain decomposition — plausibly the two-point flux machinery, `ECDGSurfaceContribution`, or boundary-condition mapping with a decomposed mesh. Untriaged beyond that.
## Coverage gap
These are the only GPU model families with **no MPI test coverage**: all `ec_*` and `esatmo*` ctest entries are serial, so this crash is invisible to CI. Suggest adding 2-rank variants of `ec_advection_3d_entropy_conservation` (entropy conservation is a very sharp halo-correctness check) and `esatmo3d_motionless` once the fault is fixed.
Found while extending the comm/compute overlap of #142 to these models (PR #142, fourth commit — that change is validated by the full serial suite, 134/134, and inherits the exchange machinery already validated multi-rank elsewhere).
Contributor guide
Research direction
Start by reproducing the fault with the listed 2-rank commands, including ec_advection_3d_entropy_conservation and esatmo3d_motionless. Read SELF_DomainDecomposition_t.f90 and trace the EC/ES-Atmo model classes, especially the two-point flux machinery, ECDGSurfaceContribution, and boundary-condition mapping. Done means the models run correctly under domain decomposition and 2-rank MPI coverage is added for the suggested tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- distributed-systems, hpc, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100