FluidNumerics / FluidNumerics/SELF

EC-DG / ES-Atmo GPU models crash with a GPU memory access fault on multi-rank runs (pre-existing; no MPI test coverage)

Open
#147 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Fortran
Stars
92
Forks
13
Avg merge
20h 38m
Merged PRs (30d)
7

Description

## Symptom

Running any of the EC-DG / ES-Atmo GPU tests on 2 MPI ranks aborts with a GPU memory access fault shortly after initialization:

```
$ mpirun -np 2 ./test/ec_advection_3d_entropy_conservation
...
SELF_DomainDecomposition_t.f90 : Rank 1 : n_elements = 14
SELF_DomainDecomposition_t.f90 : Rank 2 : n_elements = 13
Memory access fault by GPU node-3 (Agent handle: ...) on address 0x.... Reason: Unknown.
Program received signal SIGABRT
```

Reproduced with `ec_advection_2d_entropy_conservation`, `ec_advection_3d_entropy_conservation`, `esatmo3d_motionless`, and `esatmo3d_hydrostaticbalance` on 8×MI300X (ROCm 6.4.3, OpenMPI 5.0.8 GPU-aware, 2 ranks / 2 GPUs).

## This is pre-existing, not related to #142

Verified by same-node A/B across three images: the crash is **identical** on
1. the pre-#142 baseline (per-side-per-variable exchange, current main),
2. #142 through its third commit, and
3. #142 with the EC/ES-Atmo overlap restructure (fourth commit).

The halo exchange machinery is exercised multi-rank by the 12 passing MPI regression tests (BR gradient, vector DG divergence, advection-diffusion 2-D/3-D), so the fault is in something specific to the EC/ES-Atmo model classes under domain decomposition — plausibly the two-point flux machinery, `ECDGSurfaceContribution`, or boundary-condition mapping with a decomposed mesh. Untriaged beyond that.

## Coverage gap

These are the only GPU model families with **no MPI test coverage**: all `ec_*` and `esatmo*` ctest entries are serial, so this crash is invisible to CI. Suggest adding 2-rank variants of `ec_advection_3d_entropy_conservation` (entropy conservation is a very sharp halo-correctness check) and `esatmo3d_motionless` once the fault is fixed.

Found while extending the comm/compute overlap of #142 to these models (PR #142, fourth commit — that change is validated by the full serial suite, 134/134, and inherits the exchange machinery already validated multi-rank elsewhere).

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the fault with the listed 2-rank commands, including ec_advection_3d_entropy_conservation and esatmo3d_motionless. Read SELF_DomainDecomposition_t.f90 and trace the EC/ES-Atmo model classes, especially the two-point flux machinery, ECDGSurfaceContribution, and boundary-condition mapping. Done means the models run correctly under domain decomposition and 2-rank MPI coverage is added for the suggested tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
fortran
Domain
distributed-systems, hpc, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.