FluidNumerics / FluidNumerics/SELF
Move MPI and GPU device initialization out of the mesh constructors (library-level init)
- Dominant language
- Fortran
- Stars
- 92
- Forks
- 13
- Avg merge
- 20h 38m
- Merged PRs (30d)
- 7
Description
## Problem
MPI initialization and GPU device selection currently happen as a side effect of mesh construction: every mesh constructor calls `decomp%init()`, which runs `mpi_init` and, on GPU builds, assigns the rank's device via `hipSetDevice(mod(rankId, num_devices))` (`src/gpu/SELF_DomainDecomposition.f90`).
This creates a hidden ordering constraint for user programs: **any object that allocates device memory before the mesh is constructed lands on the wrong GPU.** Concretely, a program that does
```fortran
call interp%Init(...) ! Lagrange device matrices allocated on device 0
call mesh%StructuredMesh(...) ! rank 1 switches to device 1 here
```
allocates the interpolant's device operator matrices on device 0 for **all** ranks; every subsequent kernel on rank 1 then dereferences device-0 pointers, producing an HSA memory access fault on multi-GPU nodes (with XNACK disabled and no peer access).
This bit us in practice on PR #145: two mortar tests were written interpolant-first (copied from a serial template, where the order is invisible) and crashed on the Buildkite MI210 runner with `Memory access fault by GPU node-3 ... Reason: Unknown` on rank 1, while the identically-coded model-level test passed only because it happened to construct the mesh first. Fixed by reordering (commit 8644ac9), but the trap remains for any user program and any future test.
Today the constraint is undocumented and unenforced — every MPI test in `test/` follows mesh-first ordering by convention, and nothing diagnoses a violation.
## Proposal
Separate responsibilities so environment setup is explicit and order-independent:
1. **Library-level init/finalize** — e.g. `self_init()` / `self_finalize()` (or `SELF_Environment` module) that performs `mpi_init`, rank/size query, device count query, and `hipSetDevice` once, before any SELF object is created. Constructors (mesh, Lagrange, data classes) would then assume the environment is already established.
2. **`DomainDecomposition` keeps only decomposition** — element partitioning, `elemToRank`/`offsetElem`, halo tables — and reads the already-initialized environment instead of owning `mpi_init`/`hipSetDevice`/`mpi_finalize` (the finalize-on-mesh-free coupling has the same shape of problem in reverse: freeing the mesh before other objects tears down MPI under them; see also the guard that was needed in `Free_MappedScalar2D` for persistent halo requests).
3. **Transitional safety** — keep a lazy fallback (first SELF object to need the environment initializes it) for backward compatibility, plus a debug-build check that warns when device allocations precede device selection.
This would also compose naturally with #80 (making MPI optional): a single environment-init entry point is the obvious seam for selecting the single-process vs MPI code path at run time rather than inside mesh constructors.
## Notes
- Until this lands, the mesh-must-precede-interpolant rule is documented only in comments in the mortar tests (`test/mappedscalarmortarexchange_2d_linear*.f90`, `test/mappedvectordgdivergence_2d_mortar*.f90`); it deserves a mention in the getting-started docs at minimum.
- Public-API impact should be small if the lazy fallback is kept: existing programs continue to work, and `self_init()` becomes the recommended first call.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
https://claude.ai/code/session_01BwNDGyGo9zH857vgqHW2he
Contributor guide
Research direction
Start with src/gpu/SELF_DomainDecomposition.f90 and trace how mesh constructors trigger MPI and GPU setup. Review the mortar tests named in the issue and the Free_MappedScalar2D guard to understand lifecycle assumptions. Done means an explicit library-level environment entry point, compatible constructor behavior, and updated getting-started guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- distributed-systems, hpc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100