FluidNumerics / FluidNumerics/SELF

Move MPI and GPU device initialization out of the mesh constructors (library-level init)

Open
#151 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Fortran
Stars
92
Forks
13
Avg merge
20h 38m
Merged PRs (30d)
7

Description

## Problem

MPI initialization and GPU device selection currently happen as a side effect of mesh construction: every mesh constructor calls `decomp%init()`, which runs `mpi_init` and, on GPU builds, assigns the rank's device via `hipSetDevice(mod(rankId, num_devices))` (`src/gpu/SELF_DomainDecomposition.f90`).

This creates a hidden ordering constraint for user programs: **any object that allocates device memory before the mesh is constructed lands on the wrong GPU.** Concretely, a program that does

```fortran
call interp%Init(...) ! Lagrange device matrices allocated on device 0
call mesh%StructuredMesh(...) ! rank 1 switches to device 1 here
```

allocates the interpolant's device operator matrices on device 0 for **all** ranks; every subsequent kernel on rank 1 then dereferences device-0 pointers, producing an HSA memory access fault on multi-GPU nodes (with XNACK disabled and no peer access).

This bit us in practice on PR #145: two mortar tests were written interpolant-first (copied from a serial template, where the order is invisible) and crashed on the Buildkite MI210 runner with `Memory access fault by GPU node-3 ... Reason: Unknown` on rank 1, while the identically-coded model-level test passed only because it happened to construct the mesh first. Fixed by reordering (commit 8644ac9), but the trap remains for any user program and any future test.

Today the constraint is undocumented and unenforced — every MPI test in `test/` follows mesh-first ordering by convention, and nothing diagnoses a violation.

## Proposal

Separate responsibilities so environment setup is explicit and order-independent:

1. **Library-level init/finalize** — e.g. `self_init()` / `self_finalize()` (or `SELF_Environment` module) that performs `mpi_init`, rank/size query, device count query, and `hipSetDevice` once, before any SELF object is created. Constructors (mesh, Lagrange, data classes) would then assume the environment is already established.
2. **`DomainDecomposition` keeps only decomposition** — element partitioning, `elemToRank`/`offsetElem`, halo tables — and reads the already-initialized environment instead of owning `mpi_init`/`hipSetDevice`/`mpi_finalize` (the finalize-on-mesh-free coupling has the same shape of problem in reverse: freeing the mesh before other objects tears down MPI under them; see also the guard that was needed in `Free_MappedScalar2D` for persistent halo requests).
3. **Transitional safety** — keep a lazy fallback (first SELF object to need the environment initializes it) for backward compatibility, plus a debug-build check that warns when device allocations precede device selection.

This would also compose naturally with #80 (making MPI optional): a single environment-init entry point is the obvious seam for selecting the single-process vs MPI code path at run time rather than inside mesh constructors.

## Notes

- Until this lands, the mesh-must-precede-interpolant rule is documented only in comments in the mortar tests (`test/mappedscalarmortarexchange_2d_linear*.f90`, `test/mappedvectordgdivergence_2d_mortar*.f90`); it deserves a mention in the getting-started docs at minimum.
- Public-API impact should be small if the lazy fallback is kept: existing programs continue to work, and `self_init()` becomes the recommended first call.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01BwNDGyGo9zH857vgqHW2he

Contributor guide

Open the contributing guide

Research direction

Start with src/gpu/SELF_DomainDecomposition.f90 and trace how mesh constructors trigger MPI and GPU setup. Review the mortar tests named in the issue and the Free_MappedScalar2D guard to understand lifecycle assumptions. Done means an explicit library-level environment entry point, compatible constructor behavior, and updated getting-started guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
fortran
Domain
distributed-systems, hpc
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.