FluidNumerics / FluidNumerics/SELF
AMR under MPI diverges to NaN on CUDA sm_103; fixed-mesh MPI and single-rank AMR are both clean
- Dominant language
- Fortran
- Stars
- 92
- Forks
- 13
- Avg merge
- 20h 38m
- Merged PRs (30d)
- 7
Description
## Summary
`AMRController2D` driving a `LinearEuler2D` model diverges to overflow and then NaN when run on more than one MPI rank, in a CUDA `sm_103` (Blackwell Ultra, B300) double-precision build. The same model on a **fixed mesh** reproduces its single-rank result to round-off at 1, 2, 4 and 8 ranks, and the **same adaptive model on a single rank** is clean — so neither the MPI decomposition nor the AMR path alone is at fault. Only the combination fails.
All receivers go bad simultaneously, and the onset is deterministic across rank counts.
## Environment
- SELF `8b14c8abe7768a0426b5130b4ca1943521489007` (main, 2026-08-12), from the published `latest-x86-cuda130-sm103` container image
- CUDA 13.0, `sm_103`, `-DDOUBLE_PRECISION`
- Open MPI 5.0.8, `mpi_built_with_cuda_support: true`, 8x NVIDIA B300 SXM6 on one node, one rank per GPU
- `mpirun -n N --host localhost:N` inside the container (not `srun --mpi=pmix`)
## Configuration
`LinearEuler2D` on a structured square domain, side 0.94 m, degree 7, base mesh 256x256 elements, radiation BCs on all four sides, homogeneous medium (c = 1500 m/s, rho0 = 1000). A regularized point source with a ~2.35 MHz wavelet, 15 point receivers on a circle of radius 0.35 m. 15687 time steps, 120 us, 1350 output samples.
`AMRController2D%Init` with `refineThreshold = -3.0`, `coarsenThreshold = -8.0`, `ivar = 3` (pressure), `maxLevel = 1`, `nHalo = 2`; `Adapt` called every 640 steps.
## Observed
Peak absolute receiver pressure over the whole record, and the output sample at which the first non-finite value appears:
| ranks | result |
|---|---|
| 1 | clean, max abs 3.189094e-05, no NaN |
| 2 | 10308 of 20250 samples NaN, max finite 2.409294e+241, onset sample 656 of 1350 |
| 4 | 10301 NaN, max finite 1.502060e+292, onset 655 |
| 8 | 10242 NaN, max finite 7.641190e+266, onset 664 |
All 15 receivers are affected, starting at the same sample. A separate 4-rank run aborted hard with exit code 205 and no diagnostic output.
Reaching 1e250 by the time it becomes visible suggests an unstable mode present from early in the run and growing exponentially, only surfacing in the traces once it exceeds the ~1e-5 physical signal — rather than something that goes wrong at the halfway point.
## Controls
**1. Fixed mesh, same size, same ranks — passes.** With AMR disabled (uniform 256x256), the traces are identical to all printed digits at every rank count:
```
ranks 1/2/4/8: max|trace| = 1.797050e-09 (bitwise-identical across rank counts)
wall: 87.67 -> 46.86 -> 26.67 -> 16.26 s (5.4x at 8 ranks)
```
So domain decomposition, receiver ownership and the trace reduction are all fine.
**2. Amplitude gate disabled — still diverges, identically.** #163 added the relative energy floor, which was the first thing I suspected since it performs a global reduction for the energy scale. Setting `relativeEnergyFloor = 0` changes nothing:
| ranks | gate 1e-7 | gate 0 (disabled) |
|---|---|---|
| 1 | clean, 3.189094e-05 | clean, 2.463723e-05 |
| 2 | NaN from 656, peak 2.4e+241 | NaN from 656, peak 1.8e+250 |
Same onset sample with and without the gate, so #163 is not implicated.
## What I have not established
- **Whether this is arch-specific.** An earlier measurement of this same AMR-under-MPI path on `gfx942`/ROCm (at `17276de7`, i.e. before #163) was well behaved: the adaptive run differed from the single-rank run by 7.5e-09, finite and at discretization level, which is the expected behaviour given refinement is a threshold decision on a solution that already differs at round-off. I have **not** re-run that on `gfx942` at `8b14c8ab`, so I cannot separate "CUDA/sm_103 specific" from "regressed since 17276de7". Happy to run whichever of those two you would find more useful.
- Whether `maxLevel >= 2` or other thresholds change the picture; I reproduced at `maxLevel = 1` because it is the cheapest configuration that still produces nonconforming interfaces and solution migration.
## Note on the existing test
`test/` covers the adaptive path across rank counts and passes, so whatever it exercises must be too small or too short to reach this. Worth extending alongside a fix rather than only patching the symptom.
One testing gotcha that cost me a false PASS, in case it is useful for that test: a NaN makes every ordinary comparison `False`, so `if rel > tol: fail` silently passes a fully diverged run. Checking `isfinite` first, as its own failure mode, is what surfaced this.
I can test patches on the B300 hardware quickly.
Contributor guide
Research direction
Start with the adaptive-path coverage under test/ and reproduce the reported AMRController2D%Init and Adapt configuration across MPI ranks on the B300 setup. Compare the failing adaptive run with fixed-mesh and single-rank controls, then trace the AMR migration and nonconforming-interface behavior. Done means the multi-rank run remains finite and the test explicitly fails on non-finite results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- distributed-systems, hpc
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100