FluidNumerics / FluidNumerics/SELF
Lift polynomial-degree caps in the GPU AMR kernels (AMR3D_MAXNP=12, MORTAR3D_MAXNP=16, AMR2D_MAXNP=16)
- Dominant language
- Fortran
- Stars
- 92
- Forks
- 13
- Avg merge
- 20h 38m
- Merged PRs (30d)
- 7
Description
## Summary
The GPU kernels added by the AMR work statically size their shared/scratch memory, which caps the polynomial degree on GPU builds. Runs above the cap fail cleanly with an explanatory `error stop` (host-side guards), but they fail — and high polynomial degree is the point of a spectral element code. CPU builds have no such limits.
Current caps (N+1 = points per direction):
| Cap | Value | Kernel(s) | Location |
|---|---|---|---|
| `AMR3D_MAXNP` | **12** | 3D refinement indicator (modal transform, block-per-element, shared `tmp`/`uhat` of (N+1)³) | `src/gpu/SELF_Refinement.cpp:129`, guard at `src/gpu/SELF_RefinementIndicator_3D.f90:72` |
| `MORTAR3D_MAXNP` | 16 | 3D mortar flip (shared line buffer per (mortar, line) block) | `src/gpu/SELF_Mortar.cpp:251`, guards in `src/gpu/SELF_MappedScalar_3D.f90:525`, `src/gpu/SELF_MappedVector_3D.f90:357,473` |
| `AMR2D_MAXNP` | 16 | 2D indicator (per-thread scratch) and 2D solution transfer | `src/gpu/SELF_Refinement.cpp:6`, `src/gpu/SELF_SolutionTransfer.cpp:44`, guard at `src/gpu/SELF_DGModel2D.f90:545` |
The tightest is the 3D indicator: `AMR3D_MAXNP=12` means **degree ≤ 11 for GPU AMR in 3D**, chosen because two (N+1)³ fp64 shared arrays at N+1=12 already use ~27 KB of the ~64 KB LDS/SMEM budget. Degrees up to 15 are exercised elsewhere in the test suite, so the indicator cap binds first and is below what users may reasonably run.
## Proposed work
- **3D indicator**: restructure so shared memory holds a 2D slab, not the full cube — the modal transform is three sequential tensor contractions, so process one k-slab at a time (shared (N+1)² staging, register accumulation), or stage `tmp`/`uhat` in global scratch allocated once per adapt. Either removes the cubic shared-memory term and lifts the cap to ≥ 16 (ideally: no cap, with a global-memory fallback path above the fast-path size).
- **Mortar flip / 2D kernels**: raise to match the maximum degree supported elsewhere, or add the same global-memory fallback. These are line/plane buffers, so simply raising the constant is cheap; a fallback is more future-proof.
- Keep the host-side guards, updated to match whatever the fast path supports, and keep the failure mode an `error stop` with a clear message rather than silent corruption.
- Add a degree-≥ 12 GPU test for the 3D indicator (WILL_FAIL guard today; flips to a passing test when the cap lifts).
## Constraints
- Preserve reduction/contraction operation ordering within the indicator (its host/device consistency is asserted by the existing indicator tests).
- No per-call device allocations in hot paths — any global scratch is allocated once and reused (CLAUDE.md).
## References
- #164 (3D AMR GPU backends), #165 (device-resident 3D transfer — its new kernel should be written slab-wise from the start so it never inherits a cubic shared-memory cap)
Contributor guide
Research direction
Start with the constants and host-side guards in src/gpu/SELF_Refinement.cpp, SELF_RefinementIndicator_3D.f90, SELF_Mortar.cpp, SELF_MappedScalar_3D.f90, SELF_MappedVector_3D.f90, SELF_SolutionTransfer.cpp, and SELF_DGModel2D.f90. Run the existing indicator tests and inspect their consistency checks before choosing a slab or reusable global-scratch approach. Done means degree ≥12 GPU indicator coverage passes, guards remain clear, operation ordering is preserved, and hot paths perform no per-call device allocations.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- hpc, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100