FluidNumerics / FluidNumerics/SELF

Lift polynomial-degree caps in the GPU AMR kernels (AMR3D_MAXNP=12, MORTAR3D_MAXNP=16, AMR2D_MAXNP=16)

Open
#168 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Fortran
Stars
92
Forks
13
Avg merge
20h 38m
Merged PRs (30d)
7

Description

## Summary

The GPU kernels added by the AMR work statically size their shared/scratch memory, which caps the polynomial degree on GPU builds. Runs above the cap fail cleanly with an explanatory `error stop` (host-side guards), but they fail — and high polynomial degree is the point of a spectral element code. CPU builds have no such limits.

Current caps (N+1 = points per direction):

| Cap | Value | Kernel(s) | Location |
|---|---|---|---|
| `AMR3D_MAXNP` | **12** | 3D refinement indicator (modal transform, block-per-element, shared `tmp`/`uhat` of (N+1)³) | `src/gpu/SELF_Refinement.cpp:129`, guard at `src/gpu/SELF_RefinementIndicator_3D.f90:72` |
| `MORTAR3D_MAXNP` | 16 | 3D mortar flip (shared line buffer per (mortar, line) block) | `src/gpu/SELF_Mortar.cpp:251`, guards in `src/gpu/SELF_MappedScalar_3D.f90:525`, `src/gpu/SELF_MappedVector_3D.f90:357,473` |
| `AMR2D_MAXNP` | 16 | 2D indicator (per-thread scratch) and 2D solution transfer | `src/gpu/SELF_Refinement.cpp:6`, `src/gpu/SELF_SolutionTransfer.cpp:44`, guard at `src/gpu/SELF_DGModel2D.f90:545` |

The tightest is the 3D indicator: `AMR3D_MAXNP=12` means **degree ≤ 11 for GPU AMR in 3D**, chosen because two (N+1)³ fp64 shared arrays at N+1=12 already use ~27 KB of the ~64 KB LDS/SMEM budget. Degrees up to 15 are exercised elsewhere in the test suite, so the indicator cap binds first and is below what users may reasonably run.

## Proposed work

- **3D indicator**: restructure so shared memory holds a 2D slab, not the full cube — the modal transform is three sequential tensor contractions, so process one k-slab at a time (shared (N+1)² staging, register accumulation), or stage `tmp`/`uhat` in global scratch allocated once per adapt. Either removes the cubic shared-memory term and lifts the cap to ≥ 16 (ideally: no cap, with a global-memory fallback path above the fast-path size).
- **Mortar flip / 2D kernels**: raise to match the maximum degree supported elsewhere, or add the same global-memory fallback. These are line/plane buffers, so simply raising the constant is cheap; a fallback is more future-proof.
- Keep the host-side guards, updated to match whatever the fast path supports, and keep the failure mode an `error stop` with a clear message rather than silent corruption.
- Add a degree-≥ 12 GPU test for the 3D indicator (WILL_FAIL guard today; flips to a passing test when the cap lifts).

## Constraints

- Preserve reduction/contraction operation ordering within the indicator (its host/device consistency is asserted by the existing indicator tests).
- No per-call device allocations in hot paths — any global scratch is allocated once and reused (CLAUDE.md).

## References

- #164 (3D AMR GPU backends), #165 (device-resident 3D transfer — its new kernel should be written slab-wise from the start so it never inherits a cubic shared-memory cap)

Contributor guide

Open the contributing guide

Research direction

Start with the constants and host-side guards in src/gpu/SELF_Refinement.cpp, SELF_RefinementIndicator_3D.f90, SELF_Mortar.cpp, SELF_MappedScalar_3D.f90, SELF_MappedVector_3D.f90, SELF_SolutionTransfer.cpp, and SELF_DGModel2D.f90. Run the existing indicator tests and inspect their consistency checks before choosing a slab or reusable global-scratch approach. Done means degree ≥12 GPU indicator coverage passes, guards remain clear, operation ordering is preserved, and hot paths perform no per-call device allocations.

Written by the indexing model from the issue text.

Assessment

Tech stack
fortran
Domain
hpc, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.