FluidNumerics / FluidNumerics/SELF
Lazily allocate optional GPU work arrays (jas_gpu, interpWork) — ~75% of scalar-field memory is unused by gradient-free models
- Dominant language
- Fortran
- Stars
- 92
- Forks
- 13
- Avg merge
- 20h 38m
- Merged PRs (30d)
- 7
Description
## Problem
Every `MappedScalar3D%Init` (GPU backend) unconditionally allocates:
- `jas_gpu`: `(N+1)^3 * nelem * nvar * 9` — the Jacobian-weighted contravariant workspace, used only by `MappedGradient`/`MappedDGGradient`
- `interpWork1` / `interpWork2`: grid-interpolation staging sized with `M+1`, used only by `GridInterp`-type operations
- `avgBoundary` (and its device copy), used only when side averages are needed
For a model that never computes a solution gradient (e.g. LinearEuler2D/3D with `gradient_enabled = .false.`), `jas_gpu` alone is 9× the interior field per scalar object, and `DGModel3D` holds five `MappedScalar3D` objects (`solution`, `workSol`, `dSdt`, `source`, `fluxDivergence`) — none of which take gradients except `solution`.
## Measured impact (8×MI300X benchmarking for #84 / PR #142)
LinearEuler3D at degree 7, fp64 measures **~2.4 MB per element** of device memory. Roughly 75% of each scalar field's footprint is `jas_gpu` + interp work arrays. Practical consequences on 192 GB GPUs:
- 48³ elements OOMs on a single MI300X (should need ~35 GB of live state)
- 64³ requires all 8 GPUs; 80³ doesn't fit on the node at all
- 96³ and 184³ (the sizes we wanted to benchmark) are unreachable at degree 7 fp64
Lazy allocation (on first use, or gated on `gradient_enabled` / an `Init` flag) would roughly **3× the reachable problem size per GPU** for hyperbolic-only models, with no change to models that do use gradients.
## Suggested approach
- Allocate `jas_gpu`, `interpWork1/2` on first use inside `MappedGradient`/`MappedDGGradient`/`GridInterp` (one-time `hipMalloc`, kept for the object lifetime), or
- Add an optional `Init` argument / model-level flag so `DGModel3D` only requests the gradient workspace for `solution` when `gradient_enabled`.
Same applies to the 2-D classes and `MappedVector` types.
Contributor guide
Research direction
Start by locating MappedScalar3D%Init and the MappedGradient, MappedDGGradient, and GridInterp entry points. Trace the corresponding 2-D classes and MappedVector types to identify each workspace's first use. Done means optional GPU arrays are not allocated for unused operations while gradient and interpolation users retain the required workspaces; verify the affected GPU memory behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fortran
- Domain
- hpc, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100