FluidNumerics / FluidNumerics/SELF

Lazily allocate optional GPU work arrays (jas_gpu, interpWork) — ~75% of scalar-field memory is unused by gradient-free models

Open
#143 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Fortran
Stars
92
Forks
13
Avg merge
20h 38m
Merged PRs (30d)
7

Description

## Problem

Every `MappedScalar3D%Init` (GPU backend) unconditionally allocates:

- `jas_gpu`: `(N+1)^3 * nelem * nvar * 9` — the Jacobian-weighted contravariant workspace, used only by `MappedGradient`/`MappedDGGradient`
- `interpWork1` / `interpWork2`: grid-interpolation staging sized with `M+1`, used only by `GridInterp`-type operations
- `avgBoundary` (and its device copy), used only when side averages are needed

For a model that never computes a solution gradient (e.g. LinearEuler2D/3D with `gradient_enabled = .false.`), `jas_gpu` alone is 9× the interior field per scalar object, and `DGModel3D` holds five `MappedScalar3D` objects (`solution`, `workSol`, `dSdt`, `source`, `fluxDivergence`) — none of which take gradients except `solution`.

## Measured impact (8×MI300X benchmarking for #84 / PR #142)

LinearEuler3D at degree 7, fp64 measures **~2.4 MB per element** of device memory. Roughly 75% of each scalar field's footprint is `jas_gpu` + interp work arrays. Practical consequences on 192 GB GPUs:

- 48³ elements OOMs on a single MI300X (should need ~35 GB of live state)
- 64³ requires all 8 GPUs; 80³ doesn't fit on the node at all
- 96³ and 184³ (the sizes we wanted to benchmark) are unreachable at degree 7 fp64

Lazy allocation (on first use, or gated on `gradient_enabled` / an `Init` flag) would roughly **3× the reachable problem size per GPU** for hyperbolic-only models, with no change to models that do use gradients.

## Suggested approach

- Allocate `jas_gpu`, `interpWork1/2` on first use inside `MappedGradient`/`MappedDGGradient`/`GridInterp` (one-time `hipMalloc`, kept for the object lifetime), or
- Add an optional `Init` argument / model-level flag so `DGModel3D` only requests the gradient workspace for `solution` when `gradient_enabled`.

Same applies to the 2-D classes and `MappedVector` types.

Contributor guide

Open the contributing guide

Research direction

Start by locating MappedScalar3D%Init and the MappedGradient, MappedDGGradient, and GridInterp entry points. Trace the corresponding 2-D classes and MappedVector types to identify each workspace's first use. Done means optional GPU arrays are not allocated for unused operations while gradient and interpolation users retain the required workspaces; verify the affected GPU memory behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
fortran
Domain
hpc, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.