perf(KERNEL-GEMM-BF16): cuBLASLt algo stability is asserted per-process and never measured across processes
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: KERNEL-GEMM-BF16
What
src/vt/cuda/gemm_plan_cache.h and src/vt/cuda/fp8_plan_cache.h both cache the
cublasLtMatmulAlgoGetHeuristic result per call key, in-process, and both justify the
cache's bit-exactness with the same premise:
cuBLASLt algo selection is process-deterministic (the same shape selects the same algo
per the algo-latching forensic record; see gemm_algo_log.h / .agents/state.md)
—src/vt/cuda/fp8_plan_cache.h:17
That premise is per-process and it is exactly strong enough for what the caches claim:
within one process, caching changes when the heuristic runs, never what it returns.
Nothing in the tree states or measures the cross-process question: does a fresh boot on
the same device, driver and shapes select the same algo? No cache persists that selection,
so every boot re-resolves it.
Why it matters
We already proved the analogous confounder is real one lane over. On the NVFP4 CUTLASS lane,
re-tuning in every process made paired direct/fallback runs share only 18–33 of 64 selected
tactic IDs, and 5 of 6 paired 128-token output hashes differed
(.agents/specs/nvfp4-persistent-plan-cache.md). Freezing the plan map took it to 64/64
equal IDs with zero tuning misses. Variable CUTLASS reduction order was a real confounder in
every A/B on that lane until it was removed.
If cuBLASLt selection is not cross-process stable, the bf16/f32 lane carries that same
uncontrolled variable in every same-binary A/B we have run on it, and it would not show up as
an error — it would show up as spread.
What would close this
A measurement, not a feature:
- Dump the selected algo id per call key across N fresh processes on one leased device, same
shapes, same driver (gemm_algo_log.halready exists for the logging half). - Compare the selections.
- If stable — close this with that evidence, and correct the premise's citation (see below).
Persisting buys nothing and should not be built. - If unstable — it becomes a benchmark-validity repair on
KERNEL-GEMM-BF16, and the
persistent-cache machinery insrc/vt/cuda/nvfp4_persistent_cache.cppis the shape to reuse.
Explicitly not "go persist the cuBLASLt heuristic". That is only warranted in case 4.
Two stale records to repair in the same flow
Found while reading these headers; both are one-line record defects on this row, fixed here
rather than deferred:
src/vt/cuda/fp8_plan_cache.h:18-19cites.agents/state.mdas the forensic record for the
determinism premise. That file does not exist — the protocol retired the state log
("The project has no state log"). The premise's only cited evidence is unreachable.src/vt/cuda/gemm_plan_cache.h:29says the fp8 cache "ships off". It has been default ON
since #1843, which the fp8 header itself records atfp8_plan_cache.h:23-24. The comment is
stale in the opposite direction from the code.
Evidence
src/vt/cuda/gemm_plan_cache.h:1-35(premise, default-ON polarity, #1732 capture bug)src/vt/cuda/fp8_plan_cache.h:10-31(premise + citation + #1843 flip).agents/specs/nvfp4-persistent-plan-cache.md§ "W3-C3 corrected frozen-plan component result"
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/vt/cuda/gemm_plan_cache.h, src/vt/cuda/fp8_plan_cache.h, and gemm_algo_log.h; inspect how selected algorithm IDs are logged. Run the same shapes in N fresh processes on one device and compare the selections. Done means the cross-process stability is evidenced or the benchmark-validity repair is scoped, with the two stale header records corrected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, performance, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100