mudler / mudler/vllm.cpp

perf(KERNEL-GEMM-BF16): cuBLASLt algo stability is asserted per-process and never measured across processes

Open
#2,750 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: KERNEL-GEMM-BF16

What

src/vt/cuda/gemm_plan_cache.h and src/vt/cuda/fp8_plan_cache.h both cache the
cublasLtMatmulAlgoGetHeuristic result per call key, in-process, and both justify the
cache's bit-exactness with the same premise:

cuBLASLt algo selection is process-deterministic (the same shape selects the same algo
per the algo-latching forensic record; see gemm_algo_log.h / .agents/state.md)
src/vt/cuda/fp8_plan_cache.h:17

That premise is per-process and it is exactly strong enough for what the caches claim:
within one process, caching changes when the heuristic runs, never what it returns.

Nothing in the tree states or measures the cross-process question: does a fresh boot on
the same device, driver and shapes select the same algo? No cache persists that selection,
so every boot re-resolves it.

Why it matters

We already proved the analogous confounder is real one lane over. On the NVFP4 CUTLASS lane,
re-tuning in every process made paired direct/fallback runs share only 18–33 of 64 selected
tactic IDs, and 5 of 6 paired 128-token output hashes differed
(.agents/specs/nvfp4-persistent-plan-cache.md). Freezing the plan map took it to 64/64
equal IDs with zero tuning misses. Variable CUTLASS reduction order was a real confounder in
every A/B on that lane until it was removed.

If cuBLASLt selection is not cross-process stable, the bf16/f32 lane carries that same
uncontrolled variable in every same-binary A/B we have run on it, and it would not show up as
an error — it would show up as spread.

What would close this

A measurement, not a feature:

  1. Dump the selected algo id per call key across N fresh processes on one leased device, same
    shapes, same driver (gemm_algo_log.h already exists for the logging half).
  2. Compare the selections.
  3. If stable — close this with that evidence, and correct the premise's citation (see below).
    Persisting buys nothing and should not be built.
  4. If unstable — it becomes a benchmark-validity repair on KERNEL-GEMM-BF16, and the
    persistent-cache machinery in src/vt/cuda/nvfp4_persistent_cache.cpp is the shape to reuse.

Explicitly not "go persist the cuBLASLt heuristic". That is only warranted in case 4.

Two stale records to repair in the same flow

Found while reading these headers; both are one-line record defects on this row, fixed here
rather than deferred:

  • src/vt/cuda/fp8_plan_cache.h:18-19 cites .agents/state.md as the forensic record for the
    determinism premise. That file does not exist — the protocol retired the state log
    ("The project has no state log"). The premise's only cited evidence is unreachable.
  • src/vt/cuda/gemm_plan_cache.h:29 says the fp8 cache "ships off". It has been default ON
    since #1843, which the fp8 header itself records at fp8_plan_cache.h:23-24. The comment is
    stale in the opposite direction from the code.

Evidence

  • src/vt/cuda/gemm_plan_cache.h:1-35 (premise, default-ON polarity, #1732 capture bug)
  • src/vt/cuda/fp8_plan_cache.h:10-31 (premise + citation + #1843 flip)
  • .agents/specs/nvfp4-persistent-plan-cache.md § "W3-C3 corrected frozen-plan component result"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/vt/cuda/gemm_plan_cache.h, src/vt/cuda/fp8_plan_cache.h, and gemm_algo_log.h; inspect how selected algorithm IDs are logged. Run the same shapes in N fresh processes on one device and compare the selections. Done means the cross-process stability is evidenced or the benchmark-validity repair is scoped, with the two stale header records corrected.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.