mudler / mudler/vllm.cpp

feat(KERNEL-GEMM-NVFP4-W4A4): ship a pinned GB10 tactic draw so a fresh checkout does not re-tune

Open
#2,752 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: KERNEL-GEMM-NVFP4-W4A4

What

We ship the persistent-tactic-cache machinery and no cache. ResolvePersistentCacheOptions
(src/vt/cuda/nvfp4_persistent_cache.cpp:543) defaults the cache ON and resolves it under
$XDG_CACHE_HOME/$HOME/.cache/vllm.cpp/nvfp4_autotune/..., so every fresh checkout on every
host re-tunes all 64 plans on its first serve, and gets whatever draw it gets.

There is no committed artifact and no documented way to obtain one, so the reproducibility win
the cache was built for is available only to a host that has already run once, and is
lost on any machine that has not.

Proposal

Commit one measured GB10 draw as a repository artifact, in the native document format
SerializeNativeCache already emits, plus a documented install path.

The path identity already supports this safely. nvfp4_persistent_cache.cpp:585-600 keys the
cache directory on sm arch, GPU name, device ordinal, CUDA runtime and driver, CUTLASS
version, tactic-descriptor digest + tactic set version, output dtype, fp4/scale layout and the
timing recipe. ParseNativeCache validates the metadata against the running configuration, so a
draw installed onto a host it does not describe is rejected, not silently used — the
mismatch mode this artifact could otherwise introduce is already closed.

Scope

  • one native-format draw for GB10 / sm_121, measured under a lease, with its sha256;
  • its PersistentCacheMetadataFingerprint recorded beside it, so identity is checkable without
    running;
  • the install step documented (this is the artifact half of the docs issue on
    ENG-PUBLIC-DOC-PROJECTIONS);
  • a note that the draw is a starting point for that exact configuration, not a guarantee.

Honest framing of the value

This is a reproducibility and warmup artifact, not a speed artifact. The measured
frozen-plan steady-state component on this lane is c2 1.0045x / c16 1.0050x — strict
result FAILED at 39/40 timing + 1/8 memory. What the draw buys is that two people benchmarking
this tree on the same silicon start from the same kernel plan, and that the first serve does not
pay a tuning pass. Any claim beyond that needs its own measurement.

Blocked-by consideration: if the draw-quality investigation on this same row finds that draws
differ materially in speed, which draw we ship stops being arbitrary and this issue should
wait for that answer rather than pin a lucky or unlucky sample.

Evidence

  • src/vt/cuda/nvfp4_persistent_cache.cpp:543-600 (default-on resolution + path identity)
  • src/vt/cuda/nvfp4_persistent_cache.h:105-135 (serialize / parse / atomic publish contract)
  • .agents/specs/nvfp4-persistent-plan-cache.md § "W3-C3 corrected frozen-plan component result"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/vt/cuda/nvfp4_persistent_cache.cpp:543-600 and nvfp4_persistent_cache.h:105-135, then read the relevant section of .agents/specs/nvfp4-persistent-plan-cache.md. Produce one measured native-format GB10/sm_121 draw under a lease, with its SHA-256 and PersistentCacheMetadataFingerprint recorded beside it. Document installation and state that the draw is only a starting point for the matching configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, documentation, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.