mudler / mudler/vllm.cpp

docs/USAGE.md documents no checkpoint for Qwen3.8-27B bf16, the subject of a published benchmark whose gate exists only in prose

Open
#2,951 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BENCH-QWEN38-27B-SOTA

What

docs/USAGE.md has zero entries for Qwen/Qwen3.8 and zero for the
revision 1d4bf0f2 — verified by count, not by a bare grep miss. Meanwhile the
same checkpoint is named in three benchmark documents, including
docs/benchmarks/reproduce.md, whose entire purpose is reproduction.

CLAUDE.md §"Say which weights, and from where" designates docs/USAGE.md as the
surface that records file name, size, HuggingFace repo and revision, with a
sha256 for a quantized artifact. It says code shipped without them is "something
nobody can feed". Here it is worse: a published benchmark row rests on a
checkpoint no reader can identify.

USAGE.md today records only derived artifacts for this family — EXL3, ModelOpt
NVFP4, GGUF, DFlash2 drafters. The bf16 checkpoint the benchmark actually ran on
is absent.

Why it surfaced now

While costing the Qwen3.8-27B bf16 token gate it emerged that the gate does not
exist as code
. The landing commit 0f58cbdb5 is records-only — 8 files, all
under .agents/ and docs/ — and .agents/specs/qwen38-27b-bf16-gate.md:71 says
so itself: "This row changes no src/, include/ or tests/ file, so it ports
no test."

Confirmed independently: zero qwen38/q38 directories under
tests/parity/goldens/, and git log --all --diff-filter=A shows no such path
ever existed on any branch. No gate test, no golden, no capture script. The
"4/7 prompts STRICT 16/16" result exists only in prose, and the capture.json the
spec cites lived on the share, which the spec itself calls "not a tracked surface".

So the undocumented checkpoint and the unreproducible gate are the same problem
seen twice: nothing about this benchmark can be re-run by anyone.

Two things this issue does NOT ask for

  1. It does not ask for the gate to be written. That is a campaign — a capture
    harness, two fp32 adjudication probes, a gate test, a golden directory, a
    manifest, then two resumable GB10 jobs at ~3-5 h device time for a 52 GB /
    18-shard checkpoint whose recorded cold start is 780 s. It needs its own row.
  2. It does not ask anyone to invent a pin. A repo id alone is not a pin, and
    guessing a revision would be worse than the gap. Whoever ran the benchmark
    knows which bytes they used; that is the value being recovered.

An unresolved discrepancy that blocks a staged-input assertion

.agents/specs/qwen38-27b-bf16-gate.md:583-589 records the checkpoint at
55,586,114,863 bytes; the campaign's NOTES.txt records 55,586,040,114
a 74,749-byte difference nobody has settled. Until that is resolved it cannot
become a staged-input assertion, and crc32.txt covers only the 8 small files,
not the 18 shards.

Suggested minimum

A docs/USAGE.md row for the bf16 checkpoint: repo, revision, shard count, total
size, and the sha256 of at least the index. That alone makes the published
benchmark identifiable, without waiting on the gate campaign.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with docs/USAGE.md and compare the Qwen3.8 references in docs/benchmarks/reproduce.md and the other benchmark documents. Read .agents/specs/qwen38-27b-bf16-gate.md and the campaign’s NOTES.txt to recover and reconcile the repository, revision, shard count, size, and index SHA256. Done means a verified bf16 checkpoint row documents the published benchmark input.

Written by the indexing model from the issue text.

Assessment

Tech stack
markdown
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.