docs/USAGE.md documents no checkpoint for Qwen3.8-27B bf16, the subject of a published benchmark whose gate exists only in prose
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: BENCH-QWEN38-27B-SOTA
What
docs/USAGE.md has zero entries for Qwen/Qwen3.8 and zero for the
revision 1d4bf0f2 — verified by count, not by a bare grep miss. Meanwhile the
same checkpoint is named in three benchmark documents, including
docs/benchmarks/reproduce.md, whose entire purpose is reproduction.
CLAUDE.md §"Say which weights, and from where" designates docs/USAGE.md as the
surface that records file name, size, HuggingFace repo and revision, with a
sha256 for a quantized artifact. It says code shipped without them is "something
nobody can feed". Here it is worse: a published benchmark row rests on a
checkpoint no reader can identify.
USAGE.md today records only derived artifacts for this family — EXL3, ModelOpt
NVFP4, GGUF, DFlash2 drafters. The bf16 checkpoint the benchmark actually ran on
is absent.
Why it surfaced now
While costing the Qwen3.8-27B bf16 token gate it emerged that the gate does not
exist as code. The landing commit 0f58cbdb5 is records-only — 8 files, all
under .agents/ and docs/ — and .agents/specs/qwen38-27b-bf16-gate.md:71 says
so itself: "This row changes no src/, include/ or tests/ file, so it ports
no test."
Confirmed independently: zero qwen38/q38 directories under
tests/parity/goldens/, and git log --all --diff-filter=A shows no such path
ever existed on any branch. No gate test, no golden, no capture script. The
"4/7 prompts STRICT 16/16" result exists only in prose, and the capture.json the
spec cites lived on the share, which the spec itself calls "not a tracked surface".
So the undocumented checkpoint and the unreproducible gate are the same problem
seen twice: nothing about this benchmark can be re-run by anyone.
Two things this issue does NOT ask for
- It does not ask for the gate to be written. That is a campaign — a capture
harness, two fp32 adjudication probes, a gate test, a golden directory, a
manifest, then two resumable GB10 jobs at ~3-5 h device time for a 52 GB /
18-shard checkpoint whose recorded cold start is 780 s. It needs its own row. - It does not ask anyone to invent a pin. A repo id alone is not a pin, and
guessing a revision would be worse than the gap. Whoever ran the benchmark
knows which bytes they used; that is the value being recovered.
An unresolved discrepancy that blocks a staged-input assertion
.agents/specs/qwen38-27b-bf16-gate.md:583-589 records the checkpoint at
55,586,114,863 bytes; the campaign's NOTES.txt records 55,586,040,114 —
a 74,749-byte difference nobody has settled. Until that is resolved it cannot
become a staged-input assertion, and crc32.txt covers only the 8 small files,
not the 18 shards.
Suggested minimum
A docs/USAGE.md row for the bf16 checkpoint: repo, revision, shard count, total
size, and the sha256 of at least the index. That alone makes the published
benchmark identifiable, without waiting on the gate campaign.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with docs/USAGE.md and compare the Qwen3.8 references in docs/benchmarks/reproduce.md and the other benchmark documents. Read .agents/specs/qwen38-27b-bf16-gate.md and the campaign’s NOTES.txt to recover and reconcile the repository, revision, shard count, size, and index SHA256. Done means a verified bf16 checkpoint row documents the published benchmark input.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- markdown
- Domain
- documentation
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100