Glm4MoeLite's only vs-vLLM assertion cannot fail: the gap artifact is identically zero and the goldens match 69/128
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: MODEL-TEXT-glm4-moe-lite-glm4-moe-lite-for-causal-lm
What
docs/FEATURES.md:155 advertises Glm4MoeLiteForCausalLM as "near-tie 8/8 vs
vLLM 0.25.0". The 8/8 is what the test prints. It is produced by a predicate
that cannot fail, on an artifact that is identically zero.
Measured, at ca07f6e94
The committed gap artifact,
tests/parity/goldens/glm4_moe_lite_greedy/neartie_gap_mnats.npy:
$ python3 -c "import numpy;a=numpy.load('tests/parity/goldens/glm4_moe_lite_greedy/neartie_gap_mnats.npy');print(a.shape,a.dtype,a.min(),a.max(),(a==0).all())"
(8, 16) int32 0 0 True
All 128 values are exactly 0. The only vs-vLLM assertion in
tests/vllm/models/test_glm4_moe_lite_paged_engine.cpp reads that array and
tests mn > kNearTieMnats with constexpr int32_t kNearTieMnats = 500; (:61,
:308). 0 > 500 is false at every position, so prompt_ok is true for all 8
prompts unconditionally and CHECK_MESSAGE(prompt_ok, ...) (:336) has no
failure mode. It is a positive control that provably cannot trip.
What the goldens actually say, comparing the committed greedy_ids.npy (the
vLLM oracle) against our_ids.npy:
positions match 69 of 128 (53.9%)
prompts exact 1 of 8 (prompt 7 only)
per-prompt mismatch [2, 6, 6, 12, 12, 11, 10, 0]
Prompts 3 and 4 differ in 12 of their 16 tokens. Nothing asserts against that.
Two further weaknesses in the same file
- The exact-equality assertion at
:283-287compares againstour_ids.npy,
labelled in the file itself as "ANCHOR: the committed gaps describe OUR
engine's exact sequence". That is a self-regression check, not an oracle
check. - The header at
:22-24says a divergence counts as a near-tie "ONLY where
that gap is withinkNearTieMnatsAND our token is inside vLLM's top-K". The
second conjunct is not implemented:greedy_dist.npyis read only for the
self-determinism count at:140-163, never for top-K membership. .agents/model-matrix.md:120records "vLLM K=5 self-deterministic -> STRICT
bar". PerCLAUDE.md§Gates, a distributional gate is licensed only when the
oracle's greedy decode is non-deterministic. It is deterministic here, so the
STRICT bar is what is owed, and the shipped gate asserts neither.
Not fixed in the flow that found it
Found while repairing docs/FEATURES.md's dead citations (#2825). The doc repair
publishes the real numbers in place of "near-tie 8/8". Repairing the GATE is a
different unit of work: it needs a red-first argument about which bar applies,
and regenerating an artifact that is currently all zeros.
What would close it
A gap artifact that is not identically zero, or a STRICT assertion consistent
with the deterministic oracle -- and a mutation proving the predicate can fail.
Related: #2825, #2794 (goldens predate the current pin).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with tests/vllm/models/test_glm4_moe_lite_paged_engine.cpp, the committed files under tests/parity/goldens/glm4_moe_lite_greedy/, and CLAUDE.md §Gates. Run the existing Glm4MoeLite parity test and inspect the oracle, gap, and top-K inputs. Done means the gate uses the applicable bar, the artifact is regenerated, and a mutation proves the assertion can fail.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, numpy, python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100