mudler / mudler/vllm.cpp

Glm4MoeLite's only vs-vLLM assertion cannot fail: the gap artifact is identically zero and the goldens match 69/128

Open
#2,839 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: MODEL-TEXT-glm4-moe-lite-glm4-moe-lite-for-causal-lm

What

docs/FEATURES.md:155 advertises Glm4MoeLiteForCausalLM as "near-tie 8/8 vs
vLLM 0.25.0"
. The 8/8 is what the test prints. It is produced by a predicate
that cannot fail, on an artifact that is identically zero.

Measured, at ca07f6e94

The committed gap artifact,
tests/parity/goldens/glm4_moe_lite_greedy/neartie_gap_mnats.npy:

$ python3 -c "import numpy;a=numpy.load('tests/parity/goldens/glm4_moe_lite_greedy/neartie_gap_mnats.npy');print(a.shape,a.dtype,a.min(),a.max(),(a==0).all())"
(8, 16) int32 0 0 True

All 128 values are exactly 0. The only vs-vLLM assertion in
tests/vllm/models/test_glm4_moe_lite_paged_engine.cpp reads that array and
tests mn > kNearTieMnats with constexpr int32_t kNearTieMnats = 500; (:61,
:308). 0 > 500 is false at every position, so prompt_ok is true for all 8
prompts unconditionally and CHECK_MESSAGE(prompt_ok, ...) (:336) has no
failure mode. It is a positive control that provably cannot trip.

What the goldens actually say, comparing the committed greedy_ids.npy (the
vLLM oracle) against our_ids.npy:

positions match     69 of 128   (53.9%)
prompts exact        1 of 8      (prompt 7 only)
per-prompt mismatch [2, 6, 6, 12, 12, 11, 10, 0]

Prompts 3 and 4 differ in 12 of their 16 tokens. Nothing asserts against that.

Two further weaknesses in the same file

  1. The exact-equality assertion at :283-287 compares against our_ids.npy,
    labelled in the file itself as "ANCHOR: the committed gaps describe OUR
    engine's exact sequence". That is a self-regression check, not an oracle
    check.
  2. The header at :22-24 says a divergence counts as a near-tie "ONLY where
    that gap is within kNearTieMnats AND our token is inside vLLM's top-K". The
    second conjunct is not implemented: greedy_dist.npy is read only for the
    self-determinism count at :140-163, never for top-K membership.
  3. .agents/model-matrix.md:120 records "vLLM K=5 self-deterministic -> STRICT
    bar". Per CLAUDE.md §Gates, a distributional gate is licensed only when the
    oracle's greedy decode is non-deterministic. It is deterministic here, so the
    STRICT bar is what is owed, and the shipped gate asserts neither.

Not fixed in the flow that found it

Found while repairing docs/FEATURES.md's dead citations (#2825). The doc repair
publishes the real numbers in place of "near-tie 8/8". Repairing the GATE is a
different unit of work: it needs a red-first argument about which bar applies,
and regenerating an artifact that is currently all zeros.

What would close it

A gap artifact that is not identically zero, or a STRICT assertion consistent
with the deterministic oracle -- and a mutation proving the predicate can fail.

Related: #2825, #2794 (goldens predate the current pin).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with tests/vllm/models/test_glm4_moe_lite_paged_engine.cpp, the committed files under tests/parity/goldens/glm4_moe_lite_greedy/, and CLAUDE.md §Gates. Run the existing Glm4MoeLite parity test and inspect the oracle, gap, and top-K inputs. Done means the gate uses the applicable bar, the artifact is regenerated, and a mutation proves the assertion can fail.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, numpy, python
Domain
machine-learning, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.