The neartie gap artifacts are identically zero in 12 of 18 golden dirs while our_ids diverges: not measurements
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: - (owned under ## Owed in .agents/specs/glm4-moe-lite-router-f32.md)
neartie_gap_mnats.npy is the artifact every near-tie-banded model gate reads to
decide whether a divergence is admissible. In 12 of the 18 golden directories
that carry one it is identically zero at every position, while our_ids.npy
diverges from greedy_ids.npy at 13 to 83 positions in the same directory.
Measured at c796fea41:
dir gap min/max nonzero our_ids vs greedy_ids
deepseek_v2_greedy 0/250 1/128 36/128 divergent
glm4_moe_lite_greedy 0/0 0/128 59/128 divergent
internlm2_greedy_1_8b 0/0 0/256 42/256 divergent
internlm3_greedy_8b 0/0 0/256 22/256 divergent
llama_greedy_1b 0/0 0/256 13/256 divergent
minicpm3_greedy_4b 0/0 0/256 20/256 divergent
minicpm_greedy_2b 0/0 0/256 34/256 divergent
mistral_greedy_7b 0/0 0/256 6/256 divergent
olmo2_greedy_1b 0/94 2/256 15/256 divergent
phi2_greedy_2_7b 0/250 6/256 75/256 divergent
phi4_14b_greedy 0/0 0/256 37/256 divergent
phi4_mini_greedy 0/1000 9/256 83/256 divergent
qwen35_greedy_0_8b 0/125 2/256 8/256 divergent
qwen3_32b_nvfp4a16_greedy 0/0 0/96 43/96 divergent
qwen3_greedy_0_6b 0/125 1/256 55/256 divergent
qwen3_greedy_4b 0/250 2/256 43/256 divergent
qwen3coder_greedy 0/0 0/96 1/96 divergent
yi_greedy_coder_1_5b 0/125 2/256 38/256 divergent
Reproduce with numpy over tests/parity/goldens/*/; no checkpoint and no GPU
are needed.
Why zero is not a possible measurement at a FIRST divergence
The generating scripts (scripts/*-neartie-gap.py) teacher-force the pinned
oracle on OUR token sequence and record
max(0, argmax_logprob - our_token_logprob) in milli-nats. At a prompt's first
divergent position our prefix is byte-identical to the oracle's, so the oracle's
teacher-forced argmax there is by construction the token its own free-running
greedy decode emitted — greedy_ids[i,j] — and ours is a different token. The
gap must therefore be strictly positive unless the two logprobs are exactly
equal in the oracle's own float output.
For glm4_moe_lite_greedy it reads 0 at all seven first divergences (prompts
0..6, positions 14, 6, 6, 4, 4, 4, 6), and the scripts' 99_999_000
outside-top-20 sentinel appears nowhere in the file. our_ids.npy is written by
the same script in the same call as the gap array, so the script demonstrably
ran to completion; the zeros are what it computed, not a file it never wrote.
The same argument applies to every all-zero directory above.
Consequence
Wherever a gate's admissibility test is gap > band, an all-zero artifact makes
that test unconditionally true and the assertion has no failure mode. #2839
measured exactly that for Glm4MoeLiteForCausalLM, where the artifact is
(8,16) int32, min 0 max 0, against a 500 mnat band.
This issue is about the ARTIFACT CLASS. It does not claim any particular gate is
wrong beyond #2839's, and it does not propose widening or narrowing a band. The
first thing it asks for is a re-capture on a host that has the checkpoints, with
the script's own diagnostic output preserved, so that whether the zeros come
from the script's use of prompt_logprobs or from the artifacts being written
without a real run can be settled by evidence rather than by argument.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with scripts/-neartie-gap.py and inspect the artifacts under tests/parity/goldens/; reproduce the reported zero gaps with numpy. Re-capture on a host with the checkpoints while preserving the script's diagnostic output, then determine whether the zeros come from prompt_logprobs or from artifacts being written without a real run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, numpy, python
- Domain
- machine-learning, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100