Qwen3-4B on Tenstorrent: both decode arms put 5/16 gate prompts beyond the 0.5-nat near-tie band
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 19h 27m
- Merged PRs (30d)
- 327
Description
Row: BACKEND-TENSTORRENT-HOST-FREE-FORWARD
Found during the #1625 capture-default flip's 4B bring-up (P150, 2026-09-03). The 4B TT near-tie pair cannot be committed: both arms are deterministic and reproduce byte-identical across runs, but against transformers-teacher-forced gaps on each arm's OWN prefix, 5 of 16 prompts land beyond the 500-mnat band — first-divergence margins 0.625-2.0 nats, a real forward difference at these cells, not a bf16 tie (0.6B TT worst: 500 mnats; the committed CUDA 4B pair's worst: 250 mnats).
| Arm | prompts > band | worst cell | cells > band |
|---|---|---|---|
| captured (post-flip default) | p3, p6, p11, p12, p15 | 1000 mnats @ (11,3) | 8 |
eager (VT_TT_DECODE_CAPTURE=0) |
p3, p5, p6, p7, p12 | 2000 mnats @ (5,6) | 8 |
The arms differ from each other in 112 of 256 cells while each reproduces byte-identical run-to-run (capture sha256 7e77e6ea, eager 8325ff10) — tie chaos on top of a shared forward difference. Most prompts are fully vLLM-endorsed (gap 0 on every cell), so the divergence is concentrated, not uniform.
Hypothesis to falsify: diffuse bf16 accumulation drift at H=2560/36L rather than a discrete kernel bug (a discrete bug usually yields many-nat or gibberish divergence). Discriminator: op-level bisection at 4B shapes — per-op output dump vs the f32 CPU reference at the first-divergence cells (capture (11,2) 875 mnats, (3,4) 625, (6,6) 875, (12,1) 750, (15,13) 625; eager worst (5,6) 2000).
Already fixed on the flip branch while bringing this up: the 4B capture path split on two capacity-vs-volume defects in the #1922 best-fit lend (commit 1872aeb16) — D2D copy and capture memset now take the call's byte count as the contract, not the host block's registered capacity.
Exit: either a committed 4B TT pair that passes its own gate (both arms), or a falsified-drift root cause with the discrete defect fixed and the pair re-derived. Until then the 4B dense gate skips loudly on Tenstorrent.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the 4B-shape op-level bisection, comparing per-op outputs against the f32 CPU reference at the listed first-divergence cells. Compare captured and eager runs, including VT_TT_DECODE_CAPTURE=0, then verify whether both arms pass the 500-mnat gate. The issue is done when a committed 4B Tenstorrent pair passes both gates or the discrete defect is fixed and the pair re-derived.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100