mudler / mudler/vllm.cpp

Qwen3-4B on Tenstorrent: both decode arms put 5/16 gate prompts beyond the 0.5-nat near-tie band

Open
#2,811 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
19h 27m
Merged PRs (30d)
327

Description

Row: BACKEND-TENSTORRENT-HOST-FREE-FORWARD

Found during the #1625 capture-default flip's 4B bring-up (P150, 2026-09-03). The 4B TT near-tie pair cannot be committed: both arms are deterministic and reproduce byte-identical across runs, but against transformers-teacher-forced gaps on each arm's OWN prefix, 5 of 16 prompts land beyond the 500-mnat band — first-divergence margins 0.625-2.0 nats, a real forward difference at these cells, not a bf16 tie (0.6B TT worst: 500 mnats; the committed CUDA 4B pair's worst: 250 mnats).

Arm prompts > band worst cell cells > band
captured (post-flip default) p3, p6, p11, p12, p15 1000 mnats @ (11,3) 8
eager (VT_TT_DECODE_CAPTURE=0) p3, p5, p6, p7, p12 2000 mnats @ (5,6) 8

The arms differ from each other in 112 of 256 cells while each reproduces byte-identical run-to-run (capture sha256 7e77e6ea, eager 8325ff10) — tie chaos on top of a shared forward difference. Most prompts are fully vLLM-endorsed (gap 0 on every cell), so the divergence is concentrated, not uniform.

Hypothesis to falsify: diffuse bf16 accumulation drift at H=2560/36L rather than a discrete kernel bug (a discrete bug usually yields many-nat or gibberish divergence). Discriminator: op-level bisection at 4B shapes — per-op output dump vs the f32 CPU reference at the first-divergence cells (capture (11,2) 875 mnats, (3,4) 625, (6,6) 875, (12,1) 750, (15,13) 625; eager worst (5,6) 2000).

Already fixed on the flip branch while bringing this up: the 4B capture path split on two capacity-vs-volume defects in the #1922 best-fit lend (commit 1872aeb16) — D2D copy and capture memset now take the call's byte count as the contract, not the host block's registered capacity.

Exit: either a committed 4B TT pair that passes its own gate (both arms), or a falsified-drift root cause with the discrete defect fixed and the pair re-derived. Until then the 4B dense gate skips loudly on Tenstorrent.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the 4B-shape op-level bisection, comparing per-op outputs against the f32 CPU reference at the listed first-divergence cells. Compare captured and eager runs, including VT_TT_DECODE_CAPTURE=0, then verify whether both arms pass the 500-mnat gate. The issue is done when a committed 4B Tenstorrent pair passes both gates or the discrete defect is fixed and the pair re-derived.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.