27B dense keep-quant arm: decode residency exhausts device DRAM, e2e gate unmet

Row: BACKEND-TENSTORRENT-KEEPQUANT

What is owed

The 27B dense arm (Qwen3.8-27B-Q4_K_M.gguf, sha256 7e78da5d…fe169, 17,106,775,008 B)
passed its loader wiring, MTP drafter skip, and reachability test in wave-3b-2, but its
end-to-end gate is unmet: keep-quant decode exhausts device DRAM before one 16-token
generation completes on the production entry point (examples/vllm-bench). This issue
owns that gate. The row spec (.agents/specs/tenstorrent-keepquant.md) lists it under
## Owed.

Evidence (8 runs, 2026-09-07)

• Failing allocations: 1,073,725,440 B under the 2048 budget and 134,184,960 B under the
  32 MiB VT_TT_KEEPQUANT_CHUNK_BYTES cap; both die in the dense keep-quant decode chain
  (DecodeKeepQuantWordsF32 → ttnn::where f32 planes).
• Free-at-failure declines monotonically across runs: 244 MB → 46 MB → 12.7 MB per bank.
  The allocator ends at ~34 GB allocated against a 32 GB device with a largest free block
  of 3.7 MB — fragmentation, not a single oversized request.
• Budget 512 fails identically, so the failure is not activation-sized.
• Three mitigations failed: plane budget cut; VT_TT_KEEPQUANT_CHUNK_BYTES=32MiB (got 85%
  through the first prefill, 2,489 decode calls); the knob as a hard cap over the wide-N
  ceil(N/8) trace bound.
• Design residency was surveyed at ~22-23 GB on 32 GB (15.92 GB packed + 2.37 GB embed
  twin + 2.37 GB output twin + chunk tiles + activations). The observed ~34 GB is ~11 GB
  above it.

Suspected structural contributors

1. The ~2.5 GB resident bf16 embedding twin in EnsureEmbedTableDevice.
2. f32 plane transients from the per-chunk decode fragmenting the allocator.
3. Possible keep-quant words double-staging against the design residency.

These are design levers (plane dtype, persistent workspace, twin residency), not wiring
gaps. Fixing them is a row-level change and needs its own spec.

Ready lane

The oracle denominator is validated and reusable: pinned llama.cpp b10451
(llama-completion, ignores the blk.64 MTP tensors), byte-identical harness, 16-prompt
greedy_ids.npy committed. Once residency fits, the gate needs only the TT-side run:
16-prompt greedy parity in the ratified 500-mnat band with 0 forward-divergent tokens.

Exit condition

The 27B arm passes its e2e gate through a production entry point on the identical
workload, or the arm is formally re-scoped in the row spec with the reason recorded.

Refs: #2959 (row), #3030 (capture falsification chain), #3031 (int8-dot lever).

Read .agents/specs/tenstorrent-keepquant.md and the 27B row context, then inspect the production entry point at examples/vllm-bench. Use the committed greedy_ids.npy and pinned llama.cpp b10451 oracle to reproduce the TT-side run. Done means the identical 16-prompt workload passes in the 500-mnat band with 0 forward-divergent tokens, or the row is formally re-scoped with its reason recorded.