[Linux/Kraken] Decode throughput ~50% of official benchmarks across models (35B MoE and dense 1B alike); prefill near-par

[Linux/Kraken] Decode throughput ~50% of official benchmarks across models (35B MoE and dense 1B alike); prefill near-par

Summary

On a Ryzen AI 7 350 (Kraken Point) system running Ubuntu 26.04 with a stack aligned item-by-item to the official benchmark setup — FLM v0.9.45, the same version as the official Qwen 3.6 benchmark page — decode throughput is consistently ~50% of the official numbers published for the same silicon, while prefill reaches roughly 70–85% of official (interpolated for prompt length). The deficit:

• affects a dense 1B model and the 35B MoE alike,
• is invariant to --ctx-len (131072 vs 4096: no change on 1B decode),
• is nearly invariant to power mode (turbo moves 35B decode by only +7.7%),
• leaves prefill largely intact on both models.

Environment

Prefill, by contrast: 53.16 vs 75.18 @1k official for the 35B (~0.85× after interpolating for the ~0.7k prompt length), and 787 tok/s on the 1B — both look healthy.

Analysis

The prefill/decode asymmetry, reproduced on both a dense 1B and the 35B MoE, is the key fingerprint. Two mechanisms fit the data; we cannot distinguish them from user space:

Hypothesis A — fixed per-token host-side cost. The excess per-token time scales with model depth (16 layers for the 1B, several-fold deeper plus per-token expert routing for the 35B MoE), consistent with a fixed host sync/orchestration cost on the order of ~1 ms per layer per token. Prefill amortizes this across the whole prompt; decode pays it on every token.

Hypothesis B — NPU-side effective DRAM bandwidth ~halved. Back-computing from the 1B (~0.75 GB weights/token ÷ 32.1 ms ≈ 23 GB/s effective NPU read bandwidth) also predicts the 35B numbers if per-token traffic is ~3.7 GB — i.e., the same model that would yield the official figures at ~47 GB/s. Host-visible DRAM bandwidth is at the bus ceiling (see Environment), so if this is the mechanism, it is specific to the NPU DMA path, not the memory subsystem.

Both hypotheses are compatible with prefill being near-par (prefill is compute-bound).

Ruled out by measurement: DRAM configuration (dual-channel 5600 confirmed; host read at bus limit), memlock, IOMMU translation (passthrough), CPU governor (EPP=performance), stale in-tree driver (DKMS module active), firmware (1.1.2.64), FLM version mismatch (v0.9.45 = official), context-length allocation (128k vs 4k identical), power mode (turbo tested).

Why this looks stack-generation-specific: the Strix Point Linux datapoint above (healthy 60–89 tok/s on FLM v0.9.35 / kernel 6.17) suggests Linux per se was not always taxed like this. Remaining candidates for bisection: FLM 0.9.35 → 0.9.45 changes on the Linux path, in-tree 6.17 amdxdna vs kernel-7.0/DKMS driver, XRT 2.21.75, or a Kraken-specific code path.

Questions

1. Which OS were the official Kraken benchmarks (Qwen 3.6 page) measured on — Windows or Linux?
2. Is Power Mode: Default in xrt-smi examine -r platform during FLM inference expected (i.e., is FLM's --pmode applied out-of-band from what xrt-smi reports)?
3. Are per-token host round trips on the Linux XRT path known to cost more than on Windows in the current release?

Happy to run debug builds, collect traces, or bisect FLM/driver/XRT versions on this machine — it is available as a test platform.

No source file or test is named. Reproduce the 1B and 35B decode/prefill measurements with FLM v0.9.45, then bisect the listed FLM, amdxdna/kernel, and XRT versions using traces or debug builds. Done means identifying a confirmed cause and a reproducible regression boundary.