Qwen3-8B on XDNA2 (Strix Halo): only ~10 tok/s decode + slow TTFT vs GPU - is this expected?

title: "Qwen3-8B on XDNA2 (Strix Halo): only ~10 tok/s decode + slow TTFT vs GPU - is this expected?"
labels: ["question", "performance"]

Environment

• Device: ASUS ProArt PX13 (HN7306EA), Ryzen AI MAX+ 395 (Strix Halo)
• NPU: XDNA2, driver amdxdna 0.8.0, firmware amdnpu/17f0_11/npu_7.sbin, kernel 7.1.9 (Arch)
• FLM: v1.0.3 (via Lemonade 11.9.0, fastflowlm_1.0.3_linux)
• Model: qwen3-8b-FLM (Qwen3-8B-NPU2, q4nx, 5.6 GB)

Observed performance

Decode on the NPU is ~10.4 tok/s for Qwen3-8B (86s for 887 tokens). This seems very slow compared to what one would hope from a 50 TOPS NPU, and the same model on the same machine via GPU (llama.cpp ROCm) runs at 33.4 tok/s.

Also notable: TTFT (time to first token) with a ~600-token prompt is 1.93s on NPU vs 0.34s on GPU — the NPU is slower to first token, contrary to some published claims that NPU prefill/TTFT is faster.

Power during NPU inference: SoC total ~27W, ~11.5W net over idle.

Questions

1. Is ~10 tok/s decode on Qwen3-8B expected for XDNA2 (Strix Halo) with FLM v1.0.3? Independent tests I found report similar (8.2 t/s), so I suspect this is the architectural reality, but I'd like confirmation.
2. Why is TTFT slower on NPU than GPU here? Is NPU prefill throughput actually limited, or is there a startup/dispatch overhead?
3. Any known tuning knobs (power state, batch, model variant, newer FLM/driver) that materially improve decode throughput on XDNA2 Linux?

Context

For reference, my full benchmark (CPU vs NPU vs GPU, incl. TDP-capped runs and energy-per-token) is here: same machine, Qwen3-8B:

• GPU (ROCm): 33.4 tok/s, TTFT 0.34s
• NPU (FLM): 10.4 tok/s, TTFT 1.93s
• CPU: 12.8 tok/s

Happy to provide more logs or run specific benchmarks (e.g. flm CLI directly, prefill-only timings).

Start by reproducing the reported Qwen3-8B-NPU2 benchmark on the ASUS PX13 with FLM v1.0.3, then compare direct flm CLI timings with the reported GPU and CPU results. Measure decode and roughly 600-token prefill/TTFT separately, and check whether newer FLM, driver, power-state, batch, or model-variant settings change the result. Done means documenting whether the behavior is expected and identifying any materially effective tuning knob.