Qwen3.6-35B-A3B Linux v1.0.4: ~2.7s fixed per-request TTFT = ~4,900 serialized per-expert NPU execs — validated coarsening fix (concatenated-expert GEMM)

Qwen3.6-35B-A3B on Linux (FLM v1.0.4): ~2.7 s fixed per-request TTFT = ~4,900 serialized small NPU execs from per-expert MoE dispatch — with a validated coarsening fix (concatenated-expert GEMM)

Summary

flm serve qwen3.6-moe:35b-a3b on Linux has a fixed ~2.7 s per-request TTFT cost for small prompts (10-token prompt: ~2.7–3.0 s to first token; ~300-token: 5.85 s → marginal prefill ~105–110 tok/s is at spec). strace/perf/disassembly attribute it to the MoE prefill path dispatching ~4,900 tiny serial NPU execs per request (submit-then-block, no overlap). Decode (16.5–16.8 tok/s) is at the published spec; only the per-request fixed cost is pathological, and it dominates exactly the small-prompt/agent-loop regime.

Environment

• APU: AMD Ryzen AI MAX+ 395 (Strix Halo, XDNA2 NPU5), 128 GB LPDDR5-8000, Ubuntu 26.04, amdxdna context_limit=64
• Runtime: FLM v1.0.4 (flm serve qwen3.6-moe:35b-a3b --port 8098), Q4_K weights (flm_q4k_high_precision)
• Server-reported split for a 19-token prompt: prefill_duration_ttft: 2.93 s, decoding_speed_tps: 16.48

Evidence

1. Driver-level (strace, one 10-token request, 3.75 s window, single thread issues everything):
   • ~4,890 DRM_IOCTL_AMDXDNA_EXEC_CMD and ~4,830 DRM_IOCTL_SYNCOBJ_TIMELINE_WAIT — 1:1, submit-then-block, zero pipelining
   • ~5,400 each of CREATE_BO / GEM_CLOSE / GET_BO_INFO — per-op buffer lifecycle
   • Time-weighted: 95.2% of ioctl time is the syncobj waits (1.958 s): 96% of waits cluster at 0.10–0.40 ms (median 0.217 ms, p90 0.243 ms — flat across thousands of uniform ops → real serial small-kernel time, not launch overhead). Only 188 waits > 2 ms.
2. perf: at stock env 56% of cycles in libgomp = worker spin (fixable via env; applied OMP_WAIT_POLICY=passive GOMP_SPINCOUNT=0 → CPU 2.32 → 0.36 cores/request, TTFT unchanged — spin is background waste, not the latency cause).
3. Binary analysis (libqwen3_6_moe_npu.so, not stripped): per-op dispatch is per-expert:
   • qwen3_6_moe_npu::Impl::prefill 0x77280 → _prefill_with_mm 0x76320 → qwen3_6_moe_expert_prefill_context::forward 0x897f0, which loops experts individually with full buffer+run lifecycle per iteration (15× buffer dtor, 7× delete, 6× run dtor in-function) → ~122 ops/layer-block × 40 blocks ≈ 4,900/request.
   • 22 OpenMP sites: 21 GOMP_parallel (17 fixed num_threads=4, 4 env) + 1 GOMP_parallel_sections (num_threads=3) in expert/linear prefill contexts + cpu_func — fixed counts ignore OMP_NUM_THREADS.
4. Overlap attempts fail (measured):
   • Exec-burst probe on a validated GEMM (M=128 K=2048 N=8192, kernel "MLIR_AIE"): 32 back-to-back launches = 1.07× speedup; submit_all = 155 ms → each launch blocks until the previous exec completes (per-kernel single command slot; explicit run.start() rejected "bad command state"). amdxdna/XRT is strictly serial per kernel/hwctx.
   • Two full-model FLM instances (separate hwctx): concurrent requests took 43 s TTFT each vs ~3 s solo (full-array execs multiplex on the single AIE array with severe contention). Not a parallelism path.

The fix (concatenated-expert dispatch) — validated kernel already exists

The fine granularity is FLM dispatching the top-8+shared experts individually. A concatenated-expert GEMM (one run per operator per layer-block) would cut ~122 → ~10–20 ops/block, est. floor ~2.7 s → ~0.3–0.7 s. We validated this exact kernel on-box: final_i8_MOE_GU_qwen3.6-moe_35b.xclbin (concatenated expert GU, N = TOP_K×2×IM_EXP) matches CPU math (rel RMSE < 0.05). CPU-side per-expert GLU (GOMP regions in expert_prefill_context::forward) should likewise become one parallel region over all experts.

What we tried that did NOT move the floor (so others don't repeat)

--pmode turbo; keep-alive reuse; hwctx context_limit=64; OMP_WAIT_POLICY=passive/GOMP_SPINCOUNT=0/OMP_NUM_THREADS=8 (env applied, no TTFT effect); BO pooling (ceiling ~0.07 s — BO ioctls are 3.5% of syscall time); async submission (impossible: per-hwctx serial); multi-instance fan-out (43 s contention).

Request

Please look at (a) making the MoE prefill dispatch concatenated/per-expert-group runs instead of per-expert runs, and (b) batching buffer-object reuse across the per-expert loop. Happy to test patches; the full corpus (inventories, disasm, classmaps, op-gen map, burst probe source) is available on request.

Full structured report also maintained locally (extraction corpus with 22-site GOMP table, driver trace artifacts, and fix-design docs). Reference: #636 (Windows, v0.9.45 — same "no persistent state / dispatch granularity" theme; this is the Linux v1.0.4 root-cause + validated fix direction).

Start by locating qwen3_6_moe_npu::Impl::prefill, _prefill_with_mm, and qwen3_6_moe_expert_prefill_context::forward in the MoE prefill implementation. Reproduce the reported small-prompt TTFT and inspect the per-expert buffer and run lifecycle. Done means preserving CPU math while reducing serialized dispatches through grouped or concatenated expert execution, with benchmark evidence showing lower fixed TTFT.