[Model Support]: Unofficial MiniCPM5-2B Q4NX pack for XDNA 2

Summary

Unofficial FastFlowLM / Q4NX build of openbmb/MiniCPM5-2B running on the AMD XDNA 2 NPU (no CPU/GPU decode).

• Model Weights & Precompiled AIE Kernels: huggingface.co/julianmb/MiniCPM5-2B-NPU2
• Dedicated Standalone Repro & Pipeline Repo: github.com/julianmb/minicpm5-xdna2
• Conversion Scripts & Tests: github.com/julianmb/npuhalo/tree/main/ports/minicpm5-2b
• OpenBMB Community Discussion: github.com/OpenBMB/MiniCPM/discussions/372

⚠️ Status / Triage Notice (42-Layer Runlist Timeout):

• Prefill: Passes cleanly on XDNA 2 (chunk 1/1 with 38 tokens).
• Decode: Trips {"error":"runlist failed execution (ERT_CMD_STATE_TIMEOUT)"} on the first token in libqwen3_npu.so.
• Root Cause: As confirmed and isolated by @Platano78, the qwen3 engine currently batches all layer forwards into a single monolithic xrt::runlist (working up to 36 layers on qwen3:4b, while gemma4e handles 42 layers on the same hardware).
• Reproducer: A self-contained 10-second reproducer is provided in scripts/reproduce_ert_timeout.py. Chunking decode forward passes into sub-runlists (e.g. 21 + 21 layers) in libqwen3_npu.so should unlock MiniCPM5 and other 40+ layer models.

Hardware Compatibility

• Verified Testbed: AMD Ryzen AI Max+ 395 (Strix Halo, 50 TOPS XDNA 2, /dev/accel/accel0)
• Expected Compatibility: Any AMD XDNA 2 silicon:
  • Strix Halo: Ryzen AI Max+ 395, 390, 385, PRO series
  • Strix Point: Ryzen AI 9 HX 375, HX 370, 365, PRO 300 series
  • Krackan / Kraken Point: Ryzen AI 7, Ryzen AI 5
• Runtime: FastFlowLM (tested with v1.0.1, v1.0.2, v1.0.4)

Measured & Projected Performance

• Prefill Speed (TTFT): 81.5 – 128.1 tok/s (~420 ms TTFT) — Verified passing
• Projected Decode Speed: ~63.1 – 63.6 tok/s on XDNA 2 (extrapolated from isolated single-layer GEMM latencies on the 1.7B twin; live generation currently blocked by 42-layer runlist timeout)
• Power Draw: ~2–4 W (leaves 100% of the iGPU and CPU free)
• Footprint: ~1.88 GB in NPU memory (Q4_1 / Q4NX layout)

What is in the Pack

• model.q4nx (~1.88 GB, Q4_1 / Q4NX)
• config.json
• Precompiled AIE kernels: attn.xclbin, mm.xclbin, layer.xclbin, dequant.xclbin
• MiniCPM5 tokenizer + chat template

How It Was Ported (GQA Firmware Workaround)

MiniCPM5-2B has 16 Query heads and 2 Key/Value heads ($16:2 = 8:1$ GQA ratio). The FastFlowLM AIE firmware (libmha.so) does not have a native $8:1$ kernel for $d_{head}=128$:

1. $4\times$ KV Head Replication: Replicated the 2 KV heads $4\times$ along dimension 0 into 8 KV heads ($16:8 = 2:1$ GQA ratio). Under Grouped Query Attention, this maintains exact mathematical equivalence while matching the native _gen_mha_seq_d128_q2 AIE kernel.
2. Qwen3 Runtime Routing: Dynamically dispatched $d_{head}=128$ via libqwen3_npu.so when intermediate_size == 6144.
3. Identity QK-Norm Injection: Synthetic unit RMSNorm tensors ($\gamma = 1.0$) were injected across all 42 layers in model.q4nx.

How to Run

mkdir -p ~/.config/flm/models
git clone https://huggingface.co/julianmb/MiniCPM5-2B-NPU2 ~/.config/flm/models/MiniCPM5-2B-NPU2

# Register under "models" in ~/.config/flm/model_list.json:
# "minicpm5:2b": {
#   "path": "~/.config/flm/models/MiniCPM5-2B-NPU2",
#   "model_type": "qwen3",
#   "tokenizer": "~/.config/flm/models/MiniCPM5-2B-NPU2"
# }

flm serve minicpm5:2b --host 127.0.0.1 --port 8001

Start with scripts/reproduce_ert_timeout.py in the linked minicpm5-xdna2 repository, then inspect the qwen3 engine producing libqwen3_npu.so and its decode runlist handling. Reproduce the 42-layer ERT_CMD_STATE_TIMEOUT and verify that splitting the decode forward pass into sub-runlists allows MiniCPM5 generation to complete on XDNA 2.