flm-real segfaults on NPU firmware timeout when GPU is concurrently active via ROCm (Strix Halo) — companion to amd/xdna-driver#1605

Summary

Running flm serve (NPU) concurrently with a llama.cpp llama-server instance on the ROCm0 device (same iGPU) on Strix Halo reliably causes the NPU firmware to stop responding mid-command. The XRT layer throws a timeout exception which flm-real does not catch, so the process segfaults instead of returning a clean error to the caller. Filed as a companion to amd/xdna-driver#1605 — a driver collaborator (maxzhen) confirmed the underlying firmware timeout needs to be root-caused from FLM's side first (call stack into XRT + inferencing command/argument buffer content at the time of the hang), and that the segfault-on-timeout itself is FLM's responsibility to fix (uncaught exception from XRT).

Environment

• Hardware: AMD Ryzen AI Max+ 395 "Strix Halo" (Radeon 8060S iGPU, XDNA2 NPU), BIOS 03.03 (INSYDE)
• Distro/kernel: CachyOS, reproduced on both 7.1.5-1-cachyos and 7.1.8-1-cachyos
• NPU firmware version: 1.1.2.65
• XRT: 2.21.75-10.1, xrt-plugin-amdxdna 2.21.75-2.1
• Served via: lemonade-server (lemond 11.6.0) → FastFlowLM (flm-real), model qwen3.6-moe:35b-a3b, --ctx-len 32768
• Concurrent GPU workload: llama-server (llama.cpp b10333) serving a separate model on --device ROCm0 (HIP backend) on the same Radeon 8060S

Symptom

1. NPU model loads and serves requests normally in isolation.
2. Once a second, unrelated model is actively generating via llama.cpp on ROCm0 (same iGPU) at the same time, the NPU command queue stalls. lemond's watchdog observed:

   [Warn] (BackendWatchdog) FastFlowLM health probe failed 1/3 after 94s without observable progress
   

3. Shortly after, the kernel logs a firmware timeout (amdxdna ... aie2_set_cmd_timeout, identical Context ID: 13 / Context PC: 0x28b060ad every single time across 6 captured occurrences), and flm-real crashes:

   flm-real[<pid>]: segfault at ... in libxrt_coreutil.so.2.21.75 (or libqwen2_npu.so)
   

   systemd-coredump confirms SIGSEGV, backtrace bottoming out inside XRT/FLM's own runtime (no further symbols).
4. lemond detects the dead backend (CURL error: Server returned nothing) and auto-restarts flm-real, which reloads the model from scratch.

This is fully reproducible: 6 crashes captured across two different kernel versions, at both a very large NPU context length (240000) and a conservative one (32768) — ruling out context size as the trigger. The one variable that correlates exactly is the GPU side actively running over ROCm at the same time.

Timeline / correlation with ROCm

This box previously ran two models concurrently (one GPU, one NPU) stably for an extended period. The crashes began immediately after a system llama-cpp package providing ROCm/HIP support was installed, and llama-server started auto-selecting ROCm0 for GPU inference instead of Vulkan0 (both devices are available on this system). Full details, logs, and package-update timeline are in amd/xdna-driver#1605.

What I can provide

• Full kernel log around each crash, coredump backtraces (already posted in #1605)
• xrt-smi examine output
• Willing to test with logging/verbosity flags, a debug build, or specific FLM versions if that helps narrow down the call stack maxzhen asked for on the driver side
• Have not yet tested whether switching the GPU side to Vulkan0 avoids the timeout (currently prioritizing GPU throughput over that test) — can do this test if it would help isolate the issue

Ask

Per the driver-side discussion, could someone help identify:

1. What FLM/XRT call is in flight when the firmware stops responding (to help the driver team narrow down the FW-level root cause)?
2. Whether flm-real should be catching the XRT timeout exception and returning a clean error/retry instead of crashing?

Reproduce the crash with flm-real serving an NPU model while llama-server uses ROCm0, then inspect the kernel logs and coredump backtrace. Trace the in-flight XRT call and inferencing command or argument buffer at timeout, and verify that flm-real returns a clean error instead of segfaulting.