IOCTL regression using ASR and embedding models simultaneously (Krackan Point)

Hardware: AMD Ryzen AI 350 (Krackan Point) chip with XDNA2 support on a Framework 13 mainboard. Running NixOS with the nix-amd-ai flake to install FastFlowLM, Lemonade, and other backends.

You can see https://github.com/noamsto/nix-amd-ai/issues/79 for earlier discussion and bisecting of this issue. TL;DR: after FastFlowLM 0.9.44, FLM stopped being able to load and run any models on my Krackan Point system, with an IOCTL-related error. The downstream repository author helped me bisect the issue and ruled out it being a difference in kernel version, since downgrading to 0.9.44 on my up-to-date (kernel 7.1.3) system allowed models to run again. Note that the same error appeared regardless of the model chosen, and regardless of whether it was run with sudo or not:

❯ flm run gemma4-it:e4b
[FLM]  Loading model: /home/example/.config/flm/models/Gemma4-E4B-IT-NPU2
[ERROR]  Failed to load model: DRM_IOCTL_AMDXDNA_CREATE_HWCTX IOCTL failed (err=-22): Invalid argument

The kernel logs also included several related lines:

[84319.443131] amdxdna 0000:c2:00.1: [drm] *ERROR* aie2_send_mgmt_msg_wait: command opcode 0x2 failed, status 0x2000003
[84319.443143] amdxdna 0000:c2:00.1: [drm] *ERROR* aie2_hwctx_init: Alloc hw resource failed, ret -22
[84319.443198] amdxdna 0000:c2:00.1: [drm] *ERROR* amdxdna_drm_create_hwctx_ioctl: Init hwctx failed, ret -22

Other relevant output:

❯ flm validate
[Linux]  Kernel: 7.1.3
[Linux]  NPU: /dev/accel/accel0 with 8 columns
[Linux]  NPU FW Version: 1.1.2.64
[Linux]  amdxdna version: 0.8
[Linux]  Memlock Limit: infinity

c2:00.1 Signal processing controller [1180]: Advanced Micro Devices, Inc. [AMD] Strix/Krackan/Strix Halo Neural Processing Unit [1022:17f0] (rev 20)
	Subsystem: Framework Computer Inc. Device [f111:000b]
	Kernel driver in use: amdxdna
	Kernel modules: amdxdna

Plus some troubleshooting steps from an older kernel and FLM version:

❯ journalctl -k -b | grep -iE 'amdxdna|pasid|carveout|sva'
Aug 05 14:01:09 pike kernel: amdxdna 0000:c2:00.1: [drm] Load firmware amdnpu/17f0_10/npu_7.sbin
Aug 05 14:01:09 pike kernel: amdxdna 0000:c2:00.1: enabling device (0000 -> 0002)
Aug 05 14:01:09 pike kernel: [drm] Initialized amdxdna_accel_driver 0.6.0 for 0000:c2:00.1 on minor 0

❯ cat /sys/bus/pci/devices/0000:c2:00.1/iommu_group/type
─────┬───────────────────────────────────────────────────────────────────────────────
     │ File: /sys/bus/pci/devices/0000:c2:00.1/iommu_group/type
─────┼───────────────────────────────────────────────────────────────────────────────
   1 │ identity
─────┴──────────────────────────────────────────────────────────────────────────────

❯ sudo grep -H . /sys/module/amdxdna/parameters/*
/sys/module/amdxdna/parameters/aie2_max_col:128
/sys/module/amdxdna/parameters/force_cmdlist:Y

Reproduce the failure with flm run on the AMD Ryzen AI 350/Krackan Point setup, comparing FastFlowLM 0.9.44 with the later version that fails. Start with the DRM_IOCTL_AMDXDNA_CREATE_HWCTX error, the amdxdna kernel log entries, and flm validate output. Done means identifying the regression and restoring model loading for the affected NPU configuration.