[Issue]: KFD runlist oversubscription at exactly 8 CP queue slots causes sclk pin + preemption hangs on RDNA2 (gfx1030) under 3+ concurrent ROCm processes — root cause + fix (amdgpu.num_kcq)

Title

[Issue]: KFD runlist oversubscription at exactly 8 CP queue slots causes sclk pin + preemption hangs on RDNA2 (gfx1030) under 3+ concurrent ROCm processes — root cause + fix (amdgpu.num_kcq)

Environment

• GPU: AMD Radeon PRO V620 (Navi 21 / gfx1030)
• OS: Ubuntu 24.04.4 LTS
• Kernel: 6.8.0-137-generic
• ROCm: 7.2.4
• Workload: 2-3 concurrent llama.cpp server processes (HIP backend), each an independent long-lived process holding its own HSA queues — not a single multi-stream process.

Symptom

Under sustained 3-process concurrency on one card:

• GPU core clock (sclk) pins at maximum continuously, even when all three processes are otherwise idle between requests (~42 W idle-but-pinned vs a true idle floor well under that).
• Reported GPU busy% reads ~99% even when no process is actively decoding — the busy metric does not reflect real work.
• Under load, threads intermittently hang stuck in ioctl on /dev/kfd inside libhsa-runtime64, requiring a process kill to recover.

Root cause (traced against kernel 6.8 amdgpu/amdkfd source, verified against sysfs on this box)

Navi 21 exposes 16 HQD compute-queue slots on MEC0. The amdgpu kernel driver reserves amdgpu.num_kcq of those as kernel compute rings — default 8 — leaving exactly 8 CP queue slots for all ROCm userspace processes on the card combined.

Each concurrent HIP process consumes multiple CP queues (in our testing, ~3 per process: 2 HIP-visible + 1 ROCr-internal queue that isn't covered by GPU_MAX_HW_QUEUES). At 3 concurrent processes this reaches or exceeds the 8-queue budget. KFD's runlist-size calculation (pm_calc_rlib_size()) detects the oversubscription and emits a chained runlist instead of a flat one; the MEC hardware scheduler then round-robins the queue sets on a fixed timer quantum indefinitely. This means the CP front-end never truly idles (the sclk pin, and the misleading busy% reading), and the constant evict/preempt cycling this produces exercises gfx10's known-fragile preemption path — which is where we see the hangs.

This budget is considerably tighter than AMD's own Instinct-class oversubscription guidance implies (that guidance describes 24 queues; gfx9 has 8 queues/pipe where RDNA2 has 4).

Fix, measured

1. Partial mitigation: GPU_MAX_HW_QUEUES=2 per process. Caps the HIP-visible queue count per process (AMD's own workload-tuning guidance already recommends =2 for single-stream llama.cpp-style workloads), no throughput cost measured for a single compute stream. This raises the ceiling from ~daily failures to only failing under full 3-process co-residency, because the ROCr-internal queue isn't covered by this cap (still ~3 queues/process floor).
2. Full fix: amdgpu.num_kcq=2 kernel module parameter, raising the userspace CP queue budget from 8 to 14. With this + the GPU_MAX_HW_QUEUES=2 cap, 3 fully concurrent processes measured at 9/14 queues used, zero oversubscription lines in dmesg, and — notably — the card reached a genuine 3-process idle state for the first time (sclk 0, ~7 W, 0% busy), confirming the busy%/clock-pin symptom above was entirely the oversubscription artifact, not real GPU load.

Question for maintainers

Is the num_kcq=8 default (reserving half of Navi 21's 16 HQD slots for kernel rings) intended to leave RDNA2 users this little multi-process headroom, or is this closer to a bringup-era default that hasn't been revisited for the "several independent long-lived ROCm processes sharing one consumer/workstation card" use case? Related reports we found describe the same symptom class on RDNA3/RDNA4 (pinned clocks, hangs that don't reproduce on Vulkan) but none isolated this specific mechanism:

• ROCm/TheRock#5793 (gfx1102, resolved via sched_policy=1 rather than the queue budget)
• ROCm/ROCm#5706 (gfx1201, unresolved on ROCm, users moved to Vulkan)
• ROCm/amdgpu#153 (RDNA3, multiple HW queues via MES, 100% GPU usage)

Happy to provide the full sysfs dump, dmesg oversubscription lines, or test additional configurations if useful.

Start by reviewing the kernel 6.8 amdgpu/amdkfd source around pm_calc_rlib_size() and the amdgpu.num_kcq parameter. Reproduce the three-process workload, then inspect sysfs and dmesg for queue usage and oversubscription lines. Done means determining whether the default is intentional and validating any proposed queue-budget change without the reported hangs.