ROCm / ROCm/rocm-cli

[Issue]: AsyncEventsLoop busy-spins at 100% of one CPU core after ComfyUI inference on Ryzen AI MAX+ 395 (gfx1151)

Open
#325 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
40
Forks
9
Avg merge
4d 20h
Merged PRs (30d)
59

Description

Problem Description

After ComfyUI completes an image generation, the ROCr runtime leaves one host CPU logical core continuously busy at approximately 100% while the application is otherwise idle.

perf attributes virtually all sampled CPU cycles to:

rocr::core::Runtime::AsyncEventsLoop(void*)

in:

libhsa-runtime64.so.1

The process remains at 100% CPU indefinitely until ComfyUI/the Python process is stopped or restarted.

Expected behavior

After an inference request completes and ComfyUI is idle, ROCr helper threads should block or use negligible CPU. One host CPU core should not remain fully utilized indefinitely.

Actual behavior

After generating one image with ComfyUI and waiting for the UI to become idle:

  • The python main.py process remains at approximately 100% CPU.
  • The busy thread may be scheduled on different logical CPUs over time.
  • The issue persists indefinitely while the process remains alive.
  • The behavior occurs after each image-generation session.
  • The application itself is idle; ComfyUI's main asyncio thread is blocked in select() and its prompt worker is waiting on its work queue.
  • The CPU time is spent in ROCr's AsyncEventsLoop, not in ComfyUI Python code.

This causes unnecessary CPU power usage, heat/fan activity, and permanently consumes one CPU core while the inference server is idle.

OS:
NAME="Ubuntu"
VERSION="26.04 LTS (Resolute Raccoon)"
CPU:
model name : AMD RYZEN AI MAX+ 395 w/ Radeon 8060S
GPU:
Name: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S
Marketing Name: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S
Name: gfx1151
Marketing Name: AMD Radeon Graphics
Name: amdgcn-amd-amdhsa--gfx1151
Name: amdgcn-amd-amdhsa--gfx11-generic
Name: aie2p
Marketing Name: RyzenAI-npu5

=== ROCm packages ===
amdgpu-core 1:7.2.70201-2303469.24.04
amdgpu-install 30.30.1.0.30300100-2303411.24.04
hip-base
hipblas 3.2.0.70201-8124.04
hipblas-common
hipblas-common-dev 1.4.0.70201-81
24.04
hipblas-dev 3.2.0.70201-8124.04
hipblaslt 1.2.2.70201-81
24.04
hipblaslt-dev 1.2.2.70201-8124.04
hipcc 1.1.1.70201-81
24.04
hipcub
hipcub-dev 4.2.0.70201-8124.04
hip-dev 7.2.53211.70201-81
24.04
hip-doc 7.2.53211.70201-8124.04
hipfft 1.0.22.70201-81
24.04
hipfft-dev 1.0.22.70201-8124.04
hipfort
hipfort-dev 0.7.1.70201-81
24.04
hipify-clang 22.0.0.70201-8124.04
hiprand 3.1.0.70201-81
24.04
hiprand-dev 3.1.0.70201-8124.04
hip-rocclr
hip-runtime-amd 7.2.53211.70201-81
24.04
hip-samples 7.2.53211.70201-8124.04
hipsolver 3.2.0.70201-81
24.04
hipsolver-dev 3.2.0.70201-8124.04
hipsparse 4.2.0.70201-81
24.04
hipsparse-dev 4.2.0.70201-8124.04
hipsparselt 0.2.6.70201-81
24.04
hipsparselt-dev 0.2.6.70201-8124.04
hipstdpar
hiptensor 2.2.0.70201-81
24.04
hiptensor-dev 2.2.0.70201-8124.04
hip-thrust
hsa-amd-aqlprofile 1.0.0.70201-81
24.04
hsa-ext-rocr-dev
hsakmt-roct
hsakmt-roct-dev
hsa-rocr 1.18.0.70201-8124.04
hsa-rocr-dev 1.18.0.70201-81
24.04
rocm 7.2.1.70201-8124.04
rocm-cmake 0.14.0.70201-81
24.04
rocm-core 7.2.1.70201-8124.04
rocm-dbgapi 0.77.4.70201-81
24.04
rocm-debug-agent 2.1.0.70201-8124.04
rocm-developer-tools 7.2.1.70201-81
24.04
rocm-device-libs 1.0.0.70201-8124.04
rocm-gdb 16.3.70201-81
24.04
rocm-hip 7.2.1.70201-8124.04
rocminfo 1.0.0.70201-81
24.04
rocm-llvm 22.0.0.26084.70201-8124.04
rocm-opencl 2.0.0.70201-81
24.04
rocm-opencl-dev 2.0.0.70201-8124.04
rocm-opencl-icd
rocm-opencl-icd-loader
rocm-opencl-sdk 7.2.1.70201-81
24.04
rocm-openmp 7.2.1.70201-8124.04
rocm-smi-lib 7.8.0.70201-81
24.04

=== ROCr library package ===
hsa-rocr: /opt/rocm-7.2.1/lib/libhsa-runtime64.so.1

=== GPU ===
Runtime Version: 1.18
HSA Agents
Agent 1
Name: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S
Marketing Name: AMD RYZEN AI MAX+ 395 w/ Radeon 8060S
Vendor Name: CPU
Agent 2
Name: gfx1151
Marketing Name: AMD Radeon Graphics
Vendor Name: AMD
Name: amdgcn-amd-amdhsa--gfx1151
Name: amdgcn-amd-amdhsa--gfx11-generic
Agent 3
Name: aie2p
Marketing Name: RyzenAI-npu5
Vendor Name: AMD

=== Kernel AMDGPU ===
srcversion: 3C579E7F273939F4A3F71D3
vermagic: 7.0.0-30-generic SMP preempt mod_unload modversions

=== Container image digest ===
ID=sha256:384aa1fecef6a841832e0d5552949977330308d8c25e212a94f5e8dfcc061cae
Digests=kyuz0/amd-strix-halo-comfyui@sha256:384aa1fecef6a841832e0d5552949977330308d8c25e212a94f5e8dfcc061cae

=== Container HSA environment ===
HSA_ENABLE_MWAITX=1

=== PyTorch / HIP ===

Reproduction steps

  1. Start the ComfyUI container with access to /dev/kfd and /dev/dri.
  2. Start ComfyUI with a ROCm-capable PyTorch build.
  3. Generate one image successfully.
  4. Wait until generation has fully completed and ComfyUI is otherwise idle.
  5. Monitor the host:
PID=$(docker inspect -f '{{.State.Pid}}' comfyui)
pidstat -p "$PID" 1 10
  1. The Python process continuously uses approximately 100% of one logical CPU:
Average:        0   <PID>  99.90    0.00    0.00    0.00   99.90     -  python
  1. Profile it from the host:
PID=$(docker inspect -f '{{.State.Pid}}' comfyui)

sudo perf record -F 99 -p "$PID" -g -- sleep 30
sudo perf report --stdio --no-children

perf result

A representative profile:

Samples: 2K of event 'cpu/cycles/P'

Event count (approx.): 148883194143

Overhead Command Shared Object Symbol

........ ....... ..................... ...............................................

99.83%  python   libhsa-runtime64.so.1  [.] rocr::core::Runtime::AsyncEventsLoop(void*)
        |
        ---rocr::core::Runtime::AsyncEventsLoop(void*)

 0.03%  python   [kernel.kallsyms]      [k] amdgpu_cgs_read_register
 0.03%  python   [kernel.kallsyms]      [k] read_tsc
 0.03%  python   [kernel.kallsyms]      [k] tick_nohz_handler
 0.03%  python   [kernel.kallsyms]      [k] _find_next_and_bit
 0.03%  python   [kernel.kallsyms]      [k] native_irq_return_iret
 0.00%  python   libc.so.6              [.] __errno_location
 0.00%  python   ld-linux-x86-64.so.2   [.] _dl_tlsdesc_return
 0.00%  python   [kernel.kallsyms]      [k] __rcu_read_lock
 0.00%  python   [kernel.kallsyms]      [k] perf_event_update_userpage
 0.00%  python   [kernel.kallsyms]      [k] calc_timer_values
 0.00%  python   [kernel.kallsyms]      [k] local_clock
 0.00%  python   [kernel.kallsyms]      [k] native_write_msr
 0.00%  python   [kernel.kallsyms]      [k] perf_event_idx_default

In another capture without symbol resolution, the hot addresses were also in libhsa-runtime64.so.1:

     4.38%  python   libhsa-runtime64.so.1  [.] 0x00000000001217f1
     3.84%  python   libhsa-runtime64.so.1  [.] 0x0000000000121413
     3.54%  python   libhsa-runtime64.so.1  [.] 0x0000000000121433

Configuration tested

I explicitly tested:

HSA_ENABLE_MWAITX=1

The behavior persisted: after inference, AsyncEventsLoop still used approximately one full CPU core.

I did not set HSA_ENABLE_INTERRUPT=0, because ROCr documents that it disables hardware interrupts and changes completion detection to memory polling, which would not be an appropriate workaround for a busy-polling issue.

Relevant observations

  • The issue remains after disabling the ComfyUI-AMDGPUMonitor custom node.
  • With that monitor disabled, ComfyUI Python-level threads are idle according to py-spy:
    • Main thread: select() / asyncio event loop
    • Prompt worker: waiting on queue
  • Kernel-level perf consistently identifies ROCr AsyncEventsLoop as the source of CPU cycles.
  • The issue is reproducible after successful inference and does not require an active generation.

Request

Could AMD please confirm whether continuous busy spinning in rocr::core::Runtime::AsyncEventsLoop(void*) after an idle inference workload is expected on gfx1151 / Ryzen AI MAX+ 395?

If it is not expected, please advise whether there is:

  • a runtime fix,
  • a supported ROCr environment-variable workaround,
  • a required kernel/AMDGPU/ROCm package version combination,
  • or additional diagnostic information needed to isolate the cause.
Operating System

Ubuntu 26.04 LTS (Resolute Raccoon)

CPU

AMD RYZEN AI MAX+ 395 w/ Radeon 8060S

GPU

AMD RYZEN AI MAX+ 395 w/ Radeon 8060S

ROCm Version

ROCm 7.15

ROCm Component

No response

Steps to Reproduce

Reproduction steps

  1. Start the ComfyUI container with access to /dev/kfd and /dev/dri.
  2. Start ComfyUI with a ROCm-capable PyTorch build.
  3. Generate one image successfully.
  4. Wait until generation has fully completed and ComfyUI is otherwise idle.
  5. Monitor the host:
PID=$(docker inspect -f '{{.State.Pid}}' comfyui)
pidstat -p "$PID" 1 10
  1. The Python process continuously uses approximately 100% of one logical CPU:
Average:        0   <PID>  99.90    0.00    0.00    0.00   99.90     -  python
  1. Profile it from the host:
PID=$(docker inspect -f '{{.State.Pid}}' comfyui)

sudo perf record -F 99 -p "$PID" -g -- sleep 30
sudo perf report --stdio --no-children
(Optional for Linux users) Output of /opt/rocm/bin/rocminfo --support
rocminfo --support output
Paste output here
Additional Information

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the post-inference idle state using the ComfyUI container, then run the provided pidstat and perf commands to confirm CPU use in rocr::core::Runtime::AsyncEventsLoop(void*). Check the listed ROCm, kernel, GPU, and HSA_ENABLE_MWAITX configuration while isolating the runtime behavior. Done means determining whether this is expected and identifying a supported fix, workaround, or required version combination.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, linux, python, pytorch
Domain
devops, operating-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.