Curious stall during vae decode while idle

I set the workflow to auto queue and everything works as expected as long as I'm active at the computer, tho once I leave the computer idle, let's say over night, the thing gets stuck at vae decode for several hours. This is a Linux box running kernel 6.1 and rocm 5.4 on top of xf86-video-amdgpu-23.0 driver.

So my first take was to disable screen blanking and dpms without success. Changing gpu power profile from auto to performance and cpu governor from ondemand to performance didn't help either. Creating artificial cpu load was pointless as well. Finally running glxgears seemed to help in so far that the stall was reduced to maybe 15 minutes. If I enable --highvram as well in combination with running glxgears then vae decode seems to work as expected.

I'm aware blaming Comfy might be wrong but I have no clue what to blame at this point ;)