Compute nodes don't get terminated despite Slurm indicating idle~ status

CycleCloud version: 8.6.2-3276
Slurm version: 22.05.11

Autoscaling down after the job queue gets empty worked for me successfully numerous times until it didn't after full occupancy of the cluster lasting several days. All jobs were then killed, the compute nodes transitioned to `idle~`, but CycleCloud didn't deprovision the VMs. How can I investigate the cause of this behavior?

Start by reviewing the interaction between CycleCloud 8.6.2-3276 and Slurm 22.05.11 when jobs are killed and compute nodes enter idle~ status. Reproduce the behavior after sustained full cluster occupancy and inspect the relevant logs to identify why deprovisioning is not triggered. Done means the cause is documented and node termination resumes after the queue becomes empty.

Written by the indexing model from the issue text.