Dynamic nodes stuck in `IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING` are permanently unschedulable, invisible to `sinfo -R`, and never remediated by clustermgtd

This was a tough one for me to debug, had to have help from claude.ai. Here is the issue as summarized by claude.

## Summary

A dynamic (cloud) compute node can end up in state
`IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING`. Slurm excludes such a node from scheduling, so it
is never selected for a job and `ResumeProgram` is never invoked — no EC2 API call is ever
made for it. clustermgtd does not recognize this state as unhealthy, so nothing ever clears
it. The node is invisible to normal monitoring: `sinfo -R` reports nothing, there is no
reason string, and the base state reads `IDLE`.

The result is silent, permanent capacity loss. On our cluster 250 of ~1720 dynamic nodes were
in this state across 30 compute resources, with 11 compute resources having **zero** usable
nodes. Two production queues had been running at half capacity for at least nine days. Jobs
pended with `Nodes required for job are DOWN, DRAINED or reserved for jobs in higher priority
partitions`, which points an operator at the wrong cause — nothing was DOWN or DRAINED.

## Environment

- ParallelCluster 3.13.2 (`aws-parallelcluster-cookbook-3.13.2`)
- Slurm 24.05.8
- Dynamic nodes only. Compute resources spanning on-demand and spot, many instance families.

## The effect is deterministic and easy to demonstrate

`sbatch --test-only` shows the node is not merely busy, it is not a candidate at all. Three
compute resources, same cluster, same moment:

```
# all 10 nodes in the latched state
$ sbatch --test-only -p od-c7a-2xl -c 1 --mem=1G --wrap="true"
allocation failure: Requested node configuration is not available

# nodes 1-2 clean, nodes 3-10 latched -> only a clean node is ever offered
$ sbatch --test-only -p od-m7a-xl -c 1 --mem=1G --wrap="true"
sbatch: Job ... to start at ... on nodes od-m7a-xl-dy-od-16-gb-4-cores-1

# all nodes clean
$ sbatch --test-only -p od-r7a-2xl -c 1 --mem=1G --wrap="true"
sbatch: Job ... to start at ... on nodes od-r7a-2xl-dy-od-64-gb-8-cores-2
```

Meanwhile the affected nodes look healthy by every normal check:

```
$ sinfo -R
REASON USER TIMESTAMP NODELIST
# empty

$ scontrol show node od-r7i-12xl-dy-od-384-gb-24-cores-6
State=IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING
Reason= # none
BootTime=None SlurmdStartTime=None
```

## Why nothing recovers it

Two independent gaps that compose:

1. **Slurm cannot clear the flag.** `NOT_RESPONDING` is cleared when a node answers a ping,
and powered-down cloud nodes are deliberately never pinged — correctly, there is no
instance to ping. For a powered-down node the flag is therefore cleared by an event that
can never occur. Arguably the flag is also semantically meaningless for a node Slurm
itself powered off.

2. **clustermgtd does not consider it unhealthy.** Its health checks key on base state and on
EC2 reality. This node reads base state `IDLE` with no backing instance, which is exactly
what a correctly powered-down dynamic node looks like. It is in none of the remediation
sets. clustermgtd ran continuously for two months on our cluster, looping every 60
seconds, and never flagged any of the 250 nodes.

Compounding this, ParallelCluster sets `idle_on_node_suspend` itself:

```
/opt/slurm/etc/slurm.conf:39
SlurmctldParameters=idle_on_node_suspend,power_save_min_interval=30,node_reg_mem_percent=75
```

That converts the base state from `DOWN` to `IDLE` on suspend. A node left at
`DOWN+...+NOT_RESPONDING` would have appeared in `sinfo -R` and in clustermgtd's unhealthy
list. Flipping it to `IDLE` removes the only signal an operator would have had, which is why
this is silent rather than merely annoying.

The state also survives a slurmctld restart, since it is restored from `StateSaveLoc`:

```
[2026-08-20T11:32:21.535] Recovered state of 1936 nodes
[2026-08-20T11:32:21.535] Down nodes: sp-r7i-l-dy-sp-16-gb-1-cores-2
```

1936 nodes recovered, one reported down, 250 silently unschedulable.

## What I could not determine

I cannot tell you what *creates* the state, and I want to be upfront that on 3.13.2 every
path I traced self-heals correctly:

- **Spot reclamation.** Traced seven nodes end to end. clustermgtd logged `no corresponding
instance in EC2` / `EC2 instance availability for node ... has timed out` / `Setting
unhealthy dynamic nodes to down and power_down`, and all seven landed at clean
`IDLE+CLOUD+POWERED_DOWN`. No latch.
- **Resume capacity failure.** Produces `update_node: node ... reason set to:
(Code:UnfulfillableCapacity)Failure when resuming nodes` then `state set to DOWN` — visible
in `sinfo -R`, and reset by the insufficient-capacity timeout. Working as intended.
- **slurmctld restart.** See above, does not create it.

The affected nodes had **zero** events in six-plus weeks of retained logs (`clustermgtd` back
to Jul 8, `slurm_suspend.log` back to Jul 25, `slurm_resume.log`, `slurmctld.log`). So these
were most likely created by an older version or a one-off control-plane interruption and then
never swept up. My origin logs have long since rotated.

## Why this is still actionable

Because the absence of a repro does not reduce the impact. Even if the bug that creates this
state is already fixed, **there is no remediation for nodes already in it.** It survives
cluster updates and daemon restarts, and it is invisible. A cluster affected before an upgrade
stays permanently degraded afterwards with no signal at all. Ours lost half of two queues for
at least nine days; one user's job waited 39.9 hours and then failed with `ReqNodeNotAvail`.

## Requests

1. **Remediate it.** Have clustermgtd clear a stray `NOT_RESPONDING` flag on a powered-down
dynamic node that has no backing instance. This is unambiguously safe: there is no
instance, nothing is running, and the flag conveys no information about a machine that
does not exist.
2. **Failing that, surface it.** A node that Slurm will not schedule should be visible
somewhere — a reason string, a clustermgtd warning, anything. Silence is the expensive part.
3. **Consider whether `idle_on_node_suspend` should apply** to a node carrying
`NOT_RESPONDING`, given it is what suppresses the `DOWN` state that would otherwise have
made this observable.

Related in class, though a different trigger and a much louder symptom: #7470, where
clustermgtd likewise does not recognize a node state it did not create
(`DOWN+CLOUD+MAINTENANCE+RESERVED+NOT_RESPONDING`).

## Workaround, for anyone who finds this

Detect:

```
scontrol show nodes --oneliner \
| awk '{for(i=1;i<=NF;i++){if($i ~ /^NodeName=/)n=substr($i,10); if($i ~ /^State=/)s=substr($i,7)}
if(s=="IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING") print n}'
```

Fix:

```
scontrol update nodename= state=power_down_force
```

The nodes cycle `POWER_DOWN` -> `POWERING_DOWN` -> clean `IDLE+CLOUD+POWERED_DOWN` within a
couple of clustermgtd passes, and pending jobs start once they land. On our cluster five jobs
that had been pending 11.5 hours started within a few minutes of the reset.

Note for anyone scripting this: match the state by whole `+`-delimited token, not substring —
`POWERED_DOWN` contains `DOWN`. And never reset a node in `POWERING_UP`, where
`NOT_RESPONDING` is legitimate and transient while the instance boots.

Start with clustermgtd's dynamic-node health checks and the state behavior described for /opt/slurm/etc/slurm.conf, then inspect how it handles IDLE+CLOUD+POWERED_DOWN+NOT_RESPONDING nodes without an EC2 instance. Use the scontrol state and power_down_force commands from the report to validate the scenario. Done means affected nodes are remediated or surfaced with an actionable signal without resetting legitimate POWERING_UP nodes.

Written by the indexing model from the issue text.