✨ Implement Slurm availability synchronization after the design decision

🤖 AI text below 🤖

Elevator pitch

Keep resource-hungry cluster jobs pending according to an explicit, tested availability policy rather than discovering predictable provider failures inside the payload.

Current state

The earlier proposal reserved all static license tokens using LICENSE_ONLY reservations. That remains a candidate, not an approved algorithm or failure policy.

Work and decisions

• Wait for the parent decision on facts, identity, policy, freshness, and recovery.
• Implement only the chosen Slurm mechanism, reusing provider-neutral checks where their semantics match.
• Preserve administrator-owned resources and explicitly define behavior for running jobs.
• Provide dry-run diagnostics, idempotence, ownership, and deployment/recovery instructions.
• Do not present a periodic timer without a proven failure/staleness policy as a complete safety guarantee.

Dependencies and release boundary

Blocked by 💡 Design QDMI availability-based Slurm scheduling. Static work: ✨ Centralize static-license QDMI deployment and optional SPANK injection. Provider facts are separate child studies.

Acceptance and validation

• The approved design is linked before implementation starts.
• A real isolated Slurm fixture proves pending/eligible transitions, running-job behavior, ownership, service failure, stale observations, and recovery.
• No provider or permission error silently grants availability.
• Implementation remains unscheduled until the design is accepted.

Non-goals

Blocking static Slurm deployment; treating LICENSE_ONLY or an accepting-jobs boolean as already selected; changing configured capacity without an explicit design.

Required counterexamples

Use explicit counterexamples: a service may accept queued submissions outside an execution window; an online or BUSY device may reject further submissions; an IDLE device may still deny a particular principal. Queue depth and IDLE/BUSY alone do not establish submission acceptance. Distinguish observed provider facts, identity/access, execution readiness, and administrator admission policy. Test unknown/stale observations and recovery without presenting a snapshot as a reservation.