Implement Slurm submission and allocation runtime

Priority Level

High

Task Summary

Implement the structured Slurm control plane and allocation-local runtime that execute a resolved Data Designer Slurm plan with deterministic steps, readiness reporting, and one cleanup owner.

Technical Details & Implementation Plan

• Render batch directives and submission arguments only from validated structured fields.
• Submit, query, and cancel jobs through a typed command client with normalized scheduler observations.
• Preserve accounting lag as an explicit nonterminal observation until the configured reconciliation deadline.
• Stage a versioned, checksummed runtime bundle and render one thin generation entrypoint per attempt.
• Verify the runtime bundle and resolved plan before allocation-local execution.
• Centralize plan reads, step construction, per-node process launch, process identity tracking, readiness transitions, endpoint publication, and cleanup.
• Construct every allocation-local step through one common runner using argument vectors rather than shell fragments.
• Start the zero-GPU client preflight before model services, then launch coordinated deployments, wait for readiness, publish logical endpoints, run the client, validate candidate results, and finalize.
• Route normal exit, server/rank failure, client failure, signals, cancellation, timeout, and partial startup through idempotent cleanup.

Acceptance criteria

• Structured rendering rejects control characters and never evaluates user-provided shell text.
• Submission and observation parsers handle arrays, active states, terminal states, malformed output, tool failures, and accounting lag.
• Runtime checksum, plan checksum, GPU-shape, host, path, and port failures occur before server launch.
• Required server-step exit fails the attempt and leaves no managed child process behind.
• Readiness updates use the reviewed attempt-readiness writer and do not introduce a second schema.
• Signal, cancellation, partial-start, readiness-timeout, follower-failure, and cleanup re-entry tests pass against fake infrastructure.
• No unrestricted ambient environment forwarding is used.

Out of scope

• Authored configuration and plan compilation.
• Image registry implementation.
• Persistent retry, winner, and collection policy.
• In-memory resident monitoring.

Investigation / Context

This is the scheduler and allocation-runtime lane for #850. Submission returns after Slurm accepts work; later commands observe persisted state from fresh processes.

Agent Plan / Findings

Land the command client and renderer first, followed by runtime staging, common step/process helpers, readiness integration, and failure-injection tests.

Planned PR slices

1. Command client and renderer

   • Status: Incomplete — PR #892
   • Implement typed submit/query/cancel commands, normalized scheduler parsers, safe batch directives, and the thin script renderer.
   • This slice supports the M0 plan/render proof and supplies scheduler primitives for #867 and M1.

2. One-node allocation runtime

   • Status: Incomplete — PR not opened
   • Add checksummed runtime staging, resolved-plan verification, the common step runner, process supervision, readiness transitions, client sequencing, finalization, and cleanup.
   • Integrate the completed contracts and APIs from #866, #867, #875, #876, and the foundational state-writer slice of #869.
   • This slice supports the M2 proof.

3. Distributed execution and failure hardening

   • Status: Incomplete — PR not opened
   • Complete multi-node lane execution, follower-failure handling, cancellation, signals, timeouts, partial startup, backpressure, cleanup re-entry, and no-orphan coverage.
   • This slice supports the post-M2 proofs and completes the issue.

Each slice is an independently reviewable PR against feat/slurm-execution. Intermediate PRs reference Part of #868; only the final PR uses Closes #868.

Dependencies

Blocked by the shared-contract and fake-infrastructure foundation work tracked by #850.