Run, observe, and analyze Slurm benchmarks

Priority Level

High

Task Summary

Implement deterministic benchmark expansion, batch execution, fresh-process observation, and analysis for the optional Slurm integration.

Technical Details & Implementation Plan

• Expand strict benchmark intent into deterministic child run configurations and immutable benchmark records.
• Submit child runs through the public Slurm execution service and return without resident monitoring.
• Persist benchmark-to-run identity so later processes can observe and analyze the same children.
• Refresh child state through normalized scheduler and persisted-state evidence.
• Compute stable aggregate analysis from validated child results while preserving failed, incomplete, and missing-run classifications.
• Expose equivalent Python service and CLI operations for benchmark run and analyze workflows.

Acceptance criteria

• Equivalent benchmark input expands to the same ordered child runs and digests.
• Benchmark submission returns after scheduling children and does not require a resident controller.
• Analysis works from a fresh process and never guesses success from incomplete evidence.
• Missing, failed, partial, stale, and scheduler-inconsistent child runs remain explicit in results.
• Local/fake tests cover expansion, submission, refresh, mixed outcomes, and deterministic analysis.
• Real-cluster acceptance validates at least one multi-run benchmark workflow before release.

Out of scope

• Interactive dashboards or resident monitoring.
• New benchmark algorithms unrelated to Slurm execution.
• Generic scheduler or platform adapters.

Investigation / Context

This is the benchmark implementation lane in #850. #865 and #872 explicitly exclude benchmark implementation, while #870 treats benchmark analysis as a final acceptance scenario.

Agent Plan / Findings

Reuse the same immutable planning, execution, and state contracts as ordinary runs; benchmark records should add hierarchy and analysis intent rather than a second control plane.

Dependencies

Depends on shared benchmark records in #873, fake infrastructure in #872, deterministic planning in #875, the client worker in #876, the public service foundation and operational run/observe capabilities from 874#1 and 874#2, the one-node runtime capability from 868#2, and the persistence, winner, observation, and reconciliation capabilities from 869#1, 869#2, and 869#3. It does not depend on distributed/failure hardening in 868#3, retry/collection in 869#4, or the later 874#3, 874#4, and 874#5 slices.