Expose Slurm services, CLI, packaging, and documentation

Priority Level

High

Task Summary

Implement the public Python services, command-line surface, packaging integration, and user documentation for Slurm batch execution while preserving a lightweight base Data Designer installation.

Technical Details & Implementation Plan

• Expose package-owned service APIs for profile management, execution, observation, cancellation, retry, collection, images, and benchmarks.
• Implement the data-designer slurm command tree described by #850 using lazy CLI extension discovery.
• Keep command handlers thin: load strict input, call a service, render a stable result, and normalize errors.
• Complete optional-extra and release metadata without importing the Slurm package from base-only commands.
• Add Fern documentation for installation, configuration, execution, status, retry, collection, images, benchmarks, and troubleshooting.
• Verify examples and docs against built wheels rather than a source checkout.

Acceptance criteria

• Base-only installation does not install, import, or display Slurm functionality.
• Selecting a Slurm command loads the optional extension and exposes the documented command tree.
• Every public command has an equivalent Python service operation and stable result/error behavior.
• Root CLI import and help performance remain within the existing budgets.
• Built-wheel tests cover base-only, extra-based, and leaf-package installation paths.
• Fern documentation matches the shipped command and configuration surfaces.

Out of scope

• Scheduler command implementation.
• Plan compilation, serving, image internals, client execution, state persistence, or benchmark algorithms owned by their respective lanes.
• Interactive sessions or resident monitoring.

Investigation / Context

#852 and #853 established the optional package and lazy command-discovery foundations. This task owns the complete public surface that consumes the implementation lanes tracked by #850.

Agent Plan / Findings

Build services first, then make CLI commands and documentation thin consumers of those reviewed public operations.

Planned PR slices

1. Public service foundation

   • Status: Complete - PR #894
   • Implement typed service facades and stable result/error contracts against reviewed contracts and deterministic fakes.
   • Provide the service boundary needed for the M0 proof.

2. Execute, status, cancel, and image wiring

   • Status: Complete - PRs #926 and #933
   • Integrate configuration, planning, serving, image, runtime, client-worker, and state capabilities from #875, #866, 867#3, 868#2, #876, 869#1, 869#2, and 869#3.
   • Add thin profile, execute, status, cancel, and image commands.
   • This slice provides the operational run/observe surface needed for M1, M2, and #877.

3. Retry and merge wiring

   • Status: Complete - PR #931
   • Integrate retry and collection services after 869#4 provides the completed retry/collection capability.
   • Add thin retry and merge commands with stable result and error rendering.
   • This slice supports M3c.

4. Benchmark and workflow integration

   • Status: Complete - PR #932
   • Integrate benchmark service and CLI operations after #877 provides benchmark expansion, observation, and analysis.
   • Complete the corresponding user-workflow documentation.
   • This slice supports M3d.

5. Documentation, packaging, and release integration

   • Status: Incomplete — PR not opened
   • Complete Fern navigation and documentation, built-wheel examples, dependency auditing, installation coverage, and four-package release integration.
   • This slice completes the issue after the implementation APIs stabilize.

Each slice is an independently reviewable PR against feat/slurm-execution. Intermediate PRs reference Part of #874; only the final PR uses Closes #874.

Dependencies

Depends on #852 and #853 and the shared contracts in #873. Those foundation prerequisites are complete. Slice 874#1 can begin against reviewed contracts and deterministic fakes. Slice 874#2 consumes #875, #866, 867#3, 868#2, #876, 869#1, 869#2, and 869#3. Slice 874#3 consumes 869#4. Slice 874#4 consumes #877. #877 depends only on 874#1 and the operational run/observe surface from 874#2, not completion of #874.