Persist allocation-local performance telemetry for Slurm runs and benchmarks

Priority Level

High

Task Summary

Persist bounded, allocation-local performance telemetry for Data Designer Slurm runs and make validated summaries available to benchmark analysis.

The Slurm integration already persists deterministic execution plans, image identities, dependency locks, run state, output counts, and aggregate benchmark timing. It does not currently collect the existing Data Designer OpenTelemetry endpoint or per-replica serving metrics during a run. Benchmark analysis can report rows per second and GPU hours, but it cannot explain utilization gaps or distinguish client scheduling, transport, and serving bottlenecks.

Technical Details & Implementation Plan

• Define a versioned, bounded performance-telemetry artifact associated with one run attempt.
• Collect metrics only inside the allocation and stop collection with the attempt. Do not require a resident external controller.
• Consume the existing Data Designer OpenTelemetry endpoint when enabled.
• Collect supported serving metrics per resolved backend without exposing unvalidated endpoints.
• Persist sample coverage, gaps, and source identity so missing telemetry remains explicit.
• Produce a validated summary containing, where available:
  • completed records and records per second;
  • request count and requests per second;
  • prompt and generated token counts and rates;
  • active model requests and request-duration distributions;
  • running and waiting serving requests;
  • KV-cache utilization and active-replica coverage;
  • scheduler/request-admission pressure summaries;
  • effective HTTP transport limits and pool waiters when the client exposes them.
• Record runtime-observed serving identity, including serving version and selected attention backend when the serving runtime exposes them. Preserve unknown rather than inferring runtime state from intended configuration.
• Extend benchmark analysis to consume only validated attempt telemetry and populate its existing optional request/token fields.
• Keep prompts, completions, row values, credentials, environment variables, hostnames, and environment-specific paths out of telemetry artifacts.
• Provide stable artifacts or an export boundary that external visualization systems can consume without adding a dashboard dependency to the package.
• Add deterministic fake-endpoint tests for complete samples, partial samples, unavailable metrics, malformed responses, bounded retention, interruption, and final summary validation.

Investigation / Context

Part of #850 and a follow-up to #877.

The current Slurm planning code fully materializes RunConfig and validates collisions for an explicitly configured OpenTelemetry port, but compatibility defaults disable the endpoint with otel_metrics_port=None. The runtime does not currently scrape or persist Data Designer metrics.

The benchmark implementation added by #877 reconstructs successful runs from durable state and computes boot time, generation time, wall time, rows per second, and GPU-hour recommendations. Its result schema also has optional request_count and token_count fields, but current observation derives only record and timing measurements.

#727 tracks durable engine capacity diagnostics for public async runs. This task should consume stable engine metrics and artifacts rather than duplicate scheduler instrumentation. Recent performance findings are tracked by #945 and #946; transport-pool metrics may depend on the client surface added for #946.

This task is intentionally separate from #877 because that issue scopes benchmark analysis to point-in-time persisted run evidence and excludes interactive dashboards or resident monitoring.

Agent Plan / Findings

The existing implementation already provides the identity and persistence foundations needed for telemetry:

• immutable resolved plans and topology;
• digest-bound client and serving images;
• dependency and installed-distribution records;
• attempt lifecycle and readiness records;
• validated client results and output manifests;
• fresh-process benchmark observation and analysis.

The missing seam is an attempt-owned metrics collector plus a stable telemetry artifact and summary contract. Implement the artifact and fake collector first, wire allocation-local collection into attempt lifecycle second, then extend benchmark observation and reports.

Dependencies

Part of #850. Follow-up to #877. Reuse the engine observability work tracked by #727. Transport-specific fields depend on the supported client transport-metrics surface tracked by #946.

Start with the existing Slurm planning code, attempt lifecycle, durable state, and benchmark observation and analysis described in the issue. Implement and test the attempt-owned artifact and fake collector first, then trace allocation-local collection into lifecycle records and validated benchmark inputs. Done means deterministic tests cover complete, partial, unavailable, malformed, bounded, interrupted, and validated-summary cases without exposing sensitive values.