Persist structured checkpoint-I/O policy and runtime-image identity in startup telemetry
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
Motivation
The startup metrics added by #18608 provide model-loading timings, but effective checkpoint-I/O policy currently depends on parsing server logs. Missing logs or multiple primary/draft loading events can leave attribution incomplete. Metric rows also lack the launch-time runtime-image reference, making matched-image comparisons depend on external joins.
Proposed scope
- Forward the canonical finalized primary checkpoint-I/O status through existing startup metadata and /server_info, including requested/selected/activated/effective and fallback reason. Keep primary and draft attribution separate; unsupported/unobserved paths remain unknown.
- Prefer structured status in the CI collector, retaining explicitly labeled legacy-log fallback for older runtimes. Add separate timing and policy completeness indicators.
- Stamp the actual test-container launch reference at Kubernetes/Slurm launch sites, including architecture-correct runtime images, and persist it with provenance. Record an OCI digest only when actually supplied by a digest-pinned launch reference; tagged references do not prove a resolved digest.
- Add focused tests and document comparison semantics. No new CI stages or model launches.
Non-goals
No policy-allocation or loader behavior changes, native counterfactual eligibility probes, checkpoint fingerprints, slowest-rank timing/collectives, failed-start persistence, or launch-to-first-token instrumentation. Resolving the running digest for mutable image tags is separate infrastructure work.
Related: #18608, #17393. Independent of the 50/50 split in #19170.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace the existing startup metadata and /server_info paths, then inspect the CI collector and Kubernetes/Slurm launch sites. Confirm how primary and draft attribution, legacy-log fallback, completeness indicators, and launch-reference provenance are represented; add the focused tests and documentation needed to verify comparison semantics without adding CI stages or model launches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, python
- Domain
- infrastructure, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100