Next release: data safety, scoring correctness, recovery and package validation checklist

Goal

Use this as the next-release quality checklist, with a bounded set of verified defects and existing review work rather than opening duplicate refactor requests. Suggested ordering below is for maintainer review; no release version or date is assumed.

Suggested release blockers: data safety and truthful results

• #361 — reproduction cleanup deletes unrelated user files in an existing --work-dir. Fix in the focused PR linked below; test PASS, FAIL, download/judge errors, interruption and overlapping invocations.
• #362 — reproduction ignores sample coverage and compares unrequested rubric columns as zero. Define valid full versus partial reproduction before returning PASS.
• #363 — a new judge model can silently reuse old verdicts and relabel the summary. Require truthful provenance/cache matching.

Recovery and existing CI coverage

• #364 — corrupt judge JSON is skipped before worker recovery. Share cache validation, use atomic verdict writes, and define partial-error reporting.
• #365 — runtime-server/ changes select no downstream native harness builds. Correct/test the existing dependency selector.

Review existing work instead of duplicating it

• #329 / #297: batch resume must be driven by run metadata and preserve prior results.
• #319 / #300: shared judge transport behavior.
• #354 / #301: shared Stage-1 interceptor semantics and parity tests.
• #340 / #243: report Stage 1 and Stage 2 together with explicit denominators.
• #313 / #296: portable rescore paths and explicit invalid-input failures.
• #347: record run/corpus/agent provenance; coordinate with evaluator cache provenance in #363.

Maintainers can explicitly defer a candidate, with the reason and remaining limitation recorded here. This checklist does not declare every existing feature PR a release blocker.

Validate the actual release candidate

• Run the existing host tests, Ruff, Pyright, corpus validation and Markdown checks on the combined candidate commit; results from separate PR heads do not prove the combined candidate is sound.
• Build wheel and sdist, run metadata checks, then install each in a clean environment outside the source checkout. Verify packaged CLI --help, bundled corpus discovery/JSON loading, and a one-task offline adapter conversion. This catches source-checkout-only imports and missing package data.
• Ensure changes to native runtime dependencies trigger the correct existing image build matrix (#365).
• Update only Unreleased during development; choose the release version/date and reconcile pyproject.toml, tag and changelog when maintainers cut the release.
• Release notes distinguish verified functionality from untested runtime/provider combinations and state any deferred correctness limitations.

Harbor scope

Keep #331 as the Harbor integration tracker; review #353, #346 and #360 there, with #358/#359 covering remaining lifecycle/upstream acceptance work. Do not reopen deferred Harbor CI #350 or make image publication a new prerequisite. Loader checks are not browser/provider E2E or parity evidence.

Audit evidence

The five new bugs were checked against main 9dd9d44; reproduce/rescore sources match the current working branch. Offline temporary-directory/mock reproductions verified user-file deletion, sample-count acceptance, stale-cache attribution and corrupt-cache skipping. The CI selector was exercised with a runtime-server-only input. No live model calls, release publication or real user-data deletion were performed.

@Perry2004 please review the suggested blocker ordering, the linked fixes, and which items must be completed versus explicitly deferred for the next release.

Start by reviewing linked issues #361–#365 and the existing host tests, Ruff, Pyright, corpus validation, and Markdown checks. Then inspect the combined release candidate workflow, package metadata, wheel and sdist validation, and native dependency selector. Done means blockers are fixed or explicitly deferred, the packaged CLI and corpus work from clean environments, and release evidence and notes reflect tested limitations.