partial_archive_url resume skips errored instances instead of retrying them

Problem

When using partial_archive_url to resume a partial run, the runner skips all instances that have an entry in the output file — including instances that failed with infrastructure errors (runtime timeouts, 429s, 502s). There is no way to distinguish "completed successfully" from "completed with error" during the resume check.

How the resume works today

_get_instances_for_attempt() in benchmarks/utils/evaluation.py reads output.critic_attempt_N.jsonl and skips any instance ID already present. An instance that failed with a runtime timeout still has an entry (with test_result.result: None), so it gets skipped.

What happened

We had a swtbench ACP run (eval run 23627243393) where 221 out of 433 instances failed due to infrastructure errors (runtime timeouts, 429 Too Many Requests, 502 Bad Gateway). The agent successfully wrote patches for all 433 instances, but the eval harness couldn't spin up runtimes to verify 221 of them.

When we tried to resume with partial_archive_url pointing at the original archive, the runner saw all 433 entries as "completed" and skipped straight to the eval phase — reproducing the exact same 221 errors.

Workaround: strip error entries from the archive

We had to manually clean the archive before using it as a partial:

1. Extract results.tar.gz
2. Load output.report.json to get the list of error_ids
3. Strip entries with those IDs from all JSONL files (output.jsonl, output.critic_attempt_*.jsonl, output_errors.jsonl, output.swtbench.jsonl)
4. Remove corresponding conversation archives from conversations/
5. Remove output.report.json (will be regenerated)
6. Repack and upload to a new GCS path
7. Retrigger with partial_archive_url pointing to the cleaned archive

Python snippet for the stripping step:

import json, os

basedir = "eval_outputs/.../output_dir"

# Get error IDs from the report
with open(f"{basedir}/output.report.json") as f:
    error_ids = set(json.load(f)["error_ids"])

# Strip error entries from all JSONL files
for fname in os.listdir(basedir):
    if fname.startswith("output") and fname.endswith(".jsonl"):
        path = os.path.join(basedir, fname)
        with open(path) as fh:
            lines = fh.readlines()
        kept = [l for l in lines if json.loads(l).get("instance_id") not in error_ids]
        with open(path, "w") as fh:
            fh.writelines(kept)

# Remove conversation archives for error instances
conv_dir = os.path.join(basedir, "conversations")
for conv_file in os.listdir(conv_dir):
    if conv_file.replace(".tar.gz", "") in error_ids:
        os.remove(os.path.join(conv_dir, conv_file))

os.remove(f"{basedir}/output.report.json")

Suggested fix

Add a retry_errors option (or make it the default) to the resume logic. When loading a partial archive, check whether each instance entry represents a successful completion or an error, and only skip successfully completed instances.

Possible implementation: in _get_instances_for_attempt(), when reading existing output entries, check if test_result.result is None or if the instance ID appears in error_ids from the report, and exclude those from the "already completed" set.

Related

• evaluation PR #326 — feat: resume partial runs from archive
• evaluation PR #344 — fix for partial archive extraction
• Affected run: swtbench 23627243393 (221/433 infra errors, all due to runtime timeouts/429s/502s)

Start in benchmarks/utils/evaluation.py at _get_instances_for_attempt(), then inspect the partial archive's JSONL entries and output.report.json handling. Confirm that successful instances remain skipped while entries with test_result.result set to None or IDs in error_ids are eligible for retry. Done means resuming a partial archive retries infrastructure-error instances without rerunning successful ones.