PolicyEngine / PolicyEngine/policybench
Batch rerun on a completed run dir replays early rounds and regresses the CSV
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2
- Forks
- 3
- Avg merge
- 12h 7m
- Merged PRs (30d)
- 13
Description
Problem
Rerunning eval-no-tools-batch against an output dir whose run already completed does not extend it — it replays the stored batch state from round 0 ("round 1: resuming batch batch_…"), stops before the later rounds, and overwrites by_model/<model>.csv with fewer repair rounds applied, silently discarding the completed run's later repairs.
Observed (gpt-5.5 default-effort run, 2026-07-05):
- First invocation: rounds 0–5, final CSV 1,984 rows with 23 missing predictions.
- Second invocation of the identical command on the same dir (intending to continue repairs with
POLICYBENCH_MAX_REPAIR_ROUNDS=8): re-collected rounds 0–2 from the stored OpenAI batch ids, printed "scheduling repair round" after round 2, then wrote the CSV and exited — leaving 51 missing predictions (rounds 3–5's successful repairs discarded).
Whether the raised POLICYBENCH_MAX_REPAIR_ROUNDS (#97) was honored by the resume path is also unclear from the logs — the process stopped scheduling after round 2 despite 18 units still violating.
Expected
Either (a) a completed run dir is a no-op ("already complete: 1,984 rows, 23 missing — pass --force to redo"), or (b) resume continues from the LAST persisted round and only schedules new repair rounds for currently-missing units, never producing a CSV with more missing cells than the one it replaces.
Workaround used
Archived the dir and ran fresh in a new output dir. Evidence preserved at results/local/gpt55_default_run_20260705_attempt1/ (both CSVs' round provenance in batches/).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the eval-no-tools-batch resume path and how persisted state under batches/ is read and written. Compare the stored round provenance with the regenerated by_model/.csv using results/local/gpt55_default_run_20260705_attempt1/. Done means rerunning a completed directory is a no-op or resumes from the last persisted round without increasing missing predictions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100