PolicyEngine / PolicyEngine/policybench

Batch rerun on a completed run dir replays early rounds and regresses the CSV

Open
#102 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2
Forks
3
Avg merge
12h 7m
Merged PRs (30d)
13

Description

Problem

Rerunning eval-no-tools-batch against an output dir whose run already completed does not extend it — it replays the stored batch state from round 0 ("round 1: resuming batch batch_…"), stops before the later rounds, and overwrites by_model/<model>.csv with fewer repair rounds applied, silently discarding the completed run's later repairs.

Observed (gpt-5.5 default-effort run, 2026-07-05):

  • First invocation: rounds 0–5, final CSV 1,984 rows with 23 missing predictions.
  • Second invocation of the identical command on the same dir (intending to continue repairs with POLICYBENCH_MAX_REPAIR_ROUNDS=8): re-collected rounds 0–2 from the stored OpenAI batch ids, printed "scheduling repair round" after round 2, then wrote the CSV and exited — leaving 51 missing predictions (rounds 3–5's successful repairs discarded).

Whether the raised POLICYBENCH_MAX_REPAIR_ROUNDS (#97) was honored by the resume path is also unclear from the logs — the process stopped scheduling after round 2 despite 18 units still violating.

Expected

Either (a) a completed run dir is a no-op ("already complete: 1,984 rows, 23 missing — pass --force to redo"), or (b) resume continues from the LAST persisted round and only schedules new repair rounds for currently-missing units, never producing a CSV with more missing cells than the one it replaces.

Workaround used

Archived the dir and ran fresh in a new output dir. Evidence preserved at results/local/gpt55_default_run_20260705_attempt1/ (both CSVs' round provenance in batches/).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the eval-no-tools-batch resume path and how persisted state under batches/ is read and written. Compare the stored round provenance with the regenerated by_model/.csv using results/local/gpt55_default_run_20260705_attempt1/. Done means rerunning a completed directory is a no-op or resumes from the last persisted round without increasing missing predictions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.