PolicyEngine / PolicyEngine/policybench
Make Claude runs parallel-safe by moving the hard timeout to the chunk subprocess
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2
- Forks
- 3
- Avg merge
- 12h 7m
- Merged PRs (30d)
- 13
Description
Claude models are forced to --parallel 1 (SERIAL_ONLY_MODEL_PREFIXES in chunked_eval.py) because the last-resort wall-clock timeout in eval_no_tools._run_request_with_wall_timeout uses SIGALRM, which Python only delivers on the main thread. In worker threads the guard silently disappears, and a provider call that outlives its client timeout wedges the run.
Each chunk already executes as a subprocess (chunked_eval.run_chunk → subprocess.run). Moving the hard bound there makes parallelism safe without signals:
- pass
timeout=tosubprocess.run, sized asrows_in_chunk × wall_timeout + slack - treat
TimeoutExpiredas an infrastructure failure: kill the chunk, leave it incomplete, let resume retry it - drop the SIGALRM path (or keep it main-thread-only as a secondary bound)
- lift the serial restriction so
claude-*can run--parallel 4+
A full Claude model run currently takes ~2–3h serial at chunk size 5; at --parallel 4 it's ~30–40 min. Ran into this again adding Sonnet 5 (2026-07-02).
🤖 Generated with Claude Code
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with chunked_eval.py, especially run_chunk and its subprocess.run call, then inspect eval_no_tools._run_request_with_wall_timeout and SERIAL_ONLY_MODEL_PREFIXES. Implement the subprocess hard timeout so an expired chunk is killed and left incomplete for resume, remove or limit the SIGALRM path, and verify that claude-* runs can use parallel workers.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 58/100