PolicyEngine / PolicyEngine/policybench

Make Claude runs parallel-safe by moving the hard timeout to the chunk subprocess

Open
#84 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
2
Forks
3
Avg merge
12h 7m
Merged PRs (30d)
13

Description

Claude models are forced to --parallel 1 (SERIAL_ONLY_MODEL_PREFIXES in chunked_eval.py) because the last-resort wall-clock timeout in eval_no_tools._run_request_with_wall_timeout uses SIGALRM, which Python only delivers on the main thread. In worker threads the guard silently disappears, and a provider call that outlives its client timeout wedges the run.

Each chunk already executes as a subprocess (chunked_eval.run_chunksubprocess.run). Moving the hard bound there makes parallelism safe without signals:

  • pass timeout= to subprocess.run, sized as rows_in_chunk × wall_timeout + slack
  • treat TimeoutExpired as an infrastructure failure: kill the chunk, leave it incomplete, let resume retry it
  • drop the SIGALRM path (or keep it main-thread-only as a secondary bound)
  • lift the serial restriction so claude-* can run --parallel 4+

A full Claude model run currently takes ~2–3h serial at chunk size 5; at --parallel 4 it's ~30–40 min. Ran into this again adding Sonnet 5 (2026-07-02).

🤖 Generated with Claude Code

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with chunked_eval.py, especially run_chunk and its subprocess.run call, then inspect eval_no_tools._run_request_with_wall_timeout and SERIAL_ONLY_MODEL_PREFIXES. Implement the subprocess hard timeout so an expired chunk is killed and left incomplete for resume, remove or limit the SIGALRM path, and verify that claude-* runs can use parallel workers.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.