Please open-source the `generate_completions.py` used for the v2 completions

Start with evaluation/compute_pass_at_1.py, evaluation/execute_benchmark.py, the shipped completions/ directory, and the last generate_completions.py version in git history. Compare the documented scoring flow with the v2 model set and identify the missing generation details. Done means publishing the exact script or README specification covering prompts, sampling, response parsing, retries, and n=5 behavior.

Please open-source the generate_completions.py used for the v2 completions

Thanks for releasing DevBench! The scoring side is fully reproducible —
running evaluation/compute_pass_at_1.py over the shipped completions/
reproduces the paper's numbers closely.

The generation side is not, because the script that produced the current
completions seems to be missing from the repo. So we can only re-score the
static files you shipped, not generate completions for a new model under the
same protocol or re-derive the paper's Pass@1 from scratch.

What's missing

• generate_completions.py is not in the tree at the pinned v2 revision
  (831495a, "hardened v2: 1,800 tasks, 9 current-gen models").
• The last version in git history targets an older model set (gpt-4o,
  gpt-4.1*, claude-3-7/4-sonnet, deepseek-v3/v3.1) — not the shipped v2
  models (gpt-5.5/5.4-mini/5.4-nano, claude-opus-4-7, claude-sonnet-4-6,
  deepseek-v4-pro, llama-4-maverick, mistral-medium-3.5, qwen3.6-27b).
• The README says the FIM prompt is "defined in evaluation/execute_benchmark.py",
  but that file only contains the execution/scoring harness — no
  prompt-construction or model-call code.

Why it blocks reproduction

With the documented max_tokens=800, current reasoning models spend the budget
on reasoning tokens and return empty content on many tasks (~40%+ empty in
my runs), whereas your shipped completions have a very low empty rate. So a
different generation setup must have been used, and we can't match it blindly.

Request

Could you open-source the exact generation script (or just specify it in the
README), including:

1. The exact per-language system + user / FIM prompt template.
2. Sampling params actually used, especially how reasoning models were
   handled (default settings? what max_tokens?).
3. Response parsing / code-extraction logic.
4. Empty-output/retry handling and the n=5 sampling details.

Even a lightly-cleaned version would make generation reproducible. Thanks!