ACP Codex GPT-5.4 swebenchmultimodal: missing output.report.json

Summary

The latest ACP Codex GPT-5.4 swebenchmultimodal run (23983222379) does not produce an output.report.json summary file. Instead, resolved/not-resolved status is only available via 102 individual per-instance report.json files under logs/run_evaluation/.

Details

• Run ID: 23983222379
• GCS Artifacts: results.tar.gz
• SDK commit: 6626a36
• Agent: codex-acp v0.11.1

What's in the tarball

output.jsonl              ✅  (102 instances)
cost_report.jsonl         ✅
output.report.json        ❌  MISSING
logs/instance_*.log       ✅  (204 files — 102 instance + 102 output)
logs/run_evaluation/      ✅  (102 per-instance report.json + test_output.txt + patch.diff)

The per-instance eval reports confirm 21/102 resolved (20.6%), so the evaluation did run — it just didn't aggregate into a summary output.report.json.

Comparison with other benchmarks in the same batch

Note: swebenchmultimodal used a different SDK commit (6626a36) than the other three (4ad68fd), and was dispatched in a separate batch.

Impact

• Results are valid — all 102 instances were inferred and evaluated
• Resolved count (21) must be aggregated manually from per-instance reports
• validate_artifacts.py may not detect this since it checks for output.report.json existence — runs missing it could silently skip the resolved count check

Related

• OpenHands/benchmarks#576 — ACP master table tracking

Start with validate_artifacts.py and the artifact layout described for the swebenchmultimodal run. Trace how per-instance report.json files are handled compared with output.report.json, then verify that the run produces an aggregated summary and that validation accounts for its resolved count.