swebench-eval modal runs fail after successful evaluation because final run report is never written

Summary

Several SWE-Bench eval monitor runs are failing with the same post-processing error even though the harness itself finishes successfully.

Examples:

• https://openhands-eval-monitor.vercel.app/?status=error&run=swebench%2Flitellm_proxy-converse-nemotron-super-3-120b%2F23924261929%2F
• https://openhands-eval-monitor.vercel.app/?status=error&run=swebench%2Flitellm_proxy-converse-nemotron-super-3-120b%2F23923477151%2F
• https://openhands-eval-monitor.vercel.app/?status=error&run=swebench%2Flitellm_proxy-nvidia-nemotron-3-super-120b-a12b%2F23872514888%2F
• https://openhands-eval-monitor.vercel.app/?status=error&run=swebench%2Flitellm_proxy-minimax-MiniMax-M2-5%2F23856153722%2F
• https://openhands-eval-monitor.vercel.app/?status=error&run=swebench%2Flitellm_proxy-claude-sonnet-4-5-20250929%2F23854010466%2F
• https://openhands-eval-monitor.vercel.app/?status=error&run=swebench%2Flitellm_proxy-gemini-3-flash-preview%2F23853545809%2F
• https://openhands-eval-monitor.vercel.app/?status=error&run=swebench%2Flitellm_proxy-gemini-3-flash-preview%2F23851904790%2F

All of them show the same pattern in error_detail:

1. SWE-Bench evaluation completed successfully
2. then Script failed: [Errno 2] No such file or directory: '.../OpenHands.<run_id>.json'

Root cause

This looks like a benchmarks-side bug rather than an evaluation-infra problem.

benchmarks/benchmarks/swebench/eval_infer.py assumes the harness will always write the final run report file:

report_filename = f"{MODEL_NAME_OR_PATH}.{args.run_id}.json"
shutil.move(str(report_path), str(dest_report_path))

But with swebench==4.1.0, swebench.harness.run_evaluation returns early in the modal branch:

if modal:
    if not dataset:
        print("No instances to run.")
    else:
        validate_modal_credentials()
        run_instances_modal(predictions, dataset, full_dataset, run_id, timeout)
    return

That early return skips the later make_run_report(...) call, so the per-instance modal reports exist but the final OpenHands.<run_id>.json file never gets created.

Notes

• The same unconditional move logic also exists in benchmarks/benchmarks/swebenchmultilingual/eval_infer.py.
• We are adding a defensive recovery workaround in OpenHands/evaluation, but the real fix belongs here.

Suggested fix

One of:

• write the final run report after modal evaluation in the benchmarks wrapper, or
• update the wrapper to synthesize the final report from logs/run_evaluation/<run_id>/.../report.json when the aggregate file is missing.

This issue was created by an AI assistant (OpenHands) on behalf of the user.

Start in benchmarks/benchmarks/swebench/eval_infer.py and compare its modal evaluation path with the report move logic. Check the corresponding flow in benchmarks/benchmarks/swebenchmultilingual/eval_infer.py, then run or inspect a modal SWE-Bench evaluation using the linked failure pattern. Done means successful modal evaluations leave the expected final run report in place in both wrappers.