OpenHands / OpenHands/benchmarks
Track structured timing for phased image assembly cleanup
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 124
- Forks
- 90
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
Problem
The phased SWE/SWT-bench image assembly logs include human-readable timing in lines like:
[assembly] OK ... total=109.8s build=87.2s push=6.2s
but the emitted manifest.jsonl records do not populate structured timing fields for assembly. In the full SWE-bench validation run below, fields such as duration_seconds, build_seconds, and related timing fields were null, so analyzing performance required scraping the raw GitHub Actions log.
Validation run: https://github.com/OpenHands/benchmarks/actions/runs/24810843867
That run succeeded (500/500 base images and 500/500 assembled images), but performance/debug analysis was more manual than it should be:
- Average assembly total: ~131.8s/image
- Average docker build: ~106.9s/image
- Average push: ~5.8s/image
- Implied cleanup/other post-build gap: ~19.1s/image
docker system pruneproduced 170 warning-only rc=1 failures, likely prune-lock contention
Proposal
Populate structured timing fields in assembly manifest records, including at least:
- total assembly duration
- docker build duration
- docker push duration
- docker rmi duration
- docker system prune duration
- docker builder prune duration
- cleanup status / return codes for rmi, system prune, and builder prune
This would make future tuning less guessy and avoid scraping CI logs for basic timing and cleanup behavior.
Acceptance criteria
manifest.jsonlforassemble_all_agent_imagesincludes non-null timing fields for each assembled image.- Cleanup subprocess outcomes are represented in a structured way.
- Existing summary tooling either preserves these fields or can surface aggregate build/push/cleanup timing.
- Tests cover the new timing fields for a successful assembly.
Context
This came out of the SWE-bench disk/OOM follow-up to the SWT-bench cleanup work in PR #672 and SWE-bench validation PR #690.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the assemble_all_agent_images entry point and the code that writes manifest.jsonl, then trace how build, push, rmi, system prune, and builder prune subprocesses are executed. Review the existing summary tooling and tests before deciding where timing and cleanup outcomes belong. Done means successful assembly records contain non-null timing fields, structured cleanup results are preserved, and tests cover the success path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, github-actions, python
- Domain
- build-system, ci-cd, devops
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100