TIGER-AI-Lab / TIGER-AI-Lab/ClawBench
Harbor: preserve verifier evidence and scoring when Kernel cleanup fails
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 795
- Forks
- 58
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 18
Description
Reproducible control-flow defect
harbor_adapter.test_script("kernel") emits set -euo pipefail and calls kernel-browser.py finalize before copying /data to /logs/verifier/data, running verify.py, or cleaning up email.
In kernel-browser.py:cmd_finalize, a BrowserRuntimeError from provider.cleanup(...) returns 1. Consequently a provider deletion outage aborts the verifier script before scoring or copying verifier evidence. The failure branch writes state/private metadata but skips _write_lifecycle, so the normal /data lifecycle artifact is also absent. This is established by code inspection, not a live provider test.
Acceptance criteria
- Separate scoring, replay collection, browser deletion, and email cleanup outcomes; guarantee cleanup attempts and preserve diagnostic artifacts even if another phase fails.
- Persist a public lifecycle failure artifact before returning on every provider-error path.
- Define how Harbor represents cleanup infrastructure failure without silently converting it into agent failure or hiding it behind an ordinary success reward.
- Fault-inject replay, deletion, verifier, and artifact-copy failures; assert evidence and final exit/result semantics.
- Cover failure after successful setup but before verification, where the setup EXIT trap has already been removed; document provider TTL fallback if cancellation cannot invoke cleanup.
Child of #331. This is a lifecycle/result-contract fix, separate from #346's readiness work. It does not reopen deferred CI #350 or require publishing an image.
@Perry2004 please review the failure semantics and cleanup ownership before implementation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with harbor_adapter.test_script("kernel") and kernel-browser.py:cmd_finalize, then trace the calls to verify.py, /data, and /logs/verifier/data. Map the separate scoring, replay, deletion, email, and artifact-copy outcomes before deciding the lifecycle failure contract. Done means fault-injection coverage for each failure path, preserved public evidence, and documented TTL fallback behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend-api-design, testing
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100