TIGER-AI-Lab / TIGER-AI-Lab/ClawBench

Harbor: preserve verifier evidence and scoring when Kernel cleanup fails

Open
#358 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
795
Forks
58
Avg merge
2d 21h
Merged PRs (30d)
18

Description

Reproducible control-flow defect

harbor_adapter.test_script("kernel") emits set -euo pipefail and calls kernel-browser.py finalize before copying /data to /logs/verifier/data, running verify.py, or cleaning up email.

In kernel-browser.py:cmd_finalize, a BrowserRuntimeError from provider.cleanup(...) returns 1. Consequently a provider deletion outage aborts the verifier script before scoring or copying verifier evidence. The failure branch writes state/private metadata but skips _write_lifecycle, so the normal /data lifecycle artifact is also absent. This is established by code inspection, not a live provider test.

Acceptance criteria

  • Separate scoring, replay collection, browser deletion, and email cleanup outcomes; guarantee cleanup attempts and preserve diagnostic artifacts even if another phase fails.
  • Persist a public lifecycle failure artifact before returning on every provider-error path.
  • Define how Harbor represents cleanup infrastructure failure without silently converting it into agent failure or hiding it behind an ordinary success reward.
  • Fault-inject replay, deletion, verifier, and artifact-copy failures; assert evidence and final exit/result semantics.
  • Cover failure after successful setup but before verification, where the setup EXIT trap has already been removed; document provider TTL fallback if cancellation cannot invoke cleanup.

Child of #331. This is a lifecycle/result-contract fix, separate from #346's readiness work. It does not reopen deferred CI #350 or require publishing an image.

@Perry2004 please review the failure semantics and cleanup ownership before implementation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with harbor_adapter.test_script("kernel") and kernel-browser.py:cmd_finalize, then trace the calls to verify.py, /data, and /logs/verifier/data. Map the separate scoring, replay, deletion, email, and artifact-copy outcomes before deciding the lifecycle failure contract. Done means fault-injection coverage for each failure path, preserved public evidence, and documented TTL fallback behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend-api-design, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.