TIGER-AI-Lab / TIGER-AI-Lab/ClawBench
Stage-1 interceptor matching is duplicated between runtime-server and edgebench_judge, and the live copy has zero tests
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 795
- Forks
- 58
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 18
Description
The Stage-1 interceptor decision — the benchmark's deterministic ground truth — exists in two hand-maintained copies:
src/clawbench/runtime/runtime-server/server.py:173-185(_const_fields_match) and:449-477(url / method / body / params gate) — the real thing, running in-container.src/clawbench/eval/edgebench_judge.py:55-94— a re-implementation whose comment says it "mirrors runtime-server".
Only the mirror has tests (tests/test_edgebench_judge.py). server.py is 878 lines with zero tests and is excluded from pyright (pyproject.toml:74 excludes src/clawbench/runtime). So the copy that decides every published Intercepted number is the unverified one, and any drift between the two means offline judging disagrees with what actually happened during runs.
Ask:
- Extract the matching predicate into a shared module (e.g.
runtime/shared/matching.py) imported by both. - Add a parity test over a fixture matrix of
url_pattern/method/body/paramscases asserting both paths agree — including the regex edge cases from #258 (empty pattern matches everything; malformed pattern raises inside the CDP loop and silently kills interception for the rest of the run).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with src/clawbench/runtime/runtime-server/server.py:173-185 and :449-477, then compare the mirror in src/clawbench/eval/edgebench_judge.py:55-94 and tests/test_edgebench_judge.py. Extract the shared matching behavior and add a fixture matrix covering URL, method, body, params, empty patterns, and malformed patterns. Done means both paths agree, including the regex edge cases, with coverage for the runtime-side decision.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, testing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100