openclaw / openclaw/shellbench
fix(native-eval): make OpenClaw Tool Search controls explicit
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 139
- Forks
- 30
- Avg merge
- 7m
- Merged PRs (30d)
- 6
Description
Summary
ShellBench's OpenClaw native harness does not currently encode the disabled
Tool Search control explicitly. When a run omits openclaw_tool_search_mode,
the generated OpenClaw config omits tools.toolSearch, so the installed
OpenClaw default decides the mode.
Impact
An intended Tool Search "off" ablation can silently execute with code mode
enabled. That contaminates score, token, tool-call, and latency comparisons
because the control and treatment are not actually distinct.
This occurred in an exact candidate run on July 29, 2026: the run index had no
mode, but the exported OpenClaw trace reported codeModeEngaged: true.
Expected behavior
- No mode means the harness writes
tools.toolSearch: false. directoryandcodeare explicit supported values.- The selected value survives fleet dispatch, manifests, and automatic reruns.
- Empty fleet environment values normalize to unset metadata, not
"".
Verification
Add focused runner and fleet tests covering explicit off, code mode, dispatch,
manifest normalization, and rerun preservation.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the OpenClaw native runner, fleet dispatch, manifest normalization, and automatic rerun handling described in the issue, then run the existing focused tests. Done means omitted mode writes an explicit false value, directory and code remain supported, dispatch and reruns preserve the selection, and empty fleet values become unset metadata.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100