openclaw / openclaw/shellbench

fix(native-eval): make OpenClaw Tool Search controls explicit

Open
#61 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

clawsweeper:linked-pr-open clawsweeper:no-new-fix-pr clawsweeper:source-repro impact:other issue-rating: 🦞 diamond lobster P2
Dominant language
Python
Stars
139
Forks
30
Avg merge
7m
Merged PRs (30d)
6

Description

Summary

ShellBench's OpenClaw native harness does not currently encode the disabled
Tool Search control explicitly. When a run omits openclaw_tool_search_mode,
the generated OpenClaw config omits tools.toolSearch, so the installed
OpenClaw default decides the mode.

Impact

An intended Tool Search "off" ablation can silently execute with code mode
enabled. That contaminates score, token, tool-call, and latency comparisons
because the control and treatment are not actually distinct.

This occurred in an exact candidate run on July 29, 2026: the run index had no
mode, but the exported OpenClaw trace reported codeModeEngaged: true.

Expected behavior

  • No mode means the harness writes tools.toolSearch: false.
  • directory and code are explicit supported values.
  • The selected value survives fleet dispatch, manifests, and automatic reruns.
  • Empty fleet environment values normalize to unset metadata, not "".

Verification

Add focused runner and fleet tests covering explicit off, code mode, dispatch,
manifest normalization, and rerun preservation.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the OpenClaw native runner, fleet dispatch, manifest normalization, and automatic rerun handling described in the issue, then run the existing focused tests. Done means omitted mode writes an explicit false value, directory and code remain supported, dispatch and reruns preserve the selection, and empty fleet values become unset metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.