openclaw / openclaw/shellbench
feat(native-eval): export normalized tool discovery telemetry
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 139
- Forks
- 30
- Avg merge
- 7m
- Merged PRs (30d)
- 6
Description
Summary
Export normalized native-harness discovery telemetry separately from final tool
calls so ShellBench can compare deferred tool loading without treating missing
events as zero.
Motivation
The current research audit exports ATIF tool calls, but it loses harness-native
discovery detail. OpenClaw's tool_search_code result already contains catalog
size and search, describe, and call counts; the normalized CSV records only the
wrapper call. Codex native tool search exists in its runtime protocol, but
codex exec --json does not currently expose those events. Claude Code and
Hermes do not provide an equivalent event in the traces used by this runner.
This makes tool-search comparisons ambiguous: a missing count can mean zero
operations, unsupported discovery, or an exporter fidelity gap.
Proposed approach
- Add a discovery-event research table with
search,describe,load, and
calloperations plus source harness, target ids, latency when available,
success, and trace fidelity. - Extract OpenClaw counters and selected calls from the structured
tool_search_coderesult details. - Mark Codex as
unobservableuntil its native JSON stream exports ToolSearch
items, and mark harnesses without a discovery surface asunsupported. - Keep discovery operations separate from final tool calls and report runner
SHA plus trajectory fidelity with every row.
Alternatives considered
Inferring discovery from wrapper names or recording absent events as zero was
rejected because both produce false cross-harness comparisons.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the current ATIF tool-call export and the structured tool_search_code result handling. Add a separate discovery-event table with the proposed operations, harness status, targets, latency, success, source SHA, and trajectory fidelity; verify that OpenClaw counters are extracted while Codex and unsupported harnesses remain explicitly distinguishable from zero.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100