openclaw / openclaw/shellbench

feat(native-eval): export normalized tool discovery telemetry

Open
#59 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

clawsweeper:linked-pr-open clawsweeper:needs-product-decision clawsweeper:no-new-fix-pr impact:other issue-rating: 🌊 off-meta tidepool P3
Dominant language
Python
Stars
139
Forks
30
Avg merge
7m
Merged PRs (30d)
6

Description

Summary

Export normalized native-harness discovery telemetry separately from final tool
calls so ShellBench can compare deferred tool loading without treating missing
events as zero.

Motivation

The current research audit exports ATIF tool calls, but it loses harness-native
discovery detail. OpenClaw's tool_search_code result already contains catalog
size and search, describe, and call counts; the normalized CSV records only the
wrapper call. Codex native tool search exists in its runtime protocol, but
codex exec --json does not currently expose those events. Claude Code and
Hermes do not provide an equivalent event in the traces used by this runner.

This makes tool-search comparisons ambiguous: a missing count can mean zero
operations, unsupported discovery, or an exporter fidelity gap.

Proposed approach

  • Add a discovery-event research table with search, describe, load, and
    call operations plus source harness, target ids, latency when available,
    success, and trace fidelity.
  • Extract OpenClaw counters and selected calls from the structured
    tool_search_code result details.
  • Mark Codex as unobservable until its native JSON stream exports ToolSearch
    items, and mark harnesses without a discovery surface as unsupported.
  • Keep discovery operations separate from final tool calls and report runner
    SHA plus trajectory fidelity with every row.

Alternatives considered

Inferring discovery from wrapper names or recording absent events as zero was
rejected because both produce false cross-harness comparisons.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the current ATIF tool-call export and the structured tool_search_code result handling. Add a separate discovery-event table with the proposed operations, harness status, targets, latency, success, source SHA, and trajectory fidelity; verify that OpenClaw counters are extracted while Codex and unsupported harnesses remain explicitly distinguishable from zero.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.