anthropics / anthropics/skills

skill-creator run_eval.py: parallel identical command clones contaminate trigger detection, collapsing recall to 0

Offen
#1,427 2 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
176k
Forks
20.8k
Ø Merge
7 Std. 21 Min.
Gemergte PRs (30 T.)
5

Beschreibung

## Summary

`skill-creator`'s trigger evaluation (`scripts/run_eval.py`) systematically under-counts triggers — in my environment recall collapsed to exactly **0% across 27+ positive samples and three very different candidate descriptions** (precision stayed 100%, accuracy pinned at 50%). After packet-level debugging with `--output-format stream-json` captures, the root cause is a design issue in how triggering is detected, not stream-format drift:

**Parallel workers create N identical-description command clones in the same shared `.claude/commands/` directory, but each worker only counts a trigger if the model invokes *its own* hashed clone name. The model sees ~N identical entries and picks one arbitrarily, so each worker's match probability is ≤1/N and measured recall collapses toward 0.**

## Mechanism

`run_single_query()` writes a clone named `-skill-.md` into `/.claude/commands/`, then counts a trigger only when the accumulated `tool_use` input JSON contains that exact hashed name (`clean_name in accumulated_json`).

With the default `--num-workers 10` and `--runs-per-query 3`, ~10 clones with **identical descriptions but different hashes** coexist in the *same* commands directory at any moment (all workers share one `project_root`). Every `claude -p` session sees all of them. When a query should trigger, the model picks one of the N identical entries — almost never the one belonging to the worker that spawned this particular session.

Two aggravating factors:

1. **Orphaned clones**: if `run_loop.py`/`run_eval.py` is killed, clone files are not cleaned up (the `finally` only unlinks the current worker's file). Stale clones from previous runs inflate N further, and they may carry *previous-iteration* descriptions, silently contaminating later iterations.
2. **Canonical-name capture**: if the skill under test is referenced by its real name in the user's `CLAUDE.md` (common for infra skills), the model calls `Skill("")` rather than any clone, so the hashed-name match can never hit. Probably worth a note in the docs — such skills can't be meaningfully description-evaluated in place.

## Reproduction (verified on Claude Code CLI, 2026-07-11)

1. Create 5+ clone files in `~/.claude/commands/` exactly as `run_single_query()` does (same description, different 8-hex suffixes).
2. Run one positive query via `claude -p "" --output-format stream-json --verbose --include-partial-messages`.
3. Observe the `tool_use` input: the model calls **one** `Skill("-skill-")` — in my captures it repeatedly picked a *stale* clone left over from an earlier probe, i.e. any individual worker's `clean_name in accumulated_json` check returns `False` even though the description **did** trigger.

Control experiment: with exactly **one** clone present, the model invokes that clone's full hashed name and detection works correctly — confirming the stream-event parsing (`stream_event` → `content_block_start` → `input_json_delta`) is fine on current CLI builds. The failure is purely the N-clone ambiguity.

## Suggested fixes

Any one of these would fix the collapse; (1)+(2) together seem cleanest:

1. **Match the clone-name prefix instead of the exact hash**: `f"{skill_name}-skill-" in accumulated_json`. Within one eval iteration all concurrent clones carry the same description, so any clone hit is a valid trigger signal for that description.
2. **Sweep stale clones** (`-skill-*.md`) at eval start, and register a cleanup handler so killed runs don't leave orphans behind.
3. Alternatively, give each worker an isolated throwaway `project_root` so listings don't overlap (heavier; also note user-level `~/.claude/commands/` is shared regardless when `project_root` resolves to `$HOME`).

## Minor related fragility

`run_single_query()` returns `False` as soon as the first `tool_use` block is anything other than `Skill`/`Read` (run_eval.py L133-141). In tool-heavy environments (many MCP servers, hooks) the model may legitimately call another tool first and *then* consult the skill — those runs are counted as non-triggers. Worth reconsidering, e.g. keep listening until `message_stop` instead of returning early.

Happy to submit a PR for (1)+(2) if that's welcome.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.