anthropics / anthropics/claude-plugins-official
skill-creator eval harness (run_eval.py) reports 0 triggers for every skill on Claude Code 2.1.x
- Dominant language
- Python
- Stars
- 36.3k
- Forks
- 4.1k
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 539
Description
## Summary
`skills/skill-creator/scripts/run_eval.py` measures whether a skill description causes
Claude to invoke the skill. On current Claude Code CLIs (verified on **2.1.168**) it
reports a 0% trigger rate for every skill and every query, which silently invalidates
both `run_eval.py` and the `run_loop.py` description optimizer built on top of it. Two
independent bugs:
## Bug 1 — the harness registers a slash-command, but the CLI only fires the `Skill` tool for skills
`run_single_query()` writes the description under test to
`.claude/commands/.md`, then runs `claude -p ""` and watches the
stream-json output for a `Skill` tool call. In current CLIs, commands (`/name`) and
skills (`.claude/skills//SKILL.md`) are separate mechanisms: a natural-language
query only ever produces a `Skill` tool call for a real skill, never for a command
file. The harness is watching for an event its own setup can never produce.
**Repro/evidence:** with a real skill installed (`.claude/skills/…`), a headless run
(`claude -p "de-slop this: …" --output-format stream-json`) emits
`tool_use | Skill | `. With the same description registered as a command
file, no query ever emits a `Skill` tool call.
**Fix:** write the temp skill to `.claude/skills//SKILL.md` with
`name: ` in the frontmatter (the unique `-skill-` name already
prevents collisions with installed skills), and clean up with `shutil.rmtree`.
## Bug 2 — race: `claude -p` is spawned before the fresh skill is indexed
Even with Bug 1 fixed, the harness `Popen`s `claude -p` immediately after writing the
skill file. The subprocess starts before the newly written skill is discoverable and
never sees it → still 0 triggers. A short settle delay (`time.sleep(1.5)`) between
writing the SKILL.md and spawning the subprocess makes detection reliable.
**Repro/evidence:** manually creating a uniquely named skill, sleeping 1s, then running
`claude -p` reliably produces the `Skill` tool call; without the pause the skill is
never invoked.
## Bug 3 — the first tool call is treated as the verdict, so auto-memory makes every query a false miss
`run_single_query()` returns False as soon as the first streamed tool call is anything
other than the skill trigger. Sessions with the auto-memory feature enabled `Read` their
`MEMORY.md` before responding, so in any project where memory exists, every query —
including obvious positives — returns False on that first unrelated `Read`. Because the
memory read is model-chosen, this starts as intermittent flakiness and becomes a
permanent 0% recall once the project accumulates memory.
**Fix:** unrelated tool calls and non-matching `Read`s should keep the watch going; only
a completed `Skill` tool call is a routing decision (matching skill = triggered, a
different skill = not triggered), with the final `result` event or the timeout as the
no-trigger verdict.
## Bug 4 — exact-name matching under-counts with parallel workers
Each `ProcessPoolExecutor` worker registers its own uniquely-named temp copy
(`-skill-`) of the SAME description in the same project. A nested session
therefore sees N identical skills differing only in hex suffix and picks one
arbitrarily; matching the spawning worker's exact name under-counts triggers by roughly
a factor of N. Since every concurrent sibling within one `run_eval()` call carries the
description under test, detection should match the base prefix `-skill-`
instead of the exact unique name (the prefix still excludes an installed copy of the
base skill, which has no `-skill-` suffix).
## Suggested additional hardening
If a skill with the same *base* name as the skill under test is already installed in
the project the eval runs from, Claude may invoke the installed copy (whose name
differs from the unique temp name) → false miss. A warning when
`/.claude/skills/` exists would save users a confusing
debugging session.
## Environment
- Claude Code CLI 2.1.168, macOS (darwin 25.5)
- skill-creator plugin from claude-plugins-official marketplace, July 2026
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in skills/skill-creator/scripts/run_eval.py, especially run_single_query(), and note that run_loop.py depends on it. Reproduce with claude -p --output-format stream-json using command files versus .claude/skills//SKILL.md. Done means the harness detects real Skill tool calls reliably after indexing delay, ignores unrelated tools, and handles parallel temp skill names.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli, testing, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100