anthropics / anthropics/skills
skill-creator: description optimizer reports ~0% recall for skills that actually do trigger
- Lingua principale
- Python
- Stelle
- 176k
- Fork
- 20.8k
- Merge medio
- 7h 21m
- PR unite (30g)
- 5
Descrizione
# skill-creator: description optimizer reports ~0% recall for skills that actually do trigger
Two bugs in `example-skills/skills/skill-creator/scripts/run_eval.py` cause trigger evaluation to record a skill as "not triggered" when it demonstrably *is* triggered. Both fail in the same direction — they only ever turn a real trigger into a miss — so `run_loop.py` optimizes against a false signal and rewrites descriptions that were already fine.
## Symptom
Running the description optimization loop on a freshly written skill:
```
Train: 18/36 correct, precision=100% recall=0% accuracy=50%
Test : 13/24 correct, precision=100% recall=8% accuracy=54%
```
Every positive query scored `rate=0/3`. Capturing the raw `claude -p` stream for one of those queries and replaying it through `run_single_query`'s own detection logic returns `True` — the model called `Skill` with the expected `clean_name`. So the queries trigger; the harness fails to record it.
## Bug 1: the drained tail is never parsed
`run_single_query` reads the subprocess stream incrementally. When `process.poll()` shows the child has exited, it reads the remainder into `buffer` and then breaks out of the loop — but parsing only happens in the branch below, on freshly `os.read`-ed chunks. Whatever was in that final read is discarded unparsed, and the function falls through to `return triggered` (still `False`).
```python
while time.time() - start_time < timeout:
if process.poll() is not None:
remaining = process.stdout.read()
if remaining:
buffer += remaining.decode("utf-8", errors="replace")
break # <-- buffer never parsed
...
```
This hits whenever the child's output lands in one go, which is the common case for a query the model answers in a single turn.
## Bug 2: parallel workers share one commands directory
All workers write their temporary skill copy into the same `/.claude/commands/`, and each `claude -p` runs with `cwd=project_root`. With `--num-workers N`, every run therefore sees its own copy *plus* the N-1 copies belonging to the other in-flight runs — N near-identical descriptions competing for the same query. Detection requires the run's own `clean_name`, so a run has roughly a 1/N chance of being credited.
That is exactly the observed `recall=8%` at the default `--num-workers 10`. This one is nastier than bug 1 because it yields a plausible-looking nonzero number rather than an obvious zero, and it scales with worker count, so it reads as a description quality problem.
## Fix
Both are addressed in `run_eval.py`:
1. Factor the line-parsing block into a closure and call it on the drained remainder as well as on each streamed chunk, returning as soon as a verdict is reached.
2. Give each run its own `tempfile.mkdtemp()` project root, write the command file there, pass it as `cwd`, and `shutil.rmtree` it in the `finally`.
After the fix, on the same skill and eval set (20 queries x 3 runs, `--num-workers 10`):
```
Train: 36/36 correct, precision=100% recall=100% accuracy=100%
Test : 24/24 correct, precision=100% recall=100% accuracy=100%
```
A side effect of fix 2 worth noting: with a shared project root, the temp skill copies land in the *invoking* session's own commands directory, so they show up in that session's skill list while the eval runs. Per-run temp roots remove that leakage too.
Happy to open a PR with the patch if useful.
Environment: Claude Code 2.1.221, macOS (darwin 25.5.0), Python 3.12.
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.