anthropics / anthropics/skills

skill-creator: trigger detection reports 0% recall for every skill

Offen
#1,721 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
176k
Forks
20.8k
Ø Merge
7 Std. 21 Min.
Gemergte PRs (30 T.)
5

Beschreibung

## Summary

`skill-creator`'s trigger evaluation reports `precision=100% recall=0%` for every skill, regardless of how good the description is. `run_loop` then "improves" the description against evidence that every positive query failed, and reports the result as an optimisation.

This fails silently. Nothing errors, and the output looks like a legitimate measurement of a bad description.

## Cause

`run_single_query()` in `skills/skill-creator/scripts/run_eval.py` decides whether the skill triggered by watching the `claude -p` event stream. Three paths return "did not trigger" before the evidence can arrive.

**1. Any other tool appearing first is treated as proof of failure** (line ~137):

```python
if tool_name in ("Skill", "Read"):
pending_tool_name = tool_name
accumulated_json = ""
else:
return False
```

A realistic query makes the model look around first — `Bash`, `Glob`, `Grep`. The harness sees that first tool, returns `False`, and kills the process. This bites hardest on exactly the queries the skill-creator guide asks authors to write: concrete ones, with file paths and backstory.

**2. The first `content_block_stop` ends the whole judgement:**

```python
elif se_type in ("content_block_stop", "message_stop"):
if pending_tool_name:
return clean_name in accumulated_json
```

A `Read` of an unrelated file before the `Skill` call closes its own block, and the match is decided against that block's accumulated input. A later `Skill` call is never seen.

**3. The non-streaming fallback returns inside its own loop:**

```python
for content_item in message.get("content", []):
...
return triggered
```

The `return` is inside the `for`, so only the first `tool_use` in an assistant message is examined.

## Evidence

Found while tuning the description of `reveal-doc-style`, an internal house-style skill for
protocol and regulatory prose at Reveal Genomics — it carries the writing rules and the
tracked-changes tooling used on a diagnostic assay's bridge study plan.

Raw stream for a query the harness scored as *not* triggering:

```
query: "i went through the bridge study plan last night and made a bunch of edits in
word. can you work out what i changed and roll it into section 7"
tool_use order: ['Skill', 'Bash', 'Bash', 'Bash', 'Read']
input_json_delta: '{"skill": "reveal-doc-style-skill-test01'
```

That query is representative of the whole set: concrete, with a real document and a real section
number, which is exactly what makes the model orient itself with `Bash` before or around the
`Skill` call.

The skill was the first thing the model reached for, and the harness still scored it 0.

Measured on a 20-query trigger set for `reveal-doc-style` — 11 should-trigger, 9 deliberately
near-miss negatives (a warehouse query, a docx-to-markdown conversion, a slide deck, a literature
lookup) — serially, one run per query:

| | shipped | patched |
|---|---|---|
| positives detected | 0 / 11 | 5 / 11 (original description) |
| positives detected | — | 9 / 11 (revised description) |
| false alarms | 0 / 9 | 0 / 9 |

The patched harness is what showed the original description was under-triggering, and what confirmed a revision fixed it. The shipped harness reported both as identically worthless.

## Fix

Keep scanning until the `result` event: don't treat other tools as disproof, reset per-block state instead of concluding from it, and return from the fallback only after the whole message has been walked. Happy to send a PR if useful.

## Also, possibly separate

With `--num-workers` above 1 the patched harness still returned zeros on queries that pass serially. Not diagnosed; all numbers above are `--num-workers 1`. Worth checking whether concurrent `claude -p` processes sharing one `.claude/commands/` directory interfere.

## Environment

Claude Code on Windows, running the harness under WSL. `claude plugin` CLI available; `anthropic-skills` plugin.

Beitragsleitfaden

Für dieses Repository ist kein Beitragsleitfaden indexiert

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.