skill-creator: trigger eval reports ~0% recall for almost any skill

# skill-creator: trigger eval reports ~0% recall for almost any skill

`scripts/run_eval.py` reports that a skill never triggers, for essentially any skill and any
description. `run_loop.py` then optimizes against that constant signal for its full iteration
budget, producing a "best description" chosen from noise.

The tell is a run whose numbers are **precision 100%, recall 0%, accuracy exactly 50%** on a
balanced eval set — every positive fails, and every negative "passes" only because nothing ever
fires.

```
Iteration 1/5
Train: 18/36 correct, precision=100% recall=0% accuracy=50% (182.6s)
Test : 12/24 correct, precision=100% recall=0% accuracy=50% (0.0s)
...
Best score: 4/8 (iteration 1)
```

Three independent causes compound. Each alone is enough to pin recall at zero, which is why the
first two fixes I tried each looked like they'd done nothing.

## Environment

- Claude Code on macOS (darwin 25.5.0)
- skill-creator from `anthropics/claude-plugins-official`, cache `fe10fbc4a554`
- Reproduced on `claude-opus-5`
- `run_eval.py` is byte-identical between cache `f394cf31d246` (2026-07-26) and `fe10fbc4a554`
(current), so this is not a recently-introduced regression.

## Cause 1 — the 30s default timeout expires before the skill is invoked

`--timeout` defaults to `30` and `--num-workers` to `10` (`run_eval.py:264-265`). A real agent run
that consults a skill takes far longer than that: in my capture, the target skill was invoked at
roughly the 200-second mark of a single unparallelized run. With ten agents contending, nearly every
run is killed before the skill is ever reached, and a killed run scores as "did not trigger".

This is the dominant cause. Raising `--timeout 240 --num-workers 4`, with no other change, moved a
control skill from 0/4 to 2/4 on positives.

## Cause 2 — detection returns on the *first* Skill/Read call, not the matching one

`run_eval.py:137-154`:

```python
if se_type == "content_block_start":
cb = se.get("content_block", {})
if cb.get("type") == "tool_use":
tool_name = cb.get("name", "")
if tool_name in ("Skill", "Read"):
pending_tool_name = tool_name
accumulated_json = ""
else:
return False # (a)

elif se_type in ("content_block_stop", "message_stop"):
if pending_tool_name:
return clean_name in accumulated_json # (b)
```

`(a)` gives up if the agent's first tool call is anything but `Skill`/`Read` — `Glob`, `Bash`, a
`TodoWrite`, or an MCP file tool that a `CLAUDE.md` steers toward. `(b)` gives up if the first
`Skill` call is some *other* skill.

`(b)` is the common path, because agents routinely consult a process skill before the domain one.
With the `superpowers` plugin installed, its SessionStart hook ("invoke relevant skills BEFORE any
response") makes a superpowers skill the near-universal first call. Recorded stream from a query
that should trigger:

```
1 Skill {"skill": "superpowers:systematic-debugging"}
2 Skill {"skill": "quokka-ledger-ops-skill-deadbeef"} <- the target, never reached
```

The target is invoked, and the harness reports `False`.

## Cause 3 — name collision when the skill under test is already installed

`run_single_query` writes a synthetic command `-skill-.md` into
`.claude/commands/` and detects on that name. If the real skill is installed and visible — a
user-global skill in `~/.claude/skills/` is visible from every cwd — the agent invokes the real
skill instead, whose name lacks the uuid suffix, and the match fails:

```
1 Skill {"skill": "my-installed-skill"} <- real skill; clean_name never matches
```

This also means candidate descriptions are never actually under test: the agent is reading the real
skill's on-disk description, not the candidate. Worth at minimum a documented precondition, or
detecting on both names and erroring when the installed skill wins.

## Reproduction

Deterministic, no API calls. Replays a recorded stream through the real detection code by pinning
`uuid4` and putting a fake `claude` on `PATH` that cats a fixture:

```python
# replay.py
import os, sys, types, uuid
from pathlib import Path

harness, fixture, skill = sys.argv[1], sys.argv[2], sys.argv[3]
os.environ["FIXTURE"] = fixture
os.environ["PATH"] = "/tmp/fakebin:" + os.environ["PATH"] # fakebin/claude = `exec cat "$FIXTURE"`
uuid.uuid4 = lambda: types.SimpleNamespace(hex="deadbeef" + "0" * 24)

sys.path.insert(0, harness)
from scripts import run_eval

print(run_eval.run_single_query(
query="replayed", skill_name=skill, skill_description="replayed",
timeout=30, project_root=str(Path.home()), model=None))
```

Against a fixture where the target skill is the second `Skill` call:

| harness | result | correct |
| --- | --- | --- |
| upstream | `False` | no — the skill was invoked |
| with the patch below | `True` | yes |

I can attach the fixture if useful.

## Suggested fixes

**Cause 1** — raise the default timeout substantially (180s+), or scale it with `--num-workers`. A
timeout expiry is currently indistinguishable from a genuine non-trigger; logging them separately
would have made this obvious immediately.

**Cause 2** — keep scanning subsequent tool calls instead of returning on the first, bounded so long
runs still terminate. Patch that flips the reproduction above from `False` to `True`:

```diff
+MAX_TOOL_CALLS = 8
@@
pending_tool_name = None
accumulated_json = ""
+ tool_calls_seen = 0
@@
if cb.get("type") == "tool_use":
tool_name = cb.get("name", "")
+ tool_calls_seen += 1
if tool_name in ("Skill", "Read"):
pending_tool_name = tool_name
accumulated_json = ""
else:
- return False
+ pending_tool_name = None
+ accumulated_json = ""
+ if tool_calls_seen > MAX_TOOL_CALLS:
+ return False
@@
- elif se_type in ("content_block_stop", "message_stop"):
- if pending_tool_name:
- return clean_name in accumulated_json
- if se_type == "message_stop":
- return False
+ elif se_type == "content_block_stop":
+ if pending_tool_name and clean_name in accumulated_json:
+ return True
+ pending_tool_name = None
+ accumulated_json = ""
@@
elif tool_name == "Read" and clean_name in tool_input.get("file_path", ""):
triggered = True
- return triggered
+ if triggered:
+ return True
```

The last hunk also fixes the `assistant` fallback returning after inspecting only the first
`tool_use` block in a message.

**Cause 3** — document the precondition, or have `run_loop.py` move the installed skill aside for the
duration and restore it (needs care: an interrupted run must not leave the user's skill missing).

**Separately**, `run_loop.py` could refuse to iterate when a baseline evaluation yields zero true
positives. Five iterations against an all-zero signal is pure waste, and the resulting
"best_description" is indistinguishable from a random pick.

## Not a bug

`Test : ... (0.0s)` in `run_loop.py` output looks alarming but is only a display artifact —
`run_loop.py:86-87` evaluates train and test in one batch for parallelism and attributes the whole
elapsed time to train. Flagging it so nobody else chases it.

Start with scripts/run_eval.py, especially the tool-call detection around lines 137-154 and the defaults around lines 264-265, then reproduce the deterministic case with replay.py and its recorded fixture. Compare the upstream result with the expected detection of a later target Skill call, and inspect scripts/run_loop.py for baseline handling and installed-skill interactions. Done means the reported trigger cases are evaluated correctly without treating timeout or name-collision failures as valid zero-recall results.

Written by the indexing model from the issue text.