skill-creator trigger eval scores 0% recall when the skill is already installed (and under parallel workers)

## Summary

`skill-creator`'s trigger eval (`scripts/run_eval.py`) reports **0% recall for every candidate description** (while precision stays 100%) whenever either of two conditions holds. Both are structural in the detection logic, not description quality. Verified on Claude Code CLI v2.1.204 by capturing the full `stream-json` output of eval sessions.

## Root cause 1: the skill under test is already installed

`run_single_query` registers a temp copy of the skill as `.claude/commands/-skill-.md` and counts a trigger only if the `Skill`/`Read` tool input contains that unique hashed name.

If the real skill is installed (e.g. `~/.claude/skills//`), the session sees **both** the real skill and the temp copy. The model reliably invokes the canonical name:

```json
{"skill": "topical-authority", "args": "..."} // real skill, triggered at ~10s
```

`"topical-authority-skill-a1b2c3d4" in accumulated_json` is False, so a genuine trigger is scored as a miss. Every should-trigger query fails the same way, recall is exactly 0%, and all should-not-trigger queries pass, so the loop looks healthy (precision 100%) while measuring nothing. This is the common case for the documented workflow of improving an *existing* skill's description.

Side effect: because detection never returns True early, the child session keeps running and actually starts **executing the real skill** (real Bash/tool calls in the user's environment) until the per-query timeout kills it, for every should-trigger query x runs_per_query x iterations.

## Root cause 2: parallel workers register identical competing copies

With `--num-workers N` (default 10), up to N sessions run concurrently and each registers its own temp copy in the same `.claude/commands/` dir. All copies carry the **same candidate description**, differing only in hash. Every session sees all of them, and the model picks one arbitrarily, so a worker scores a trigger only when the model happens to pick *its* hash (~1/N chance). This depresses recall even when the skill is not installed.

## Reproduction

1. Have the skill installed under `~/.claude/skills//`.
2. Run `scripts/run_eval.py` (or `run_loop.py`) on that skill with any eval set containing should-trigger queries.
3. Observe recall 0%, precision 100% on every iteration.
4. Capture one query's stream (`claude -p "" --output-format stream-json --verbose --include-partial-messages` with a temp command registered): the `Skill` tool_use appears within seconds, with the **real** skill name as input.

## Fix (verified locally)

Two small changes to `run_single_query` / `run_eval`:

1. **Shadow the installed skill during the eval batch.** Rename `~/.claude/skills/` (and `/.claude/skills/`) to a path *outside* the skills dir for the duration of the batch, restore in `finally`. Note: renaming in place inside `skills/` is not enough; a renamed dir is still discovered and registered under its new name.

2. **Match the shared prefix, not the per-worker hash.** Within one eval batch all temp copies carry the same candidate description, so a trigger on any of them is a valid trigger for that description:

```python
match_token = f"{skill_name}-skill-" # instead of clean_name
```

used in the three detection points (input_json_delta accumulation, content_block_stop check, and the full-assistant-message fallback).

After both changes, the same eval set goes from 0% recall to correct detection (should-trigger queries detected within ~10s, should-not-trigger still quiet), and the installed skill is restored cleanly.

Patch against current `skills/skill-creator/scripts/run_eval.py`:

```diff
--- a/skills/skill-creator/scripts/run_eval.py
+++ b/skills/skill-creator/scripts/run_eval.py
@@ -6,6 +6,7 @@
"""

import argparse
+import contextlib
import json
import os
import select
@@ -19,6 +20,38 @@
from scripts.utils import parse_skill_md


+@contextlib.contextmanager
+def shadow_installed_skill(skill_name: str, project_root: Path):
+ """Temporarily hide installed copies of the skill under test.
+
+ If the skill is already installed (user or project skills dir), claude -p
+ sees both the real skill and the temp eval command and invokes the real
+ one by its canonical name. Detection matches only the temp copy's unique
+ name, so every real trigger scores as a miss and recall collapses to 0%.
+ Renaming the installed copy for the duration of the eval leaves the temp
+ copy (carrying the candidate description) as the only triggerable target.
+ """
+ candidates = [
+ Path.home() / ".claude" / "skills" / skill_name,
+ Path(project_root) / ".claude" / "skills" / skill_name,
+ ]
+ renamed = []
+ try:
+ for d in candidates:
+ if d.is_dir():
+ # Move OUT of the skills dir: a renamed dir still inside
+ # skills/ is discovered and registered under its new name.
+ target = d.parent.parent / (d.name + ".eval-shadow")
+ if not target.exists():
+ d.rename(target)
+ renamed.append((d, target))
+ yield
+ finally:
+ for orig, target in renamed:
+ if target.exists() and not orig.exists():
+ target.rename(orig)
+
+
def find_project_root() -> Path:
"""Find the project root by walking up from cwd looking for .claude/.

@@ -50,6 +83,11 @@
"""
unique_id = uuid.uuid4().hex[:8]
clean_name = f"{skill_name}-skill-{unique_id}"
+ # Parallel workers register identical-description copies under different
+ # hashes, all visible to every session; the model picks one arbitrarily.
+ # Within one eval batch the description is the same for all copies, so a
+ # trigger on ANY of them is a valid trigger for the candidate description.
+ match_token = f"{skill_name}-skill-"
project_commands_dir = Path(project_root) / ".claude" / "commands"
command_file = project_commands_dir / f"{clean_name}.md"

@@ -144,12 +182,12 @@
delta = se.get("delta", {})
if delta.get("type") == "input_json_delta":
accumulated_json += delta.get("partial_json", "")
- if clean_name in accumulated_json:
+ if match_token in accumulated_json:
return True

elif se_type in ("content_block_stop", "message_stop"):
if pending_tool_name:
- return clean_name in accumulated_json
+ return match_token in accumulated_json
if se_type == "message_stop":
return False

@@ -161,9 +199,9 @@
continue
tool_name = content_item.get("name", "")
tool_input = content_item.get("input", {})
- if tool_name == "Skill" and clean_name in tool_input.get("skill", ""):
+ if tool_name == "Skill" and match_token in tool_input.get("skill", ""):
triggered = True
- elif tool_name == "Read" and clean_name in tool_input.get("file_path", ""):
+ elif tool_name == "Read" and match_token in tool_input.get("file_path", ""):
triggered = True
return triggered

@@ -195,7 +233,8 @@
"""Run the full eval set and return results."""
results = []

- with ProcessPoolExecutor(max_workers=num_workers) as executor:
+ with shadow_installed_skill(skill_name, project_root), \
+ ProcessPoolExecutor(max_workers=num_workers) as executor:
future_to_info = {}
for item in eval_set:
for run_idx in range(runs_per_query):
```