skill-creator: description-eval harness cannot detect the trigger route it installs, and leaks its per-run command files

Two defects in `plugins/skill-creator/skills/skill-creator/scripts/run_eval.py` attack the
**measurement**, not the skill's advice. The first floors every candidate description's score
at the same number, so the optimizer loop reports "no wording helps" regardless of wording.
The second lets stale candidate descriptions from killed runs contaminate later runs. A third,
smaller issue — the plugin manifest has no `version` field — means a fix can never reach
installed caches even after it merges.

All line numbers below are against current `main`
(`2dc97c0d12ab25539cf5f85294c068511269ab09`, 2026-08-16); the installed cache copy verified
byte-identical to it.

## Environment

- Claude Code 2.1.233, Linux
- Plugin installed from the `claude-plugins-official` marketplace
- Cache path: `~/.claude/plugins/cache/claude-plugins-official/skill-creator/unknown/`
— note the literal `unknown` version component (see Defect 3)
- Harness: `skills/skill-creator/scripts/run_eval.py` (last upstream change to
`plugins/skill-creator`: `2a40fd2e7c52`, 2026-04-23)

## Defect 1: the detector cannot see the invocation route the harness itself sets up

`run_single_query` registers each candidate description as a **slash command** — a per-run
file written to `.claude/commands/-skill-.md` (`run_eval.py` lines 51–68):

```python
unique_id = uuid.uuid4().hex[:8]
clean_name = f"{skill_name}-skill-{unique_id}"
project_commands_dir = Path(project_root) / ".claude" / "commands"
command_file = project_commands_dir / f"{clean_name}.md"
...
command_file.write_text(command_content)
```

When Claude then triggers on that command, the tool_use in the stream is named
`SlashCommand`. But the detector's stream path accepts only `Skill` and `Read`, and returns
`False` for **anything else** (lines 133–141):

```python
if se_type == "content_block_start":
cb = se.get("content_block", {})
if cb.get("type") == "tool_use":
tool_name = cb.get("name", "")
if tool_name in ("Skill", "Read"):
pending_tool_name = tool_name
accumulated_json = ""
else:
return False
```

The string `SlashCommand` does not appear anywhere under `scripts/` (verified by grep). So
the one invocation route the harness installs is the one route it cannot score: a successful
trigger via the installed command file is recorded as a miss.

A second, independent bug sits in the fallback path for full assistant messages: the
`return triggered` is **inside** the `for` loop over content items (lines 157–168), so the
function returns after examining the first `tool_use` item regardless of what follows:

```python
elif event.get("type") == "assistant":
message = event.get("message", {})
for content_item in message.get("content", []):
if content_item.get("type") != "tool_use":
continue
tool_name = content_item.get("name", "")
tool_input = content_item.get("input", {})
if tool_name == "Skill" and clean_name in tool_input.get("skill", ""):
triggered = True
elif tool_name == "Read" and clean_name in tool_input.get("file_path", ""):
triggered = True
return triggered # <-- inside the loop: only the first tool_use is examined
```

A compounding factor: the worker discards the `claude` subprocess's stderr entirely
(`stderr=subprocess.DEVNULL`, line 88), so any per-query CLI failure also scores silently as
"not triggered" rather than as an error.

**Measured consequence** (2026-08-10/11): four different candidate descriptions all scored
**exactly 8/14** on a 14-query eval set. With should-trigger queries structurally unable to
register a hit, only the no-trigger queries can pass — the score is a floor determined by the
eval-set composition, not by the description. The honest-looking conclusion the loop hands
back is "no wording helps," which is precisely what a floored metric produces. The
`run_loop.py` optimizer then iterates `improve_description` against this noise.

**Suggested fix**: accept `SlashCommand` as a valid invocation in both detector paths — in
the stream path, treat `tool_name == "SlashCommand"` like `Skill`/`Read` and match
`clean_name` in its accumulated input; in the fallback path, check
`tool_input.get("command", "")` — and dedent `return triggered` out of the loop so every
content item is examined before returning.

## Defect 2: the per-run command file leaks when a worker dies

Cleanup of the per-run command file is a `finally` **inside the worker function**
(`run_eval.py` lines 179–181):

```python
finally:
if command_file.exists():
command_file.unlink()
```

Workers run in a `ProcessPoolExecutor` (line 198). A worker process that is killed —
parent interrupted or killed mid-run, pool torn down — never executes its `finally`, and
the file it wrote stays behind in the user's `.claude/commands/`, still carrying whatever
candidate description that run was testing. Nothing ever sweeps the directory: each run
uses a fresh `uuid` suffix, so stale files never collide and never get overwritten.

**Measured consequence** (2026-08-10/11): 13 stale `-skill-.md` files
accumulated in a project's `.claude/commands/` across interrupted runs. Transcripts from a
later eval showed Claude invoking a **stale** command by name — i.e. a run scored against a
description it was not testing. Combined with Defect 1 this is invisible: the stale
invocation is a `SlashCommand` tool_use, which the detector reports as a plain miss.

**Suggested fix**: move cleanup out of the worker — a pre-run sweep of
`.claude/commands/-skill-*.md` in `run_eval()` before submitting work, plus a
supervisor-level `finally` in the parent (which survives worker kills) that removes all
files created for the batch.

## Defect 3: no `version` field in plugin.json, so a fix can never reach installed caches

`.claude-plugin/plugin.json` carries `name`, `description`, and `author` — no `version`.
The install cache is keyed by version, so the plugin lands at
`~/.claude/plugins/cache/claude-plugins-official/skill-creator/unknown/` — the version
component is literally `unknown`. A cache keyed by version can never observe that `unknown`
changed: once the defects above are fixed upstream, installed copies will keep serving the
old files. **Suggested fix**: add a `version` field and bump it with the fix.

## Note

None of this touches the skill's written guidance, which is sound. The defects sit in the
instrument: the eval harness misreports whether a description triggered, and the
optimization loop then optimizes — and reports conclusions — against that misreading. A
user following the (correct) advice to "measure rather than guess" gets measurements that
are worse than guessing, delivered with the confidence of a harness.

One previously observed symptom is deliberately omitted: a rejection of bracketed model ids
(e.g. `claude-opus-5[1m]`) passed via `--model`. Re-tested on Claude Code 2.1.233, the CLI
now accepts the bracketed id and `modelUsage` confirms that model served the request, so it
no longer reproduces; the harness itself performs no model validation either way
(`run_loop.py` line 255 forwards the string verbatim).

Start with plugins/skill-creator/skills/skill-creator/scripts/run_eval.py, especially run_single_query, the stream and assistant-message detector paths, and ProcessPoolExecutor cleanup. Then inspect run_loop.py and .claude-plugin/plugin.json to understand optimizer use and plugin versioning. Done means command invocations are measured correctly, all batch-created command files are removed after interruptions, and installed caches can receive the fix.

Written by the indexing model from the issue text.