skill-creator: description-optimization loop burned 100% of a Max subscription in ~3 minutes — false-negative eval bug + no cost guardrails

The description-optimization loop in skill-creator (scripts/run_loop.py → scripts/run_eval.py) consumed 100% of a Claude Max subscription's quota in roughly 3 minutes (121 claude -p invocations before crashing in iteration 2). Two independent problems compounded:

1. A correctness bug in run_eval.py produces false negatives (0% recall), which makes the loop believe the description never triggers, so it keeps iterating and burning quota on a problem that doesn't exist.
2. The optimization flow has no cost guardrails: SKILL.md instructs the agent to run the eval with the session's model, and the defaults multiply into hundreds of potential invocations with no warning or confirmation.

Bug 1: early returns in run_eval.py cause false negatives

run_single_query() decides "not triggered" far too early, in three places (scripts/run_eval.py (https://github.com/anthropics/skills/blob/main/skills/skill-creator/scripts/run_eval.py)):

- Lines 140–141: if the first tool_use block is anything other than Skill/Read (e.g. Claude runs a Grep, Glob, or TodoWrite before loading the skill), it hits else: return False and the run is scored as not-triggered — even though the skill loads one block later.
- Lines 150–154: message_stop returns False if no Skill/Read block was pending, so any first assistant message that ends without touching the skill fails the run, even when a later turn would invoke it.
- Line 168: the fallback branch returns after inspecting only the first tool_use item in the message.

In practice, models frequently do a quick search/read or write a todo list before invoking a skill, so should-trigger queries systematically score 0/3. Observed effect: precision=100% recall=0% on every iteration, which the improvement loop then "fixes" by rewriting a description that was never broken.

Suggested fix: only return early on a positive match; on non-matching blocks or message_stop, clear the pending state and keep consuming the stream. Score "not triggered" only at the result event or timeout. (Applied locally, works as expected.)

Bug 2: no cost guardrails on the optimization loop

- SKILL.md (line ~390) says: "Use the model ID from your system prompt (the one powering the current session)". When the session runs on a large reasoning model, every eval call generates thousands of thinking tokens that count against the user's subscription quota.
- Defaults multiply aggressively: --num-workers 10, --runs-per-query 3, --max-iterations 5, ~20-query eval set → up to ~300 claude -p invocations per optimization run. 10 parallel workers also produced a wave of 429s.
- Neither SKILL.md nor the scripts mention cost, quota, or ask for confirmation before launching. The words "cost"/"quota" don't appear in the optimization section at all.
- These subprocess invocations don't appear in the user's web history, so the quota drain looks inexplicable from the user's side.

Suggested fixes:
- Print an estimated invocation count (queries × runs × iterations) and require explicit confirmation before starting.
- Instruct/default the eval calls to a small model (haiku), or at minimum warn loudly when the chosen model is a reasoning-class model, and document the fidelity trade-off.
- Lower --num-workers default (10 concurrent sessions against one subscription trips rate limits anyway).

Environment

- skill-creator from this repo, installed via plugin marketplace (June 2026 snapshot; bugs verified present at current main)
- Claude Code CLI on macOS, Claude Max subscription
- Loop invoked as documented in SKILL.md § "Description Optimization", Step 3

Start with scripts/run_eval.py, especially run_single_query() around the cited early returns at lines 140-168, and reproduce the false-negative behavior with the documented description optimization flow. Then read scripts/run_loop.py and SKILL.md around the Description Optimization section and line ~390 for the cost-guardrail problem. Done means non-matching stream events no longer score early failure, and the loop warns or confirms before large eval runs.

索引モデルが issue の本文から書いたものです。