anthropics / anthropics/claude-plugins-official

skill-creator: run_eval trigger detection reports ~0 on current Claude Code (slash-command injection mismatch) + Windows select() bug

Open
#3,172 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
36.3k
Forks
4.1k
Avg merge
2d 14h
Merged PRs (30d)
539

Description

## Summary

In `plugins/skill-creator`, the **trigger-accuracy** tooling (`scripts/run_eval.py`, used by `scripts/run_loop.py`) under-reports trigger rates to ~0 on current Claude Code (tested 2.1.181 / 2.1.185), because its skill-injection and its trigger-detection no longer match how skills are surfaced/invoked. A well-formed skill that reliably triggers in production scores ≈0 with `run_eval`. There is also a separate Windows portability bug that makes the optimizer Unix-only.

Commit tested: `545162ba19782cb08668a5d1bb51a8e0368872ad`.

## Bug 1 — injection/detection mismatch (under-reports to ~0)

`run_eval.run_single_query` injects the skill as a **slash command** under `/.claude/commands/.md` (lines ~51–68), then detects a trigger only when the model's **first** streamed tool call is `Skill` or `Read` referencing that name (lines ~128–153):

```python
if cb.get("type") == "tool_use":
tool_name = cb.get("name", "")
if tool_name in ("Skill", "Read"):
pending_tool_name = tool_name
...
else:
return False # first tool isn't Skill/Read -> "not triggered"
```

On current Claude Code, a natural-language prompt does **not** invoke a custom slash command, and the model consults a real skill via the `Skill` tool — which only appears when the skill is installed as `.claude/skills//SKILL.md`, not as a command. So the detector observes either no tool use or a non-Skill first tool, and records "not triggered" for skills that actually trigger fine.

### Repro / evidence

Same skill (a research skill with a detailed description), same 20-query eval set, current claude, opus-tier model:

| Method | Positive trigger rate |
|---|---|
| `run_eval.py` (slash-command injection) | **1/30** runs |
| Inject as real skill `.claude/skills//SKILL.md`, watch for `Skill` tool | **3/3** strong positives; full set: **0.95 accuracy, 1.0 positive recall, 0.90 specificity** |

The slash-command path is blind to the real trigger; the real-skill path measures it correctly.

### Suggested fix

Inject the skill the way it actually ships — `/.claude/skills//SKILL.md` — and keep the existing `Skill`/`Read` first-tool detection (it works once the model can actually invoke the skill). Optionally also recognize the `SlashCommand` tool if you keep a command-based variant.

## Bug 2 — `select.select()` on a pipe breaks on Windows

`run_eval.py` (~line 108) polls the subprocess stdout **pipe** with `select.select([process.stdout], [], [], 1.0)`. On Windows `select()` accepts only sockets, so every query raises `OSError: [WinError 10038] An operation was attempted on something that is not a socket`, is swallowed, and reported as a 0.0 trigger rate. This makes `run_eval`/`run_loop` Unix-only.

### Suggested fix

Replace the `select`-based loop with a cross-platform reader (e.g. iterate `for line in process.stdout:` or a reader thread with a timeout). A line-buffered iterator over stdout preserves the early-exit behavior and works on Windows.

## Environment

- Claude Code 2.1.181 (bundled) / 2.1.185 (standalone), opus-4-8.
- Repro'd on Windows 11 (Bug 2) and WSL Ubuntu-24.04 (Bug 1, where `select` works).

Happy to send a PR for either fix if useful.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with plugins/skill-creator/scripts/run_eval.py, especially run_single_query, the skill injection path, trigger detection, and subprocess output loop; then check how scripts/run_loop.py consumes its results. Reproduce the evaluation on a Unix-like system and Windows, using the issue's current Claude Code versions and test skill. Done means real installed skills are detected accurately and evaluation runs without the Windows pipe error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
developer-experience, testing, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.