anthropics / anthropics/skills

[skill-creator] Trigger-eval harness silently broken on Windows: select() on pipes, probe files leak into global ~/.claude/commands, parallel twins cap recall at ~1/N

オープン
#1,692 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
176k
フォーク
20.8k
平均マージ
7時間 21分
マージ済み PR(30日)
5

説明

## Summary

The skill-creator description-optimization harness (`skills/skill-creator/scripts/run_eval.py` + `run_loop.py`) is structurally broken on Windows. Worse than crashing, it **fails silently**: every probe dies or mismeasures, the loop reports `recall=0%` (or a hard ~1/N cap), and then happily "improves" the description against pure noise for up to 5 iterations. Each probe is a full `claude -p` session that loads the user's entire MCP/skills context, so a user running the loop as documented burns a large share of their plan's usage limit on measurements that were never valid.

All four defects below were hit in one real session (Claude Code 2.1.247 desktop, Windows 11 Pro, Python 3.14.6, `anthropic-skills` plugin build) and confirmed present in this repo at current `main`.

## Defect 1 — `select.select()` on a subprocess pipe → every probe dies with WinError 10038

`run_eval.py:108`:

```python
ready, _, _ = select.select([process.stdout], [], [], 1.0)
```

On Windows, `select()` supports **sockets only**. Every single probe raises `[WinError 10038] An operation was attempted on something that is not a socket`, gets swallowed as `Warning: query failed`, and counts as "not triggered". The loop then prints a clean-looking `precision=100% recall=0%` table and proceeds to "improve" the description.

**Fix that worked locally:** replace the select loop with a daemon reader thread pumping `process.stdout.read1(8192)` into a `queue.Queue`, and `queue.get(timeout=1.0)` in the main loop. Cross-platform, keeps the early-kill-on-trigger behavior.

## Defect 2 — `find_project_root()` walks up to `$HOME` and writes probe files into the user's **global** `~/.claude/commands`

`run_eval.py:22` walks up from `Path.cwd()` until it finds a `.claude` directory. The documented invocation (`cd && python -m scripts.run_loop ...`) starts from a plugin directory with no `.claude`, so on a default setup the walk reaches the user's home — which **does** contain `~/.claude` — and `run_eval.py:53` then writes every probe command file into the user's global `~/.claude/commands/`.

Consequences observed:
- Probe skills (`-skill-`) appeared in the available-skills list of **every other session on the machine**, including unrelated interactive ones.
- A killed/interrupted run never reaches the `finally` cleanup: 10 orphaned probe command files were left permanently in the global commands dir.

## Defect 3 — parallel workers expose N twin copies to every probe → recall is capped at ~1/N

`run_eval.py:198` runs probes through `ProcessPoolExecutor` (default ~10 workers). Because all workers share the same project root (Defect 2 makes it global), each probe session sees ~10 identical skills with different UUID-suffixed names and invokes an arbitrary one — while the harness only counts a match on its **own** UUID name.

Measured effect: two consecutive iterations with materially different descriptions both scored exactly `recall=11%` train / `8%` test (~1/10 — collision probability, not signal). After isolating each probe in its own throwaway root, the same description scored 60/60.

**Fix that worked locally:** in `run_single_query`, ignore the shared `project_root` argument and use `tempfile.mkdtemp(prefix="skill-trigger-")` per probe, creating `.claude/commands/` inside it, with `shutil.rmtree(project_root, ignore_errors=True)` in `finally`. This also fixes Defect 2's leak and orphaned files.

## Defect 4 — `read_text()` without encoding → crash on non-Latin eval sets

`run_loop.py:261`:

```python
eval_set = json.loads(Path(args.eval_set).read_text())
```

On Windows this decodes with cp1252 and immediately raises `UnicodeDecodeError` for any UTF-8 eval set containing non-Latin queries (Arabic in our case — exactly the multilingual eval queries the skill's own docs recommend). Same pattern exists for other `read_text()`/`write_text()` calls. Workaround: `PYTHONUTF8=1`; proper fix: `encoding="utf-8"` everywhere.

## Repro

1. Windows 11, Claude Code CLI ≥ 2.1.x logged in, Python 3.13+.
2. Create any eval set with Arabic queries → Defect 4 crash.
3. Retry with `PYTHONUTF8=1` → Defect 1: wall of WinError 10038 warnings, `recall=0%`.
4. Patch Defect 1, hide nothing else → watch probe files appear in `~/.claude/commands` (Defect 2) and recall plateau at ~1/workers regardless of description (Defect 3).

## Suggested upstream fixes

- Thread-pump (or `asyncio` with Proactor) instead of `select()` on pipes.
- Per-probe isolated temp project root, cleaned in `finally` (fixes 2 + 3 together).
- Explicit `encoding="utf-8"` on all text I/O.
- A loud failure when >50% of probes error out — a harness that can't measure should stop the loop, not feed `recall=0%` into the improver and spend the user's quota on noise.

Happy to open a PR with the three local patches if useful — they took the harness from 100% invalid probes to a clean 60/60 run on the same machine.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。