anthropics / anthropics/skills

[skill-creator] Trigger-eval harness silently broken on Windows: select() on pipes, probe files leak into global ~/.claude/commands, parallel twins cap recall at ~1/N

Open
#1,692 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
176k
Forks
20.9k
Avg merge
7h 21m
Merged PRs (30d)
5

Description

## Summary

The skill-creator description-optimization harness (`skills/skill-creator/scripts/run_eval.py` + `run_loop.py`) is structurally broken on Windows. Worse than crashing, it **fails silently**: every probe dies or mismeasures, the loop reports `recall=0%` (or a hard ~1/N cap), and then happily "improves" the description against pure noise for up to 5 iterations. Each probe is a full `claude -p` session that loads the user's entire MCP/skills context, so a user running the loop as documented burns a large share of their plan's usage limit on measurements that were never valid.

All four defects below were hit in one real session (Claude Code 2.1.247 desktop, Windows 11 Pro, Python 3.14.6, `anthropic-skills` plugin build) and confirmed present in this repo at current `main`.

## Defect 1 — `select.select()` on a subprocess pipe → every probe dies with WinError 10038

`run_eval.py:108`:

```python
ready, _, _ = select.select([process.stdout], [], [], 1.0)
```

On Windows, `select()` supports **sockets only**. Every single probe raises `[WinError 10038] An operation was attempted on something that is not a socket`, gets swallowed as `Warning: query failed`, and counts as "not triggered". The loop then prints a clean-looking `precision=100% recall=0%` table and proceeds to "improve" the description.

**Fix that worked locally:** replace the select loop with a daemon reader thread pumping `process.stdout.read1(8192)` into a `queue.Queue`, and `queue.get(timeout=1.0)` in the main loop. Cross-platform, keeps the early-kill-on-trigger behavior.

## Defect 2 — `find_project_root()` walks up to `$HOME` and writes probe files into the user's **global** `~/.claude/commands`

`run_eval.py:22` walks up from `Path.cwd()` until it finds a `.claude` directory. The documented invocation (`cd && python -m scripts.run_loop ...`) starts from a plugin directory with no `.claude`, so on a default setup the walk reaches the user's home — which **does** contain `~/.claude` — and `run_eval.py:53` then writes every probe command file into the user's global `~/.claude/commands/`.

Consequences observed:
- Probe skills (`-skill-`) appeared in the available-skills list of **every other session on the machine**, including unrelated interactive ones.
- A killed/interrupted run never reaches the `finally` cleanup: 10 orphaned probe command files were left permanently in the global commands dir.

## Defect 3 — parallel workers expose N twin copies to every probe → recall is capped at ~1/N

`run_eval.py:198` runs probes through `ProcessPoolExecutor` (default ~10 workers). Because all workers share the same project root (Defect 2 makes it global), each probe session sees ~10 identical skills with different UUID-suffixed names and invokes an arbitrary one — while the harness only counts a match on its **own** UUID name.

Measured effect: two consecutive iterations with materially different descriptions both scored exactly `recall=11%` train / `8%` test (~1/10 — collision probability, not signal). After isolating each probe in its own throwaway root, the same description scored 60/60.

**Fix that worked locally:** in `run_single_query`, ignore the shared `project_root` argument and use `tempfile.mkdtemp(prefix="skill-trigger-")` per probe, creating `.claude/commands/` inside it, with `shutil.rmtree(project_root, ignore_errors=True)` in `finally`. This also fixes Defect 2's leak and orphaned files.

## Defect 4 — `read_text()` without encoding → crash on non-Latin eval sets

`run_loop.py:261`:

```python
eval_set = json.loads(Path(args.eval_set).read_text())
```

On Windows this decodes with cp1252 and immediately raises `UnicodeDecodeError` for any UTF-8 eval set containing non-Latin queries (Arabic in our case — exactly the multilingual eval queries the skill's own docs recommend). Same pattern exists for other `read_text()`/`write_text()` calls. Workaround: `PYTHONUTF8=1`; proper fix: `encoding="utf-8"` everywhere.

## Repro

1. Windows 11, Claude Code CLI ≥ 2.1.x logged in, Python 3.13+.
2. Create any eval set with Arabic queries → Defect 4 crash.
3. Retry with `PYTHONUTF8=1` → Defect 1: wall of WinError 10038 warnings, `recall=0%`.
4. Patch Defect 1, hide nothing else → watch probe files appear in `~/.claude/commands` (Defect 2) and recall plateau at ~1/workers regardless of description (Defect 3).

## Suggested upstream fixes

- Thread-pump (or `asyncio` with Proactor) instead of `select()` on pipes.
- Per-probe isolated temp project root, cleaned in `finally` (fixes 2 + 3 together).
- Explicit `encoding="utf-8"` on all text I/O.
- A loud failure when >50% of probes error out — a harness that can't measure should stop the loop, not feed `recall=0%` into the improver and spend the user's quota on noise.

Happy to open a PR with the three local patches if useful — they took the harness from 100% invalid probes to a clean 60/60 run on the same machine.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with skills/skill-creator/scripts/run_eval.py and run_loop.py, reproducing the Windows failure with a UTF-8 eval set containing Arabic queries. Trace the subprocess output handling, project-root discovery, parallel probe isolation and text I/O, then verify probes remain isolated and cleaned up, multilingual input loads, and excessive probe failures stop the loop instead of producing misleading scores.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.