anthropics / anthropics/claude-code

skill-creator: description optimization loop silently measures nothing on Windows (select() on pipe, then cp1252 report crash)

Ouverte
#89,575 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
area:skills bug has repro platform:windows
Langage dominant
Python
Étoiles
145k
Forks
23.1k
Métriques de merge des PR
Métriques de PR en attente

Description

## Summary

The `skill-creator` description-optimization loop (`scripts/run_loop.py` → `scripts/run_eval.py`)
is unusable on Windows. Two independent defects: every eval query fails silently, and the report
writer then crashes. The combination is bad because the loop still exits 0 and prints a plausible
result — `recall=0%` for every candidate description — which reads as "your skill never triggers"
rather than "nothing was measured".

Environment: Windows 11, Python 3.13.1, Claude Code 2.1.119, run from Git Bash.

## Bug 1 — `select.select()` on a subprocess pipe (Windows: sockets only)

`scripts/run_eval.py` polls the `claude -p` subprocess with:

```python
ready, _, _ = select.select([process.stdout], [], [], 1.0)
...
chunk = os.read(process.stdout.fileno(), 8192)
```

On Windows `select.select()` accepts sockets only, so this raises
`OSError: [WinError 10038] An operation was attempted on something that is not a socket`.

The exception is swallowed and reported as `Warning: query failed: ...`, once per query per run.
With 20 queries × 3 runs that is 60 identical warning lines, easily mistaken for network flakiness.

**Impact:** no query is ever evaluated. Every candidate description scores identically
(`precision=100% recall=0%`), so the "best" description is chosen by an arbitrary tie-break. The
run reports success.

**Repro:** run `python -m scripts.run_loop --eval-set --skill-path --model `
on Windows. `claude -p` itself works fine when invoked directly, which confirms the fault is in the
polling, not the CLI.

**Suggested fix:** replace the `select`-based polling with a reader thread feeding a queue —
portable and behaviourally identical:

```python
class _LineReader:
def __init__(self, stream):
self.q, self.eof = queue.Queue(), False
threading.Thread(target=self._pump, args=(stream,), daemon=True).start()

def _pump(self, stream):
try:
for raw in iter(stream.readline, b""):
self.q.put(raw)
finally:
self.q.put(None)

def read(self, timeout):
try:
item = self.q.get(timeout=timeout)
except queue.Empty:
return b""
if item is None:
self.eof = True
return b""
return item
```

then in the loop:

```python
chunk = reader.read(1.0)
if not chunk:
if reader.eof:
break
continue
```

Verified: with this change the loop evaluates queries correctly on Windows.

## Bug 2 — HTML report written with the locale encoding

`scripts/run_loop.py` writes the live report with `Path.write_text(...)` and no `encoding`, so
Windows uses cp1252. The report contains `✗` (U+2717), which cp1252 cannot encode:

```
UnicodeEncodeError: 'charmap' codec can't encode character '✗' in position 12931
```

This aborts the whole run after the evaluation work is already done, so the results are lost.

**Suggested fix:** pass `encoding="utf-8"` on every `write_text` that emits report HTML or JSON.
`PYTHONUTF8=1` works as a user-side workaround but should not be required.

## Suggestion — the harness measures a command, not a skill

Separate from the two defects. `run_eval.py` emulates the skill by writing a file into
`.claude/commands/` and checking whether Claude invokes it. In `claude -p` a slash command is not
invoked spontaneously, so this may under-report triggering even once the defects above are fixed.

Concretely: after fixing both bugs, the harness still reported `recall=0%` for my skill. Probing
the *installed* skill directly — same model, same queries, `claude -p` with the skill present in
`~/.claude/skills/` — it loaded on 5 of 5 queries and answered from its contents. So the harness
result and reality disagreed completely.

If the intent is to measure skill triggering, testing against an actually-installed skill would
reflect what users experience. At minimum it would be worth documenting that the numbers are
relative, not absolute.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Piste de recherche

Start with scripts/run_eval.py and trace the claude -p output polling on Windows, then inspect scripts/run_loop.py where the live HTML or JSON report is written. Reproduce the run with the provided command and eval set. Done means queries produce real measurements and the report completes on Windows without losing results to encoding errors.

Rédigé par le modèle d'indexation à partir du texte de l'issue.

Évaluation

Stack technique
python
Domaine
testing-qa, tooling
Type d'issue
Bug
Difficulté
3/5
Temps estimé
1-2 jours
Activité
Active
Clarté
Clairement spécifiée
Accessibilité débutants
76/100

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.