anthropics / anthropics/claude-code

skill-creator: description optimization loop silently measures nothing on Windows (select() on pipe, then cp1252 report crash)

Đang mở
#89,575 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
area:skills bug has repro platform:windows
Ngôn ngữ chính
Python
Star
145k
Fork
23.1k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

## Summary

The `skill-creator` description-optimization loop (`scripts/run_loop.py` → `scripts/run_eval.py`)
is unusable on Windows. Two independent defects: every eval query fails silently, and the report
writer then crashes. The combination is bad because the loop still exits 0 and prints a plausible
result — `recall=0%` for every candidate description — which reads as "your skill never triggers"
rather than "nothing was measured".

Environment: Windows 11, Python 3.13.1, Claude Code 2.1.119, run from Git Bash.

## Bug 1 — `select.select()` on a subprocess pipe (Windows: sockets only)

`scripts/run_eval.py` polls the `claude -p` subprocess with:

```python
ready, _, _ = select.select([process.stdout], [], [], 1.0)
...
chunk = os.read(process.stdout.fileno(), 8192)
```

On Windows `select.select()` accepts sockets only, so this raises
`OSError: [WinError 10038] An operation was attempted on something that is not a socket`.

The exception is swallowed and reported as `Warning: query failed: ...`, once per query per run.
With 20 queries × 3 runs that is 60 identical warning lines, easily mistaken for network flakiness.

**Impact:** no query is ever evaluated. Every candidate description scores identically
(`precision=100% recall=0%`), so the "best" description is chosen by an arbitrary tie-break. The
run reports success.

**Repro:** run `python -m scripts.run_loop --eval-set --skill-path --model `
on Windows. `claude -p` itself works fine when invoked directly, which confirms the fault is in the
polling, not the CLI.

**Suggested fix:** replace the `select`-based polling with a reader thread feeding a queue —
portable and behaviourally identical:

```python
class _LineReader:
def __init__(self, stream):
self.q, self.eof = queue.Queue(), False
threading.Thread(target=self._pump, args=(stream,), daemon=True).start()

def _pump(self, stream):
try:
for raw in iter(stream.readline, b""):
self.q.put(raw)
finally:
self.q.put(None)

def read(self, timeout):
try:
item = self.q.get(timeout=timeout)
except queue.Empty:
return b""
if item is None:
self.eof = True
return b""
return item
```

then in the loop:

```python
chunk = reader.read(1.0)
if not chunk:
if reader.eof:
break
continue
```

Verified: with this change the loop evaluates queries correctly on Windows.

## Bug 2 — HTML report written with the locale encoding

`scripts/run_loop.py` writes the live report with `Path.write_text(...)` and no `encoding`, so
Windows uses cp1252. The report contains `✗` (U+2717), which cp1252 cannot encode:

```
UnicodeEncodeError: 'charmap' codec can't encode character '✗' in position 12931
```

This aborts the whole run after the evaluation work is already done, so the results are lost.

**Suggested fix:** pass `encoding="utf-8"` on every `write_text` that emits report HTML or JSON.
`PYTHONUTF8=1` works as a user-side workaround but should not be required.

## Suggestion — the harness measures a command, not a skill

Separate from the two defects. `run_eval.py` emulates the skill by writing a file into
`.claude/commands/` and checking whether Claude invokes it. In `claude -p` a slash command is not
invoked spontaneously, so this may under-report triggering even once the defects above are fixed.

Concretely: after fixing both bugs, the harness still reported `recall=0%` for my skill. Probing
the *installed* skill directly — same model, same queries, `claude -p` with the skill present in
`~/.claude/skills/` — it loaded on 5 of 5 queries and answered from its contents. So the harness
result and reality disagreed completely.

If the intent is to measure skill triggering, testing against an actually-installed skill would
reflect what users experience. At minimum it would be worth documenting that the numbers are
relative, not absolute.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Hướng nghiên cứu

Start with scripts/run_eval.py and trace the claude -p output polling on Windows, then inspect scripts/run_loop.py where the live HTML or JSON report is written. Reproduce the run with the provided command and eval set. Done means queries produce real measurements and the report completes on Windows without losing results to encoding errors.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
testing-qa, tooling
Loại issue
Lỗi
Độ khó
3/5
Thời gian dự kiến
1-2 ngày
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Đặc tả rõ ràng
Mức phù hợp với người mới
76/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.