anthropics / anthropics/claude-plugins-official
skill-creator: description optimizer reports scores it never measured (recall structurally 0%) + 2 Windows blockers
- 主要言語
- Python
- スター
- 36.2k
- フォーク
- 4.1k
- PR マージ指標
- PR 指標を取得中
説明
## Summary
The `skill-creator` description-optimization loop (`scripts/run_loop.py` → `scripts/run_eval.py`) reports plausible-looking scores while **never actually measuring whether a skill triggers**. On top of that, two separate bugs make it fail outright on Windows.
I hit all of these while optimizing a real skill's description, so the details below are measured, not theoretical.
**Environment:** Windows 11, Python 3.14.5, Claude Code CLI (`claude -p` confirmed working standalone), `claude-plugins-official` → `skill-creator`.
---
## 1. Trigger detection never fires — recall is structurally 0% (likely all platforms)
`run_eval.py` registers the candidate skill by writing a **slash command** file:
```python
# run_eval.py ~L52-68
clean_name = f"{skill_name}-skill-{unique_id}"
project_commands_dir = Path(project_root) / ".claude" / "commands"
command_file = project_commands_dir / f"{clean_name}.md"
...
command_file.write_text(command_content)
```
…but it only counts a trigger when the **`Skill` tool** is invoked with that name:
```python
# run_eval.py ~L186-189
if tool_name == "Skill" and clean_name in tool_input.get("skill", ""):
triggered = True
elif tool_name == "Read" and clean_name in tool_input.get("file_path", ""):
triggered = True
```
These are two different mechanisms. An entry in `.claude/commands/` is a slash command (invoked via `SlashCommand` / typing `/name`), not a skill in `available_skills` — so the `Skill` tool essentially never fires for it, and `triggered` can never become `True`.
**Observed:** across 5 candidate descriptions (1 baseline + 4 generated), every iteration reported exactly:
```
Train: 18/36 correct, precision=100% recall=0% accuracy=50%
Test : 12/24 correct, precision=100% recall=0% accuracy=50%
```
Every `should_trigger: true` query failed; every `should_trigger: false` query "passed" trivially by nothing ever firing. Accuracy pins at exactly the negative fraction, so all candidates tie and `best_description` is effectively arbitrary.
**Control (same query, real installed skill, no harness):**
```bash
claude -p "" --output-format stream-json --verbose
```
```json
{"name":"Skill","input":{"skill":"runtime-truth","args":"..."}}
```
So the skill and description were fine all along — the harness could not see it. This is the most serious issue: **the tool silently reports a metric it cannot compute**, and a user following the documented workflow would apply an unvalidated description believing it was measured. I could not verify whether macOS/Linux behaves differently, but the registration/detection mismatch looks platform-independent.
**Suggested fix:** register the candidate as an actual skill (e.g. a temp skill dir with `SKILL.md` frontmatter carrying the candidate description) so it appears in `available_skills`; or, if the slash-command approach is intentional, also count `SlashCommand` invocations. Either way, a guard that errors out when recall is 0% across all candidates would have surfaced this immediately instead of emitting confident numbers.
---
## 2. `select.select()` on a pipe — Windows blocker
```python
# run_eval.py ~L108
ready, _, _ = select.select([process.stdout], [], [], 1.0)
```
On Windows `select.select()` accepts **only sockets**, so every query raises:
```
Warning: query failed: [WinError 10038] An operation was attempted on something that is not a socket
```
Every query fails, and because failures are swallowed as warnings the run still produces a scored report.
**Suggested fix:** read the pipe on a background thread into a `queue.Queue` (preserves incremental streaming and early trigger-detection short-circuit), or use `process.communicate(timeout=…)` if streaming isn't required. Patch that worked for me:
```python
import threading, queue as _queue
_chunks = _queue.Queue()
def _pipe_reader(fd, q):
try:
while True:
data = os.read(fd, 8192)
if not data:
break
q.put(data)
finally:
q.put(None)
threading.Thread(target=_pipe_reader,
args=(process.stdout.fileno(), _chunks), daemon=True).start()
while time.time() - start_time < timeout:
try:
chunk = _chunks.get(timeout=1.0)
except _queue.Empty:
if process.poll() is not None:
break
continue
if chunk is None:
break
buffer += chunk.decode("utf-8", errors="replace")
```
---
## 3. `UnicodeEncodeError` writing the HTML report — Windows blocker
```
File "scripts/run_loop.py", line 151, in run_loop
live_report_path.write_text(generate_html(partial_output, ...))
UnicodeEncodeError: 'charmap' codec can't encode character '✗' in position 14035
```
`Path.write_text()` uses the locale encoding (cp1252 on Windows), and the report contains `✗`/`✓`.
**Suggested fix:** `write_text(..., encoding="utf-8")` at every report/JSON write site (`--report none` does not avoid it, since the *live* report is written regardless).
---
## 4. Minor: orphaned command files are left behind
`run_eval.py` unlinks the temp command file in a `finally`, but when the run dies (crash in #3, or the user interrupts) the files survive. After my runs, 10 files remained in `.claude/commands/`:
```
runtime-truth-skill-199fa8fd.md
runtime-truth-skill-27daf233.md
… (10 total)
```
They then show up in `available_skills` in *every subsequent session*, cluttering the list and competing with the real skill during triggering. Worth a startup sweep of stale `*-skill-.md` files, or writing them to a temp dir outside `.claude/commands/`.
---
## Impact
Issues #2 and #3 make the description optimizer unusable on Windows out of the box. Issue #1 is worse than unusable: it produces confident, well-formatted metrics for something it never measured, and the documented workflow ends with "apply `best_description`" — so the failure mode is silently shipping an unvalidated description.
Happy to open a PR with the Windows fixes (#2/#3) if useful; #1 needs a maintainer decision on which registration mechanism is intended.
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
評価
この issue はまだ評価されていません。