anthropics / anthropics/skills

skill-creator run_eval.py reports 0% trigger rate for working descriptions

オープン
#1,559 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
176k
フォーク
20.8k
平均マージ
7時間 21分
マージ済み PR(30日)
5

説明

`skills/skill-creator/scripts/run_eval.py` reports a 0% trigger rate for descriptions that do trigger. Three separate causes. With all three addressed, a description that scored 0% scores 75–100%.

### 1. A non-skill first tool call ends the run

`run_eval.py:137-141`

```python
if tool_name in ("Skill", "Read"):
pending_tool_name = tool_name
accumulated_json = ""
else:
return False
```

With `--include-partial-messages`, the model commonly opens with a `Bash` orientation call (`git status`, `ls`) before invoking a skill. That `return False` ends the run before the skill can fire, so every query scores 0.

### 2. A different skill firing first also ends the run

`run_eval.py:150-154`

```python
elif se_type in ("content_block_stop", "message_stop"):
if pending_tool_name:
return clean_name in accumulated_json
```

If any other `Skill`/`Read` block completes first, the run returns a verdict immediately. Only a positive match is conclusive here; a non-match should reset and keep reading.

### 3. Concurrent runs share one commands directory

`run_eval.py:53` builds `/.claude/commands/` for every run, and `run_all` uses a `ProcessPoolExecutor`. Filenames are unique, so nothing is overwritten — but with N workers, N near-identical skill files are visible to every `claude -p` at once. The model picks among them and usually not the one being measured.

Measured on one query, same description:

| workers | trigger rate |
|---|---|
| 1 | 100% |
| 4 | 0% |
| 6 | 0% |

### Also worth documenting

Trigger rate depends on the working directory resembling a project the skill applies to. In an empty scratch directory the model explores with `Bash` and invokes nothing, so every description scores 0 regardless of quality. A rebase skill measured 0% in an empty dir and 100% in a git repo with migrations present.

### Patch

Attached below. Fixes 1 and 2 keep reading the stream instead of returning early. Fix 3 gives each run its own root under `.eval-runs//`, symlinking the project's top-level entries so the working directory still looks real, and removes it in `finally`.

After: 0% → 75% at 4 workers, 100% serial.

run_eval.py diff

```diff
@@ import shutil added to imports @@

unique_id = uuid.uuid4().hex[:8]
clean_name = f"{skill_name}-skill-{unique_id}"
- project_commands_dir = Path(project_root) / ".claude" / "commands"
+ run_root = Path(project_root) / ".eval-runs" / unique_id
+ project_commands_dir = run_root / ".claude" / "commands"
command_file = project_commands_dir / f"{clean_name}.md"

try:
project_commands_dir.mkdir(parents=True, exist_ok=True)
+ for entry in Path(project_root).iterdir():
+ if entry.name in (".claude", ".eval-runs"):
+ continue
+ link = run_root / entry.name
+ if not link.exists():
+ link.symlink_to(entry, target_is_directory=entry.is_dir())

@@ subprocess @@
- cwd=project_root,
+ cwd=str(run_root),

@@ content_block_start @@
else:
- return False
+ pending_tool_name = None
+ accumulated_json = ""

@@ content_block_stop @@
elif se_type in ("content_block_stop", "message_stop"):
if pending_tool_name:
- return clean_name in accumulated_json
- if se_type == "message_stop":
- return False
+ if clean_name in accumulated_json:
+ return True
+ pending_tool_name = None
+ accumulated_json = ""

@@ cleanup @@
finally:
- if command_file.exists():
- command_file.unlink()
+ shutil.rmtree(run_root, ignore_errors=True)
+ try:
+ run_root.parent.rmdir()
+ except OSError:
+ pass
```

Happy to open a PR if that is useful.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。