anthropics / anthropics/skills
skill-creator run_eval.py reports 0% trigger rate for working descriptions
- 主要言語
- Python
- スター
- 176k
- フォーク
- 20.8k
- 平均マージ
- 7時間 21分
- マージ済み PR(30日)
- 5
説明
`skills/skill-creator/scripts/run_eval.py` reports a 0% trigger rate for descriptions that do trigger. Three separate causes. With all three addressed, a description that scored 0% scores 75–100%.
### 1. A non-skill first tool call ends the run
`run_eval.py:137-141`
```python
if tool_name in ("Skill", "Read"):
pending_tool_name = tool_name
accumulated_json = ""
else:
return False
```
With `--include-partial-messages`, the model commonly opens with a `Bash` orientation call (`git status`, `ls`) before invoking a skill. That `return False` ends the run before the skill can fire, so every query scores 0.
### 2. A different skill firing first also ends the run
`run_eval.py:150-154`
```python
elif se_type in ("content_block_stop", "message_stop"):
if pending_tool_name:
return clean_name in accumulated_json
```
If any other `Skill`/`Read` block completes first, the run returns a verdict immediately. Only a positive match is conclusive here; a non-match should reset and keep reading.
### 3. Concurrent runs share one commands directory
`run_eval.py:53` builds `/.claude/commands/` for every run, and `run_all` uses a `ProcessPoolExecutor`. Filenames are unique, so nothing is overwritten — but with N workers, N near-identical skill files are visible to every `claude -p` at once. The model picks among them and usually not the one being measured.
Measured on one query, same description:
| workers | trigger rate |
|---|---|
| 1 | 100% |
| 4 | 0% |
| 6 | 0% |
### Also worth documenting
Trigger rate depends on the working directory resembling a project the skill applies to. In an empty scratch directory the model explores with `Bash` and invokes nothing, so every description scores 0 regardless of quality. A rebase skill measured 0% in an empty dir and 100% in a git repo with migrations present.
### Patch
Attached below. Fixes 1 and 2 keep reading the stream instead of returning early. Fix 3 gives each run its own root under `.eval-runs//`, symlinking the project's top-level entries so the working directory still looks real, and removes it in `finally`.
After: 0% → 75% at 4 workers, 100% serial.
run_eval.py diff
```diff
@@ import shutil added to imports @@
unique_id = uuid.uuid4().hex[:8]
clean_name = f"{skill_name}-skill-{unique_id}"
- project_commands_dir = Path(project_root) / ".claude" / "commands"
+ run_root = Path(project_root) / ".eval-runs" / unique_id
+ project_commands_dir = run_root / ".claude" / "commands"
command_file = project_commands_dir / f"{clean_name}.md"
try:
project_commands_dir.mkdir(parents=True, exist_ok=True)
+ for entry in Path(project_root).iterdir():
+ if entry.name in (".claude", ".eval-runs"):
+ continue
+ link = run_root / entry.name
+ if not link.exists():
+ link.symlink_to(entry, target_is_directory=entry.is_dir())
@@ subprocess @@
- cwd=project_root,
+ cwd=str(run_root),
@@ content_block_start @@
else:
- return False
+ pending_tool_name = None
+ accumulated_json = ""
@@ content_block_stop @@
elif se_type in ("content_block_stop", "message_stop"):
if pending_tool_name:
- return clean_name in accumulated_json
- if se_type == "message_stop":
- return False
+ if clean_name in accumulated_json:
+ return True
+ pending_tool_name = None
+ accumulated_json = ""
@@ cleanup @@
finally:
- if command_file.exists():
- command_file.unlink()
+ shutil.rmtree(run_root, ignore_errors=True)
+ try:
+ run_root.parent.rmdir()
+ except OSError:
+ pass
```
Happy to open a PR if that is useful.
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
評価
この issue はまだ評価されていません。