anthropics / anthropics/skills
skill-creator: parallel trigger-eval workers share one commands dir, collapsing measured trigger rate to ~1/num_workers
- Linguagem predominante
- Python
- Estrelas
- 176k
- Forks
- 20.8k
- Merge médio
- 7h 21min
- PRs com merge (30d)
- 5
Descrição
## Summary
`skill-creator`'s trigger-eval harness writes every worker's temp command file into a **single shared directory**, so parallel workers see each other's files. Each `claude -p` process is presented with N identical skills differing only by UUID, picks one, and is scored as "triggered" only if it happens to pick its own.
The measured trigger rate therefore collapses toward `1/num_workers` **independently of the description being tested**. Since `--num-workers` defaults to 10, `run_loop.py` optimizes against roughly 10%-scaled noise.
## Affected code
`skills/skill-creator/scripts/run_eval.py`, `run_single_query()`:
```python
project_commands_dir = Path(project_root) / ".claude" / "commands"
command_file = project_commands_dir / f"{clean_name}.md"
```
`project_root` comes from `find_project_root()`, which walks up from cwd. It is the *same path for every worker*, and workers run concurrently via `ProcessPoolExecutor`. Detection then requires the run's own `clean_name`:
```python
if clean_name in accumulated_json:
return True
```
A run where the model triggers correctly but selects a sibling worker's identical skill is recorded as a non-trigger.
## Evidence
Same eval set (20 queries), same skill, same description, same model, varying only `--num-workers`:
| `--num-workers` | Recall | Precision |
|---|---|---|
| 10 (default) | 6% | 100% |
| 3 | 20%, then 0% | 100% |
| **1** | **80%** | 90% |
Recall tracks `1/N`. The description was never the variable.
Two independent confirmations:
1. An isolated manual probe — one command file, one `claude -p`, no concurrency — triggered on the first attempt for the query `"boot the whole ineedabaker stack…"`. Replaying `run_eval.py`'s own stream-parsing logic over that transcript returns `True`. The harness scored that identical query `0/3` under default settings, twice.
2. During a `--num-workers 3` run, the host surfaced two simultaneously registered skills differing only by UUID (`run-ineedabaker-skill-5d2f8649`, `run-ineedabaker-skill-e9edc6a8`). The directory is empty after each run, so cleanup works — the collision is purely concurrent.
## Impact
`run_loop.py` selects `best_description` by comparing scores that are dominated by this race. In our run it preferred a rewrite that was not better, and reported 6% recall for a description that measures 80% when evaluated correctly. Any conclusion drawn from the default configuration is unreliable.
## Reproduction
```bash
cd skills/skill-creator
python -m scripts.run_eval --eval-set evals.json --skill-path \
--num-workers 10 --runs-per-query 1 --verbose # recall collapses
python -m scripts.run_eval --eval-set evals.json --skill-path \
--num-workers 1 --runs-per-query 1 --verbose # recall recovers
```
## Suggested fix
Isolate each worker rather than sharing one commands directory — give every invocation its own temporary project root (`tempfile.mkdtemp()` containing `.claude/commands/`) and pass it as `cwd` to the subprocess. That keeps parallelism while guaranteeing exactly one candidate skill is visible per run.
Serializing (`--num-workers 1`) also works but makes the 5-iteration loop far slower than the docs imply.
Worth noting separately: `--timeout` defaults to 30s while 10 workers contend, which independently truncates slower runs. Isolation fixes the race; the timeout default may deserve its own look.
## Environment
- `anthropics/skills` @ `3b3fad96af16a10759d930941b4520ba0c40edae`
- Claude Code 2.1.251, macOS 15 (Darwin 25.6.0), Python 3.13
- Model: `claude-opus-5`
Guia de contribuição
Nenhum guia de contribuição indexado para este repositório
Avaliação
Esta issue ainda não foi avaliada.