anthropics / anthropics/skills

skill-creator: parallel trigger-eval workers share one commands dir, collapsing measured trigger rate to ~1/num_workers

Aberta
#1,706 1 comentário 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Python
Estrelas
176k
Forks
20.8k
Merge médio
7h 21min
PRs com merge (30d)
5

Descrição

## Summary

`skill-creator`'s trigger-eval harness writes every worker's temp command file into a **single shared directory**, so parallel workers see each other's files. Each `claude -p` process is presented with N identical skills differing only by UUID, picks one, and is scored as "triggered" only if it happens to pick its own.

The measured trigger rate therefore collapses toward `1/num_workers` **independently of the description being tested**. Since `--num-workers` defaults to 10, `run_loop.py` optimizes against roughly 10%-scaled noise.

## Affected code

`skills/skill-creator/scripts/run_eval.py`, `run_single_query()`:

```python
project_commands_dir = Path(project_root) / ".claude" / "commands"
command_file = project_commands_dir / f"{clean_name}.md"
```

`project_root` comes from `find_project_root()`, which walks up from cwd. It is the *same path for every worker*, and workers run concurrently via `ProcessPoolExecutor`. Detection then requires the run's own `clean_name`:

```python
if clean_name in accumulated_json:
return True
```

A run where the model triggers correctly but selects a sibling worker's identical skill is recorded as a non-trigger.

## Evidence

Same eval set (20 queries), same skill, same description, same model, varying only `--num-workers`:

| `--num-workers` | Recall | Precision |
|---|---|---|
| 10 (default) | 6% | 100% |
| 3 | 20%, then 0% | 100% |
| **1** | **80%** | 90% |

Recall tracks `1/N`. The description was never the variable.

Two independent confirmations:

1. An isolated manual probe — one command file, one `claude -p`, no concurrency — triggered on the first attempt for the query `"boot the whole ineedabaker stack…"`. Replaying `run_eval.py`'s own stream-parsing logic over that transcript returns `True`. The harness scored that identical query `0/3` under default settings, twice.
2. During a `--num-workers 3` run, the host surfaced two simultaneously registered skills differing only by UUID (`run-ineedabaker-skill-5d2f8649`, `run-ineedabaker-skill-e9edc6a8`). The directory is empty after each run, so cleanup works — the collision is purely concurrent.

## Impact

`run_loop.py` selects `best_description` by comparing scores that are dominated by this race. In our run it preferred a rewrite that was not better, and reported 6% recall for a description that measures 80% when evaluated correctly. Any conclusion drawn from the default configuration is unreliable.

## Reproduction

```bash
cd skills/skill-creator
python -m scripts.run_eval --eval-set evals.json --skill-path \
--num-workers 10 --runs-per-query 1 --verbose # recall collapses
python -m scripts.run_eval --eval-set evals.json --skill-path \
--num-workers 1 --runs-per-query 1 --verbose # recall recovers
```

## Suggested fix

Isolate each worker rather than sharing one commands directory — give every invocation its own temporary project root (`tempfile.mkdtemp()` containing `.claude/commands/`) and pass it as `cwd` to the subprocess. That keeps parallelism while guaranteeing exactly one candidate skill is visible per run.

Serializing (`--num-workers 1`) also works but makes the 5-iteration loop far slower than the docs imply.

Worth noting separately: `--timeout` defaults to 30s while 10 workers contend, which independently truncates slower runs. Isolation fixes the race; the timeout default may deserve its own look.

## Environment

- `anthropics/skills` @ `3b3fad96af16a10759d930941b4520ba0c40edae`
- Claude Code 2.1.251, macOS 15 (Darwin 25.6.0), Python 3.13
- Model: `claude-opus-5`

Guia de contribuição

Nenhum guia de contribuição indexado para este repositório

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.