anthropics / anthropics/skills

skill-creator: parallel trigger-eval workers share one commands dir, collapsing measured trigger rate to ~1/num_workers

Open
#1,706 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
176k
Forks
20.9k
Avg merge
7h 21m
Merged PRs (30d)
5

Description

## Summary

`skill-creator`'s trigger-eval harness writes every worker's temp command file into a **single shared directory**, so parallel workers see each other's files. Each `claude -p` process is presented with N identical skills differing only by UUID, picks one, and is scored as "triggered" only if it happens to pick its own.

The measured trigger rate therefore collapses toward `1/num_workers` **independently of the description being tested**. Since `--num-workers` defaults to 10, `run_loop.py` optimizes against roughly 10%-scaled noise.

## Affected code

`skills/skill-creator/scripts/run_eval.py`, `run_single_query()`:

```python
project_commands_dir = Path(project_root) / ".claude" / "commands"
command_file = project_commands_dir / f"{clean_name}.md"
```

`project_root` comes from `find_project_root()`, which walks up from cwd. It is the *same path for every worker*, and workers run concurrently via `ProcessPoolExecutor`. Detection then requires the run's own `clean_name`:

```python
if clean_name in accumulated_json:
return True
```

A run where the model triggers correctly but selects a sibling worker's identical skill is recorded as a non-trigger.

## Evidence

Same eval set (20 queries), same skill, same description, same model, varying only `--num-workers`:

| `--num-workers` | Recall | Precision |
|---|---|---|
| 10 (default) | 6% | 100% |
| 3 | 20%, then 0% | 100% |
| **1** | **80%** | 90% |

Recall tracks `1/N`. The description was never the variable.

Two independent confirmations:

1. An isolated manual probe — one command file, one `claude -p`, no concurrency — triggered on the first attempt for the query `"boot the whole ineedabaker stack…"`. Replaying `run_eval.py`'s own stream-parsing logic over that transcript returns `True`. The harness scored that identical query `0/3` under default settings, twice.
2. During a `--num-workers 3` run, the host surfaced two simultaneously registered skills differing only by UUID (`run-ineedabaker-skill-5d2f8649`, `run-ineedabaker-skill-e9edc6a8`). The directory is empty after each run, so cleanup works — the collision is purely concurrent.

## Impact

`run_loop.py` selects `best_description` by comparing scores that are dominated by this race. In our run it preferred a rewrite that was not better, and reported 6% recall for a description that measures 80% when evaluated correctly. Any conclusion drawn from the default configuration is unreliable.

## Reproduction

```bash
cd skills/skill-creator
python -m scripts.run_eval --eval-set evals.json --skill-path \
--num-workers 10 --runs-per-query 1 --verbose # recall collapses
python -m scripts.run_eval --eval-set evals.json --skill-path \
--num-workers 1 --runs-per-query 1 --verbose # recall recovers
```

## Suggested fix

Isolate each worker rather than sharing one commands directory — give every invocation its own temporary project root (`tempfile.mkdtemp()` containing `.claude/commands/`) and pass it as `cwd` to the subprocess. That keeps parallelism while guaranteeing exactly one candidate skill is visible per run.

Serializing (`--num-workers 1`) also works but makes the 5-iteration loop far slower than the docs imply.

Worth noting separately: `--timeout` defaults to 30s while 10 workers contend, which independently truncates slower runs. Isolation fixes the race; the timeout default may deserve its own look.

## Environment

- `anthropics/skills` @ `3b3fad96af16a10759d930941b4520ba0c40edae`
- Claude Code 2.1.251, macOS 15 (Darwin 25.6.0), Python 3.13
- Model: `claude-opus-5`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in skills/skill-creator/scripts/run_eval.py at run_single_query(), then inspect find_project_root() and the subprocess invocation. Reproduce the recall difference with --num-workers 10 versus 1 from the issue; done means concurrent runs no longer expose sibling command files, preserve parallelism, and clean up their temporary directories.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
73/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.