anthropics / anthropics/claude-plugins-official
skill-creator: run_eval.py omits --strict-mcp-config, so the operator's MCP servers are charged against a 30s per-query timeout
- 主要言語
- Python
- スター
- 36.3k
- フォーク
- 4.1k
- 平均マージ
- 2日 14時間
- マージ済み PR(30日)
- 539
説明
## Summary
`skills/skill-creator/scripts/run_eval.py` builds its `claude -p` command without `--strict-mcp-config`, so every subprocess boots **the operator's entire MCP server set** before doing any model work. With `--timeout` defaulting to 30s, startup alone can consume the budget and the query is recorded as "did not trigger".
The consequence worth flagging: **the harness measures the operator's MCP configuration as much as the skill's description.** The same skill, same description, same eval set scores differently on two machines depending on what each has configured. Results are not comparable across users, and anyone with a rich MCP setup is silently penalised.
## Root cause
`run_single_query` in `scripts/run_eval.py`, current marketplace build (synced 2026-08-25):
```python
cmd = [
"claude",
"-p", query,
"--output-format", "stream-json",
"--verbose",
"--include-partial-messages",
]
```
No `--strict-mcp-config`, so the subprocess inherits the user's full MCP configuration. Meanwhile:
```python
parser.add_argument("--timeout", type=int, default=30, help="Timeout per query in seconds")
```
A Playwright MCP server launches a browser per subprocess. Multiply by `--num-workers` (default 10) and the machine is starting ten browsers to answer ten one-line prompts.
## Measurement
On a machine with a normal plugin set including Playwright, a single query took **85s wall**, of which roughly 10s was actual model work. Every query in that run reported as a miss against the 30s default.
## Fix
One line:
```diff
cmd = [
"claude",
"-p", query,
"--output-format", "stream-json",
"--verbose",
"--include-partial-messages",
+ "--strict-mcp-config",
]
```
This is also correct on the merits independent of the timeout. The trigger eval measures whether a *description* causes a skill to fire; the operator's MCP servers are not part of that question, and excluding them makes the measurement reproducible across machines.
## Why this is not one of the existing reports
I checked the open skill-creator eval issues before filing. This is a distinct mechanism from all of them:
- **#1749** — parallel workers cross-pollinate via shared `.claude/commands/`. Concurrency bug; reproduces with no MCP servers configured.
- **#2066**, **#1357 (Bug 2)** — stream parser returns `False` on the first non-`Skill`/`Read` tool. Parser bug; the subprocess starts fine.
- **#1357 (Bug 1)** — `using-superpowers` intercepts the first `Skill` call.
- **#1357 (Bug 3)**, **#2003** — an already-installed real skill wins over the synthetic command.
- **#3921**, **#2505**, **#3483** — Windows-specific (`select()` on pipes, encoding, CLI resolution).
- **#4425**, **#4692**, **#3172**, **#2678** — trigger route not detected on current Claude Code builds.
None of them involves subprocess startup cost. This one is visible as **wall-clock time per query far exceeding the model work**, and it disappears entirely on a machine with no MCP servers configured — which is probably why it has not surfaced: it is invisible in a clean CI environment and only bites operators with real setups.
It also **compounds** with the others rather than replacing them. In our case #1749 and this bug together produced a completely inverted result, and each needed its own fix before the numbers meant anything.
## Environment
- skill-creator from `claude-plugins-official`, marketplace synced 2026-08-25
- macOS Darwin 27.x
- Playwright, context7, and several other MCP servers configured
Happy to open a PR.
コントリビューションガイド
このリポジトリのコントリビューションガイドは索引されていません
調査の方向性
Open skills/skill-creator/scripts/run_eval.py and inspect run_single_query, along with the argument parser's --timeout default. Verify the generated claude -p command uses strict MCP configuration, then run the evaluation with configured MCP servers and confirm queries are no longer consumed by MCP startup before the timeout.
索引モデルが issue の本文から書いたものです。
評価
- 技術スタック
- python
- 領域
- cli, testing
- issue の種類
- バグ
- 難易度
- 1/5
- 見積もり時間
- 1時間未満
- 活発さ
- 活発
- 明瞭さ
- 明確に書かれている
- 初心者へのやさしさ
- 92/100