anthropics / anthropics/claude-plugins-official
skill-creator: run_eval.py omits --strict-mcp-config, so the operator's MCP servers are charged against a 30s per-query timeout
- Ngôn ngữ chính
- Python
- Star
- 36.3k
- Fork
- 4.1k
- Merge trung bình
- 2 ngày 14 giờ
- Pull request đã merge (30 ngày)
- 539
Mô tả
## Summary
`skills/skill-creator/scripts/run_eval.py` builds its `claude -p` command without `--strict-mcp-config`, so every subprocess boots **the operator's entire MCP server set** before doing any model work. With `--timeout` defaulting to 30s, startup alone can consume the budget and the query is recorded as "did not trigger".
The consequence worth flagging: **the harness measures the operator's MCP configuration as much as the skill's description.** The same skill, same description, same eval set scores differently on two machines depending on what each has configured. Results are not comparable across users, and anyone with a rich MCP setup is silently penalised.
## Root cause
`run_single_query` in `scripts/run_eval.py`, current marketplace build (synced 2026-08-25):
```python
cmd = [
"claude",
"-p", query,
"--output-format", "stream-json",
"--verbose",
"--include-partial-messages",
]
```
No `--strict-mcp-config`, so the subprocess inherits the user's full MCP configuration. Meanwhile:
```python
parser.add_argument("--timeout", type=int, default=30, help="Timeout per query in seconds")
```
A Playwright MCP server launches a browser per subprocess. Multiply by `--num-workers` (default 10) and the machine is starting ten browsers to answer ten one-line prompts.
## Measurement
On a machine with a normal plugin set including Playwright, a single query took **85s wall**, of which roughly 10s was actual model work. Every query in that run reported as a miss against the 30s default.
## Fix
One line:
```diff
cmd = [
"claude",
"-p", query,
"--output-format", "stream-json",
"--verbose",
"--include-partial-messages",
+ "--strict-mcp-config",
]
```
This is also correct on the merits independent of the timeout. The trigger eval measures whether a *description* causes a skill to fire; the operator's MCP servers are not part of that question, and excluding them makes the measurement reproducible across machines.
## Why this is not one of the existing reports
I checked the open skill-creator eval issues before filing. This is a distinct mechanism from all of them:
- **#1749** — parallel workers cross-pollinate via shared `.claude/commands/`. Concurrency bug; reproduces with no MCP servers configured.
- **#2066**, **#1357 (Bug 2)** — stream parser returns `False` on the first non-`Skill`/`Read` tool. Parser bug; the subprocess starts fine.
- **#1357 (Bug 1)** — `using-superpowers` intercepts the first `Skill` call.
- **#1357 (Bug 3)**, **#2003** — an already-installed real skill wins over the synthetic command.
- **#3921**, **#2505**, **#3483** — Windows-specific (`select()` on pipes, encoding, CLI resolution).
- **#4425**, **#4692**, **#3172**, **#2678** — trigger route not detected on current Claude Code builds.
None of them involves subprocess startup cost. This one is visible as **wall-clock time per query far exceeding the model work**, and it disappears entirely on a machine with no MCP servers configured — which is probably why it has not surfaced: it is invisible in a clean CI environment and only bites operators with real setups.
It also **compounds** with the others rather than replacing them. In our case #1749 and this bug together produced a completely inverted result, and each needed its own fix before the numbers meant anything.
## Environment
- skill-creator from `claude-plugins-official`, marketplace synced 2026-08-25
- macOS Darwin 27.x
- Playwright, context7, and several other MCP servers configured
Happy to open a PR.
Hướng dẫn đóng góp
Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này
Hướng nghiên cứu
Open skills/skill-creator/scripts/run_eval.py and inspect run_single_query, along with the argument parser's --timeout default. Verify the generated claude -p command uses strict MCP configuration, then run the evaluation with configured MCP servers and confirm queries are no longer consumed by MCP startup before the timeout.
Do mô hình lập chỉ mục viết ra từ nội dung của issue.
Đánh giá
- Công nghệ
- python
- Lĩnh vực
- cli, testing
- Loại issue
- Lỗi
- Độ khó
- 1/5
- Thời gian dự kiến
- Dưới một giờ
- Mức độ hoạt động
- Sôi nổi
- Độ rõ ràng
- Đặc tả rõ ràng
- Mức phù hợp với người mới
- 92/100