anthropics / anthropics/claude-plugins-official

skill-creator: run_eval.py omits --strict-mcp-config, so the operator's MCP servers are charged against a 30s per-query timeout

Đang mở Phù hợp với người mới
#5,624 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
36.3k
Fork
4.1k
Merge trung bình
2 ngày 14 giờ
Pull request đã merge (30 ngày)
539

Mô tả

## Summary

`skills/skill-creator/scripts/run_eval.py` builds its `claude -p` command without `--strict-mcp-config`, so every subprocess boots **the operator's entire MCP server set** before doing any model work. With `--timeout` defaulting to 30s, startup alone can consume the budget and the query is recorded as "did not trigger".

The consequence worth flagging: **the harness measures the operator's MCP configuration as much as the skill's description.** The same skill, same description, same eval set scores differently on two machines depending on what each has configured. Results are not comparable across users, and anyone with a rich MCP setup is silently penalised.

## Root cause

`run_single_query` in `scripts/run_eval.py`, current marketplace build (synced 2026-08-25):

```python
cmd = [
"claude",
"-p", query,
"--output-format", "stream-json",
"--verbose",
"--include-partial-messages",
]
```

No `--strict-mcp-config`, so the subprocess inherits the user's full MCP configuration. Meanwhile:

```python
parser.add_argument("--timeout", type=int, default=30, help="Timeout per query in seconds")
```

A Playwright MCP server launches a browser per subprocess. Multiply by `--num-workers` (default 10) and the machine is starting ten browsers to answer ten one-line prompts.

## Measurement

On a machine with a normal plugin set including Playwright, a single query took **85s wall**, of which roughly 10s was actual model work. Every query in that run reported as a miss against the 30s default.

## Fix

One line:

```diff
cmd = [
"claude",
"-p", query,
"--output-format", "stream-json",
"--verbose",
"--include-partial-messages",
+ "--strict-mcp-config",
]
```

This is also correct on the merits independent of the timeout. The trigger eval measures whether a *description* causes a skill to fire; the operator's MCP servers are not part of that question, and excluding them makes the measurement reproducible across machines.

## Why this is not one of the existing reports

I checked the open skill-creator eval issues before filing. This is a distinct mechanism from all of them:

- **#1749** — parallel workers cross-pollinate via shared `.claude/commands/`. Concurrency bug; reproduces with no MCP servers configured.
- **#2066**, **#1357 (Bug 2)** — stream parser returns `False` on the first non-`Skill`/`Read` tool. Parser bug; the subprocess starts fine.
- **#1357 (Bug 1)** — `using-superpowers` intercepts the first `Skill` call.
- **#1357 (Bug 3)**, **#2003** — an already-installed real skill wins over the synthetic command.
- **#3921**, **#2505**, **#3483** — Windows-specific (`select()` on pipes, encoding, CLI resolution).
- **#4425**, **#4692**, **#3172**, **#2678** — trigger route not detected on current Claude Code builds.

None of them involves subprocess startup cost. This one is visible as **wall-clock time per query far exceeding the model work**, and it disappears entirely on a machine with no MCP servers configured — which is probably why it has not surfaced: it is invisible in a clean CI environment and only bites operators with real setups.

It also **compounds** with the others rather than replacing them. In our case #1749 and this bug together produced a completely inverted result, and each needed its own fix before the numbers meant anything.

## Environment

- skill-creator from `claude-plugins-official`, marketplace synced 2026-08-25
- macOS Darwin 27.x
- Playwright, context7, and several other MCP servers configured

Happy to open a PR.

Hướng dẫn đóng góp

Chưa lập chỉ mục được hướng dẫn đóng góp cho kho mã nguồn này

Hướng nghiên cứu

Open skills/skill-creator/scripts/run_eval.py and inspect run_single_query, along with the argument parser's --timeout default. Verify the generated claude -p command uses strict MCP configuration, then run the evaluation with configured MCP servers and confirm queries are no longer consumed by MCP startup before the timeout.

Do mô hình lập chỉ mục viết ra từ nội dung của issue.

Đánh giá

Công nghệ
python
Lĩnh vực
cli, testing
Loại issue
Lỗi
Độ khó
1/5
Thời gian dự kiến
Dưới một giờ
Mức độ hoạt động
Sôi nổi
Độ rõ ràng
Đặc tả rõ ràng
Mức phù hợp với người mới
92/100

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.