anthropics / anthropics/claude-plugins-official

skill-creator: run_eval.py omits --strict-mcp-config, so the operator's MCP servers are charged against a 30s per-query timeout

未关闭 适合新手
#5,624 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
36.3k
派生
4.1k
平均合并
2 天 14 小时
30 天内合并 PR
539

描述

## Summary

`skills/skill-creator/scripts/run_eval.py` builds its `claude -p` command without `--strict-mcp-config`, so every subprocess boots **the operator's entire MCP server set** before doing any model work. With `--timeout` defaulting to 30s, startup alone can consume the budget and the query is recorded as "did not trigger".

The consequence worth flagging: **the harness measures the operator's MCP configuration as much as the skill's description.** The same skill, same description, same eval set scores differently on two machines depending on what each has configured. Results are not comparable across users, and anyone with a rich MCP setup is silently penalised.

## Root cause

`run_single_query` in `scripts/run_eval.py`, current marketplace build (synced 2026-08-25):

```python
cmd = [
"claude",
"-p", query,
"--output-format", "stream-json",
"--verbose",
"--include-partial-messages",
]
```

No `--strict-mcp-config`, so the subprocess inherits the user's full MCP configuration. Meanwhile:

```python
parser.add_argument("--timeout", type=int, default=30, help="Timeout per query in seconds")
```

A Playwright MCP server launches a browser per subprocess. Multiply by `--num-workers` (default 10) and the machine is starting ten browsers to answer ten one-line prompts.

## Measurement

On a machine with a normal plugin set including Playwright, a single query took **85s wall**, of which roughly 10s was actual model work. Every query in that run reported as a miss against the 30s default.

## Fix

One line:

```diff
cmd = [
"claude",
"-p", query,
"--output-format", "stream-json",
"--verbose",
"--include-partial-messages",
+ "--strict-mcp-config",
]
```

This is also correct on the merits independent of the timeout. The trigger eval measures whether a *description* causes a skill to fire; the operator's MCP servers are not part of that question, and excluding them makes the measurement reproducible across machines.

## Why this is not one of the existing reports

I checked the open skill-creator eval issues before filing. This is a distinct mechanism from all of them:

- **#1749** — parallel workers cross-pollinate via shared `.claude/commands/`. Concurrency bug; reproduces with no MCP servers configured.
- **#2066**, **#1357 (Bug 2)** — stream parser returns `False` on the first non-`Skill`/`Read` tool. Parser bug; the subprocess starts fine.
- **#1357 (Bug 1)** — `using-superpowers` intercepts the first `Skill` call.
- **#1357 (Bug 3)**, **#2003** — an already-installed real skill wins over the synthetic command.
- **#3921**, **#2505**, **#3483** — Windows-specific (`select()` on pipes, encoding, CLI resolution).
- **#4425**, **#4692**, **#3172**, **#2678** — trigger route not detected on current Claude Code builds.

None of them involves subprocess startup cost. This one is visible as **wall-clock time per query far exceeding the model work**, and it disappears entirely on a machine with no MCP servers configured — which is probably why it has not surfaced: it is invisible in a clean CI environment and only bites operators with real setups.

It also **compounds** with the others rather than replacing them. In our case #1749 and this bug together produced a completely inverted result, and each needed its own fix before the numbers meant anything.

## Environment

- skill-creator from `claude-plugins-official`, marketplace synced 2026-08-25
- macOS Darwin 27.x
- Playwright, context7, and several other MCP servers configured

Happy to open a PR.

贡献指南

这个仓库没有索引到贡献指南

调研方向

Open skills/skill-creator/scripts/run_eval.py and inspect run_single_query, along with the argument parser's --timeout default. Verify the generated claude -p command uses strict MCP configuration, then run the evaluation with configured MCP servers and confirm queries are no longer consumed by MCP startup before the timeout.

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
cli, testing
Issue 类型
缺陷
难度
1/5
预计耗时
1 小时以内
活跃度
活跃
描述清晰度
描述清楚
新手友好度
92/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。