anthropics / anthropics/claude-plugins-official

skill-creator: run_eval.py omits --strict-mcp-config, so the operator's MCP servers are charged against a 30s per-query timeout

Open Beginner friendly
#5,624 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
36.3k
Forks
4.1k
Avg merge
2d 14h
Merged PRs (30d)
539

Description

## Summary

`skills/skill-creator/scripts/run_eval.py` builds its `claude -p` command without `--strict-mcp-config`, so every subprocess boots **the operator's entire MCP server set** before doing any model work. With `--timeout` defaulting to 30s, startup alone can consume the budget and the query is recorded as "did not trigger".

The consequence worth flagging: **the harness measures the operator's MCP configuration as much as the skill's description.** The same skill, same description, same eval set scores differently on two machines depending on what each has configured. Results are not comparable across users, and anyone with a rich MCP setup is silently penalised.

## Root cause

`run_single_query` in `scripts/run_eval.py`, current marketplace build (synced 2026-08-25):

```python
cmd = [
"claude",
"-p", query,
"--output-format", "stream-json",
"--verbose",
"--include-partial-messages",
]
```

No `--strict-mcp-config`, so the subprocess inherits the user's full MCP configuration. Meanwhile:

```python
parser.add_argument("--timeout", type=int, default=30, help="Timeout per query in seconds")
```

A Playwright MCP server launches a browser per subprocess. Multiply by `--num-workers` (default 10) and the machine is starting ten browsers to answer ten one-line prompts.

## Measurement

On a machine with a normal plugin set including Playwright, a single query took **85s wall**, of which roughly 10s was actual model work. Every query in that run reported as a miss against the 30s default.

## Fix

One line:

```diff
cmd = [
"claude",
"-p", query,
"--output-format", "stream-json",
"--verbose",
"--include-partial-messages",
+ "--strict-mcp-config",
]
```

This is also correct on the merits independent of the timeout. The trigger eval measures whether a *description* causes a skill to fire; the operator's MCP servers are not part of that question, and excluding them makes the measurement reproducible across machines.

## Why this is not one of the existing reports

I checked the open skill-creator eval issues before filing. This is a distinct mechanism from all of them:

- **#1749** — parallel workers cross-pollinate via shared `.claude/commands/`. Concurrency bug; reproduces with no MCP servers configured.
- **#2066**, **#1357 (Bug 2)** — stream parser returns `False` on the first non-`Skill`/`Read` tool. Parser bug; the subprocess starts fine.
- **#1357 (Bug 1)** — `using-superpowers` intercepts the first `Skill` call.
- **#1357 (Bug 3)**, **#2003** — an already-installed real skill wins over the synthetic command.
- **#3921**, **#2505**, **#3483** — Windows-specific (`select()` on pipes, encoding, CLI resolution).
- **#4425**, **#4692**, **#3172**, **#2678** — trigger route not detected on current Claude Code builds.

None of them involves subprocess startup cost. This one is visible as **wall-clock time per query far exceeding the model work**, and it disappears entirely on a machine with no MCP servers configured — which is probably why it has not surfaced: it is invisible in a clean CI environment and only bites operators with real setups.

It also **compounds** with the others rather than replacing them. In our case #1749 and this bug together produced a completely inverted result, and each needed its own fix before the numbers meant anything.

## Environment

- skill-creator from `claude-plugins-official`, marketplace synced 2026-08-25
- macOS Darwin 27.x
- Playwright, context7, and several other MCP servers configured

Happy to open a PR.

Contributor guide

No contributing guide indexed for this repository

Research direction

Open skills/skill-creator/scripts/run_eval.py and inspect run_single_query, along with the argument parser's --timeout default. Verify the generated claude -p command uses strict MCP configuration, then run the evaluation with configured MCP servers and confirm queries are no longer consumed by MCP startup before the timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, testing
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
92/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.