modelcontextprotocol / modelcontextprotocol/conformance
Standardize how scenarios report setup/execution failures
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 127
- Forks
- 101
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 7
Description
A scenario that can't execute (connect failure, fixture missing, capability not advertised) should count as a FAILURE. The escape hatches are scenario filtering (--suite/--scenario) and the expected-failures baseline (#99) — not in-scenario skipping or silent passes.
Today every scenario hand-rolls this: most wrap the body in one big try/catch and emit their primary check ID as FAILURE on any error. That's fine for single-check scenarios but gets awkward with multiple checks at different severities (#200 discussion) — which check ID gets the connect failure is arbitrary, and the error message ends up under a label that has nothing to do with the actual problem.
Options:
- Emit all of the scenario's check IDs as FAILURE with the setup error message (consistent output shape)
- Emit a dedicated
<scenario-name>-setupcheck (semantically cleaner, but adds non-spec-mapped IDs) - A
client-helper.tsutility that scenarios call on setup failure, so the convention is enforced in one place
The helper is probably the best and then we can evolve the behavior from there. Having all the check-id's emitted requires a scenario knowing all its checks, which might be useful but is a larger change. The -setup check is probably reasonable as a first cut for this.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the scenario implementations and the proposed client-helper.ts utility, then compare how setup and execution failures are currently reported. Decide on the setup-failure convention described in the issue and verify that scenarios report failures consistently while preserving --suite, --scenario, and expected-failures behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- testing
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100