modelcontextprotocol / modelcontextprotocol/conformance
WARNING severity is inconsistent between client and server runners (client warnings can block Tier 1, server warnings cannot)
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 127
- Forks
- 101
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 7
Description
Problem
The two runners treat WARNING checks differently:
- Client runner (
src/runner/client.ts):overallFailure = failed > 0 || warnings > 0 || ...— any WARNING fails the scenario run. - Server runner (
src/runner/server.ts): the pass/fail denominator counts only SUCCESS and FAILURE; warnings are reported but never affect the result.
Since tier-check requires pass_rate = 1.0 for Tier 1 on both conformance and client_conformance (src/tier-check/tier-logic.ts), the same severity mapping (SHOULD → WARNING) is a Tier 1 blocker on the client side but cosmetic on the server side. Whether an SDK's tier is affected by a SHOULD-violation currently depends on which side of the wire the scenario tests, which is hard to defend in a tiering dispute.
Related: src/expected-failures.ts counts WARNING as failure in baseline mode, which is a third distinct behavior.
Ask
Pick one semantic and apply it in both runners (and baseline mode). Options:
- Warnings never fail runs or tiers anywhere (server behavior today); SHOULD compliance becomes report-only.
- Warnings fail everywhere (client behavior today); Tier 1 then requires SHOULD compliance on both sides.
- Keep warnings non-failing for the run but surface a separate SHOULD-compliance rate in tier-check so tiering can reference it explicitly.
Context: noticed while landing #322, whose DELETE-status check is a WARNING because the spec pins no success status (#429). No current Tier 1 SDK emits warnings on either side, so any of the above can land without demoting anyone today.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read src/runner/client.ts, src/runner/server.ts, src/expected-failures.ts, and src/tier-check/tier-logic.ts to compare how WARNING affects runs, baselines, and tier rates. Run the relevant conformance and tier-check tests, then confirm which warning semantic is chosen and verify that all three paths apply it consistently without changing current Tier 1 results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- testing-qa
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100