modelcontextprotocol / modelcontextprotocol/conformance
Policy: do SHOULD-level (WARNING) checks count toward Tier-1 conformance?
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 127
- Forks
- 101
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 7
Description
SEP-1730 says Tier-1 requires "all conformance tests pass" / "100% compliance" but doesn't define whether SHOULD-level checks (which emit WARNING) count. Our own tooling disagrees with itself:
| Layer | Behavior | Location |
|---|---|---|
| SEP-1730 | undefined | seps/1730 |
tier-check |
WARNING ignored in pass_rate (only SUCCESS/FAILURE counted) |
src/tier-check/checks/test-conformance-results.ts:55-57 |
| CI runner | warnings > 0 → overallFailure |
src/runner/client.ts:202 |
Today every Tier-1 SDK (Go, TypeScript, Python) shows 0 warnings. So the de-facto policy is "WARNINGs must be fixed," but tier-check would still award Tier-1 if one slipped through.
What a WARNING actually means here: since conformance runs against a maximally-configured everything-client, a WARNING signals "the SDK cannot be configured to satisfy this SHOULD" (missing API surface), not "a typical app skips it." That's a stronger statement than a SHOULD usually carries.
Decide and document one of:
- (a) Tier-1 requires zero WARNINGs → align
tier-checkwithclient.ts:202; scenario authors keep using WARNING for SHOULD and it's effectively required - (b) Tier-1 tolerates WARNINGs → drop
warnings > 0fromclient.ts:202overallFailure; WARNINGs become genuinely advisory - (c) Distinguish "SDK can't" (FAILURE) from "SDK can but everything-client didn't opt in" (WARNING) — probably impractical
Since it's roughly "0 cost" to do it today, and having SHOULD checks be required for Tier 1 makes the ecosystem stronger, I'm leaning towards (a).
Whichever we pick should be recorded in AGENTS.md and potentially proposed as a clarification to SEP-1730.
Came up reviewing #200 (SEP-2164), where the same SHOULD-vs-FAILURE question determines whether a wrong error code blocks Tier-1. Related: #243.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read SEP-1730, src/tier-check/checks/test-conformance-results.ts:55-57, and src/runner/client.ts:202, then review the related context in #200 and #243. Determine which WARNING policy the project adopts and document that decision in AGENTS.md, with a SEP-1730 clarification if required; done means the documented policy matches the relevant tooling behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- documentation, testing-qa
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100