modelcontextprotocol / modelcontextprotocol/conformance

Policy: do SHOULD-level (WARNING) checks count toward Tier-1 conformance?

Open
#245 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
127
Forks
101
Avg merge
6d 1h
Merged PRs (30d)
7

Description

SEP-1730 says Tier-1 requires "all conformance tests pass" / "100% compliance" but doesn't define whether SHOULD-level checks (which emit WARNING) count. Our own tooling disagrees with itself:

Layer Behavior Location
SEP-1730 undefined seps/1730
tier-check WARNING ignored in pass_rate (only SUCCESS/FAILURE counted) src/tier-check/checks/test-conformance-results.ts:55-57
CI runner warnings > 0overallFailure src/runner/client.ts:202

Today every Tier-1 SDK (Go, TypeScript, Python) shows 0 warnings. So the de-facto policy is "WARNINGs must be fixed," but tier-check would still award Tier-1 if one slipped through.

What a WARNING actually means here: since conformance runs against a maximally-configured everything-client, a WARNING signals "the SDK cannot be configured to satisfy this SHOULD" (missing API surface), not "a typical app skips it." That's a stronger statement than a SHOULD usually carries.

Decide and document one of:

  • (a) Tier-1 requires zero WARNINGs → align tier-check with client.ts:202; scenario authors keep using WARNING for SHOULD and it's effectively required
  • (b) Tier-1 tolerates WARNINGs → drop warnings > 0 from client.ts:202 overallFailure; WARNINGs become genuinely advisory
  • (c) Distinguish "SDK can't" (FAILURE) from "SDK can but everything-client didn't opt in" (WARNING) — probably impractical

Since it's roughly "0 cost" to do it today, and having SHOULD checks be required for Tier 1 makes the ecosystem stronger, I'm leaning towards (a).

Whichever we pick should be recorded in AGENTS.md and potentially proposed as a clarification to SEP-1730.

Came up reviewing #200 (SEP-2164), where the same SHOULD-vs-FAILURE question determines whether a wrong error code blocks Tier-1. Related: #243.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read SEP-1730, src/tier-check/checks/test-conformance-results.ts:55-57, and src/runner/client.ts:202, then review the related context in #200 and #243. Determine which WARNING policy the project adopts and document that decision in AGENTS.md, with a SEP-1730 clarification if required; done means the documented policy matches the relevant tooling behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
documentation, testing-qa
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.