modelcontextprotocol / modelcontextprotocol/conformance
SEP-2575: remaining conformance tests
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 127
- Forks
- 101
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 7
Description
Status as of the 70b2800d traceability refresh (run against typescript-sdk@main). 8 of 26 declared requirements are tested, 18 are untested. Of the untested 18, 7 have a scenario already written but the reference SDK can't run it yet, and 11 have no scenario at all.
A. Scenario written — blocked on typescript-sdk support
src/scenarios/client/request-metadata.ts (#270) emits all 7 of these, but everythingClient.ts on typescript-sdk@main has no request-metadata handler — it logs Unknown scenario and exits before any HTTP request is made, so no checks are emitted. These will flip to tested once the SDK has a SEP-2575-aware conformance client, or we fix the checker to be more correct.
So no scenario action needed on these, this is a tracking bug.
| Check | Requirement | Level |
|---|---|---|
sep-2575-client-populates-meta |
Every client request includes protocolVersion, clientInfo, clientCapabilities in _meta |
MUST |
sep-2575-http-client-sends-version-header |
Every POST includes MCP-Protocol-Version header |
MUST |
sep-2575-http-version-header-matches-meta |
Header value matches _meta.…/protocolVersion |
MUST |
sep-2575-client-declares-roots-capability |
Roots-capable clients declare roots in _meta.…/clientCapabilities |
MUST (cond.) |
sep-2575-client-declares-sampling-capability |
Sampling-capable clients declare sampling |
MUST (cond.) |
sep-2575-client-declares-elicitation-capability |
Elicitation-capable clients declare elicitation |
MUST (cond.) |
sep-2575-client-retry-supported-version |
Client retries with a mutually-supported version after UnsupportedProtocolVersionError |
SHOULD |
B. No scenario yet — seems straightforward to write
| Check | Requirement | Level | Notes |
|---|---|---|---|
sep-2575-server-sends-subscription-ack |
notifications/subscriptions/acknowledged is the first message on a subscriptions/listen stream |
MUST | Open a listen stream, assert first frame |
sep-2575-server-tags-subscription-id |
Listen-stream notifications carry _meta.…/subscriptionId |
MUST | Open listen stream, trigger a notification, inspect _meta |
sep-2575-server-honors-notification-filter |
Server doesn't send notification types the client didn't request | MUST NOT | Open with a narrow filter, trigger out-of-filter event, assert silence; needs a trigger tool on the SUT server, or could be a lightweight check on recieved notification types |
sep-2575-http-server-no-independent-requests-on-stream |
Request stream contains only IncompleteResult, never independent JSON-RPC requests |
MUST NOT | Call a tool that needs sampling/elicitation, scan stream frames |
sep-2575-server-no-log-without-loglevel |
No notifications/message for requests that didn't set _meta.…/logLevel |
MUST NOT | Call a logging tool with no logLevel, assert no log frames. This one is a bit weird because it should be true on every request that you don't get logs, but having a tool that logs w/ capability but doesn't log w/o seems good. |
sep-2575-server-sends-prompts-list-changed-on-subscription |
List-changed-capable servers notify listen streams with promptsListChanged: true |
SHOULD | Needs the SUT server to expose a way to mutate its prompt list |
sep-2575-server-sends-tools-list-changed-on-subscription |
Same, for tools | SHOULD | Same. |
C. No scenario yet — observability / spec-wording questions
These describe internal server state rather than wire behavior, or depend on the SUT having a way to trigger the precondition. These we can likely exclude, or revisit their spec language if we want them to be testable. Maybe there are more creative scenarios than I can think of to test these, but I'm inclined to exclude them.
| Check | Requirement | Level | Concern |
|---|---|---|---|
sep-2575-server-stateless-no-prior-context |
Server doesn't treat connection/process identity as a session proxy | MUST NOT | Partially covered already: sep-2575-request-meta-invalid-missing-{meta,protocol-version,client-info,client-capabilities} verify the server rejects requests with incomplete _meta (i.e. won't fall back to remembered state) — but that's per-request, not cross-connection. A full scenario should also verify request #2 on a fresh connection succeeds with full _meta. Spec could phrase in terms of an observable consequence. |
sep-2575-server-stateless-no-connection-reuse-required |
Related operations work across separate connections | MUST NOT | Similar to above, this one seems difficult to test. We either should exclude it or update the spec language to make it clearer what we're |
sep-2575-http-server-disconnect-is-cancel |
Closing the SSE stream is treated as cancellation | MUST | "Treated as" is internal state. Once the stream closes there's no channel to observe it; only indirectly testable via …-stops-on-cancel. |
sep-2575-http-server-stops-on-cancel |
Server stops work and sends no further messages for the cancelled request | SHOULD + MUST NOT | "Stop work as soon as practical" is unobservable from a black-box harness. The "no further messages" half is testable only if the server might also fan out to a subscriptions/listen stream. |
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the traceability refresh and the existing scenario in src/scenarios/client/request-metadata.ts, then inspect everythingClient.ts to understand the current TypeScript SDK limitation. Review the seven proposed scenarios in section B against existing conformance patterns and identify which can be implemented without new SUT capabilities. Done means adding the feasible scenarios or recording why the remaining checks are excluded or deferred.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- api, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100