modelcontextprotocol / modelcontextprotocol/conformance
Run local conformance against any SDK at any ref (unblocks removing vendored examples)
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 127
- Forks
- 101
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 7
Description
Context
The conformance repo currently vendors a typescript-sdk client and server in examples/{clients,servers}/typescript/ so its own CI can test-the-tests. That's drifted from typescript-sdk's canonical conformance fixtures and creates a version-bump dance whenever a new scenario lands ahead of SDK support.
Per Discord discussion the lean is option (1): delete the vendored examples, keep only negative-path fixtures here, and rely on the real SDK repos for the passing implementation. Nate's condition, which I agree with: this only works if we make it "super easy to clone and test an SDK."
That condition is the blocker, and it overlaps with the inner-loop pain when writing scenarios (#237 review): the implementation that passes a new scenario is usually still on an unmerged SDK branch, and getting from "scenario edited" to "ran against typescript-sdk@that-branch" is currently a manual pnpm link dance documented only for typescript-sdk.
Goal
One command that runs the local conformance build against <sdk>@<ref>, for any official SDK, locally and in CI.
./some-script.sh --sdk typescript-sdk@main
./some-script.sh --sdk python-sdk@some-branch --scenario sep-2164-resource-not-found
./some-script.sh --matrix typescript-sdk@main,python-sdk@main,go-sdk@v1.2.0
What's needed
Each SDK already encodes "how to build me and start the conformance fixture" in its .github/workflows/conformance.yml, but the shapes differ — python-sdk pins an npm version + bash script, typescript-sdk uses npm scripts, go-sdk and others use this repo's composite action. We need a small declarative contract the runner can consume directly. Either:
- Parse what's already in each SDK's workflow (fragile), or
- Each SDK publishes a
conformance.config.{json,yaml}withclone/build/server-command/client-command. Same file can carry the per-SDK expected-failures baseline (#99) and the scenario→fixture mapping (#114 comment).
CI here
Once ./some-script.sh --sdk works, this repo's CI replaces the vendored examples with run --sdk typescript-sdk@<pinned-sha> (a known-good ref, not main, so SDK regressions don't break conformance CI). A separate non-blocking matrix job can run against @main of each official SDK and comment results on the PR.
Negative-path fixtures (servers that deliberately violate the spec) stay in this repo since no SDK should ship those.
First step
The first step for this is making it easy to run the typescript-sdk to preserve existing behavior, and then we can attempt to generalize.
Related: #114, #243, #248, #237.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the vendored examples in examples/{clients,servers}/typescript/ and the TypeScript SDK's .github/workflows/conformance.yml to understand the existing behavior. First make the local runner execute conformance against a TypeScript SDK ref, then verify that a pinned SDK ref can replace the vendored examples without changing current results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- testing, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100