modelcontextprotocol / modelcontextprotocol/conformance

Run local conformance against any SDK at any ref (unblocks removing vendored examples)

Open
#250 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
127
Forks
101
Avg merge
6d 1h
Merged PRs (30d)
7

Description

Context

The conformance repo currently vendors a typescript-sdk client and server in examples/{clients,servers}/typescript/ so its own CI can test-the-tests. That's drifted from typescript-sdk's canonical conformance fixtures and creates a version-bump dance whenever a new scenario lands ahead of SDK support.

Per Discord discussion the lean is option (1): delete the vendored examples, keep only negative-path fixtures here, and rely on the real SDK repos for the passing implementation. Nate's condition, which I agree with: this only works if we make it "super easy to clone and test an SDK."

That condition is the blocker, and it overlaps with the inner-loop pain when writing scenarios (#237 review): the implementation that passes a new scenario is usually still on an unmerged SDK branch, and getting from "scenario edited" to "ran against typescript-sdk@that-branch" is currently a manual pnpm link dance documented only for typescript-sdk.

Goal

One command that runs the local conformance build against <sdk>@<ref>, for any official SDK, locally and in CI.

./some-script.sh --sdk typescript-sdk@main
./some-script.sh --sdk python-sdk@some-branch --scenario sep-2164-resource-not-found
./some-script.sh --matrix typescript-sdk@main,python-sdk@main,go-sdk@v1.2.0

What's needed

Each SDK already encodes "how to build me and start the conformance fixture" in its .github/workflows/conformance.yml, but the shapes differ — python-sdk pins an npm version + bash script, typescript-sdk uses npm scripts, go-sdk and others use this repo's composite action. We need a small declarative contract the runner can consume directly. Either:

  • Parse what's already in each SDK's workflow (fragile), or
  • Each SDK publishes a conformance.config.{json,yaml} with clone/build/server-command/client-command. Same file can carry the per-SDK expected-failures baseline (#99) and the scenario→fixture mapping (#114 comment).

CI here

Once ./some-script.sh --sdk works, this repo's CI replaces the vendored examples with run --sdk typescript-sdk@<pinned-sha> (a known-good ref, not main, so SDK regressions don't break conformance CI). A separate non-blocking matrix job can run against @main of each official SDK and comment results on the PR.

Negative-path fixtures (servers that deliberately violate the spec) stay in this repo since no SDK should ship those.

First step

The first step for this is making it easy to run the typescript-sdk to preserve existing behavior, and then we can attempt to generalize.

Related: #114, #243, #248, #237.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the vendored examples in examples/{clients,servers}/typescript/ and the TypeScript SDK's .github/workflows/conformance.yml to understand the existing behavior. First make the local runner execute conformance against a TypeScript SDK ref, then verify that a pinned SDK ref can replace the vendored examples without changing current results.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.