modelcontextprotocol / modelcontextprotocol/conformance

Shift config of "known-sdks.ts" to SDK-side config file

Open
#460 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
127
Forks
101
Avg merge
6d 1h
Merged PRs (30d)
7

Description

I really like the idea of easing the options handling and execution of the conformance runner, but immediately fell into the pitfall that this requires cross repo PRs and also decouples the versioning a bit inconvenient if changes are needed.

of course the sdk map would stay in this repo, but after checkout it could check for a conformance.yaml with the relevant settings:

# modelcontextprotocol/go-sdk
# conformance.yaml
build: 'go build -o ./.conformance-server ./conformance/everything-server && go build -o ./.conformance-client ./conformance/everything-client'
client:
    command: './.conformance-client'
server:
    command: './.conformance-server -http=localhost:3000 -stateless=false'
    url: 'http://localhost:3000'
expectedFailures: 'conformance/baseline.yml'
specOverrides:
    '2026-07-28':
        server: { command: './.conformance-server -http=localhost:3000' }

WDYT?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading known-sdks.ts and the conformance runner's current options-handling path; the issue proposes moving SDK-specific settings into an SDK-side conformance.yaml while keeping the SDK map in this repository. Compare the proposed build, client, server, expectedFailures, and specOverrides fields with the current configuration, and consider how existing baseline.yml handling is affected. Done means the runner can obtain these settings from the checked-out SDK without requiring cross-repository configuration changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.