modelcontextprotocol / modelcontextprotocol/conformance

WIP: generic scenario steering — plumbing-only client scenarios shouldn't need per-SDK fixture handlers

Open
#431 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
127
Forks
101
Avg merge
6d 1h
Merged PRs (30d)
7

Description

Parking a design note; details to be fleshed out (there may be an earlier exploration of this on a branch elsewhere — to be dug up and linked).

Pain (fresh example from #335)

All five Tier 1 SDK conformance clients failed the new json-schema-2020-12-preservation scenario with "Unknown scenario" — yet when we prototyped the missing handler in each fixture (10-35 lines each), every SDK passed 9/9. The scenario needs no SDK behavior change at all; the per-scenario fixture handler is pure dispatch plumbing, duplicated across N repos, and it rots (see #345 for in-repo drift of the same kind).

Idea

Scenarios of this shape declare their client-side steps as data over the context-passing channel from #51 (implemented), e.g. steps like "tools/list", "call tool X with the schema you got for tool Y". Each SDK fixture implements one small generic interpreter, once; after that, new plumbing-only scenarios need zero fixture changes anywhere. Scenarios needing real client behavior (elicitation decisions, sampling, auth flows) keep bespoke handlers.

Related: #51 (context channel), #345 (handler drift), #335 (motivating case, with working handler patches for all five SDKs).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the context-passing channel described in #51, then inspect the working per-SDK handlers from #335 and the drift noted in #345. Define the generic scenario-step data and interpreter boundary, while preserving bespoke handlers for scenarios requiring real client behavior. Done means plumbing-only scenarios can run across SDK fixtures without per-scenario handler changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.