modelcontextprotocol / modelcontextprotocol/conformance

CI does not smoke-test client CLI path; everything-client drift on elicitation-sep1034-client-defaults and sse-retry

Open
#345 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
TypeScript
Stars
127
Forks
101
Avg merge
6d 1h
Merged PRs (30d)
7

Description

Describe the bug

CI runs npm test (vitest) but never exercises the documented client-mode CLI path (node dist/index.js client --command "…"). Regressions in the bundled reference client (examples/clients/typescript/everything-client.ts) are therefore not caught until someone runs that path manually against the in-repo fixture (README.md, CONTRIBUTING.md). SDK_INTEGRATION.md documents the same CLI for external SDK conformance clients; it does not validate the in-repo everything-client, so following it would not catch this drift either.

Two concrete examples in the --suite core scenarios on current main:

  1. elicitation-sep1034-client-defaults — the handler is registered under the wrong name (elicitation-defaults), and the client advertises elicitation.applyDefaults instead of elicitation.form.applyDefaults (required by @modelcontextprotocol/sdk 1.29+ for default merging).
  2. sse-retry — no handler is registered in everything-client, even though the scenario is in the core suite.

The scenario implementations and checks appear correct; upstream typescript-sdk's conformance client already registers both handlers with the correct names and capability shapes.

To Reproduce

  1. npm ci && npm run build
  2. Elicitation drift:
node dist/index.js client \
  --command "npx tsx examples/clients/typescript/everything-client.ts" \
  --scenario elicitation-sep1034-client-defaults
  1. SSE retry missing handler:
node dist/index.js client \
  --command "npx tsx examples/clients/typescript/everything-client.ts" \
  --scenario sse-retry
  1. Full core suite (both failures, among others):
node dist/index.js client \
  --command "npx tsx examples/clients/typescript/everything-client.ts" \
  --suite core

Expected behavior

The bundled everything-client should register elicitation-sep1034-client-defaults (with elicitation.form.applyDefaults: true) and sse-retry, matching AGENTS.md guidance to extend the everything-client for passing examples.

CI should smoke-test the documented client CLI path (at least --suite core) so reference-client drift is caught automatically — reusing the existing client command (AGENTS.md: no parallel test runner).

Logs

Elicitation — client subprocess exits before connecting:

Unknown scenario: elicitation-sep1034-client-defaults

(If the handler were registered under the old name elicitation-defaults instead, elicitation checks fail because defaults are not applied.)

SSE retry:

Unknown scenario: sse-retry

Core suite summary (representative):

✗ elicitation-sep1034-client-defaults: 0 passed, 5 failed
✗ sse-retry: 0 passed, 1 failed
Total: 238 passed, 6 failed, 0 warnings

Compare with typescript-sdk main (both scenarios pass):

npm start -- sdk typescript-sdk --mode client --scenario elicitation-sep1034-client-defaults
npm start -- sdk typescript-sdk --mode client --scenario sse-retry

Additional context

  • Related (not in scope for this issue): #250 proposes eventually removing vendored examples/{clients,servers}/typescript/ and running CI via sdk typescript-sdk@<pinned-sha> instead. That is not implemented today; this issue covers the incremental fix for the current documented workflow.
  • Proposed fix (for discussion): add an npm script + CI step running --suite core via the existing client subcommand; align everything-client.ts with upstream typescript-sdk for both handlers above.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with npm ci && npm run build, then run the documented client commands for elicitation-sep1034-client-defaults, sse-retry, and --suite core. Inspect examples/clients/typescript/everything-client.ts alongside the scenario checks and AGENTS.md guidance. Done means both handlers pass and CI smoke-tests the existing client command for the core suite.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
ci-cd, cli, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.