modelcontextprotocol / modelcontextprotocol/conformance
CI does not smoke-test client CLI path; everything-client drift on elicitation-sep1034-client-defaults and sse-retry
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 127
- Forks
- 101
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 7
Description
Describe the bug
CI runs npm test (vitest) but never exercises the documented client-mode CLI path (node dist/index.js client --command "…"). Regressions in the bundled reference client (examples/clients/typescript/everything-client.ts) are therefore not caught until someone runs that path manually against the in-repo fixture (README.md, CONTRIBUTING.md). SDK_INTEGRATION.md documents the same CLI for external SDK conformance clients; it does not validate the in-repo everything-client, so following it would not catch this drift either.
Two concrete examples in the --suite core scenarios on current main:
elicitation-sep1034-client-defaults— the handler is registered under the wrong name (elicitation-defaults), and the client advertiseselicitation.applyDefaultsinstead ofelicitation.form.applyDefaults(required by@modelcontextprotocol/sdk1.29+ for default merging).sse-retry— no handler is registered in everything-client, even though the scenario is in the core suite.
The scenario implementations and checks appear correct; upstream typescript-sdk's conformance client already registers both handlers with the correct names and capability shapes.
To Reproduce
npm ci && npm run build- Elicitation drift:
node dist/index.js client \
--command "npx tsx examples/clients/typescript/everything-client.ts" \
--scenario elicitation-sep1034-client-defaults
- SSE retry missing handler:
node dist/index.js client \
--command "npx tsx examples/clients/typescript/everything-client.ts" \
--scenario sse-retry
- Full core suite (both failures, among others):
node dist/index.js client \
--command "npx tsx examples/clients/typescript/everything-client.ts" \
--suite core
Expected behavior
The bundled everything-client should register elicitation-sep1034-client-defaults (with elicitation.form.applyDefaults: true) and sse-retry, matching AGENTS.md guidance to extend the everything-client for passing examples.
CI should smoke-test the documented client CLI path (at least --suite core) so reference-client drift is caught automatically — reusing the existing client command (AGENTS.md: no parallel test runner).
Logs
Elicitation — client subprocess exits before connecting:
Unknown scenario: elicitation-sep1034-client-defaults
(If the handler were registered under the old name elicitation-defaults instead, elicitation checks fail because defaults are not applied.)
SSE retry:
Unknown scenario: sse-retry
Core suite summary (representative):
✗ elicitation-sep1034-client-defaults: 0 passed, 5 failed
✗ sse-retry: 0 passed, 1 failed
Total: 238 passed, 6 failed, 0 warnings
Compare with typescript-sdk main (both scenarios pass):
npm start -- sdk typescript-sdk --mode client --scenario elicitation-sep1034-client-defaults
npm start -- sdk typescript-sdk --mode client --scenario sse-retry
Additional context
- Related (not in scope for this issue): #250 proposes eventually removing vendored
examples/{clients,servers}/typescript/and running CI viasdk typescript-sdk@<pinned-sha>instead. That is not implemented today; this issue covers the incremental fix for the current documented workflow. - Proposed fix (for discussion): add an npm script + CI step running
--suite corevia the existingclientsubcommand; aligneverything-client.tswith upstream typescript-sdk for both handlers above.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with npm ci && npm run build, then run the documented client commands for elicitation-sep1034-client-defaults, sse-retry, and --suite core. Inspect examples/clients/typescript/everything-client.ts alongside the scenario checks and AGENTS.md guidance. Done means both handlers pass and CI smoke-tests the existing client command for the core suite.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- ci-cd, cli, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100