modelcontextprotocol / modelcontextprotocol/conformance

tier-audit docs coverage: evaluate the feature list against the spec revision an SDK targets

Open
#441 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
127
Forks
101
Avg merge
6d 1h
Merged PRs (30d)
7

Description

The tier audit's documentation check scores SDKs against the canonical 48-feature list in .claude/skills/mcp-sdk-tier-audit/references/feature-list.md. The list carries no lifecycle metadata, and the rubric in docs-coverage-prompt.md allows only PASS, PARTIAL, or FAIL, so a feature the spec itself has removed still scores FAIL when an SDK neither implements nor documents it.

As of the 2026-07-28 revision, 11 of the 48 rows are removed or deprecated: 14 and 15 (resources/subscribe/unsubscribe, removed by SEP-2575), 23 (sampling, deprecated by SEP-2577), 29 (notifications/elicitation/complete, absent from the final 2026-07-28 schema), 30 and 31 (roots, deprecated by SEP-2577), 32 and 33 (logging, SEP-2577/SEP-2575), 36 (ping, removed by SEP-2575), and 39 and 40 (legacy HTTP+SSE, deprecated in 2025-03-26). An SDK that tracks the current revision is therefore capped at 37/48 and can never meet Tier 1's "all non-experimental features documented with examples", even when every feature it ships is documented with examples. An SDK that keeps deprecated surface scores higher, which inverts the incentive the SEP-2596 lifecycle is meant to create.

The referee already handles this correctly for conformance: scenario filtering by removedIn at --spec-version is what keeps a 2026-07-28 SDK from being failed on initialize or logging/setLevel. The docs rubric is the one check still scoring against the union of all revisions.

Proposal:

  1. Add lifecycle columns to feature-list.md (introduced/deprecated/removed, with the SEP that did it), mirroring the scenarios' removedIn metadata.
  2. Add a rule to docs-coverage-prompt.md: when the audited SDK targets spec revision R, a feature removed at or before R scores N/A and leaves the denominator. A feature deprecated at or before R that the SDK deliberately declines also scores N/A when the omission is documented with a stated reason.

Observed in practice auditing an SDK that targets 2026-07-28 only: 37/48 with zero PARTIAL rows and all 11 FAILs being removed or deprecated features, each already accounted for in the SDK's own feature index with the SEP that retired it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with .claude/skills/mcp-sdk-tier-audit/references/feature-list.md and docs-coverage-prompt.md, then compare their current rubric with the referee's scenario filtering by removedIn at --spec-version. Add lifecycle metadata and revision-aware N/A guidance for removed or deliberately declined deprecated features, preserving the documented SEP references. Done means the 2026-07-28 audit no longer counts retired features in its denominator.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
documentation, testing-qa
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.