modelcontextprotocol / modelcontextprotocol/conformance

Spec tracking check should validate the submitted SDK release, not first-release-after-spec

Open
#167 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
127
Forks
101
Avg merge
6d 1h
Merged PRs (30d)
7

Description

Problem

The current spec tracking check (src/tier-check/checks/spec-tracking.ts) finds the first SDK release published after the latest spec release and measures the time gap between them. This doesn't validate anything meaningful for tiering:

  1. It doesn't know which SDK version is being submitted for evaluation
  2. The "first release after spec" could be a totally unrelated bugfix/patch
  3. It's disconnected from the spec version the submission claims conformance against

For example, the Java SDK tiering submission shows "9d gap" — but that just means some SDK release happened 9 days after the latest spec release, not that the submitted version (1.0.0) was released within 30 days of the spec version (2025-06-18) it's being evaluated against.

Proposed Change

The spec tracking check should:

  1. Accept the submitted SDK version (or release tag) as input
  2. Accept the target spec version being claimed
  3. Verify the submitted SDK release exists on GitHub
  4. Measure the gap between the target spec release date and the submitted SDK release date
  5. Pass if the SDK release was within 30 days of that spec version's release

This ties the check to the actual submission rather than being a context-free scrape of GitHub releases.

Context

Came up during review of the Java SDK Tier 2 assessment (https://github.com/modelcontextprotocol/modelcontextprotocol/issues/2301).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/tier-check/checks/spec-tracking.ts and trace how the tiering submission currently supplies release and spec information. Define the inputs and validation flow needed for the submitted SDK release, then verify the release exists and that the measured gap uses the claimed spec version and 30-day threshold; the Java SDK assessment in issue #2301 provides the motivating case.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, typescript
Domain
testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.