modelcontextprotocol / modelcontextprotocol/conformance

Expose expected-vs-emitted check coverage per scenario

Open
#451 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
127
Forks
101
Avg merge
6d 1h
Merged PRs (30d)
7

Description

A scenario can emit only a thin subset of its checks while the reported denominator makes that subset look complete.

The concrete case from mcpkit: auth/authorization-server-migration reported 2 passed, 1 failed for months. The initial wire handshake failed, so roughly 76 other checks never emitted. PR #327 makes the setup failure explicit, but the result still cannot show how much of the scenario did not run.

The harder case has no setup failure: setup succeeds, only accept-path checkpoints emit, and the thin scenario result passes.

Add a per-scenario expected-vs-emitted coverage signal to the upstream results. It should be visible in both terminal summaries and machine-readable output. A count is enough to expose the gap; stable IDs would also identify the missing checks.

Today Scenario and ClientScenario do not declare their expected check IDs, and some checks are dynamic or version-gated. The implementation will need a stable expected set or another baseline that excludes checks which are not applicable to the selected spec version or extension.

Acceptance criteria:

  • Every executed scenario reports expected and emitted check counts, or equivalent coverage data.
  • Missing eligible checks remain visible when every emitted check passes.
  • Machine-readable results expose the coverage signal for downstream consumers.
  • Spec-version, extension, and documented applicability gates do not create false gaps.
  • Tests cover both a setup failure and a passing thin-shadow result.
  • SDKs do not need to maintain their own per-scenario count snapshots.

@panyam offered to implement this in #327.

Prior art: https://github.com/panyam/mcpkit/pull/1115

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace how Scenario and ClientScenario define or execute checks, then follow result construction into terminal summaries and machine-readable output. Review the setup-failure and passing thin-scenario cases first. Done means eligible expected and emitted coverage is reported consistently, applicability gates do not create false gaps, and both cases have tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.