modelcontextprotocol / modelcontextprotocol/conformance

tier-check reports 0/30 server conformance despite all tests passing

Open
#182 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
127
Forks
101
Avg merge
6d 1h
Merged PRs (30d)
7

Description

Summary

Running tier-check against a server that passes 30/30 conformance tests reports 0/30 (0%) server conformance.

Steps to Reproduce

# Start server (tower-mcp conformance server on port 3001)
cargo run --manifest-path examples/conformance-server/Cargo.toml &

# Regular conformance suite: 39/39 pass (30 scenarios, some with multiple checks)
npx @modelcontextprotocol/conformance@0.1.15 server --url http://127.0.0.1:3001/mcp
# Output: Total: 39 passed, 0 failed

# tier-check: reports 0/30
npx @modelcontextprotocol/conformance@0.1.15 tier-check --repo joshrotenberg/tower-mcp --conformance-server-url http://127.0.0.1:3001/mcp

Expected

tier-check should report 30/30 (100%) server conformance, matching the regular server command results.

Actual

Conformance:
                 2025-03-26 2025-06-18 2025-11-25              All*
✗ Server                  —       0/26       0/30        0/30 (0%)

Environment

  • @modelcontextprotocol/conformance v0.1.15
  • Server: tower-mcp conformance-server (passes all 30 scenarios / 39 checks)
  • macOS, Node 20

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the conformance server and compare the tier-check command with the regular server command in @modelcontextprotocol/conformance v0.1.15. Trace how each command counts server scenarios, then verify that tier-check reports 30/30 (100%) when all 30 scenarios pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
node.js, typescript
Domain
testing-qa, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.