modelcontextprotocol / modelcontextprotocol/conformance
tier-check reports 0/30 server conformance despite all tests passing
Open
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 127
- Forks
- 101
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 7
Description
Summary
Running tier-check against a server that passes 30/30 conformance tests reports 0/30 (0%) server conformance.
Steps to Reproduce
# Start server (tower-mcp conformance server on port 3001)
cargo run --manifest-path examples/conformance-server/Cargo.toml &
# Regular conformance suite: 39/39 pass (30 scenarios, some with multiple checks)
npx @modelcontextprotocol/conformance@0.1.15 server --url http://127.0.0.1:3001/mcp
# Output: Total: 39 passed, 0 failed
# tier-check: reports 0/30
npx @modelcontextprotocol/conformance@0.1.15 tier-check --repo joshrotenberg/tower-mcp --conformance-server-url http://127.0.0.1:3001/mcp
Expected
tier-check should report 30/30 (100%) server conformance, matching the regular server command results.
Actual
Conformance:
2025-03-26 2025-06-18 2025-11-25 All*
✗ Server — 0/26 0/30 0/30 (0%)
Environment
@modelcontextprotocol/conformancev0.1.15- Server: tower-mcp conformance-server (passes all 30 scenarios / 39 checks)
- macOS, Node 20
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the issue with the conformance server and compare the tier-check command with the regular server command in @modelcontextprotocol/conformance v0.1.15. Trace how each command counts server scenarios, then verify that tier-check reports 30/30 (100%) when all 30 scenarios pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- node.js, typescript
- Domain
- testing-qa, tooling
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100