modelcontextprotocol / modelcontextprotocol/conformance

feat(tier-check): support configurable label taxonomy and aliases for Tier 1 repo checks

Open
#482 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
127
Forks
101
Avg merge
6d 1h
Merged PRs (30d)
7

Description

Is your feature request related to a problem? Please describe.

The MCP SDK Tiering specification (SEP-1730 / docs/community/sdk-tiers.mdx) requires specific labels for issue triage (bug, enhancement, question), status (needs confirmation, needs repro, ready for work, good first issue, help wanted), and priority (P0P3).

Currently, tier-check enforces these through exact hardcoded string matches:

  • Matches lowercase names strictly against TYPE_LABELS, STATUS_LABELS, and PRIORITY_LABELS.
  • Hardcodes labels: 'P0' when querying GitHub's API to calculate critical issue resolution times.

Many organizations and SDK ecosystems (e.g. Google/DevRel, cloud providers, and established OSS projects) use namespaced labels like priority: p0, type: bug, and status: waiting for response to support organization-wide issue taxonomies, automation bots, and issue-mirroring bridges (e.g., Jira, Google Issue Tracker).

Even when an SDK repository fully adheres to the required semantic taxonomy and respects the triage/P0 SLAs, tier-check fails due to exact string mismatch. Forcing these repos to adopt bare strings creates unnecessary migration friction for maintainers seeking Tier 1 / Tier 2 certification.

Describe the solution you'd like.

Allow tier-check to recognize configurable label aliases or mappings so repositories can pass tier checks while preserving their established label conventions.

  1. Configuration Mechanism:
    Support an optional label alias mapping, configured either via:

    • A repository-level config file (e.g., .github/mcp-conformance.json or .mcp/tier-check.json).
    • The built-in SDK config in src/sdk-runner/known-sdks.ts (or via a CLI flag like --labels-config).

    Example configuration:

    {
      "labels": {
        "aliases": {
          "bug": ["type: bug"],
          "enhancement": ["type: feature request"],
          "question": ["type: question"],
          "needs confirmation": ["status: feedback wanted", "status: investigating"],
          "needs repro": ["status: waiting for response", "needs more info"],
          "ready for work": ["ready for work", "needs work"],
          "help wanted": ["status: help wanted", "help wanted"],
          "P0": ["priority: p0"],
          "P1": ["priority: p1"],
          "P2": ["priority: p2"],
          "P3": ["priority: p3"]
        }
      }
    }
    
  2. Evaluation in checkLabels:
    When evaluating whether a required label is present, check if the repo contains the standard label name or any of its configured aliases.

  3. Evaluation in checkP0Resolution:
    Update checkP0Resolution to query the configured P0 label (e.g., priority: p0 instead of the hardcoded 'P0'), ensuring that P0 resolution metrics reflect the repository's actual critical issues.

  4. Default Behavior:
    If no mapping is provided, retain the exact current behavior with zero breaking changes.

Describe alternatives you've considered

  1. Renaming labels in each SDK repository:
    Breaks existing workflows, bot automation, and internal issue-sync tooling across repos that rely on structured label prefixes (priority:*, type:*, status:*).

  2. Duplicate labels (e.g., both P0 and priority: p0):
    Creates contributor confusion, label clutter, and requires maintainers to double-tag every issue to satisfy both external checks and internal automations.

  3. Relying solely on GitHub Issue Types:
    checkLabels already supports GitHub Issue Types for bug/enhancement/question, but GitHub Issue Types do not cover status or priority (P0P3), leaving the main gap unaddressed.

  4. Simple prefix-stripping:
    While some differences are purely prefix/case conventions (e.g. priority: p0 vs P0, type: bug vs bug), many existing repos use established semantic synonyms for the same triage stages:

    • enhancementtype: feature request
    • needs confirmationstatus: feedback wanted or status: investigating
    • needs reprostatus: waiting for response or needs more info
    • ready for workneeds work

    Supporting a 1-to-many alias dictionary ensures existing repositories with mature workflows don't have to break internal automations or rename established labels.

Additional Context

  • We are seeing this while preparing the MCP Toolbox repositories for Tier 1 conformance. All required triage and priority categories are tracked, but naming conventions follow standard namespaced prefixes (e.g., priority: p0, status: feedback wanted).
  • We are happy to submit a PR to implement this alias support in src/tier-check/ if this approach aligns with the team.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read docs/community/sdk-tiers.mdx and the tier-check implementation under src/tier-check/, focusing on checkLabels and checkP0Resolution; compare how src/sdk-runner/known-sdks.ts currently supplies SDK configuration. Confirm the chosen alias/configuration path preserves exact current behavior when absent, recognizes configured aliases, and uses the configured P0 label for resolution metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.