microsoft / microsoft/mcp

Enforce performance regression budgets in CI

Open
#3,123 0 comments 0 reactions 0 assignees View on GitHub
enhancement Performance
Dominant language
C#
Stars
3.7k
Forks
624
Avg merge
2d 20h
Merged PRs (30d)
220

Description

## Outcome

Material Azure MCP performance regressions are detected before release, while normal benchmark noise does not create unreliable blocking checks.

## In scope

- Store approved baselines and historical trends
- Compare pull request or scheduled results against regression budgets
- Add warning and blocking thresholds with actionable diagnostics
- Require explicit evidence and approval to accept or replace a degraded baseline
- Publish performance summaries for release decisions
- Ratchet absolute targets after baseline qualification

## Out of scope

- Blocking changes before benchmark stability is demonstrated
- Automatically accepting a new degraded baseline
- Treating statistically insignificant changes as regressions

## Acceptance criteria and budgets

- [ ] Changes above 5% produce a non-blocking warning with evidence
- [ ] p50 or p95 latency regression above 10% blocks after qualification
- [ ] p99 latency regression above 20% blocks after qualification
- [ ] Throughput reduction above 10% blocks after qualification
- [ ] Steady-state CPU or memory increase above 10% blocks after qualification
- [ ] Peak memory or concurrency scaling degradation above 15% blocks after qualification
- [ ] Timeout or server-error increase above 0.5 percentage points blocks after qualification
- [ ] A baseline qualifies only after at least 10 successful scheduled runs across 14 days
- [ ] Baseline replacement requires a linked issue, measured impact, and owner approval

## Dependencies and risks

- Stable workloads, runners, and benchmark implementations
- PR and release quality guardrails Epic #2494
- Performance Epic #2412

## Evidence for closure

- CI or scheduled regression gate
- Baseline governance documentation
- Demonstrated warning, blocking, and approved-baseline-update paths

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the CI or scheduled benchmark implementation and the related guardrails in Epics #2494 and #2412. Define how baselines, warning and blocking thresholds, and approval evidence are represented. Done means demonstrated warning, blocking, and approved-baseline-update paths, plus baseline governance documentation and release summaries.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, csharp
Domain
ci-cd, devops, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.