Enforce performance regression budgets in CI
- Dominant language
- C#
- Stars
- 3.7k
- Forks
- 624
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 220
Description
## Outcome
Material Azure MCP performance regressions are detected before release, while normal benchmark noise does not create unreliable blocking checks.
## In scope
- Store approved baselines and historical trends
- Compare pull request or scheduled results against regression budgets
- Add warning and blocking thresholds with actionable diagnostics
- Require explicit evidence and approval to accept or replace a degraded baseline
- Publish performance summaries for release decisions
- Ratchet absolute targets after baseline qualification
## Out of scope
- Blocking changes before benchmark stability is demonstrated
- Automatically accepting a new degraded baseline
- Treating statistically insignificant changes as regressions
## Acceptance criteria and budgets
- [ ] Changes above 5% produce a non-blocking warning with evidence
- [ ] p50 or p95 latency regression above 10% blocks after qualification
- [ ] p99 latency regression above 20% blocks after qualification
- [ ] Throughput reduction above 10% blocks after qualification
- [ ] Steady-state CPU or memory increase above 10% blocks after qualification
- [ ] Peak memory or concurrency scaling degradation above 15% blocks after qualification
- [ ] Timeout or server-error increase above 0.5 percentage points blocks after qualification
- [ ] A baseline qualifies only after at least 10 successful scheduled runs across 14 days
- [ ] Baseline replacement requires a linked issue, measured impact, and owner approval
## Dependencies and risks
- Stable workloads, runners, and benchmark implementations
- PR and release quality guardrails Epic #2494
- Performance Epic #2412
## Evidence for closure
- CI or scheduled regression gate
- Baseline governance documentation
- Demonstrated warning, blocking, and approved-baseline-update paths
Contributor guide
Research direction
Start by reviewing the CI or scheduled benchmark implementation and the related guardrails in Epics #2494 and #2412. Define how baselines, warning and blocking thresholds, and approval evidence are represented. Done means demonstrated warning, blocking, and approved-baseline-update paths, plus baseline governance documentation and release summaries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, csharp
- Domain
- ci-cd, devops, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100