Reduce maintenance cost of Source-Build SDK Diff Tests baselines and exclusions
- Dominant language
- No language data
- Stars
- 287
- Forks
- 145
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 10
Description
## Summary
The Source-Build SDK Diff Tests (comparing binaries/files/APIs between source-built and Microsoft-built SDKs) generate frequent baseline update PRs that require human review and merge. The maintenance cost of this process is disproportionate to the regressions it catches.
## The Problem
The `CreateBaselineUpdatePR` tooling generates PRs titled "Update Source-Build SDK Diff Tests Baselines and Exclusions" against dotnet/dotnet on a regular basis. These PRs update files in `test/Microsoft.DotNet.SourceBuild.Tests/assets/SdkContentTests/`.
### Data (Jan 1 - Jun 15, 2026)
| Metric | Value |
|--------|-------|
| Total baseline update PRs created | **49** |
| Merged | **37** |
| Closed without merge | **9** |
| Currently open | **3** |
| Average frequency | **~6.7 PRs/month (~1.6/week)** |
These PRs are authored by `dotnet-sb-bot` and primarily reviewed/merged by a single maintainer. Most are merged same-day, indicating routine approval rather than deep review catching issues.
### Sample PR
https://github.com/dotnet/dotnet/pull/7162
## Concern
1. **High frequency, low signal:** ~37 merged baseline updates in ~5.5 months means the team is spending review/merge cycles nearly twice a week on what are largely expected differences.
2. **Low regression catch rate:** The vast majority of these PRs are routine acknowledgements of known/expected differences - not regressions being caught and fixed.
3. **Evolved landscape:** Source build has matured significantly since these tests were originally implemented. The diff surface has changed, but the testing approach hasn't adapted.
## Desired Outcome
We need to find a way to lower the maintenance cost while preserving the ability to catch meaningful regressions.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.