dotnet / dotnet/source-build

Reduce maintenance cost of Source-Build SDK Diff Tests baselines and exclusions

Open
#5,598 2 comments 0 reactions 0 assignees View on GitHub
area-testing
Dominant language
No language data
Stars
287
Forks
145
Avg merge
1d 22h
Merged PRs (30d)
10

Description

## Summary

The Source-Build SDK Diff Tests (comparing binaries/files/APIs between source-built and Microsoft-built SDKs) generate frequent baseline update PRs that require human review and merge. The maintenance cost of this process is disproportionate to the regressions it catches.

## The Problem

The `CreateBaselineUpdatePR` tooling generates PRs titled "Update Source-Build SDK Diff Tests Baselines and Exclusions" against dotnet/dotnet on a regular basis. These PRs update files in `test/Microsoft.DotNet.SourceBuild.Tests/assets/SdkContentTests/`.

### Data (Jan 1 - Jun 15, 2026)

| Metric | Value |
|--------|-------|
| Total baseline update PRs created | **49** |
| Merged | **37** |
| Closed without merge | **9** |
| Currently open | **3** |
| Average frequency | **~6.7 PRs/month (~1.6/week)** |

These PRs are authored by `dotnet-sb-bot` and primarily reviewed/merged by a single maintainer. Most are merged same-day, indicating routine approval rather than deep review catching issues.

### Sample PR

https://github.com/dotnet/dotnet/pull/7162

## Concern

1. **High frequency, low signal:** ~37 merged baseline updates in ~5.5 months means the team is spending review/merge cycles nearly twice a week on what are largely expected differences.
2. **Low regression catch rate:** The vast majority of these PRs are routine acknowledgements of known/expected differences - not regressions being caught and fixed.
3. **Evolved landscape:** Source build has matured significantly since these tests were originally implemented. The diff surface has changed, but the testing approach hasn't adapted.

## Desired Outcome

We need to find a way to lower the maintenance cost while preserving the ability to catch meaningful regressions.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.