dotnet / dotnet/source-build

Improve the UX of api-diff suppression baseline and test resiliency

Open
#5,356 2 comments 0 reactions 0 assignees View on GitHub
area-testing
Dominant language
No language data
Stars
287
Forks
145
Avg merge
1d 22h
Merged PRs (30d)
10

Description

With https://github.com/dotnet/dotnet/pull/2542 we have enabled api-diffs between Microsoft and source-build SDKs. Resulting diffs are captured in https://github.com/dotnet/source-build/issues/5344

The implementation uses a built-in feature of ApiCompat task to generate a new suppression baseline, which is further compared with the old baseline. The issue exists with UX of the baseline file. It is not clear which assembly is missing the API, so in the following baseline entry example either `left` or `right` assembly are missing the API:
```

CP0001
T:System.Diagnostics.CodeAnalysis.ExperimentalAttribute
/tmp/ak2gfs0d.ycb/msft/sdk/10.0.100-rc.2.25465.105/DotnetTools/dotnet-watch/10.0.100-rc.2.25465.105/tools/net10.0/any/BuildHost-netcore/Microsoft.CodeAnalysis.Workspaces.MSBuild.BuildHost.dll
/tmp/ak2gfs0d.ycb/sb/sdk/10.0.100-rc.2.25465.105/DotnetTools/dotnet-watch/10.0.100-rc.2.25465.105/tools/net10.0/any/BuildHost-netcore/Microsoft.CodeAnalysis.Workspaces.MSBuild.BuildHost.dll

```

Unlike suppression file, the task's output is specific in calling out the actual assembly missing the type:
```
error CP0001: Type 'System.Diagnostics.CodeAnalysis.ExperimentalAttribute' exists on /tmp/zezs3i1y.dqa/sb/sdk/10.0.100-rc.2.25465.105/DotnetTools/dotnet-watch/10.0.100-rc.2.25465.105/tools/net10.0/any/BuildHost-netcore/Microsoft.CodeAnalysis.Workspaces.MSBuild.BuildHost.dll but not on /tmp/zezs3i1y.dqa/msft/sdk/10.0.100-rc.2.25465.105/DotnetTools/dotnet-watch/10.0.100-rc.2.25465.105/tools/net10.0/any/BuildHost-netcore/Microsoft.CodeAnalysis.Workspaces.MSBuild.BuildHost.dll
```

Additionally, the current model would not capture potential bugs, where a specific API, in a single change, disappears on one binary and appears on the other. This can happen if some APIs are conditioned-out on source-build and someone accidentally alters this condition by simply reversing it. The API would disappear from MSFT build, appear in source-build, but the suppression file would not have any differences, thus the test would not fail.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.