dotnet / dotnet/aspnetcore

Benchmark validation doesn't appear to be working on non-Windows OS

Open
#46,495 2 comments 0 reactions 0 assignees View on GitHub
area-infrastructure bug
Dominant language
C#
Stars
38.4k
Forks
10.9k
Avg merge
2d 10h
Merged PRs (30d)
281

Description

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Describe the bug

Benchmarks written with benchmarkdotnet are validated on build:

https://github.com/dotnet/aspnetcore/blob/f0af11263e006d0df75b32d20d85bde5870a8d9c/eng/targets/CSharp.Common.targets#L133-L142

It appears that failures are only correctly reported on Windows. This behavior was noticed when I added a benchmark that is only valid for Windows, but tests all passed.

For example, I added an exception to a benchmark's setup here - https://github.com/dotnet/aspnetcore/pull/46491 - and only Windows tests failed:

![image](https://user-images.githubusercontent.com/303201/217125198-4ae73d85-ed1d-4b89-a4e3-b6664a97c60a.png)

I can see in the logs that they are being detected and run on other OS, but no error reported:

![image](https://user-images.githubusercontent.com/303201/217125313-720a1aa9-4723-4498-aee0-821644c6923c.png)

### Expected Behavior

Validation reports problems and the build fails on Linux and macOS.

### Steps To Reproduce

https://github.com/dotnet/aspnetcore/pull/46491

### Exceptions (if any)

_No response_

### .NET Version

_No response_

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.