dotnet / dotnet/BenchmarkDotNet

When user specifies both --runtimes and --corerun the Runtime printed for CoreRun job is invalid

Open
#2,094 0 comments 0 reactions 1 assignee Claimed by @adamsitnik View on GitHub
bug
Dominant language
C#
Stars
11.5k
Forks
1.1k
Avg merge
6d 11h
Merged PRs (30d)
12

Description

With #2002 we made it possible to mix `--runtimes and --corerun`, with #1621 we started hiding the `Job` and `Toolchain` columns.

Repro:

```cmd
dotnet run -c Release -f net6.0 -- -filter *IntroDisassembly.* --runtimes net6.0 net7.0 --corerun D:\projects\runtime\artifacts\bin\testhost\net7.0-windows-Debug-x64\shared\Microsoft.NETCore.App\8.0.0\corerun.exe --job dry
```

```ini
BenchmarkDotNet=v0.13.2.20220901-develop, OS=Windows 11 (10.0.22000.856/21H2)
AMD Ryzen Threadripper PRO 3945WX 12-Cores, 1 CPU, 24 logical and 12 physical cores
.NET SDK=7.0.100-rc.1.22423.16
[Host] : .NET 6.0.8 (6.0.822.36306), X64 RyuJIT AVX2
Job-MFIBNO : .NET 8.0.0 (42.42.42.42424), X64 RyuJIT AVX2
Job-UPJICA : .NET 6.0.8 (6.0.822.36306), X64 RyuJIT AVX2
Job-QIGXWB : .NET 7.0.0 (7.0.22.42223), X64 RyuJIT AVX2
```

| Method | Runtime | Mean | Error | Ratio | Code Size |
|--------- |--------- |---------:|------:|------:|----------:|
| SumLocal | .NET 6.0 | 255.2 us | NA | 1.00 | 49 B |
| SumLocal | .NET 6.0 | 255.8 us | NA | 1.00 | 35 B |
| SumLocal | .NET 7.0 | 258.3 us | NA | 1.01 | 49 B |
| | | | | | |
| SumField | .NET 6.0 | 264.3 us | NA | 0.95 | 139 B |
| SumField | .NET 6.0 | 279.4 us | NA | 1.00 | 67 B |
| SumField | .NET 7.0 | 264.9 us | NA | 0.95 | 139 B |

As you can see, we BDN is currently printing ` .NET 6.0` for the CoreRun job (most likely because it was build targeting `net6.0` tfm). This is wrong. In such case it should contain something like "local build". In case of multiple runtimes and multiple coreruns provided, we should just display the job column id (#1621)

Reported offline by @bartonjs

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.