dotnet / dotnet/BenchmarkDotNet
Statistically tests should be able to compare methods only among a specific runtime in multi-framework bench
- Dominant language
- C#
- Stars
- 11.5k
- Forks
- 1.1k
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 12
Description
| Method | Runtime | Input | Count | Mean | Error | StdDev | Ratio | MannWhitney(5%) | RatioSD | Gen0 | Gen1 | Gen2 | Allocated | Alloc Ratio |
|---------------------- |-------------- |------------------- |------ |----------------:|--------------:|--------------:|------:|---------------- |--------:|---------:|---------:|---------:|----------:|------------:|
| **RepeatWithCounter** | **.NET 8.0** | **👍** | **3** | **22.51 ns** | **0.477 ns** | **0.423 ns** | **1.55** | **Slower** | **0.03** | **0.0204** | **-** | **-** | **128 B** | **3.20** |
| RepeatNoCounter | .NET 8.0 | 👍 | 3 | 14.51 ns | 0.219 ns | 0.194 ns | 1.00 | Base | 0.00 | 0.0063 | - | - | 40 B | 1.00 |
| RepeatPowerShell | .NET 8.0 | 👍 | 3 | 16.76 ns | 0.158 ns | 0.132 ns | 1.16 | Slower | 0.02 | 0.0063 | - | - | 40 B | 1.00 |
| RepeatDoubleBlockSize | .NET 8.0 | 👍 | 3 | 17.74 ns | 0.304 ns | 0.455 ns | 1.22 | Slower | 0.04 | 0.0063 | - | - | 40 B | 1.00 |
| RepeatWithCounter | NativeAOT 8.0 | 👍 | 3 | 27.16 ns | 0.508 ns | 0.475 ns | 1.87 | Slower | 0.02 | 0.0179 | - | - | 112 B | 2.80 |
| RepeatNoCounter | NativeAOT 8.0 | 👍 | 3 | 17.75 ns | 0.074 ns | 0.058 ns | 1.22 | Slower | 0.02 | 0.0063 | - | - | 40 B | 1.00 |
| RepeatPowerShell | NativeAOT 8.0 | 👍 | 3 | 19.38 ns | 0.435 ns | 0.565 ns | 1.34 | Slower | 0.04 | 0.0063 | - | - | 40 B | 1.00 |
| RepeatDoubleBlockSize | NativeAOT 8.0 | 👍 | 3 | 20.51 ns | 0.142 ns | 0.133 ns | 1.41 | Slower | 0.02 | 0.0063 | - | - | 40 B | 1.00 |
For example, we should be able to compare 4 methods in each runtime respectly:
| Method | Runtime | MannWhitney(5%) |
|---------------------- |-------------- |---------------- |
| RepeatWithCounter | NET 8.0 | Slower |
| RepeatNoCounter | .NET 8.0 | Base |
| RepeatPowerShell | .NET 8.0 | Slower |
| RepeatDoubleBlockSize | .NET 8.0 | Slower |
| RepeatWithCounter | NativeAOT 8.0 | ? (Slower?) |
| RepeatNoCounter | NativeAOT 8.0 | ***Base*** |
| RepeatPowerShell | NativeAOT 8.0 | ? (Slower?) |
| RepeatDoubleBlockSize | NativeAOT 8.0 | ? (Slower?) |
Contributor guide
Assessment
This issue has not been assessed yet.