dotnet / dotnet/BenchmarkDotNet
BaselineCustomAnalyzer raise warning under specific conditions
- Dominant language
- C#
- Stars
- 11.5k
- Forks
- 1.1k
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 12
Description
I’m trying to write benchmarks that measure performance difference between .NET version.
But when using following settings [`BaselineCustomAnalyzer` raise warnings](https://github.com/dotnet/BenchmarkDotNet/blob/94303f4e456f66aa2d6b6cae4a3c4eb9217ed7e7/src/BenchmarkDotNet/Analysers/BaselineCustomAnalyzer.cs#L33-L37) and `Ratio`/`Ratio` column show `?`.
1. Set baseline on Job (that running benchmark with .NET 8).
2. Set baseline on benchmark method (It's used when running benchmark with different config)
3. Enable grouping with `.AddLogicalGroupRules(BenchmarkLogicalGroupRule.ByMethod)`
#### Benchmark results
> BaselineCustomAnalyzer
> Summary -> A question mark '?' symbol indicates that it was not possible to compute the (Ratio, RatioSD) column(s) because the baseline or benchmark could not be found, or the baseline value is too close to zero.
```markdown
| Method | Toolchain | Mean | Error | StdDev | Ratio | RatioSD |
|----------------------------- |---------- |---------:|----------:|---------:|------:|--------:|
| OrderBy_EnumerableDataSource | .NET 8.0 | 153.3 ms | 10.27 ms | 0.56 ms | 1.00 | 0.00 |
| OrderBy_EnumerableDataSource | .NET 9.0 | 149.0 ms | 233.89 ms | 12.82 ms | 0.97 | 0.07 |
| OrderBy_EnumerableDataSource | .NET 10.0 | 140.8 ms | 216.04 ms | 11.84 ms | 0.92 | 0.07 |
| | | | | | | |
| OrderBy_ArrayDataSource | .NET 8.0 | 164.7 ms | 136.52 ms | 7.48 ms | ? | ? |
| OrderBy_ArrayDataSource | .NET 9.0 | 147.1 ms | 223.11 ms | 12.23 ms | ? | ? |
| OrderBy_ArrayDataSource | .NET 10.0 | 148.8 ms | 88.03 ms | 4.83 ms | ? | ? |
```
### TestCode
```csharp
internal class Program
{
static void Main(string[] args)
{
var config = DefaultConfig.Instance
.AddLogicalGroupRules(BenchmarkLogicalGroupRule.ByMethod)
.WithOrderer(new DefaultOrderer(jobOrderPolicy: JobOrderPolicy.Numeric))
.WithOptions(ConfigOptions.DisableOptimizationsValidator);
config = config.AddJob(
Job.ShortRun.WithToolchain(CsProjCoreToolchain.NetCoreApp80).AsBaseline(),
Job.ShortRun.WithToolchain(CsProjCoreToolchain.NetCoreApp90),
Job.ShortRun.WithToolchain(CsProjCoreToolchain.NetCoreApp10_0));
var summaries = BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly)
.Run(args, config)
.ToArray();
var config2 = summaries[0].BenchmarksCases[0].Config;
}
}
public class Benchmarks
{
private readonly Consumer Consumer = new();
public int N { get; set; } = 1_000_000;
private readonly IEnumerable EnumerableDataSource;
private readonly int[] ArrayDataSource;
public Benchmarks()
{
var rand = new Random(0);
var source = Enumerable.Range(1, N).Select(x => rand.Next());
EnumerableDataSource = source;
ArrayDataSource = EnumerableDataSource.ToArray();
}
[Benchmark(Baseline = true)]
public void OrderBy_EnumerableDataSource()
{
foreach (var i in EnumerableDataSource.OrderBy(x => x))
{
Consumer.Consume(i);
}
}
[Benchmark]
public void OrderBy_ArrayDataSource()
{
foreach (var i in ArrayDataSource.OrderBy(x => x))
{
Consumer.Consume(i);
}
}
}
```
### Analysis
It seems be resolved by modifying following logics.
https://github.com/dotnet/BenchmarkDotNet/blob/94303f4e456f66aa2d6b6cae4a3c4eb9217ed7e7/src/BenchmarkDotNet/Reports/BaseliningStrategy.cs#L35-L45
By adding following code that enforce to use Job's baseline setting.
```csharp
if(useDescriptors && useJobs)
benchmark.Job.Meta.Baseline
```
I've not created a PR because I'm not sure the side effects of this changes.
Contributor guide
Assessment
This issue has not been assessed yet.