dotnet / dotnet/BenchmarkDotNet

BaselineCustomAnalyzer raise warning under specific conditions

Open
#2,956 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
11.5k
Forks
1.1k
Avg merge
6d 11h
Merged PRs (30d)
12

Description

I’m trying to write benchmarks that measure performance difference between .NET version.

But when using following settings [`BaselineCustomAnalyzer` raise warnings](https://github.com/dotnet/BenchmarkDotNet/blob/94303f4e456f66aa2d6b6cae4a3c4eb9217ed7e7/src/BenchmarkDotNet/Analysers/BaselineCustomAnalyzer.cs#L33-L37) and `Ratio`/`Ratio` column show `?`.
1. Set baseline on Job (that running benchmark with .NET 8).
2. Set baseline on benchmark method (It's used when running benchmark with different config)
3. Enable grouping with `.AddLogicalGroupRules(BenchmarkLogicalGroupRule.ByMethod)`

#### Benchmark results

> BaselineCustomAnalyzer
> Summary -> A question mark '?' symbol indicates that it was not possible to compute the (Ratio, RatioSD) column(s) because the baseline or benchmark could not be found, or the baseline value is too close to zero.

```markdown
| Method | Toolchain | Mean | Error | StdDev | Ratio | RatioSD |
|----------------------------- |---------- |---------:|----------:|---------:|------:|--------:|
| OrderBy_EnumerableDataSource | .NET 8.0 | 153.3 ms | 10.27 ms | 0.56 ms | 1.00 | 0.00 |
| OrderBy_EnumerableDataSource | .NET 9.0 | 149.0 ms | 233.89 ms | 12.82 ms | 0.97 | 0.07 |
| OrderBy_EnumerableDataSource | .NET 10.0 | 140.8 ms | 216.04 ms | 11.84 ms | 0.92 | 0.07 |
| | | | | | | |
| OrderBy_ArrayDataSource | .NET 8.0 | 164.7 ms | 136.52 ms | 7.48 ms | ? | ? |
| OrderBy_ArrayDataSource | .NET 9.0 | 147.1 ms | 223.11 ms | 12.23 ms | ? | ? |
| OrderBy_ArrayDataSource | .NET 10.0 | 148.8 ms | 88.03 ms | 4.83 ms | ? | ? |
```

### TestCode

```csharp
internal class Program
{
static void Main(string[] args)
{
var config = DefaultConfig.Instance
.AddLogicalGroupRules(BenchmarkLogicalGroupRule.ByMethod)
.WithOrderer(new DefaultOrderer(jobOrderPolicy: JobOrderPolicy.Numeric))
.WithOptions(ConfigOptions.DisableOptimizationsValidator);

config = config.AddJob(
Job.ShortRun.WithToolchain(CsProjCoreToolchain.NetCoreApp80).AsBaseline(),
Job.ShortRun.WithToolchain(CsProjCoreToolchain.NetCoreApp90),
Job.ShortRun.WithToolchain(CsProjCoreToolchain.NetCoreApp10_0));

var summaries = BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly)
.Run(args, config)
.ToArray();

var config2 = summaries[0].BenchmarksCases[0].Config;
}
}

public class Benchmarks
{
private readonly Consumer Consumer = new();

public int N { get; set; } = 1_000_000;

private readonly IEnumerable EnumerableDataSource;
private readonly int[] ArrayDataSource;

public Benchmarks()
{
var rand = new Random(0);
var source = Enumerable.Range(1, N).Select(x => rand.Next());

EnumerableDataSource = source;
ArrayDataSource = EnumerableDataSource.ToArray();
}

[Benchmark(Baseline = true)]
public void OrderBy_EnumerableDataSource()
{
foreach (var i in EnumerableDataSource.OrderBy(x => x))
{
Consumer.Consume(i);
}
}

[Benchmark]
public void OrderBy_ArrayDataSource()
{
foreach (var i in ArrayDataSource.OrderBy(x => x))
{
Consumer.Consume(i);
}
}
}
```

### Analysis

It seems be resolved by modifying following logics.
https://github.com/dotnet/BenchmarkDotNet/blob/94303f4e456f66aa2d6b6cae4a3c4eb9217ed7e7/src/BenchmarkDotNet/Reports/BaseliningStrategy.cs#L35-L45

By adding following code that enforce to use Job's baseline setting.
```csharp
if(useDescriptors && useJobs)
benchmark.Job.Meta.Baseline
```

I've not created a PR because I'm not sure the side effects of this changes.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.