dotnet / dotnet/BenchmarkDotNet

RunAllJoined should consider baseline per type

Open
#1,164 2 comments 1 reaction 1 assignee Assigned to @YegorStepanov View on GitHub
enhancement
Dominant language
C#
Stars
11.5k
Forks
1.1k
Avg merge
4d 11h
Merged PRs (30d)
11

Description

If you have something like this

```csharp
public class Program
{
public static void Main(string[] args)
{
BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).RunAllJoined();
}

private static readonly Guid GuidValue = Guid.NewGuid();

private static readonly string StringValue = GuidValue.ToString();

[MemoryDiagnoser]
public class Formatting
{
[Benchmark(Baseline = true)]
public string Old()
{
return CompactGuidOld.ToCompactString(GuidValue);
}

[Benchmark]
public string New()
{
return CompactGuid.ToString(GuidValue.ToUuid());
}
}

[MemoryDiagnoser]
public class Parsing
{
[Benchmark(Baseline = true)]
public Guid Old()
{
CompactGuidOld.TryParse(StringValue, out var result);
return result;
}

[Benchmark]
public Guid New()
{
CompactGuid.TryParse(StringValue, out var result);
return result.ToGuid();
}
}
}
```

I would expect my results to show both baselines (`Formatting.Old` and `Parsing.Old`) and the other benchmarks in the same type scaled based on each baseline.

Instead, the report only shows one baseline and the rest scaled based on that:

| Type | Method | Mean | Error | StdDev | Median | Ratio | RatioSD | Gen 0 | Gen 1 | Gen 2 | Allocated |
|----------- |------- |----------:|----------:|----------:|----------:|------:|--------:|-------:|------:|------:|----------:|
| Formatting | Old | 109.65 ns | 2.5403 ns | 4.1022 ns | 109.02 ns | 1.00 | 0.00 | 0.0191 | - | - | 80 B |
| Parsing | Old | 17.03 ns | 0.2582 ns | 0.2416 ns | 17.11 ns | 0.15 | 0.01 | - | - | - | - |
| Formatting | New | 100.32 ns | 2.2110 ns | 6.2722 ns | 98.11 ns | 0.97 | 0.06 | 0.0191 | - | - | 80 B |
| Parsing | New | 24.45 ns | 0.5010 ns | 0.4184 ns | 24.44 ns | 0.22 | 0.01 | - | - | - | - |

Is this expected? Wouldn't it be better to group one baseline per type? This seems semi-related to https://github.com/dotnet/BenchmarkDotNet/issues/617, but that seems to be about categories.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.