dotnet / dotnet/BenchmarkDotNet

--join does not respect types and ratio

Open
#2,022 0 comments 0 reactions 1 assignee Assigned to @YegorStepanov View on GitHub
Dominant language
C#
Stars
11.5k
Forks
1.1k
Avg merge
6d 11h
Merged PRs (30d)
12

Description

When benchmarking multiple types with `--join` option, the resulting table does not group by types, and `Ratio` is completely messed up.

```
| Type | Method | Pending | Mean | Error | StdDev | Ratio | RatioSD | Code Size | Gen 0 | Allocated | Alloc Ratio |
|------------- |------------- |-------- |-----------:|---------:|---------:|------:|--------:|----------:|-------:|----------:|------------:|
| AsyncAwait | Callback | False | 287.8 ns | 2.01 ns | 1.88 ns | 1.00 | 0.00 | 1,561 B | 0.2651 | 832 B | 1.00 |
| ContinueWith | Callback | False | 293.9 ns | 0.70 ns | 0.66 ns | 1.02 | 0.01 | 1,572 B | 0.2651 | 832 B | 1.00 |
| AsyncAwait | ProtoPromise | False | 352.5 ns | 1.53 ns | 1.36 ns | 1.22 | 0.01 | 2,010 B | - | - | 0.00 |
| ContinueWith | ProtoPromise | False | 495.5 ns | 8.59 ns | 8.03 ns | 1.72 | 0.03 | 11,185 B | - | - | 0.00 |
| AsyncAwait | RsgPromise | False | NA | NA | NA | ? | ? | - | - | - | ? |
| ContinueWith | RsgPromise | False | 573.8 ns | 3.57 ns | 2.79 ns | 1.99 | 0.02 | 2,476 B | 0.3290 | 1032 B | 1.24 |
| AsyncAwait | Task | False | 319.3 ns | 1.35 ns | 1.05 ns | 1.11 | 0.01 | 2,496 B | 0.0610 | 192 B | 0.23 |
| ContinueWith | Task | False | 2,231.1 ns | 26.55 ns | 24.84 ns | 7.75 | 0.09 | 9,632 B | 0.3891 | 1224 B | 1.47 |
| AsyncAwait | UniTask | False | 367.2 ns | 1.68 ns | 1.57 ns | 1.28 | 0.01 | 2,425 B | - | - | 0.00 |
| ContinueWith | UniTask | False | 708.9 ns | 8.67 ns | 15.86 ns | 2.51 | 0.07 | 8,385 B | - | - | 0.00 |
| AsyncAwait | ValueTask | False | 485.4 ns | 2.04 ns | 1.59 ns | 1.68 | 0.01 | 2,991 B | - | - | 0.00 |
| ContinueWith | ValueTask | False | NA | NA | NA | ? | ? | - | - | - | ? |
```

Types should be grouped separately with `--join` and `Ratio` respected for each type.

See also #1864 which was regarding categories, but this issue is without categories.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.