dotnet / dotnet/BenchmarkDotNet

Custom names for argument/parameter values

Open
#1,634 6 comments 8 reactions 0 assignees View on GitHub
Dominant language
C#
Stars
11.5k
Forks
1.1k
Avg merge
6d 11h
Merged PRs (30d)
12

Description

Hello,
I recently came across the following issue while writing benchmarks.

Say I have a benchmark that receives one argument of type `List` through an `ArgumentsSource`, for example:
```csharp
public IEnumerable> Source => new[]
{
new List {1, 2},
new List {1, 3}
};

[Benchmark]
[ArgumentsSource(nameof(Source))]
public void Benchmark(List argument)
{
// Do stuff
}
```

The summary for this benchmark would look like this (max column width increased, measurement results are random):
```
| Method | argument | Mean | Error | StdDev |
|---------- |------------------------------------------------ |---------:|----------:|----------:|
| Benchmark | System.Collections.Generic.List`1[System.Int32] | 1.234 ms | 0.0123 ms | 0.0123 ms |
| Benchmark | System.Collections.Generic.List`1[System.Int32] | 5.678 ms | 0.0456 ms | 0.0456 ms |
```

Since the text displayed in the `argument` column is the same for both cases, and only tells of each argument's type, **there's no trivial way to distinguish between them and know which exact value was passed to each case**. This might not be the exact case in this specific example, but this is the general idea.

**Note that this is relevant not just to `List`, but to any non-primitive type, and also when using `ParamsSource` instead of `ArgumentsSource`.**

I would like to have the aforementioned ability, and one way I thought of doing this is by attaching a custom name to each value in the arguments' source. For example, if I attach the name `"One Two"` to `new List {1, 2}` and `"One Three"` to `new List {1, 3}`, the summary could look like this:
```
| Method | argument | Mean | Error | StdDev |
|---------- |------------ |---------:|----------:|----------:|
| Benchmark | One Two | 1.234 ms | 0.0123 ms | 0.0123 ms |
| Benchmark | One Three | 5.678 ms | 0.0456 ms | 0.0456 ms |
```

As far as I understood from looking at the source code and existing issues, there's currently no simple way of doing this.
I could, for example, create a type inheriting from `List` for each value and override `ToString()` with the name I want it to have, but the more values I add the more effort this would take. Furthermore, if I wanted to use a `sealed` type as an argument, this would not be possible.

Another way of distinguishing between cases that's currently possible (which I'm aware of) is by accounting for the order by which the cases appear in the summary. However, I imagine this could get confusing when working with many values or with values of different types.

I've already thought of a way to implement this (both for arguments and params), and I'm willing to open a PR, but beforehand I'd like to know if there's anything I missed (e.g. another way of doing what I described above) and if this is a feature you think fits for this library.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.