dotnet / dotnet/BenchmarkDotNet
IterationSetup for micro-benchmarks (`InvocationParam`)
- Dominant language
- C#
- Stars
- 11.5k
- Forks
- 1.1k
- Avg merge
- 6d 11h
- Merged PRs (30d)
- 12
Description
I remember seeing an issue about this, but I couldn't find it again.
Sometimes, in order to test a single responsibility, setup is necessary for each invocation. In cases where the benchmark method runs very quickly (micro-benchmarks), BDN complains about times less than 100ms due to inaccurate timings. Sometimes it's not possible to increase the number of operations while operating on the setup data (or doing so would affect the timing due to iterating over multiple setup datas).
Example:
```cs
[GenericTypeArguments(typeof(Vector4))]
[GenericTypeArguments(typeof(object))]
class AsyncBenchmark
{
TaskCompletionSource _source;
[IterationSetup]
public void IterationSetup()
{
_source = new TaskCompletionSource();
}
[Benchmark]
public void AwaitPendingTask()
{
Run(_source.Task);
_source.SetResult(default);
static async void Run(Task awaitable)
{
_ = await awaitable;
}
}
}
```
That could be changed to something like this.
```cs
[GenericTypeArguments(typeof(Vector4))]
[GenericTypeArguments(typeof(object))]
class AsyncBenchmark
{
const int invokeCount = 100_000;
TaskCompletionSource[] _sources = new TaskCompletionSource[invokeCount];
[IterationSetup]
public void IterationSetup()
{
for (int i = 0; i < invokeCount; ++i)
{
_sources[i] = new TaskCompletionSource();
}
}
[Benchmark(OperationsPerInvoke = invokeCount)]
public void AwaitPendingTask()
{
for (int i = 0; i < invokeCount; ++i)
{
var source = _sources[i];
Run(source.Task);
source.SetResult(default);
}
static async void Run(Task awaitable)
{
_ = await awaitable;
}
}
}
```
Increasing `invokeCount` until the execution time >= 100ms. But that's a lot more code to write, it adds the overhead of iterating the array, it doesn't allow BDN to choose an optimal invocation count through heuristics, and the same code might still run faster than 100ms on a different machine.
So my idea is to have another attribute similar to `ParamsSource` that will update the value on each invocation.
```cs
[GenericTypeArguments(typeof(Vector4))]
[GenericTypeArguments(typeof(object))]
class AsyncBenchmark
{
[InvocationParamSource(nameof(CreateSource))]
public TaskCompletionSource Source { get; set; }
public TaskCompletionSource CreateSource() => new TaskCompletionSource();
[Benchmark]
public void AwaitPendingTask()
{
Run(Source.Task);
Source.SetResult(default);
static async void Run(Task awaitable)
{
_ = await awaitable;
}
}
}
```
The implementation of this could still use a storage array (or other collection), setting its length to the invocation count on each iteration, filling each index with the return value from `CreateSource` as part of the setup, then iterating over that collection and updating the value before invoking the benchmark method.
Iterating over the collection and updating the value (without calling `CreateSource`) could be measured as part the overhead and subtracted from the final result, giving us very accurate microbenchmarks. And we'll be able to write IterationSetup benchmarks that are portable to different machines. :)
[Edit] To support teardown:
```cs
[InvocationParam(nameof(CreateTarget), nameof(DisposeTarget))]
public MyDisposable Target { get; set; }
public MyDisposable CreateTarget() => new MyDisposable();
public void DisposeTarget(MyDisposable value) => value.Dispose();
```
Contributor guide
Assessment
This issue has not been assessed yet.