dotnet / dotnet/BenchmarkDotNet
Add support to deduct overhead from benchmarks
- Dominant language
- C#
- Stars
- 11.5k
- Forks
- 1.1k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 11
Description
Introduction
---
I'm trying to benchmark some large-er operations in my library that unfortunately cannot be isolated in a way that will give any confidence that it will be the same in production. This is similar to benchmarking an "AddToCollection" operation, which would need a setup that clears the collection on every operation.
Programmability proposal
---
What I would love is to be able to do the following in my benchmark:
``` C#
[MemoryDiagnoser]
public class DependencyBenchmark
{
private IManageSessionOf session;
private object myEvent;
public DependencyBenchmark()
=> myEvent = new
{
Count = 2,
Name = "Hey"
};
[Overhead(RunBefore = true)] //Default. If no RunXXX properties are set, behaviour defaults to run before
public void PreCallRequirements()
{
session = SessionFor.StartSession();
}
[Overhead(RunAfter = true)]
public void PostCallRequirements()
{
session.Dispose();
}
[Benchmark]
public void AddEventBenchmark()
{
session.AddEvent(myEvent);
}
}
```
Possible process
---
This would be conceptually similar to a setup for method invokation I guess. *However*, I think they can provide consistent and accurate results with the following process:
1. If any overhead methods exist in a benchmark class then benchmark each overhead method on its own as if they were marked with `Benchmark` attribute. In the example above, Benchmark.Net would run benchmarks on both `PreCallRequirements` and `PostCallRequirements` methods indipendantly. These would result in benchmarkPre and benchmarkPost
1. Benchmark the method `AddEventBenchmark` normally, calling the two overhead methods as method invocation setup and teardown. Lets call this benchmarkMain
1. For the above benchmarks run, substract the time found in overhead benchmark runs from the time needed to run the method _with_ the bechmarks to isolate the time needed by the actual method. In essesnce, display only one row with the following data:
| Type | Method | Mean | Error | StdDev | Gen 0 | Gen 1 | Allocated |
|---|---|---|---|---|---|---|---|
| DependencyBenchmark | AddEventBenchmark | AddEventBenchmark.Mean - PreCallRequirements.Mean - PostCallRequirements.Mean | AddEventBenchmark.Error - PreCallRequirements.Error - PostCallRequirements.Error (Maybe?) |AddEventBenchmark.StdDev - PreCallRequirements.StdDev - PostCallRequirements.StdDev (Maybe?) | AddEventBenchmark.Gen0 - PreCallRequirements.Gen0 - PostCallRequirements.Gen0 | AddEventBenchmark.Gen1- PreCallRequirements.Gen1 - PostCallRequirements.Gen1 | AddEventBenchmark.Allocated - PreCallRequirements.Allocated - PostCallRequirements.Allocated |
`GloabSetup` and `IterationSetup` methods would still be called normally in all above cases.
Benefits
---
1. Allows for better isolation
1. Allows to test operations that are normally larger\polluted by setup
Possible problems
---
1. No idea (but I could dig into my old Measurement system analysis textbooks to find out :| ) what is the correct mathematical way to derive standard deviation and error in the above scenario. I understand that if this proposal is to be done, these values should really be correct and reliable
1. Longer benchmarking times for benchmarks using this feature
Contributor guide
Assessment
This issue has not been assessed yet.