dotnet / dotnet/performance

Is IterationTime=250ms enough?

Open
#1,844 5 comments 2 reactions 0 assignees View on GitHub
Dominant language
F#
Stars
773
Forks
301
Avg merge
6d 9h
Merged PRs (30d)
15

Description

I noticed some oddities in the micro-benchmarks, e.g.:

Here is a cmd script to reproduce them:
```
dotnet run -c Release -f net5.0 -- --artifacts C:\prj\arts\base --coreRun path\to\corerun.exe --filter "*Json_ToStream*"

dotnet run -c Release -f net5.0 -- --artifacts C:\prj\arts\diff --coreRun path\to\corerun.exe --filter "*Json_ToStream*" --iterationTime 300

dotnet run -p C:\prj\performance\src\tools\ResultsComparer --base C:\prj\arts\base\results --diff C:\prj\arts\diff\results --threshold 1%%
```
Output:
```
better: 14, geomean: 2.776
worse: 1, geomean: 1.045
total diff: 15

| Slower | diff/base | Base Median (ns) | Diff Median (ns) | Modality|
| --------------------------------------------------------------------------- | ---------:| ----------------:| ----------------:| --------:|
| MicroBenchmarks.Serializers.Json_ToStream.JsonNet_ | 1.05 | 692681.11 | 724083.54 | |

| Faster | base/diff | Base Median (ns) | Diff Median (ns) | Modality|
| -------------------------------------------------------------------------------- | ---------:| ----------------:| ----------------:| --------:|
| MicroBenchmarks.Serializers.Json_ToStream.Utf8Json_ | 14.95 | 2200.00 | 147.14 | |
| MicroBenchmarks.Serializers.Json_ToStream.Utf8Json_ | 9.26 | 3100.00 | 334.82 | |
| MicroBenchmarks.Serializers.Json_ToStream.Jil_ | 8.46 | 2000.00 | 236.47 | |
| MicroBenchmarks.Serializers.Json_ToStream.Jil_ | 7.21 | 3600.00 | 499.13 | |
| MicroBenchmarks.Serializers.Json_ToStream.Utf8Json_ | 2.87 | 55800.00 | 19471.35 | |
| MicroBenchmarks.Serializers.Json_ToStream.Utf8Json_ | 2.69 | 538000.00 | 200315.62 | |
| MicroBenchmarks.Serializers.Json_ToStream.DataContractJ | 2.40 | 1886150.00 | 785514.25 | |
| MicroBenchmarks.Serializers.Json_ToStream.DataContractJsonSerial | 2.17 | 189300.00 | 87054.31 | |
| MicroBenchmarks.Serializers.Json_ToStream.DataContractJ | 2.08 | 2300800.00 | 1107050.18 | |
| MicroBenchmarks.Serializers.Json_ToStream.Utf8Json_ | 1.57 | 790900.00 | 502655.54 | |
| MicroBenchmarks.Serializers.Json_ToStream.Jil_ | 1.22 | 40950.00 | 33605.36 | |
| MicroBenchmarks.Serializers.Json_ToStream.Jil_ | 1.14 | 217600.00 | 190717.34 | |
| MicroBenchmarks.Serializers.Json_ToStream.DataContractJsonSerializer_ | 1.02 | 2584.95 | 2528.13 | |
| MicroBenchmarks.Serializers.Json_ToStream.DataContractJsonSerial | 1.02 | 1097.54 | 1074.42 | |
```
Expected output:
no diffs.

It's difficult to say what went wrong, some benchmarks from this list with the default interval time (which is [250ms](https://github.com/dotnet/performance/blob/main/src/harness/BenchmarkDotNet.Extensions/RecommendedConfig.cs#L34)) took like less than a second to warmup & run. It clearly is not enough (for any benchmark) to warm up (promote to tier1) and measure steady state.

/cc @DrewScoggins @adamsitnik @AndyAyersMS

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.