google / google/benchmark

[FR] Lightweight way to pause/resume timer in benchmarks

Open
#1,087 11 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
10.4k
Forks
1.8k
Avg merge
2d 4h
Merged PRs (30d)
8

Description

Using PauseTiming/ResumeTiming can make some benchmarks take a very long time to complete. Given the following example, if the call to A() takes longer than B(), then the runtime of the benchmarks can increase to ludicrous lengths.
```
void A();
void B();

void example(benchmark::State &state) {
for (auto const _ : state) {
state.PauseTiming();
A();
state.ResumeTiming();

B();
}
}
BENCHMARK(example);
```

To fix this, I have implemented a pair of functions (BeginIgnoreTiming/EndIgnoreTiming) that can be used to ignore the timing cost of code executed between them. It keeps timing info around for when CreateRunReport() is called, where the duration ignored is subtracted from the cpu- and real-time timings, so it only affects the printed timing values.

This allows benchmarks to run as if there were no calls to pause the timings, while getting the timings of the benchmark I am actually interested in (B() in the above example).

I have included the results of some runs from a personal project of mine. The benchmark is of a similar format to the A/B example from above.

```
With no pause/resume. Results are not useable.
---------------------------------------------------------------------------------
Benchmark Time CPU Iterations
---------------------------------------------------------------------------------
ranged_system_with_components/4096 1.04 us 1.05 us 640000
ranged_system_with_components/32768 2.92 us 2.95 us 248889
ranged_system_with_components/262144 18.0 us 18.0 us 37333
ranged_system_with_components/2097152 298 us 298 us 2358
ranged_system_with_components/16777216 2372 us 2352 us 299
* Benchmark took 4716 milliseconds

With PauseTiming/resumeTiming
---------------------------------------------------------------------------------
Benchmark Time CPU Iterations
---------------------------------------------------------------------------------
ranged_system_with_components/4096 0.574 us 0.688 us 1294865
ranged_system_with_components/32768 0.616 us 0.670 us 1120000
ranged_system_with_components/262144 0.831 us 0.625 us 1000000
ranged_system_with_components/2097152 0.630 us 0.900 us 746667
ranged_system_with_components/16777216 0.810 us 1.34 us 793297
* Benchmark took 3366572 milliseconds (56 minutes)

With new lightweight pause/resume
---------------------------------------------------------------------------------
Benchmark Time CPU Iterations
---------------------------------------------------------------------------------
ranged_system_with_components/4096 0.579 us 0.534 us 497778
ranged_system_with_components/32768 0.614 us 0.659 us 213333
ranged_system_with_components/262144 0.819 us 1.26 us 37333
ranged_system_with_components/2097152 0.601 us 0.000 us 2358
ranged_system_with_components/16777216 0.749 us 0.000 us 299
* Benchmark took 4819 milliseconds
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.