google / google/benchmark

[FR] `PredictNumItersNeeded()` 1.4 correction factor

Open
#1,848 23 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
10.4k
Forks
1.8k
Avg merge
2d 4h
Merged PRs (30d)
8

Description

**Problem description**
In the function [`PredictNumItersNeeded()`](https://github.com/google/benchmark/blob/08fdf6eb84cc8a5b65d84041257c908de5879bf5/src/benchmark_runner.cc#L313) there is this `1.4` correction factor.
This causes the time running the experiment to exceed by ~40% the time specified by `--benchmark_min_time`.
Of course, `--benchmark_min_time` denotes the *minimum* amount of time to run the benchmark, but an overrun of 40% seems excessive.
This is particularly relevant in supercomputers, where CPU time is expensive.

* Why is this estimation done, instead of stopping the iterations when the accumulated "iteration time" exceeds the target time?
* Is there a reason for selecting 1.4 as a correction factor?
* In cases where the execution times are not stable, could this prediction be wrong by a large margin?

**Suggested solution**
I suggest either removing the correction factor or making it configurable (with a default value of 1.0).

**Example**
As shown in the following output (executed with `--benchmark_min_time=1s`) the real execution time is ~1.4s: $7039 \times 198481 = 1397107759$, $8775 \times 160984 = 1412634600$, ...
```
------------------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------------------------------------------------
GNU-TBB/std::adjacent_difference/double/1024/manual_time 7039 ns 7022 ns 198481 bytes_per_second=6.50968Gi/s
GNU-TBB/std::adjacent_find/double/1024/manual_time 8775 ns 8702 ns 160984 bytes_per_second=2.61075Gi/s
GNU-TBB/std::all_of/double/1024/manual_time 7704 ns 7529 ns 199585 bytes_per_second=2.97387Gi/s
GNU-TBB/std::any_of/double/1024/manual_time 4707 ns 4625 ns 301878 bytes_per_second=4.86754Gi/s
```

When executing the same code with `--benchmark_min_time=0.71s` ($1/1.4 \simeq 0.71$), the execution times are much closer to 1s: $7681 \times 134530 = 1033324930$, $9265 \times 103410 = 958093650$, ...
```
------------------------------------------------------------------------------------------------------------------------
Benchmark Time CPU Iterations UserCounters...
------------------------------------------------------------------------------------------------------------------------
GNU-TBB/std::adjacent_difference/double/1024/manual_time 7681 ns 7615 ns 134530 bytes_per_second=5.96565Gi/s
GNU-TBB/std::adjacent_find/double/1024/manual_time 9265 ns 9175 ns 103410 bytes_per_second=2.47288Gi/s
GNU-TBB/std::all_of/double/1024/manual_time 7996 ns 7572 ns 110333 bytes_per_second=2.86519Gi/s
GNU-TBB/std::any_of/double/1024/manual_time 4656 ns 4680 ns 192865 bytes_per_second=4.92025Gi/s
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.