google / google/benchmark

Report other benchmark results relative to a baseline

Open
#659 6 comments 1 reaction 0 assignees View on GitHub
enhancement help wanted
Dominant language
C++
Stars
10.4k
Forks
1.8k
Avg merge
2d 4h
Merged PRs (30d)
8

Description

Imagine the following scenario: I've build new and fancy `vector` and want to benchmark it. I could now write the benchmarks and get some numbers, but I would have absolutely no idea how good these numbers are. For simplicity, let's assume my benchmark tests `push_back`, and it tests pushing an integer 1M times. My benchmark runs in 15ms. That's just an absolute number, it doesn't give me slightest idea how this fares against `std::vector` (for example).

Therefore I'd love to see an equivalent to `BENCHMARK`, called `BENCHMARK_RELATIVE`. I would recreate the same benchmark for `std::vector`, and use it as a baseline.

```
template
static void BM_PushBack(benchmark::State &s)
{
Vec v;
v.reserve(1'000'000);
for (auto _ : s) {
v.clear();
for (int i = 0; i < 1'000'000; ++i)
v.push_back(i);
}
}

using BM_PushBackStd = BM_PushBack >;
using BM_PushBackCustom = BM_PushBack >;

BENCHMARK(BM_PushBackStd);
BENCHMARK_RELATIVE(BM_PushBackStd, BM_PushBackCustom);
```

The resulting output would somewhat like this:

```
2018-08-22 10:46:25
Run on (8 X 4000 MHz CPU s)
CPU Caches:
L1 Data 32K (x4)
L1 Instruction 32K (x4)
L2 Unified 262K (x4)
L3 Unified 8388K (x1)
----------------------------------------------------------------
Benchmark Relative Time CPU Iterations
----------------------------------------------------------------
BM_PushBackStd 30 ms 30 ms 41
BM_PushBackCustom 196.23% 15 ms 15 ms 22
```

What relative means is up to discussion, in this case I've defined `100%` as equal in relative speed, everything below `100%` is slower than baseline, and everything above is faster. `196%` means it's 1.69 times or 96% faster as baseline.

And credit where credit is due, this idea originally comes from folly's `benchmark.h`: https://github.com/facebook/folly/blob/master/folly/docs/Benchmark.md

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.