google / google/benchmark

Use of PauseTiming() and ResumeTiming() gives completely different results

Open
#842 22 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
10.4k
Forks
1.8k
Avg merge
2d 4h
Merged PRs (30d)
8

Description

```cpp
#include
#include
#include "trie.hpp"

template
void trie_find(benchmark::State& state, ExtraArgs&&... extra_args) {
std::string filename(std::forward(extra_args)...);
std::basic_ifstream file(filename);
Trie trie;
std::basic_string word;
while (std::getline(file, word)) {
trie.insert(word);
}
file.close();
for (auto _ : state) {
// state.PauseTiming();
file.open(filename);
std::basic_string word;
while (std::getline(file, word)) {
// state.ResumeTiming();
trie.find(word);
// state.PauseTiming();
}
}
}

BENCHMARK_CAPTURE(trie_find, lulzsec, "wordlists/lulzsec.txt");
BENCHMARK_CAPTURE(trie_find, cain, "wordlists/cain.txt");
BENCHMARK_CAPTURE(trie_find, hak5, "wordlists/hak5.txt");
BENCHMARK_CAPTURE(trie_find, john, "wordlists/john.txt");
BENCHMARK_CAPTURE(trie_find, bible, "wordlists/bible.txt");

BENCHMARK_MAIN();
```

```bash
g++ benchmark.cpp -lbenchmark -O3
```

Without `PauseTiming()` and `ResumeTiming()`
```bash
2019-07-25 04:44:02
Running ./a.out
Run on (4 X 2700 MHz CPU s)
CPU Caches:
L1 Data 32K (x2)
L1 Instruction 32K (x2)
L2 Unified 256K (x2)
L3 Unified 3072K (x1)
Load Average: 1.59, 0.76, 0.53
------------------------------------------------------------
Benchmark Time CPU Iterations
------------------------------------------------------------
trie_find/lulzsec 578711892 ns 576002624 ns 1
trie_find/cain 47.3 ns 47.2 ns 13669606
trie_find/hak5 37.7 ns 37.6 ns 18620521
trie_find/john 37.7 ns 37.7 ns 18679699
trie_find/bible 24.6 ns 24.5 ns 28378274
```

With `PauseTiming()` and `ResumeTiming()`
```bash
2019-07-25 04:45:41
Running ./a.out
Run on (4 X 2700 MHz CPU s)
CPU Caches:
L1 Data 32K (x2)
L1 Instruction 32K (x2)
L2 Unified 256K (x2)
L3 Unified 3072K (x1)
Load Average: 0.49, 0.64, 0.51
------------------------------------------------------------
Benchmark Time CPU Iterations
------------------------------------------------------------
trie_find/lulzsec 765193458 ns 764815007 ns 1
trie_find/cain 818307 ns 818694 ns 629
trie_find/hak5 698936 ns 698946 ns 1769
trie_find/john 701939 ns 697227 ns 1772
trie_find/bible 679012 ns 676065 ns 1701
```

I have no idea why is that happening. Can you please help me?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.