[BUG] CPU hang on multithreaded benchmarks overriding the memory manager
- Dominant language
- C++
- Stars
- 10.4k
- Forks
- 1.8k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 8
Description
**Describe the bug**
Consider a multithreaded benchmark scenario which overrides the memory manager.
In this scenario we want all the benchmark threads to synchronize at a given point in time and we use benchmark::State::threads() to fetch the number of threads that need to be synced.
This leads to a CPU hang because when the memory manager is overriden, BenchmarkRunner::DoOneRepetition() runs a final pass to gather memory statistics on a single thread while recycling a fixture that was designed for multithreaded execution.
Here is a minimal repro scenario:
```
#include
#include
#include
#include
#include
// Set this to 0 to remove the hang
#define MEMORY_MANAGER_OVERRIDE 1
/*
Minimal repro:
When performing a benchmark and overriding the memory manager, state.threads() becomes unreliable;
This happens because a memory benchmark test will be ran on a single thread, recycling a fixture that was created for a multi-threaded scenario
*/
using cscoped_lock = std::unique_lock;
class memory_manager final : public benchmark::MemoryManager {
public:
memory_manager() = default;
void Start() override final {};
void Stop(Result&) override final {};
};
class benchmark_fixture : public benchmark::Fixture {
std::mutex m_mutex;
std::condition_variable m_cv;
uint32_t m_num_threads = 0;
private:
void SetUp(const benchmark::State& _state) override final; // Waits on every thread to enter before calling to OnBenchmarkSetup
void TearDown(const benchmark::State& _state) override final; // Waits on every thread to enter before calling to OnBenchmarkTearDown
private:
void OnBenchmarkSetup() {};
void OnBenchmarkTearDown() {};
public:
virtual ~benchmark_fixture() = default;
};
void benchmark_fixture::SetUp(const benchmark::State& _state) {
const uint32_t thread_count = _state.threads();
cscoped_lock lock(m_mutex);
++m_num_threads;
const bool startup = m_num_threads == thread_count;
if (startup) {
OnBenchmarkSetup(); // The last thread to enter will perform the benchmark setup
m_cv.notify_all();
} else {
m_cv.wait(lock);
}
}
void benchmark_fixture::TearDown(const benchmark::State& _state) {
const uint32_t thread_count = _state.threads();
cscoped_lock lock(m_mutex);
--m_num_threads;
const bool teardown = m_num_threads == 0;
if (teardown) {
OnBenchmarkTearDown(); // The last thread to enter will perform the benchmark teardown
m_cv.notify_all();
} else {
m_cv.wait(lock);
}
}
BENCHMARK_DEFINE_F(benchmark_fixture, run)
(benchmark::State& _state) {
constexpr uint32_t wait_time[] = { 16, 14, 12, 10, 8, 6, 4, 2, 0 };
// Wait an arbitrary period of time to simulate work
for (auto _ : _state) {
const uint32_t thread_index = _state.thread_index();
const uint32_t wait_time_ms = wait_time[thread_index / 2];
std::this_thread::sleep_for(std::chrono::milliseconds(wait_time_ms));
}
}
BENCHMARK_REGISTER_F(benchmark_fixture, run)->Iterations(4)->ThreadRange(1, 16);
int main(int argc, char** argv) {
char arg0_default[] = "benchmark";
char* args_default = arg0_default;
if (!argv) {
argc = 1;
argv = &args_default;
}
::benchmark::Initialize(&argc, argv);
if (::benchmark::ReportUnrecognizedArguments(argc, argv)) {
return 1;
}
#if MEMORY_MANAGER_OVERRIDE
memory_manager mem_mgr;
::benchmark::RegisterMemoryManager(&mem_mgr);
#endif
::benchmark::RunSpecifiedBenchmarks();
::benchmark::Shutdown();
return 0;
}
```
Why are memory statistics gathered in a multithreaded scenario not following the same path as regular fixture runs?
Why are they recycling a fixture that was built to run in a multithreaded fashion to perform a single threaded test?
Contributor guide
Assessment
This issue has not been assessed yet.