apache / apache/arrow

[C++] Provide mechanism for disabling MemoryPoolStats at either compile- or run-time

Open
#40,781 5 comments 1 reaction 1 assignee Claimed by @chrisxu333 View on GitHub
Component: C++ good-second-issue Type: enhancement
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

### Describe the enhancement requested

This is a follow-on issue to https://github.com/apache/arrow/issues/40646.

[MemoryPoolStats](https://github.com/apache/arrow/blob/main/cpp/src/arrow/memory_pool.h#L38) is used by Arrow's memory pools to track statistics related to memory usage by having each pool's `Allocate`, `Reallocate`, and `Free` methods call in its MemoryPoolStats's `UpdateAllocatedBytes` method. The cost of this tracking can be measured by updating a benchmark such as `AllocateDeallocate` to be multi-threaded and comparing a run with `UpdateAllocatedBytes` as-is versus a run with it commented out.

With 32 threads this benchmark is 10x faster on x86 when `AllocateDeallocate` is commented out. See https://gist.github.com/amoeba/b95102829280dbe2b1f64e6c23a5f594. https://github.com/apache/arrow/pull/40647 also has much more performance information.

https://github.com/apache/arrow/pull/40647 improves the performance under microbenchmarks but the cost cannot be completely avoided. I think it's reasonable to assume that real-world applications can likely detect this cost and may wish to disable `MemoryPoolStats` either at compile- or run-time.

### Component(s)

C++

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.