Lock contention bottleneck (jemalloc vs tcmalloc)
- Dominant language
- C++
- Stars
- 5.3k
- Forks
- 570
- Avg merge
- 23h 43m
- Merged PRs (30d)
- 172
Description
Recently I've been playing with tcmalloc (this new version) and found out that once the size of allocation is not small (there is size class for it) it always acquire the `pageheap_lock` (and indeed, this is described in [doc](https://github.com/google/tcmalloc/blob/master/docs/design.md#tcmalloc-middle-end))
However this became a bottleneck with multiple threads, here is a [sample](https://gist.github.com/azat/694a9012d10fc4536d0ce17481d113d7) that shows this, it is simply:
- creates 16 threads
- allocate objects from 4k to 1M (each time size of the allocation multiplied by 4)
- tcmalloc configured with 256K pages w/o sampling
- jemalloc uses per-cpu arena
And results (you can also find this numbers in comments):
|conf|real|user|sys|
|-|-|-|-|
|jemalloc|0m10.816s|2m24.375s|0m0.230s|
|tcmalloc|0m19.837s|4m32.754s|0m3.329s|
|jemalloc capped to 256K|0m2.567s|0m32.748s|0m0.020s|
|tcmalloc capped to 256K|0m2.335s|0m28.804s|0m0.010s|
*sys time is mostly due to futex*
Plus some locking info (no need in anything better then strace, it shows the problem):
- jemalloc:
```
$ time strace -qq -fefutex -c allocator-perf-jemalloc
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00 0.448936 11223 40 futex
------ ----------- ----------- --------- --------- ----------------
100.00 0.448936 40 total
real 0m10.851s
user 2m27.460s
sys 0m0.767s
```
- tcmalloc:
```
$ time strace -qq -fefutex -c allocator-perf-tcmalloc
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00 29.896237 18 1619220 629766 futex
------ ----------- ----------- --------- --------- ----------------
100.00 29.896237 1619220 629766 total
real 0m27.448s
user 3m44.494s
sys 0m33.782s
```
Any plans on improving this?
Or maybe adding support for custom size classes? By providing some helpers to generate them (I can even generate them right now, with some small modifications)
tcmalloc version: 8738f271bd58a0decb358fd6047f3bfac1be3382
Contributor guide
Assessment
This issue has not been assessed yet.