google / google/tcmalloc

Lock contention bottleneck (jemalloc vs tcmalloc)

Open
#40 2 comments 4 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
5.3k
Forks
570
Avg merge
23h 43m
Merged PRs (30d)
172

Description

Recently I've been playing with tcmalloc (this new version) and found out that once the size of allocation is not small (there is size class for it) it always acquire the `pageheap_lock` (and indeed, this is described in [doc](https://github.com/google/tcmalloc/blob/master/docs/design.md#tcmalloc-middle-end))

However this became a bottleneck with multiple threads, here is a [sample](https://gist.github.com/azat/694a9012d10fc4536d0ce17481d113d7) that shows this, it is simply:
- creates 16 threads
- allocate objects from 4k to 1M (each time size of the allocation multiplied by 4)
- tcmalloc configured with 256K pages w/o sampling
- jemalloc uses per-cpu arena

And results (you can also find this numbers in comments):

|conf|real|user|sys|
|-|-|-|-|
|jemalloc|0m10.816s|2m24.375s|0m0.230s|
|tcmalloc|0m19.837s|4m32.754s|0m3.329s|
|jemalloc capped to 256K|0m2.567s|0m32.748s|0m0.020s|
|tcmalloc capped to 256K|0m2.335s|0m28.804s|0m0.010s|

*sys time is mostly due to futex*

Plus some locking info (no need in anything better then strace, it shows the problem):
- jemalloc:

```
$ time strace -qq -fefutex -c allocator-perf-jemalloc
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00 0.448936 11223 40 futex
------ ----------- ----------- --------- --------- ----------------
100.00 0.448936 40 total

real 0m10.851s
user 2m27.460s
sys 0m0.767s
```

- tcmalloc:

```
$ time strace -qq -fefutex -c allocator-perf-tcmalloc
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00 29.896237 18 1619220 629766 futex
------ ----------- ----------- --------- --------- ----------------
100.00 29.896237 1619220 629766 total

real 0m27.448s
user 3m44.494s
sys 0m33.782s
```

Any plans on improving this?
Or maybe adding support for custom size classes? By providing some helpers to generate them (I can even generate them right now, with some small modifications)

tcmalloc version: 8738f271bd58a0decb358fd6047f3bfac1be3382

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.