microsoft / microsoft/mimalloc

Slowdown when using mimalloc to allocate (many) buffers

Open
#1,121 7 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
13.4k
Forks
1.2k
Avg merge
4d 45m
Merged PRs (30d)
13

Description

Hello,

I am playing with Apache Arrow, which is using mimalloc (by default), and noticed some weird performance that I could not explain and that goes away when switching to the `system` allocator. I extracted the core of the problem, shown in the following:
```c++
#include
#include
#include
#include
#include

static constexpr std::size_t NumPages = 1000;
static constexpr std::size_t NumElements = 64 * 1024 + 1;

int main(int argc, char *argv[]) {
std::vector> pages;
for (std::size_t i = 0; i < NumPages; i++) {
pages.emplace_back(new float[NumElements]);
}

auto start = std::chrono::steady_clock::now();

for (std::size_t e = 0; e < NumElements; e++) {
for (std::size_t i = 0; i < NumPages; i++) {
pages[i][e] = 0;
}
}

auto end = std::chrono::steady_clock::now();
const std::chrono::duration duration = end - start;
std::cout << "run time: " << duration.count() << "\n";

return 0;
}
```

Compiling with `g++ -O2 pages.cxx -o pages` and running gives:
```
$ taskset -c 2 ./pages
run time: 0.674548
```
However, preloading `mimalloc.so` is 10x slower:
```
$ LD_PRELOAD=install/lib64/libmimalloc.so taskset -c 2 ./pages
run time: 7.09013
```

This is on Alma Linux 8 with a somewhat dated glibc 2.28 and Linux kernel 4.18.0-553.51.1.el8_10.x86_64. However, I can also reproduce a 3x slowdown on Arch Linux with a recent kernel:
```
$ taskset -c 2 ./pages
run time: 0.66908
$ LD_PRELOAD=/usr/lib64/libmimalloc.so taskset -c 2 ./pages
run time: 1.91931
```

The numbers above are from the current `main` (or `v2.2.4` when using the Arch Linux package), but I can equally reproduce with `v1.9.4` and latest `dev3`.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by building and running the pages.cxx reproducer with g++ -O2, then compare the system allocator against LD_PRELOAD=install/lib64/libmimalloc.so under the reported CPU affinity. Investigate allocator behavior for the repeated page allocations and traversal, using the Alma Linux and Arch Linux environments as comparison points. Done means identifying the cause of the slowdown and documenting or correcting it.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
operating-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.