microsoft / microsoft/mimalloc
Peak RSS regression from 2.0.6 to 2.2.4
- Dominant language
- C
- Stars
- 13.4k
- Forks
- 1.2k
- Avg merge
- 4d 45m
- Merged PRs (30d)
- 13
Description
In Arrow we [recently updated](https://github.com/apache/arrow/commit/0826be572bd9ceeecb8c00695eb63b2f6135133f) our bundled mimalloc build from 2.0.6 to 2.2.4.
Lately I noticed that reading some Parquet files I have lying around led to a much higher max RSS than on Arrow 20.0.0, so I bisected and found out that the culprit was indeed the mimalloc version bump.
My measurements are done on a 12-core 24-thread machine on Ubuntu 24.04 using the following command (repeated 3 times to ensure the numbers are stable):
```console
$ /usr/bin/time -f "%U user %E elapsed %M kB" python -c "import pyarrow.parquet as pq; pq.read_table('FILENAME')"
```
Here I present the results for 3 different mimalloc versions, and also for jemalloc and the system (glibc) allocator:
Test file | mimalloc 3.1.5 | mimalloc 2.2.4 | mimalloc 2.0.6 | jemalloc | system (glibc)
-- | -- | -- | -- |-- | --
lineitem.parquet | 1.44 user 0:00.42 elapsed 669608 kB | 1.65 user 0:00.40 elapsed **892512 kB** | 1.38 user 0:00.44 elapsed 700696 kB | 1.39 user 0:00.44 elapsed 663816 kB | 1.42 user 0:00.46 elapsed 717120 kB
1000col.parquet | 15.48 user 0:03.15 elapsed 2666164 kB | 14.79 user 0:03.16 elapsed **3224196 kB** | 15.74 user 0:03.22 elapsed 2585096 kB | 15.85 user 0:03.18 elapsed 2593136 kB | 15.92 user 0:03.73 elapsed 2576316 kB
StockUniteLegale_utf8.parquet | 24.20 user 0:02.11 elapsed 7064604 kB | 24.99 user **0:01.73 elapsed** **8934212 kB** | 23.29 user 0:02.14 elapsed 7002348 kB | 23.23 user 0:02.15 elapsed 6950332 kB | 23.28 user 0:02.23 elapsed 6734988 kB
You can see that mimalloc 2.2.4 produces much higher peak RSS than all other allocators. Elapsed time is sometimes better than the others, which hints that 2.2.4 might just be setting aside a lot more freed memory.
As an experiment I also measured with multi-threading disabled by setting the environment variable `OMP_NUM_THREADS=1` and passing `use_threads=False` to the `read_table` function call:
Test file | mimalloc 3.1.5 | mimalloc 2.2.4 | mimalloc 2.0.6 | jemalloc | system (glibc)
-- | -- | -- | -- |-- | --
lineitem.parquet | 0.94 user 0:01.21 elapsed 641460 kB | 0.90 user 0:01.20 elapsed 626100 kB | 0.90 user 0:01.21 elapsed 628768 kB | 0.92 user 0:01.20 elapsed 629676 kB | 0.93 user 0:01.21 elapsed 619656 kB
1000col.parquet | 5.41 user 0:05.98 elapsed 1927936 kB | 5.47 user 0:06.12 elapsed 1918084 kB | 5.52 user 0:06.11 elapsed 1918744 kB | 5.47 user 0:06.10 elapsed 1918464 kB | 7.55 user 0:08.57 elapsed 2125832 kB
StockUniteLegale_utf8.parquet | 14.92 user 0:18.12 elapsed 6990128 kB | 14.52 user **0:15.94 elapsed** **8899392 kB** | 14.80 user 0:17.98 elapsed 7101364 kB | 14.42 user 0:17.53 elapsed 6809584 kB | 14.44 user 0:17.54 elapsed 6848652 kB
With multi-threading (almost) disabled, we see that the peak RSS problem mostly disappears - as I would expect - except on one file `StockUniteLegale_utf8.parquet` where it is surprisingly still present.
Also, the good news is that mimalloc 3.1.5 looks fine in these measurements. But we would rather wait for it to move to stable status before switching to it.
(note: I've also reported user time, but HW multi-threading makes comparisons unreliable)
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the reported peak RSS regression with the `/usr/bin/time` Python command against the three named Parquet files, comparing mimalloc 2.0.6, 2.2.4, 3.1.5, jemalloc, and glibc. Repeat with `OMP_NUM_THREADS=1` and `use_threads=False`; done means identifying the cause and resolving or clearly characterizing the excess memory retained by mimalloc 2.2.4.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, python, ubuntu
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100