microsoft / microsoft/mimalloc

Question about the external fragmentation in a long running process

Open
#632 5 comments 1 reaction 0 assignees View on GitHub
Dominant language
C
Stars
13.4k
Forks
1.2k
Avg merge
4d 45m
Merged PRs (30d)
13

Description

Hello Daan,

The project I am [working on](https://github.com/dragonflydb/dragonfly) uses mimalloc underneath.

A user reported the following problem:
Dragonfly reports 80GiB used memory (tracked by aggregating `mi_usable_size`) but mi_malloc stats show 90GiB committed. He also says that the backend adds up roughly 0.5% committed memory each day while `used` does not change. Assuming that my "used" accounting is correct, it means the backend does not have leakage (otherwise "used" would grow as well). Dragonfly has 16 threads.

mimalloc stats output:
```
heap stats: peak total freed current unit count
reserved: 90.1 GiB 90.1 GiB 0 90.1 GiB not all freed!
committed: 90.0 GiB 90.4 GiB 412.0 MiB 90.0 GiB not all freed!
touched: 180.1 MiB 180.1 MiB 53.1 MiB 126.9 MiB not all freed!
segments: 1.4 Ki 1.4 Ki 0 1.4 Ki not all freed!
pages: 834.0 Ki 834.2 Ki 288 833.9 Ki not all freed!
commits: 508
threads: 16 16 0 16 not all freed!
searches: 0.0 avg
numa nodes: 1
elapsed: 111589.444 s
process: user: 227549.812 s, system: 255655.955 s, faults: 0, rss: 89.2 GiB, commit: 90.0 GiB
```

1. Having 12.5% increase of "committed" over "used" seems normal to me. What ratios do you observe for long-running processes?
2. I obtained a sample of area sizes (via `mi_heap_visit_blocks(.., false, ...)`) from a single heap - from a single thread out of 16. That heap has `5.9 GiB` of committed memory and 5GiB used. When looking at the unused memory I see **lots** of areas that have unused blocks. See the chart below. Why does it happen? Specifically, can mi_malloc create a new area of block size 80 if there is another one with unused memory?
"Unused" in this context is `committed - used`.

I am also attaching the detailed table of areas from which the chart was created. The first column in the table is the number of occurrences of triple ``.
![Unused memory vs block size](https://user-images.githubusercontent.com/3674760/194696048-19a20e2b-0c61-4544-a052-1da7019d9038.png)
[sorted-malloc.txt](https://github.com/microsoft/mimalloc/files/9738719/sorted-malloc.txt)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the allocator statistics and the mi_heap_visit_blocks(.., false, ...) sampling described in the issue, then inspect the relevant allocation and area-management paths. Compare committed, reserved, and usable sizes across the supplied data. Done means explaining the observed fragmentation and whether a new size-80 area can coexist with unused blocks elsewhere, with a focused regression test if a defect is found.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.