microsoft / microsoft/mimalloc
Buggy statistics reporting in stable and v2.0.2?
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 13.4k
- Forks
- 1.2k
- Avg merge
- 4d 45m
- Merged PRs (30d)
- 13
Description
Hi,
I'm using mimalloc in my linux (ubuntu18.04.5lts) application and am investigating why my RSS keeps growing. This is proving difficult because I see negative values in the "current" allocation column.
I've changed to v2.0.2 (and forced MI_DEBUG=2) and still see the problem.
My options are as follows:
```
mimalloc: option 'show_errors': 1
mimalloc: option 'show_stats': 1
mimalloc: option 'eager_commit': 1
mimalloc: option 'eager_region_commit': 1
mimalloc: option 'reset_decommits': 1
mimalloc: option 'large_os_pages': 0
mimalloc: option 'reserve_huge_os_pages': 0
mimalloc: option 'reserve_os_memory': 0
mimalloc: option 'segment_cache': 0
mimalloc: option 'page_reset': 1
mimalloc: option 'abandoned_page_reset': 1
mimalloc: option 'segment_reset': 1
mimalloc: option 'eager_commit_delay': 1
mimalloc: option 'allow_decommit': 1
mimalloc: option 'reset_delay': 1
mimalloc: option 'segment_decommit_delay': 1
mimalloc: option 'use_numa_nodes': 0
mimalloc: option 'limit_os_alloc': 0
mimalloc: option 'os_tag': 100
mimalloc: option 'max_errors': 1600
mimalloc: option 'max_warnings': 1600
mimalloc: process init: 0x7ffff7fcf740
mimalloc: debug level : 2
mimalloc: secure level: 0
```
Is that a reasonable set? The delays are very short as I want to correlate the RSS frees back to my application.
With my reproducer I see a negative commit-current, and the commit-free is almost double the commit-total:
```
Allocating 1000000 words
mimalloc: using 2 numa regions
heap stats: peak total freed current unit count
normal 6: 192 b 1.1 kb 1.1 kb 0 b 48 b 25 ok
normal 9: 320 b 800 b 800 b 0 b 80 b 10 ok
normal 23: 896 b 896 b 896 b 0 b 896 b 1 ok
normal 37: 20.0 kb 50.1 kb 50.1 kb 0 b 10.0 kb 5 ok
heap stats: peak total freed current unit count
normal: 20.5 k 52.7 k 52.7 k 0 1.2 kb 41 ok
large: 0 b 0 b 0 b 0 b 1 b ok
huge: 7.7 m 7.7 m 7.7 m 0 7.7 mb 1 ok
total: 7.7 mb 7.8 mb 7.8 mb 0 b 1 b ok
malloc req: 7.6 mb 7.6 mb 7.6 mb 0 b 1 b ok
reserved: 16.0 mb 16.0 mb 8.0 mb 8.0 mb 1 b not all freed!
committed: 8.2 mb 8.3 mb 15.8 mb -7.4 mb 1 b ok
reset: 0 b 0 b 0 b 0 b 1 b ok
touched: 7.8 mb 7.8 mb 7.8 mb 0 b 1 b ok
segments: 2 2 2 0 ok
-abandoned: 0 0 0 0 ok
-cached: 0 0 0 0 ok
pages: 4 5 5 0 ok
-abandoned: 0 0 0 0 ok
-extended: 5
-noretire: 26
mmaps: 2
commits: 3
threads: 0 0 0 0 ok
searches: 0.0 avg
numa nodes: 2
elapsed: 4.016 s
process: user: 0.021 s, system: 0.009 s, faults: 0, rss: 36.2 mb, commit: 8.2 mb
mimalloc: process done: 0x7fa265941740
```
My initial debugging suggests that my large buffer is freed via `mi_free_block_mt()` for the actual size freed, but this does not update the commit mask. Then `mi_segment_os_free()` frees with a full (~0) commit mask of memory, leading tothe double counting.
So, are my flags valid, and if so should `mt_free_block_mt()` update the commit mask?
My repro code is
```
#include
#include
#include
#include
int main(int argc, char **argv) {
if (argc<2) {
std::cerr<<"Usage stats_minimal size\n";
return 1;
}
auto n = std::strtoull(argv[1], nullptr, 0);
std::cerr<<"Allocating " << n << " words\n";
std::vector v(n);
for (std::size_t i{0}; i != n; ++i)
v[i] = i;
std::cerr<<"Allocating " << n << " words\n";
mi_stats_print(NULL);
v.resize(0);
v.shrink_to_fit(); // This is the point at which the "current commit" value becomes negative.
return 0;
}
```
compiled with
```
g++ -g --std=c++17 -Wall -I ~/work/install/mimalloc/include/mimalloc-2.0 -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free stats_minimal.cpp ~/work/build/poplar/deb/install/mimalloc/lib/mimalloc-2.0/libmimalloc.a -lpthread
```
and I'm invoking the test via
```
./a.out 1000000
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by building and running the provided stats_minimal.cpp reproducer with the shown command and options. Trace the reported path through mi_free_block_mt() and mi_segment_os_free(), focusing on how the commit mask is accounted for during vector shrink_to_fit(). Done means the reproducer no longer reports a negative current commit or commit-free greater than commit-total.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, cpp
- Domain
- operating-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100