microsoft / microsoft/mimalloc

Buggy statistics reporting in stable and v2.0.2?

Open
#441 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
13.4k
Forks
1.2k
Avg merge
4d 45m
Merged PRs (30d)
13

Description

Hi,
I'm using mimalloc in my linux (ubuntu18.04.5lts) application and am investigating why my RSS keeps growing. This is proving difficult because I see negative values in the "current" allocation column.
I've changed to v2.0.2 (and forced MI_DEBUG=2) and still see the problem.
My options are as follows:
```
mimalloc: option 'show_errors': 1
mimalloc: option 'show_stats': 1
mimalloc: option 'eager_commit': 1
mimalloc: option 'eager_region_commit': 1
mimalloc: option 'reset_decommits': 1
mimalloc: option 'large_os_pages': 0
mimalloc: option 'reserve_huge_os_pages': 0
mimalloc: option 'reserve_os_memory': 0
mimalloc: option 'segment_cache': 0
mimalloc: option 'page_reset': 1
mimalloc: option 'abandoned_page_reset': 1
mimalloc: option 'segment_reset': 1
mimalloc: option 'eager_commit_delay': 1
mimalloc: option 'allow_decommit': 1
mimalloc: option 'reset_delay': 1
mimalloc: option 'segment_decommit_delay': 1
mimalloc: option 'use_numa_nodes': 0
mimalloc: option 'limit_os_alloc': 0
mimalloc: option 'os_tag': 100
mimalloc: option 'max_errors': 1600
mimalloc: option 'max_warnings': 1600
mimalloc: process init: 0x7ffff7fcf740
mimalloc: debug level : 2
mimalloc: secure level: 0
```
Is that a reasonable set? The delays are very short as I want to correlate the RSS frees back to my application.

With my reproducer I see a negative commit-current, and the commit-free is almost double the commit-total:
```
Allocating 1000000 words
mimalloc: using 2 numa regions
heap stats: peak total freed current unit count
normal 6: 192 b 1.1 kb 1.1 kb 0 b 48 b 25 ok
normal 9: 320 b 800 b 800 b 0 b 80 b 10 ok
normal 23: 896 b 896 b 896 b 0 b 896 b 1 ok
normal 37: 20.0 kb 50.1 kb 50.1 kb 0 b 10.0 kb 5 ok

heap stats: peak total freed current unit count
normal: 20.5 k 52.7 k 52.7 k 0 1.2 kb 41 ok
large: 0 b 0 b 0 b 0 b 1 b ok
huge: 7.7 m 7.7 m 7.7 m 0 7.7 mb 1 ok
total: 7.7 mb 7.8 mb 7.8 mb 0 b 1 b ok
malloc req: 7.6 mb 7.6 mb 7.6 mb 0 b 1 b ok

reserved: 16.0 mb 16.0 mb 8.0 mb 8.0 mb 1 b not all freed!
committed: 8.2 mb 8.3 mb 15.8 mb -7.4 mb 1 b ok
reset: 0 b 0 b 0 b 0 b 1 b ok
touched: 7.8 mb 7.8 mb 7.8 mb 0 b 1 b ok
segments: 2 2 2 0 ok
-abandoned: 0 0 0 0 ok
-cached: 0 0 0 0 ok
pages: 4 5 5 0 ok
-abandoned: 0 0 0 0 ok
-extended: 5
-noretire: 26
mmaps: 2
commits: 3
threads: 0 0 0 0 ok
searches: 0.0 avg
numa nodes: 2
elapsed: 4.016 s
process: user: 0.021 s, system: 0.009 s, faults: 0, rss: 36.2 mb, commit: 8.2 mb
mimalloc: process done: 0x7fa265941740
```

My initial debugging suggests that my large buffer is freed via `mi_free_block_mt()` for the actual size freed, but this does not update the commit mask. Then `mi_segment_os_free()` frees with a full (~0) commit mask of memory, leading tothe double counting.

So, are my flags valid, and if so should `mt_free_block_mt()` update the commit mask?

My repro code is
```
#include
#include
#include
#include

int main(int argc, char **argv) {
if (argc<2) {
std::cerr<<"Usage stats_minimal size\n";
return 1;
}
auto n = std::strtoull(argv[1], nullptr, 0);
std::cerr<<"Allocating " << n << " words\n";
std::vector v(n);
for (std::size_t i{0}; i != n; ++i)
v[i] = i;
std::cerr<<"Allocating " << n << " words\n";
mi_stats_print(NULL);
v.resize(0);
v.shrink_to_fit(); // This is the point at which the "current commit" value becomes negative.
return 0;
}
```
compiled with
```
g++ -g --std=c++17 -Wall -I ~/work/install/mimalloc/include/mimalloc-2.0 -fno-builtin-malloc -fno-builtin-calloc -fno-builtin-realloc -fno-builtin-free stats_minimal.cpp ~/work/build/poplar/deb/install/mimalloc/lib/mimalloc-2.0/libmimalloc.a -lpthread
```
and I'm invoking the test via
```
./a.out 1000000
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by building and running the provided stats_minimal.cpp reproducer with the shown command and options. Trace the reported path through mi_free_block_mt() and mi_segment_os_free(), focusing on how the commit mask is accounted for during vector shrink_to_fit(). Done means the reproducer no longer reports a negative current commit or commit-free greater than commit-total.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, cpp
Domain
operating-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.