microsoft / microsoft/mimalloc
Are pre-mmaped arenas only used when allocating with a heap?
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 13.4k
- Forks
- 1.2k
- Avg merge
- 4d 45m
- Merged PRs (30d)
- 13
Description
This might be kind of hard to phrase but here goes..
I'm in a situation (on Linux) where I have to pre-map a HUGE amount of memory per NUMA node. I want that memory to then be used (for now) by `malloc` and `free` and I do NOT want mimalloc to attempt to `mmap` more memory later.
I want these arenas to be significantly bigger than 16GB but that doesn't seem to be an option for now.
To accomplish this, I am:
```
mi_option_set(mi_option_disallow_os_alloc, 1);
....
mmap(...); // no more than 16GB
if (!mi_manage_os_memory_ex(
base,
size,
true, /* is_committed */
true, /* is_large */
true, /* is_zero (mmap zeroes memory) */
i, /* numa node */
false /* exclusive */,
&pool_node_info[i].arena_ids[j])) { // NOTE: I am NOT using these arena_id's for now
goto fail;
}
```
I am later printing the address of the first and last arena per node. The output is:
```
node 0 virt 0x26c54e600000
node 0 virt 0x26c579c00000
node 1 virt 0x6b88cb600000
node 1 virt 0x6b88f6c00000
```
But later when trying to do something with the results of `posix_memalign`, I get the value `0x7fe616141000`, which is clearly outside of the above `mmap`'ed range.
This leads to two questions:
1. Am I misunderstanding the `mi_option_disallow_os_alloc` option? If the arenas are full, shouldn't I get back a `NULL`?
2. Why am I not getting something from my mapped arenas? I know I'm not out of memory yet; this is happening early on startup.
Or to put this another way, what should I be doing in order to ensure that `mmap` is ONLY called at startup and if the arenas fill up I only get NULL back? I thought I had a good handle on the API functions I need but now I'm not so sure.
Thanks!
(EDIT: I'm using version 3.2.7)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the Linux allocation path using mi_option_set(mi_option_disallow_os_alloc, 1), mi_manage_os_memory_ex, mmap, and posix_memalign. Trace whether allocations use the managed arenas and what happens when they are exhausted. Done means the option's behavior and arena-size limitation are established, with a focused test or documented correction if the observed behavior is incorrect.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, linux
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100