microsoft / microsoft/mimalloc
Arena allocator pre-maturely returns NOMEM on parent arena while space is still available in child arena
Nobody has claimed this yet.
- Dominant language
- C
- Stars
- 13.4k
- Forks
- 1.2k
- Avg merge
- 4d 45m
- Merged PRs (30d)
- 13
Description
Looks like when registering a very large arena, mimalloc internally split it into smaller sub-arenas (16 GiB each) with parent-children relationship. Then if a series of allocation exhausted the parent arena, the mimalloc API would pre-maturely return NOMEM even though there are still space in the child arenas. This looks like a bug in the macro to iterate over the sub-arenas in the arena.c code.
Repro (version: v3)
```c
#include
#include
#include
#include
#define GIB ((size_t)1024 * 1024 * 1024)
#define MIB ((size_t)1024 * 1024)
int main(void) {
const size_t arena_size = 24 * GIB;
const size_t allocation_size = 64 * MIB;
const size_t allocation_count = 272; // 17 GiB: requires a child arena.
const size_t alignment = mi_arena_min_alignment();
const size_t mapping_size = arena_size + alignment;
void* mapping = mmap(NULL, mapping_size, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0);
if (mapping == MAP_FAILED) {
perror("mmap");
return 1;
}
const uintptr_t aligned =
((uintptr_t)mapping + alignment - 1) & ~((uintptr_t)alignment - 1);
mi_arena_id_t arena_id = NULL;
if (!mi_manage_os_memory_ex((void*)aligned, arena_size,
true, /* committed */
false, /* pinned */
true, /* initially zero */
-1, /* no NUMA preference */
true, /* exclusive */
&arena_id)) {
fprintf(stderr, "failed to register the 24 GiB arena\n");
return 1;
}
mi_heap_t* heap = mi_heap_new_in_arena(arena_id);
if (heap == NULL) {
fprintf(stderr, "failed to create the arena heap\n");
return 1;
}
for (size_t i = 0; i < allocation_count; i++) {
void* p = mi_heap_malloc(heap, allocation_size);
if (p == NULL) {
fprintf(stderr,
"allocation %zu failed after %zu MiB; child arena was not used\n",
i, i * allocation_size / MIB);
return 1;
}
if (!mi_arena_contains(arena_id, p)) {
fprintf(stderr, "allocation %zu came from outside the requested arena\n", i);
return 1;
}
}
printf("allocated %zu MiB from the parent arena and its children\n",
allocation_count * allocation_size / MIB);
return 0;
}
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in arena.c, focusing on the macro that iterates over parent and child sub-arenas, and trace the allocation path from mi_heap_malloc for a heap created by mi_heap_new_in_arena. Run the provided v3 C reproducer using mi_manage_os_memory_ex and verify that all 272 allocations succeed and remain within mi_arena_contains for the registered arena.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100