microsoft / microsoft/mimalloc

Arena allocator pre-maturely returns NOMEM on parent arena while space is still available in child arena

Open
#1,403 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C
Stars
13.4k
Forks
1.2k
Avg merge
4d 45m
Merged PRs (30d)
13

Description

Looks like when registering a very large arena, mimalloc internally split it into smaller sub-arenas (16 GiB each) with parent-children relationship. Then if a series of allocation exhausted the parent arena, the mimalloc API would pre-maturely return NOMEM even though there are still space in the child arenas. This looks like a bug in the macro to iterate over the sub-arenas in the arena.c code.

Repro (version: v3)
```c
#include

#include
#include
#include

#define GIB ((size_t)1024 * 1024 * 1024)
#define MIB ((size_t)1024 * 1024)

int main(void) {
const size_t arena_size = 24 * GIB;
const size_t allocation_size = 64 * MIB;
const size_t allocation_count = 272; // 17 GiB: requires a child arena.
const size_t alignment = mi_arena_min_alignment();
const size_t mapping_size = arena_size + alignment;

void* mapping = mmap(NULL, mapping_size, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_NORESERVE, -1, 0);
if (mapping == MAP_FAILED) {
perror("mmap");
return 1;
}

const uintptr_t aligned =
((uintptr_t)mapping + alignment - 1) & ~((uintptr_t)alignment - 1);
mi_arena_id_t arena_id = NULL;
if (!mi_manage_os_memory_ex((void*)aligned, arena_size,
true, /* committed */
false, /* pinned */
true, /* initially zero */
-1, /* no NUMA preference */
true, /* exclusive */
&arena_id)) {
fprintf(stderr, "failed to register the 24 GiB arena\n");
return 1;
}

mi_heap_t* heap = mi_heap_new_in_arena(arena_id);
if (heap == NULL) {
fprintf(stderr, "failed to create the arena heap\n");
return 1;
}

for (size_t i = 0; i < allocation_count; i++) {
void* p = mi_heap_malloc(heap, allocation_size);
if (p == NULL) {
fprintf(stderr,
"allocation %zu failed after %zu MiB; child arena was not used\n",
i, i * allocation_size / MIB);
return 1;
}
if (!mi_arena_contains(arena_id, p)) {
fprintf(stderr, "allocation %zu came from outside the requested arena\n", i);
return 1;
}
}

printf("allocated %zu MiB from the parent arena and its children\n",
allocation_count * allocation_size / MIB);
return 0;
}

```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in arena.c, focusing on the macro that iterates over parent and child sub-arenas, and trace the allocation path from mi_heap_malloc for a heap created by mi_heap_new_in_arena. Run the provided v3 C reproducer using mi_manage_os_memory_ex and verify that all 272 allocations succeed and remain within mi_arena_contains for the registered arena.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
operating-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.