Lots of warnings due to failing to allocate aligned OS memory

Nobody has claimed this yet.

Hi,

I'm using mimalloc indirectly, via the mold linker. This is mimalloc 2.0.6+ds-2+b1 from debian unstable. On a new linux kernel (v6.1-rc3-158-gee6050c8af96) nearly every use of mold results in a lot of warnings. Here's some of the output with MIMALLOC_VERBOSE=1 set:

```
mimalloc: process init: 0x7f8dacc9f800
mimalloc: debug level : 2
mimalloc: secure level: 0
mimalloc: using 2 numa regions
mimalloc: option 'show_errors': 1
mimalloc: option 'show_stats': 0
mimalloc: option 'eager_commit': 1
mimalloc: option 'deprecated_eager_region_commit': 0
mimalloc: option 'deprecated_reset_decommits': 0
mimalloc: option 'large_os_pages': 0
mimalloc: option 'reserve_huge_os_pages': 0
mimalloc: option 'reserve_huge_os_pages_at': -1
mimalloc: option 'reserve_os_memory': 0
mimalloc: option 'deprecated_segment_cache': 0
mimalloc: option 'page_reset': 0
mimalloc: option 'abandoned_page_decommit': 0
mimalloc: option 'deprecated_segment_reset': 0
mimalloc: option 'eager_commit_delay': 1
mimalloc: option 'decommit_delay': 25
mimalloc: option 'use_numa_nodes': 0
mimalloc: option 'limit_os_alloc': 0
mimalloc: option 'os_tag': 100
mimalloc: option 'max_errors': 16
mimalloc: option 'max_warnings': 16
mimalloc: option 'max_segment_reclaim': 8
mimalloc: option 'allow_decommit': 1
mimalloc: option 'segment_decommit_delay': 500
mimalloc: option 'decommit_extend_delay': 2
mimalloc: warning: thread 0x7f8da56006c0: unable to allocate aligned OS memory directly, fall back to over-allocation (67108864 bytes, address: 0x7f8d99e00000, alignment: 67108864, commit: 0)
mimalloc: warning: thread 0x7f8da4a006c0: unable to allocate aligned OS memory directly, fall back to over-allocation (67108864 bytes, address: 0x7f8d9e000000, alignment: 67108864, commit: 0)
mimalloc: warning: thread 0x7f8da68006c0: unable to allocate aligned OS memory directly, fall back to over-allocation (67108864 bytes, address: 0x7f8d9e000000, alignment: 67108864, commit: 0)
mimalloc: warning: thread 0x7f8da38006c0: unable to allocate aligned OS memory directly, fall back to over-allocation (67108864 bytes, address: 0x7f8d8fe00000, alignment: 67108864, commit: 0)
mimalloc: warning: thread 0x7f8da68006c0: mi_usable_size: pointer might not point to a valid heap region: 0x7f8d94020080
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: thread 0x7f8da68006c0: (yes, the previous pointer 0x7f8d94020080 was valid after all)
mimalloc: warning: thread 0x7f8da68006c0: mi_usable_size: pointer might not point to a valid heap region: 0x7f8d94020080
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: thread 0x7f8da56006c0: mi_usable_size: pointer might not point to a valid heap region: 0x7f8d8c020080
(this may still be a valid very large allocation (over 64MiB))
mimalloc: warning: thread 0x7f8da56006c0: (yes, the previous pointer 0x7f8d8c020080 was valid after all)
mimalloc: warning: thread 0x7f8da56006c0: mimalloc: warning: thread 0x7f8da68006c0: (yes, the previous pointer 0x7f8d94020080 was valid after all)
mi_usable_size: pointer might not point to a valid heap region: 0x7f8d8c020080
(this may still be a valid very large allocation (over 64MiB))
...
```

I created a tracepoint on probe_libmimalloc:mi_os_mem_alloc. The requests for the 64MB aligned allocation come from within mimalloc itself. Either below mi_reserve_os_memory() or mi_segments_page_alloc().

Looking further, I see that an MAP_FIXED mmap fails to return memory at that address:

```
ld 237496 [005] 67637.402262: syscalls:sys_enter_mmap: addr: 0x2006c000000, len: 0x04000000, prot: 0x00000000, flags: 0x00000022, fd: 0xffffffff, off: 0x00000000
..
ld 237496 [005] 67637.402281: syscalls:sys_exit_mmap: 0x7f022fe00000
```

As far as I can tell there are no conflicting mmap requests made be mimalloc.

My vague suspicion is that this is related to linux promoting one of the pages before / after to use transparent huge pages. The region after did get converted by THP:

```
20070000000-20071000000 rw-p 00000000 00:00 0
Size: 16384 kB
...
AnonHugePages: 14336 kB
````

That region was concurrently allocated by another thread that finished before the "failing" MAP_FIXED:
```
ld 237496 [005] 67637.402262: syscalls:sys_enter_mmap: addr: 0x2006c000000, len: 0x04000000, prot: 0x00000000, flags: 0x00000022, fd: 0xffffffff
, off: 0x00000000
ld 237495 [002] 67637.402262: syscalls:sys_enter_mmap: addr: 0x20070000000, len: 0x04000000, prot: 0x00000000, flags: 0x00000022, fd: 0xffffffff, off: 0x00000000
ld 237495 [002] 67637.402272: syscalls:sys_exit_mmap: 0x20070000000
ld 237496 [005] 67637.402281: syscalls:sys_exit_mmap: 0x7f022fe00000
```

It's possible that this is just a short term issue in linux and will get fixed. But it might also be better to not allocate neighboring segments concurrently.

It might be helpful to note the requested address in the warning. Could the "mi_usable_size: pointer might not point to a valid heap region" warnings be avoided in this case?

Regards,

Andres

Start with mi_os_mem_alloc, mi_reserve_os_memory(), and mi_segments_page_alloc(), then reproduce the aligned 64 MB mmap behavior on Linux with transparent huge pages enabled. Compare concurrent neighboring allocations and the requested address reported in warnings. Done means determining the cause and preventing misleading allocation or mi_usable_size warnings without regressing valid large allocations.

Written by the indexing model from the issue text.