llvm / llvm/llvm-project

[Feature request] Detect presence of configured memory limits instead of reporting an obscure error

Open
#212,764 0 comments 0 reactions 0 assignees View on GitHub
compiler-rt:asan
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

If application changes its memory limit to anything less than 16 TiB as [per docs](https://releases.llvm.org/16.0.0/tools/clang/docs/AddressSanitizer.html#limitations) then another call to `__mmap()` from AddressSanitizer's internals leads to an obscure error that is really hard to figure out. An example of new thread being launched after a call to `set_rlimit()`:

```
==545814==ERROR: AddressSanitizer: out of memory: failed to allocate 0xe000 (57344) bytes of Create (error code: 12)
ERROR: Failed to mmap
```

It took me a lot of time to figure out the reason. Maybe a dedicated message on such error or at least a hint regarding AddressSanitizer requiring at least 16 TiB of virtual space would be helpful. So that others in my position would find the issue quicker. Otherwise it's completely baffling to see an attempt to allocate only `57344` bytes failing with **ENOMEM**.

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure by changing the memory limit with set_rlimit() and launching a new thread, then trace the AddressSanitizer path around its internal __mmap() failure. Done means a configured virtual-memory limit below the documented requirement produces a clear diagnostic or hint instead of the obscure small-allocation ENOMEM error.

Written by the indexing model from the issue text.

Assessment

Domain
compilers, operating-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.