Correctly utilize huge pages on systems that support them
- Dominant language
- C++
- Stars
- 16.7k
- Forks
- 1.6k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 126
Description
See the corresponding [forum post](https://forums.foundationdb.org/t/continuous-out-of-memory-crashes-in-small-cluster-with-modest-read-only-workload/864).
FDB will request huge pages if they're supported by the system, but still sizes its mmap allocation to 128KiB. This creates a stranded 1920KiB per magazine allocation, which rapidly OOMs the process.
So we need to rethink our huge page support. Ideally, we could instead allocate 2MiB magazines if huge pages are enabled, or 128KiB magazines if huge pages are disabled. However, it's possible to (flakily!) fail to allocate huge pages, which would leave us with a mix of 2MB and 128KB magazines. I'm unsure what effect this would have.
Contributor guide
Research direction
Start by locating FoundationDB’s huge-page and mmap allocation paths, then trace how magazine sizes are selected when huge pages are supported or unavailable. Compare the 2MiB and 128KiB allocation cases and determine the expected behavior when huge-page allocation fails. Done means supported systems avoid stranded memory without breaking fallback allocation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- databases, distributed-systems, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100