apple / apple/foundationdb

Correctly utilize huge pages on systems that support them

Open
#909 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
16.7k
Forks
1.6k
Avg merge
1d 20h
Merged PRs (30d)
126

Description

See the corresponding [forum post](https://forums.foundationdb.org/t/continuous-out-of-memory-crashes-in-small-cluster-with-modest-read-only-workload/864).

FDB will request huge pages if they're supported by the system, but still sizes its mmap allocation to 128KiB. This creates a stranded 1920KiB per magazine allocation, which rapidly OOMs the process.

So we need to rethink our huge page support. Ideally, we could instead allocate 2MiB magazines if huge pages are enabled, or 128KiB magazines if huge pages are disabled. However, it's possible to (flakily!) fail to allocate huge pages, which would leave us with a mix of 2MB and 128KB magazines. I'm unsure what effect this would have.

Contributor guide

Open the contributing guide

Research direction

Start by locating FoundationDB’s huge-page and mmap allocation paths, then trace how magazine sizes are selected when huge pages are supported or unavailable. Compare the 2MiB and 128KiB allocation cases and determine the expected behavior when huge-page allocation fails. Done means supported systems avoid stranded memory without breaking fallback allocation.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
databases, distributed-systems, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.