microsoft / microsoft/demikernel
[catmem] huge page support
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 1.2k
- Forks
- 151
- PR merge metrics
- No merged PRs in 30d
Description
Context
Huge pages offer benefits for some high resource use applications. Applications which transmit large amounts of data via shared memory may benefit from using huge pages for IPC. Per the Linux kernel documentation on huge pages, huge pages exist as files mounted under hugetlbfs filesystem. shm_open creates files on a tempfs usually under /dev/shm.
Windows also supports huge pages (called large pages); use of MapViewOfFile3 allows use of the MEM_LARGE_PAGES flag, as well as optional NUMA zone placement through ExtendedParameters.
Proposed Solution
Move the PAL away from usage of shm_create so that files can be put either on tempfs or hugetlbfs. Create a new abstraction for resolving shared memory file names on the appropriate device. Add one implementation for tempfs and one for hugetlbfs.
For Windows in conjunction with #965, add the ability to use MEM_LARGE_PAGES when mapping shared memory.
Alternative Solutions
Adding this feature is a pretty binary choice. Huge pages will likely only offer benefits for a small subset of the application space. This should be driven accordingly.
For implementation, the current PAL implementation could instead be preserved and a hugepage PAL could be added. This would result in some redundant code, but more stability for existing cases.
Edit
I missed a subtlety here: shm_open is the POSIX-compliant interface to shared memory, and huge pages don't seem to exist in the POSIX standard. In this case, preserving the existing PAL may serve value for POSIX-compliant non-Linux targets.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the PAL shared-memory implementation and its shm_open path, then review how shared-memory file names are resolved. Read the Linux hugetlbfs and Windows MapViewOfFile3 references, along with issue #965. Done means supporting tempfs and hugetlbfs selection and Windows large-page mapping without regressing existing shared-memory behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, rust
- Domain
- operating-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100