microsoft / microsoft/demikernel

[catmem] huge page support

Open
#966 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature
Dominant language
Rust
Stars
1.2k
Forks
151
PR merge metrics
No merged PRs in 30d

Description

Context

Huge pages offer benefits for some high resource use applications. Applications which transmit large amounts of data via shared memory may benefit from using huge pages for IPC. Per the Linux kernel documentation on huge pages, huge pages exist as files mounted under hugetlbfs filesystem. shm_open creates files on a tempfs usually under /dev/shm.

Windows also supports huge pages (called large pages); use of MapViewOfFile3 allows use of the MEM_LARGE_PAGES flag, as well as optional NUMA zone placement through ExtendedParameters.

Proposed Solution

Move the PAL away from usage of shm_create so that files can be put either on tempfs or hugetlbfs. Create a new abstraction for resolving shared memory file names on the appropriate device. Add one implementation for tempfs and one for hugetlbfs.

For Windows in conjunction with #965, add the ability to use MEM_LARGE_PAGES when mapping shared memory.

Alternative Solutions

Adding this feature is a pretty binary choice. Huge pages will likely only offer benefits for a small subset of the application space. This should be driven accordingly.

For implementation, the current PAL implementation could instead be preserved and a hugepage PAL could be added. This would result in some redundant code, but more stability for existing cases.

Edit

I missed a subtlety here: shm_open is the POSIX-compliant interface to shared memory, and huge pages don't seem to exist in the POSIX standard. In this case, preserving the existing PAL may serve value for POSIX-compliant non-Linux targets.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the PAL shared-memory implementation and its shm_open path, then review how shared-memory file names are resolved. Read the Linux hugetlbfs and Windows MapViewOfFile3 references, along with issue #965. Done means supporting tempfs and hugetlbfs selection and Windows large-page mapping without regressing existing shared-memory behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust
Domain
operating-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.