[RFC] Node-local shared read-only memory pages for Firecracker sandboxes

Type: Feature request

Summary

Add a new memory backend that allows Firecracker sandboxes on the same node to share immutable 2 MiB HugeTLB baseline pages.

The backend remains page-lazy. Creating a VM builds virtual mappings and a page plan, but does not download the full memory snapshot or instantiate every physical page. On the first access to an eligible source page, the Orchestrator loads one 2 MiB page from the existing immutable Build artifact into a sparse hugetlbfs backing file. Concurrent faults for the same source are coalesced, and subsequent VMs map the same physical HugeTLB page.

Each VM maps shared source ranges with MAP_PRIVATE. Reads continue to share the file-backed HugeTLB page. A guest write triggers kernel COW and creates a VM-private page without changing the shared baseline.

The MVP shares pages only when the existing snapshot header proves that the whole guest 2 MiB page comes from the same immutable Build and aligned Build offset. It does not perform content-based deduplication.

Motivation

E2B currently restores memory lazily with userfaultfd. When a sandbox faults a page, the Orchestrator reads the snapshot data and uses UFFDIO_COPY to populate anonymous private guest memory.

This avoids eagerly loading the full snapshot, but identical pages are still copied into every VM:

Current backend

Build page X
    -> UFFDIO_COPY -> VM A private 2 MiB page
    -> UFFDIO_COPY -> VM B private 2 MiB page
    -> UFFDIO_COPY -> VM C private 2 MiB page

Many sandboxes created from the same Build read the same runtime, code, library, and initialized heap pages without modifying them. Those pages already have an immutable source identity in the snapshot header.

The proposed backend changes the resident layout to:

Build page X
    -> one hugetlbfs 2 MiB physical page
         -> MAP_PRIVATE -> VM A
         -> MAP_PRIVATE -> VM B
         -> MAP_PRIVATE -> VM C

If VM B writes the page:

shared baseline page X
    -> VM A keeps sharing X
    -> VM C keeps sharing X
    -> VM B gets one private COW page

Goals

• Share one physical 2 MiB HugeTLB page across same-node VMs that reference the same immutable source page.
• Preserve page-level lazy loading.
• Preserve guest-visible read, write, discard, pause, and resume semantics.
• Reuse the current snapshot header and Build artifact format.
• Coalesce concurrent loads of the same source page.
• Fall back to the existing UFFD backend before VM execution when the shared backend is unavailable or ineligible.
• Keep the current per-Sandbox pause consistency boundary.
• Expose enough metrics to prove physical memory savings and identify regressions.

Non-goals for the MVP

• Cross-node page sharing.
• Content-hash or byte-content deduplication.
• Sharing part of a composite 2 MiB page at 4 KiB granularity.
• Host eviction or swap of VM-private pages.
• Retaining zero-reference shared pages as a warm cache.
• Shared-memory prefetch.
• Placement affinity, quota, billing, overcommit, or capacity-policy changes.
• Live migration between the current and shared backends.
• Recovery of shared cache metadata across an Orchestrator restart.
• OverlayBD integration.
• New cross-tenant policy or accounting behavior.

Source identity

An eligible shared page is identified by its immutable source, not by snapshot ID and not by page content:

SourceKey = {
    BuildId,
    BuildChecksum,
    BuildStorageOffset,
    PageSize,
}

PageSize is 2 MiB in the MVP.

The per-Build backing object is identified by:

BuildKey = {
    BuildId,
    BuildChecksum,
    PageSize,
}

The logical offset in the backing file is the existing BuildStorageOffset.

Including the Build checksum prevents the same Build ID from aliasing different immutable data. The MVP does not calculate a per-page hash because doing so requires reading the page before determining a cache hit, which defeats the main lazy-load benefit.

Different snapshots can share a page when their headers contain the same SourceKey. Snapshot identity itself is not part of the cache key.

Eligibility

A VM may use the shared backend only when all of the following are true before Firecracker starts:

• The snapshot header parses and validates successfully.
• IncompletePendingUpload is false.
• Every referenced Build has a non-zero checksum.
• All mappings pass length, bounds, alignment, and overflow checks.
• The node capability probe succeeds.
• The node-level feature flag is enabled for this create or resume operation.

If any condition fails, the whole VM uses the existing backend. A running VM does not mix the current and shared backends.

An eligible VM may still contain individual pages that cannot be physically shared. Those pages are handled privately by the new backend.

Page planning

At VM creation, the Orchestrator converts the header mapping into one PagePlan entry per guest 2 MiB page. This step reads metadata only.

SharedSource

A page is shareable when:

• The full guest 2 MiB page is continuously covered by one Build.
• The Build has a non-zero checksum.
• BuildStorageOffset is 2 MiB aligned.
• The full source range is within the Build size.
• The page contains no zero fragment or fragment from another Build.

SharedSource {
    GuestOffset,
    BuildKey,
    BuildStorageOffset,
    Length = 2 MiB,
}

Composite

A page is composite when any of the following is true:

• Its 4 KiB slices reference multiple Builds.
• Only part of the page is zero.
• The source offset is not 2 MiB aligned.
• The source is not one complete continuous 2 MiB range.

On first access, the backend assembles all slices into one VM-private HugeTLB page. A clean resident composite page is called LocalClean because it is physically private but still logically equal to its header baseline.

Composite pages can occur when 4 KiB memfile diff dedup is enabled. For example, a child Build may contain one changed 4 KiB slice while the other 511 slices still reference its parent Build.

Zero

A complete zero 2 MiB page may use a node-local shared read-only zero HugeTLB page. A write creates a private COW page. Partial zero ranges remain part of a private Composite page.

Proposed architecture

                           E2B Orchestrator
                    +---------------------------+
snapshot header --->| PagePlan                  |
                    | SharedMemoryBackend       |
UFFD events ------->| SharedMemoryManager       |
                    | Build Reader              |
                    +-------------+-------------+
                                  |
                                  | first cold SourceKey load
                                  v
object storage/local cache -> 2 MiB staging buffer
                                  |
                                  | one host memory copy
                                  v
                    sparse hugetlbfs Build backing
                                  |
                         MAP_PRIVATE mappings
                         /                  \
                  Firecracker A       Firecracker B

The proposed components are:

• PagePlan: immutable per-VM mapping from guest page to SharedSource, Composite, or Zero.
• SharedMemoryManager: process-wide source index, load singleflight, references, sparse Build backings, and reclamation.
• SharedMemoryBackend: implementation of the existing Orchestrator memory backend lifecycle.
• A new Firecracker memory backend: mixed file-backed and anonymous HugeTLB mappings plus UFFD registration and FD handoff.

VMA, backing-page, and PTE ownership

The design has two independent states:

1. Whether a SourceKey has a Ready physical backing page. This is tracked by the Orchestrator.
2. Whether a particular Firecracker process has a PTE for that page. This is managed by Linux.

For example:

SourceKey X: Ready

VM A: PTE installed -> X
VM B: no PTE yet
VM C: no PTE yet

Firecracker creates VMAs during VM setup:

Firecracker HVA range
    -> MAP_PRIVATE
    -> hugetlbfs Build FD + BuildStorageOffset

Firecracker also registers those HVA ranges with userfaultfd and passes the UFFD to the Orchestrator. A VMA records the virtual file mapping but does not require a resident physical page or PTE.

On a guest access, KVM resolves the guest physical address to the Firecracker HVA. If the HVA has no PTE, Linux emits a UFFD event containing the fault address and blocks the faulting vCPU thread.

The Orchestrator never creates or returns a PTE. It prepares page content or confirms that a backing page is Ready, then calls the appropriate UFFD ioctl. Linux locates the target process page table through the UFFD context, installs the PTE for the supplied HVA range, and wakes the blocked vCPU thread. There is no separate Firecracker API notification on the fault path.

Shared source index

The process-wide index is runtime metadata, not a new persistent database:

BuildKey
    -> sparse hugetlbfs backing FD
    -> BuildStorageOffset
         -> PageEntry {
              Absent | Loading | Ready | Backoff | Releasing
              generation
              pageRefs
              waiters
            }

The header remains the persistent source mapping. The runtime index is rebuilt after an Orchestrator restart.

Suggested logical types:

type SharedMemoryManager struct {
    buildShards []BuildShard
    loadSlots   chan struct{}
}

type BuildBacking struct {
    key        BuildKey
    fd         int
    size       int64
    handleRefs int64
    pages      map[int64]*PageEntry
}

type PageEntry struct {
    state      PageCacheState
    generation uint64
    pageRefs   int64
    waiters    map[WaiterID]*Waiter
    retryAt    time.Time
    lastErr    error
}

type PageLease struct {
    key        SourceKey
    generation uint64
    vmID       string
}

Page leases must be idempotent and generation-aware so that a delayed release from an old page instance cannot decrement the reference count of a reloaded page after Ready -> Releasing -> Absent -> Loading -> Ready.

Storage reads, decompression, UFFD ioctls, and hole punching must not run while holding a global or shard map lock.

Shared cache state machine

                     first demand                 load succeeds
    +------------- Absent -----------------> Loading -----------------> Ready
    |                    ^                      |                          |
    |                    |                      | load failure             | last actual ref released
    |                    | retry expires        v                          v
    |                    +------------------ Backoff                   Releasing
    |                                                                    |
    +--------------------------- punch hole succeeds --------------------+

• Absent: the source offset has no resident HugeTLB page.
• Loading: one loader owns the source load; concurrent requests wait.
• Ready: a complete immutable 2 MiB page is available.
• Backoff: a bounded retry delay after load failure.
• Releasing: new acquires are serialized against hole punching.

Concurrent faults for the same SourceKey use singleflight. The loader belongs to the cache entry, not to the first VM. If that VM exits while other waiters remain, the load continues.

First-load path

The MVP uses a staging buffer:

PageEntry: Absent -> Loading

Build Reader
    -> read/decompress the complete uncompressed 2 MiB page
    -> bounded ordinary-memory staging buffer
    -> allocate/populate the hugetlbfs page
    -> copy staging buffer to hugetlbfs once
    -> publish PageEntry as Ready
    -> release the staging buffer

The page must not become Ready until all 2 MiB are present. Failed reads or decompression discard the staging buffer and leave the backing offset absent.

This adds one 2 MiB host memory copy per cold SourceKey, not per VM. All later VMs map the Ready shared page without another content copy. A bounded buffer pool and load semaphore cap temporary memory at approximately:

maximum concurrent loaders * 2 MiB

UFFD fault paths

Ready shared source

When the hugetlbfs page already exists but a VM has no PTE, Linux can report a minor fault. The Orchestrator acquires a page lease and resolves it with UFFDIO_CONTINUE or UFFDIO_CONTINUE_MODE_WP. Linux installs a new PTE in that Firecracker process pointing to the existing file-backed HugeTLB page.

PTEs are not shared between VMs. The physical file-backed page is shared:

VM A PTE ----+
             +---> one hugetlbfs physical page
VM B PTE ----+

First access to an absent shared source

This path is a Phase 0 implementation gate.

Linux distinguishes:

missing fault -> UFFDIO_COPY or UFFDIO_ZEROPAGE
minor fault   -> UFFDIO_CONTINUE for an existing page-cache page

The source offset is still a hugetlbfs hole when the first fault occurs. After the Orchestrator loads the staging buffer and populates the backing page through a separate writable mapping, the implementation must prove the correct transition to a minor resolution. Candidate behaviors to test are:

• Populate the backing and issue UFFDIO_CONTINUE if the target kernel accepts the now-existing page-cache page.
• Populate the backing, wake/retry the original fault, receive a minor fault, and then issue UFFDIO_CONTINUE.
• Use another kernel-supported file-backed missing-page resolution that preserves the shared page identity.

The implementation must not blindly UFFDIO_COPY into a VM's MAP_PRIVATE destination if that creates a VM-private page and defeats sharing.

Composite page

The Orchestrator assembles the full baseline in private memory and resolves the missing fault with UFFDIO_COPY or an equivalent private-page operation. The result is LocalClean on a read, or PrivateDirty after a write.

Zero page

A full zero page can map the shared zero backing and use the same minor/continue flow. A private zero-page fallback may use UFFDIO_ZEROPAGE if necessary.

Read and write behavior

Read

Absent SharedSource
    -> load or wait for Ready
    -> install shared mapping
    -> SharedClean

First access is a write

The MVP deliberately uses the shared-first path:

Absent SharedSource
    -> establish the complete shared baseline
    -> resume the guest store
    -> MAP_PRIVATE COW
    -> PrivateDirty

The baseline cannot be skipped because a CPU store usually modifies only part of the 2 MiB page; all untouched bytes must retain their original values.

Write to SharedClean

Linux performs MAP_PRIVATE COW. The writer receives a private 2 MiB page, while other VMs retain the immutable shared page.

Write to LocalClean

The page is already physically private, so it is modified in place and becomes PrivateDirty. No second private page is required.

VM page states

Absent
    No resident PTE and no VM-private physical page.

SharedClean
    Resident and physically shared; logically equal to the immutable baseline.

LocalClean
    Resident and physically VM-private; logically equal to a Composite baseline.

PrivateDirty
    VM-private and guest-modified; must be persisted at pause.

Zero
    Logically all zero; may be absent or mapped to the shared zero page.

LocalClean should initially be derived from PagePlan, residency, dirty, and zero state rather than persisted as a new snapshot bitmap.

References and reclamation

The manager maintains two independent references.

Build handle reference

Held while a live VM PagePlan references a Build. It keeps the sparse backing inode/FD available but does not instantiate physical pages.

Page reference

Held by an installed SharedClean mapping or an in-flight waiter. It pins one resident 2 MiB shared page.

A VM whose header may reference a page, but which has not accessed it, holds no page reference.

VM A reads X: page ref = 1
VM B has only a dormant VMA: page ref remains 1
VM A exits: page ref = 0
manager punches a hole for X
VM B later reads X: X is loaded again

The MVP does not retain Ready pages with zero actual references. On the last release, it immediately punches a hole and returns the HugeTLB page. A future bounded warm cache can be added behind explicit HugeTLB watermarks.

The Build backing is deleted only when Build handle references, page references, waiters, and background work all reach zero.

Dirty tracking and conservative releases

The MVP keeps the current asynchronous UFFD write-protect dirty tracking. It does not add a synchronous userspace round trip on every first write.

After kernel COW, the Orchestrator may not immediately know that a VM no longer maps the shared baseline. It therefore retains that VM's shared page lease conservatively until one of:

• explicit discard,
• pause-time dirty classification, or
• VM exit.

This can retain a shared baseline longer than strictly necessary, but it does not block the guest write path and remains correct.

Discard semantics

An explicit guest discard permanently abandons the previous contents:

SharedClean  -> Zero
LocalClean   -> Zero
PrivateDirty -> Zero

After discard, a read returns zero, not the old baseline. A write starts from a zero baseline. A SharedClean discard releases the VM's page lease; a PrivateDirty discard can release the private physical page immediately.

This is different from future host-driven private-page eviction, which would have to preserve the data in recoverable storage and load it again on a later read.

Pause and resume

The existing per-Sandbox stable point remains:

1. Stop the VM's vCPUs.
2. Prevent new faults, COW bookkeeping, and discard processing for that VM.
3. Drain that VM's in-flight page operations.
4. Classify each page as Absent, SharedClean, LocalClean, PrivateDirty, or Zero.
5. Keep private page bytes immutable during export.
6. Persist only PrivateDirty bytes and Zero metadata.
7. Close the VM and release its page and Build references.

Persistence behavior:

State	Write bytes to the new Build	Resulting header
Absent	No; do not load it	Preserve original source
SharedClean	No	Preserve original source
LocalClean	No	Preserve original Composite source
PrivateDirty	Yes	Map to the new Build
Zero	No page bytes	Update Zero metadata

Resume builds a new PagePlan from the resulting header. Any complete aligned source page can be shared again, including pages written into the new Build by an earlier pause.

Failure handling

Before the VM runs

Capability, header, PagePlan, hugetlbfs, or Firecracker handshake failures fall back to the existing backend for the whole VM.

Runtime source load failure

• Retry with bounded attempts and backoff.
• Wake all current singleflight waiters with the same result.
• After exhaustion, fail only the affected waiting sandbox or sandboxes.
• Return the entry to Absent after backoff so future requests can retry.
• Never publish a partially populated page as Ready.

The MVP does not dynamically migrate a running shared VM to the current backend.

Orchestrator restart

The shared cache is process-local and is not recovered. Existing startup cleanup terminates orphan Firecracker processes; kernel mappings, backing FDs, and HugeTLB pages are then reclaimed.

Backpressure

Use two bounded limits:

• A node-global load semaphore.
• A per-Sandbox concurrent load cap.

Singleflight waiters for the same SourceKey do not consume additional load slots. Exact limits, retries, and timeouts should be configured from benchmark results rather than fixed in the RFC.

Required kernel and Firecracker prototype

Before implementing the production backend, an executable capability probe must validate the exact target kernel and Firecracker fork. Checking only kernel version or feature bits is insufficient.

The probe must verify:

• Sparse 2 MiB hugetlbfs backing without eager resident allocation.
• Reservation behavior and whether MAP_NORESERVE is required.
• Two MAP_PRIVATE mappings share one file-backed HugeTLB page on read.
• A write creates a private COW page without changing the other mapping.
• UFFD_FEATURE_MISSING_HUGETLBFS and UFFD_FEATURE_MINOR_HUGETLBFS.
• The first absent-source missing-to-minor resolution path.
• UFFDIO_CONTINUE and required write-protect behavior.
• Async write-protect and current pagemap dirty classification.
• Remove/unmap event behavior.
• Hole punching with dormant VMAs and no installed PTEs.
• Safe serialization between new acquires and hole punching.
• Mixed file-backed and anonymous HugeTLB runs in one guest HVA range.
• KVM memslot behavior and practical VMA-count limits.

If any required behavior is unsupported, the node reports shared_memory_supported=false and uses the current backend.

References:

• Linux userfaultfd documentation
• Firecracker snapshot support

Firecracker contract

The E2B Firecracker fork needs a distinct shared-memory backend rather than overloading the current File or UFFD configuration.

The contract needs to describe:

• Total guest memory size and HugeTLB page size.
• Build backing FDs.
• Guest HVA/offset runs mapped to backing FD/offset runs.
• Composite/private and Zero runs.
• Required UFFD modes.
• A protocol version and capability bitmap.

Prefer a bidirectional Unix socket and SCM_RIGHTS for FDs and UFFD transfer. This avoids jail-visible backing paths and path replacement races. Batch descriptors if the FD count exceeds one message's practical limit.

Firecracker should reserve the guest HVA range and map long contiguous runs where possible:

Guest HVA

| shared Build A | private Composite | shared Build B | Zero |
     MAP_PRIVATE       anonymous          MAP_PRIVATE
     file-backed       HugeTLB            file-backed

The final API schema change must update packages/shared/pkg/fc/firecracker.yml and generated clients, with an explicit compatibility failure before guest execution.

Implementation breakdown

Phase 0: feasibility prototype

• Build a standalone hugetlbfs + UFFD + KVM/Firecracker probe.
• Verify shared PFNs/HugeTLB accounting across two MAP_PRIVATE mappings.
• Verify private COW isolation.
• Resolve the first missing-to-minor fault sequence.
• Verify hole punch, dormant VMA, refault, and reload behavior.
• Verify async WP dirty detection after COW.
• Define the Firecracker FD and mapping protocol.
• Record unsupported capability reasons as stable enums.

Orchestrator: header and PagePlan

• Add whole-VM eligibility validation.
• Convert 4 KiB header mappings into 2 MiB PagePlan entries.
• Classify SharedSource, Composite, and Zero pages.
• Validate Build boundaries, alignment, checksum, and integer overflow.
• Add unit tests for multi-Build and partial-zero pages.

Orchestrator: SharedMemoryManager

• Add sharded BuildKey and SourceKey indexes.
• Create sparse hugetlbfs backing files per immutable Build.
• Add independent Build handle references and page leases.
• Add generation-safe, idempotent lease release.
• Implement Absent/Loading/Ready/Backoff/Releasing transitions.
• Implement per-SourceKey singleflight and waiter cancellation.
• Add bounded 2 MiB staging-buffer pool.
• Integrate the existing Build reader and frame decompression.
• Publish Ready only after complete staging-to-backing copy.
• Serialize new acquires against hole punching.
• Delete unused Build backings after all work and references drain.

Orchestrator: SharedMemoryBackend

• Implement the existing MemoryBackend lifecycle or introduce an explicit backend-specific descriptor where Memfd semantics do not apply.
• Route UFFD events through PagePlan.
• Implement Ready shared-source minor resolution.
• Implement the prototype-selected first-load resolution.
• Implement Composite assembly and private-page installation.
• Implement full Zero page behavior.
• Track per-VM installed leases and in-flight operations.
• Add node-global and per-Sandbox load limits.
• Disable or ignore shared-mode prefetch in the MVP.

Suggested package layout:

packages/orchestrator/pkg/sandbox/uffd/
    shared_backend.go
    shared_manager.go
    shared_page_plan.go
    shared_fault.go
    shared_probe_linux.go

Firecracker fork

• Add a versioned shared-memory backend configuration.
• Receive backing and UFFD FDs through SCM_RIGHTS.
• Reserve one contiguous guest HVA range.
• Install coalesced file-backed and anonymous HugeTLB submappings.
• Register missing, minor, write-protect, remove, and unmap behavior required by the selected prototype.
• Keep shared backing immutable from the guest through MAP_PRIVATE.
• Fail the handshake before KVM_RUN on partial or incompatible configuration.

Pause, resume, and lifecycle

• Drain only the pausing VM's in-flight shared-memory operations.
• Derive terminal page classifications at the current stable point.
• Persist only PrivateDirty bytes and Zero metadata.
• Preserve original sources for Absent, SharedClean, and LocalClean pages.
• Release shared leases only after mappings/in-flight installs can no longer use them.
• Release all Build handles on pause/exit/failure cleanup.
• Preserve existing checkpoint pause-and-resume behavior.

Feature flag and observability

• Add a node-level feature flag affecting only new creates/resumes.
• Add stable fallback-reason metrics.
• Add node physical Ready/Loading page and HugeTLB-byte metrics.
• Add hit, miss, reload, waiter, load latency, error, and punch-hole metrics.
• Add VM logical SharedClean, LocalClean, PrivateDirty, Zero, and Absent metrics.
• Add shareability coverage: shareable_hugepages / total_hugepages.
• Ensure shared physical bytes are counted once node-wide, not once per VM reference.

Testing and benchmarking

• Unit-test PagePlan, key identity, state transitions, references, ABA protection, cancellation, and pause classification.
• Verify N readers of one SourceKey consume one 2 MiB shared physical page.
• Verify each writer adds one private 2 MiB page and cannot modify other VMs.
• Verify first access as a partial write preserves untouched baseline bytes.
• Verify Composite LocalClean -> PrivateDirty without a second private copy.
• Verify discard always reads back as zero.
• Verify last-reference punch-hole followed by another VM's lazy reload.
• Verify the first faulting VM can exit while other waiters still complete.
• Verify load failures never expose half-populated data.
• Stress concurrent fault, COW, discard, pause, exit, acquire, and release.
• Run Go race tests for manager/backend lifecycle code.
• Compare cold fault P50/P95/P99 with the current UFFD backend.
• Measure N-VM HugeTLB usage, object-store reads, pause latency, and COW cost.

Acceptance criteria

• N VMs reading one SourceKey add one shared 2 MiB physical HugeTLB page, not N pages.
• Each VM writing the page receives an isolated private page; all other VMs retain the baseline.
• A dormant VMA does not pin a physical page. The last actual page-reference release returns it, and a later access reloads it correctly.
• Pause does not load Absent pages and persists only PrivateDirty bytes plus Zero metadata.
• Ineligible VMs and unsupported nodes fall back before guest execution.
• A runtime source-load failure does not corrupt other Ready entries or permanently poison the key.
• Disabling the feature flag sends new VMs to the current backend while existing shared VMs drain normally.

Rollout

1. Land and run the standalone capability prototype.
2. Implement behind a default-off node flag in development.
3. Validate correctness and benchmark against the current UFFD backend.
4. Enable on a small node canary.
5. Roll back by disabling the flag for new VMs; allow existing shared VMs to drain.

The MVP does not change placement. Build-aware placement can be evaluated separately after measuring real cache coverage and savings.

Alternatives considered

Eagerly materialize the full baseline

Simpler faults, but higher startup latency, network traffic, and resident memory for untouched pages.

Keep using anonymous UFFDIO_COPY

Preserves the current implementation but produces one physical copy per VM.

Use the standard Firecracker File backend

Useful for one complete immutable memory file, but E2B headers can reference multiple Builds, zero ranges, and Composite pages. Materializing a complete per-snapshot file would weaken source-level sharing and lazy loading.

OverlayBD

OverlayBD can provide remote block-layer lookup and caching, but it does not make multiple Firecracker PTEs alias the same final HugeTLB page. It would add a TCMU service, another cache lifecycle, and another index while the shared-page index would still be required.

Per-page content hashing

Requires reading the page before determining identity and introduces a global content-index lifecycle. The MVP uses immutable header identity instead.

Keep zero-reference pages warm

Could avoid reloads, but requires a HugeTLB budget, watermarks, eviction policy, and demand-fault priority. The MVP releases immediately.

Risks and open implementation questions

• Exact missing-to-minor behavior after a hugetlbfs hole is populated in the target kernel.
• Whether MAP_NORESERVE and the selected mapping layout provide the intended non-resident allocation behavior.
• hugetlbfs COW, UFFD WP async, and pagemap dirty semantics in the deployed kernel.
• Hole punching behavior with dormant VMAs, installed PTEs, and recently COWed mappings.
• KVM memslot and VMA scalability for mixed mappings.
• Final Firecracker protocol shape and FD batching.
• Default load concurrency, retry, timeout, and backoff values.
• Performance thresholds for canary rollout.

The first five items are production implementation gates. If the target environment cannot provide reliable behavior, the feature must remain disabled rather than weaken memory correctness or VM isolation.

No repository files or tests are named. Start by locating the existing Orchestrator memory backend and userfaultfd restore path, then trace the backend lifecycle, snapshot-header planning, and Firecracker memory setup. Done means the shared backend preserves the stated VM semantics, falls back when ineligible, coalesces page loads, and exposes the requested memory metrics.