firecracker-microvm / firecracker-microvm/firecracker

Allow MAP_SHARED for snapshot-restore mem_backend (cooperative-snapshot tooling)

Open
#5,912 1 comment 0 reactions 0 assignees View on GitHub
Status: Awaiting review
Dominant language
Rust
Stars
36.7k
Forks
2.6k
Avg merge
3d 2h
Merged PRs (30d)
53

Description

## Use case

I'm building [forkd](https://github.com/deeplethe/forkd), an open-source `fork()` primitive for microVMs aimed at AI agent fan-out. v0.4 wants a "live-fork" path where an external snapshot manager arms `UFFDIO_WRITEPROTECT` on the source VM's memory and captures dirty pages asynchronously — moving the memory write out of the BRANCH pause window. Empirically on kernel 6.14, this drops the BRANCH pause for a 1 GiB parent from ~150 ms (v0.3.4 floor on ext4) to ~3 ms (just the `UFFDIO_WRITEPROTECT` arm).

The blocker is that `MemBackendType::File` in `snapshot_load` mmaps the backing file with `MAP_PRIVATE` (see [`src/vmm/src/vstate/memory.rs::snapshot_file`](https://github.com/firecracker-microvm/firecracker/blob/main/src/vmm/src/vstate/memory.rs)). If forkd hands FC a `/proc//fd/` as `mem_backend.backend_path`, FC opens that path but the resulting mapping is `MAP_PRIVATE` — guest writes CoW to FC-private pages and never propagate back to forkd's mmap of the same memfd. The cooperative-snapshot design is impossible.

Empirically verified (forkd's [memfd-share spike](https://github.com/deeplethe/forkd/tree/main/experiments/v0.4-memfd-share-spike) — 30-line Python script). The `Uffd` backend isn't a substitute because it gives forkd the uffd fd but no read access to FC's memory (no shared mapping).

## Proposed minimal API

Opt-in `shared: bool` on `MemBackendConfig`, default `false` (unchanged behavior):

```rust
pub struct MemBackendConfig {
pub backend_path: PathBuf,
pub backend_type: MemBackendType,
/// If true and backend_type == File, mmap with MAP_SHARED.
/// Defaults to false; ignored for Uffd.
#[serde(default)]
pub shared: bool,
}
```

Implementation is ~39 lines across 4 files (MemBackendConfig + plumbing through `guest_memory_from_file` to `snapshot_file`). Build verified against `main` (commit `053f521d9`); full patch saved as a unified diff [here](https://github.com/deeplethe/forkd/blob/main/0001-feat-mem-backend-shared-option-for-MAP-SHARED.patch).

## Open API question for you

Three shapes I considered. I'd prefer your guidance before sending a PR:

1. **Opt-in field** (above). Smallest surface. My current preference.
2. **New backend type** `MemBackendType::SharedFile`. More explicit, doubles the match-arms.
3. **A `shared_mmap: bool` at the top-level `LoadSnapshotConfig`** instead of nested in `mem_backend`.

I'll send a PR in whichever shape you confirm works.

## Backing context

- Full RFC + use cases: [DESIGN-v0.4.md](https://github.com/deeplethe/forkd/blob/main/DESIGN-v0.4.md), [DESIGN-v0.4-USE-CASES.md](https://github.com/deeplethe/forkd/blob/main/DESIGN-v0.4-USE-CASES.md)
- Empirical PoC data (3 ms/GiB `UFFD_WP` arm, EPT-mediated guest writes do propagate to UFFD_WP on the host VMA, snapshot restore round-trip works): [`experiments/v0.4-*-poc/RESULTS.md`](https://github.com/deeplethe/forkd/tree/main/experiments)
- Why this proposal vs alternatives (FC fork, `VmstateOnly` snapshot type, `process_vm_readv` bypass): [DESIGN-v0.4-PHASE3-SPIKE.md](https://github.com/deeplethe/forkd/blob/main/DESIGN-v0.4-PHASE3-SPIKE.md)
- Full proposal incl. compatibility analysis and tests plan: [FIRECRACKER-UPSTREAM-PROPOSAL.md](https://github.com/deeplethe/forkd/blob/main/FIRECRACKER-UPSTREAM-PROPOSAL.md)

forkd is Apache 2.0, 720+ stars, real production-aimed project. Happy to test whatever shape lands, and to maintain the patch in forkd's repo as a user-applied diff until landed if that helps.

Thanks for reading.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.