firecracker-microvm / firecracker-microvm/firecracker

[Feature Request] Block-level copy-on-write overlay with dirty bitmap tracking for fast VM cloning and reset

Open
#5,795 3 comments 0 reactions 1 assignee Claimed by @Manciukic View on GitHub
Status: Awaiting assignee Type: Enhancement
Dominant language
Rust
Stars
36.7k
Forks
2.6k
Avg merge
3d 3h
Merged PRs (30d)
58

Description

## Problem

When running multiple Firecracker VMs from the same base rootfs, three operations carry significant overhead:

1. **Cloning** — creating a new VM requires copying the entire root disk image. For a 400MB rootfs, that's ~200ms per clone, scaling linearly with VM count.
2. **Resetting** — returning a VM's disk to a clean state requires the same full disk copy.
3. **Snapshotting disk state** — Firecracker snapshots capture VM state and memory but not disk changes. Operators must manage disk state externally, typically by copying the full disk image.

## Proposal

Add an `Overlay` variant to the `FileEngine` enum that implements block-level copy-on-write inside the VMM:

- A **shared read-only base image** serves reads for unmodified blocks
- A **per-VM sparse overlay file** captures writes
- A **dirty bitmap** (one bit per 4KB block) tracks which blocks have been written and routes reads to the correct source
- A **delta file format** with CRC64 integrity captures only dirty blocks for efficient snapshot persistence

This sits at the same layer as the existing `Sync` and `Async` file engines — transparent to the guest, no changes to the virtio protocol, no guest-side components.

### API

```json
{
"drive_id": "rootfs",
"path_on_host": "/path/to/overlay.ext4",
"base_path": "/path/to/base-rootfs.ext4",
"is_root_device": true,
"io_engine": "Overlay"
}
```

On snapshot create, an optional `block_delta_dir` captures only dirty blocks:

```json
{
"snapshot_path": "/path/to/snap.bin",
"mem_file_path": "/path/to/snap.mem",
"block_delta_dir": "/path/to/deltas/"
}
```

On restore, the same `block_delta_dir` applies the delta to a fresh overlay — enabling cloning from a snapshot without copying the full disk.

## Benchmark Results

Bare metal (AMD Ryzen 9 7950X3D, NVMe, 128GB RAM). Guest: 2 vCPUs, 256MB RAM, 396MB rootfs.

### Clone

| Metric | Sync (current) | Overlay | Improvement |
|---|---:|---:|---|
| Clone disk cost | 205ms (full 396MB copy) | 0ms (560KB delta) | **no disk copy needed** |
| Total clone cost | 430ms | 223ms | **~2x faster** |
| Clone data size | 396MB | 560KB | **700x smaller** |

### Reset

| Method | Time |
|---|---:|
| Full disk copy (current) | 190ms |
| Overlay truncate + bitmap clear | 1-2ms |
| **Speedup** | **~100x** |

### Snapshot

Snapshot time is identical (~225ms) because it is dominated by the guest memory dump. The overlay advantage is that no separate disk copy is needed for cloning — the delta file (dirty blocks only) is produced as part of the snapshot.

## Implementation

I have a working implementation on my fork.
Happy to open a PR upstream if there is interest.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.