kvcache-ai / kvcache-ai/AgentENV

Analyze boot/resume latency

Open
#125 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
3.5k
Forks
309
Avg merge
1d 9h
Merged PRs (30d)
66

Description

## Summary

A regular Ubuntu 24.04 snapshot resume, with the Firecracker pool and shared memory device warm, takes:

| Samples | Mean | p50 | Min | Max |
| ---: | ---: | ---: | ---: | ---: |
| 10 | 46.95 ms | 44.80 ms | 42.83 ms | 66.57 ms |

Instrumentation accounts for 46.26 ms, or 98.53% of the mean. This covers the common restore path used by snapshot-backed sandbox creation and paused sandbox resume. Fresh VM boot is excluded.

The benchmark measures `FirecrackerSandbox::resume_from_snapshot_config`: constructing the runtime from an already-resolved snapshot config, restoring Firecracker, waiting for envd health, initializing envd, and returning a ready sandbox.

## Reproduction Environment

Hardware and host requirements:

- x86_64 host with `/dev/kvm` access
- Linux kernel with ublk support and `/dev/ublk-control` access
- Intel Xeon Gold 5317, 48 logical CPUs, 2 NUMA nodes
- 128 GiB RAM with swap disabled
- NVMe-backed ext4 storage

Software and workload:

- Linux `6.8.0-111-generic`
- Firecracker `1.15.1-patch-v1`
- Guest kernel `vmlinux-6.1.175`
- envd `0.5.15`
- Ubuntu 24.04
- 1 vCPU, 128 MiB, no attached drives
- Uncompressed memory snapshot with direct OverlayBD enabled

To reproduce the regular result, warm the Firecracker and network pools and keep one sandbox from the same snapshot alive so the shared memory ublk device remains referenced.

## Time Breakdown

| Stage | Mean | p50 | Share |
| --- | ---: | ---: | ---: |
| Firecracker pool acquire | 0.338 ms | 0.337 ms | 0.72% |
| Tools drive link | 0.065 ms | 0.062 ms | 0.14% |
| Rootfs OverlayBD bind | 2.945 ms | 2.876 ms | 6.27% |
| Network policy | 3.667 ms | 3.662 ms | 7.81% |
| Memory OverlayBD bind | 0.034 ms | 0.033 ms | 0.07% |
| Firecracker snapshot load | 9.579 ms | 7.631 ms | 20.40% |
| MMDS update | 0.347 ms | 0.337 ms | 0.74% |
| Firecracker resume | 0.563 ms | 0.527 ms | 1.20% |
| Guest/envd health | 22.381 ms | 22.564 ms | 47.67% |
| Release background downloads | 0.626 ms | 0.644 ms | 1.33% |
| envd initialization | 5.710 ms | 5.442 ms | 12.16% |
| Unattributed | 0.691 ms | - | 1.47% |
| **Total** | **46.948 ms** | **44.800 ms** | **100%** |

Before the vCPU resumes, setup consumes 17.54 ms. Guest health and envd initialization consume another 28.72 ms after the vCPU starts.

## Analysis

The 2.95 ms rootfs OverlayBD bind consists of approximately:

- 0.54 ms to materialize runtime configuration
- 1.86 ms to open the OverlayBD image
- 0.08 ms to bind the warm device
- 0.46 ms in main-process configuration and IPC

Memory binding costs only 0.034 ms because another live sandbox retains the shared memory ublk device.

`guest_ready` is the complete `GET /health` wait while the vCPU is running. It includes guest execution, demand paging, scheduling, networking, envd processing, and occasional retry delay. It is not 22 ms of host-side polling.

A separate trace measured approximately 2.3k KVM faults and 11.60 ms of cumulative `kvm_tdp_page_fault` handler time.`envd_init` is a subsequent `POST /init` call and costs another 5.71 ms.

## Follow-up Opportunities

1. Prefetch the recorded memory working set and prefault KVM mappings before starting the vCPU using KVM_PRE_FAULT_MEMORY.
2. Cache rootfs materialization and image-open work.
3. Avoid redundant network-policy updates.

Contributor guide

Open the contributing guide

Research direction

Start at the `FirecrackerSandbox::resume_from_snapshot_config` benchmark entry point and reproduce the warm-pool resume measurements in the stated Linux environment. Trace the listed stages, especially guest health and envd initialization, then use the measured breakdown to select and validate one follow-up optimization with before-and-after latency results.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, rust, ubuntu
Domain
infrastructure, operating-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.