kvcache-ai / kvcache-ai/AgentENV
Analyze boot/resume latency
- Dominant language
- Rust
- Stars
- 3.5k
- Forks
- 309
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 66
Description
## Summary
A regular Ubuntu 24.04 snapshot resume, with the Firecracker pool and shared memory device warm, takes:
| Samples | Mean | p50 | Min | Max |
| ---: | ---: | ---: | ---: | ---: |
| 10 | 46.95 ms | 44.80 ms | 42.83 ms | 66.57 ms |
Instrumentation accounts for 46.26 ms, or 98.53% of the mean. This covers the common restore path used by snapshot-backed sandbox creation and paused sandbox resume. Fresh VM boot is excluded.
The benchmark measures `FirecrackerSandbox::resume_from_snapshot_config`: constructing the runtime from an already-resolved snapshot config, restoring Firecracker, waiting for envd health, initializing envd, and returning a ready sandbox.
## Reproduction Environment
Hardware and host requirements:
- x86_64 host with `/dev/kvm` access
- Linux kernel with ublk support and `/dev/ublk-control` access
- Intel Xeon Gold 5317, 48 logical CPUs, 2 NUMA nodes
- 128 GiB RAM with swap disabled
- NVMe-backed ext4 storage
Software and workload:
- Linux `6.8.0-111-generic`
- Firecracker `1.15.1-patch-v1`
- Guest kernel `vmlinux-6.1.175`
- envd `0.5.15`
- Ubuntu 24.04
- 1 vCPU, 128 MiB, no attached drives
- Uncompressed memory snapshot with direct OverlayBD enabled
To reproduce the regular result, warm the Firecracker and network pools and keep one sandbox from the same snapshot alive so the shared memory ublk device remains referenced.
## Time Breakdown
| Stage | Mean | p50 | Share |
| --- | ---: | ---: | ---: |
| Firecracker pool acquire | 0.338 ms | 0.337 ms | 0.72% |
| Tools drive link | 0.065 ms | 0.062 ms | 0.14% |
| Rootfs OverlayBD bind | 2.945 ms | 2.876 ms | 6.27% |
| Network policy | 3.667 ms | 3.662 ms | 7.81% |
| Memory OverlayBD bind | 0.034 ms | 0.033 ms | 0.07% |
| Firecracker snapshot load | 9.579 ms | 7.631 ms | 20.40% |
| MMDS update | 0.347 ms | 0.337 ms | 0.74% |
| Firecracker resume | 0.563 ms | 0.527 ms | 1.20% |
| Guest/envd health | 22.381 ms | 22.564 ms | 47.67% |
| Release background downloads | 0.626 ms | 0.644 ms | 1.33% |
| envd initialization | 5.710 ms | 5.442 ms | 12.16% |
| Unattributed | 0.691 ms | - | 1.47% |
| **Total** | **46.948 ms** | **44.800 ms** | **100%** |
Before the vCPU resumes, setup consumes 17.54 ms. Guest health and envd initialization consume another 28.72 ms after the vCPU starts.
## Analysis
The 2.95 ms rootfs OverlayBD bind consists of approximately:
- 0.54 ms to materialize runtime configuration
- 1.86 ms to open the OverlayBD image
- 0.08 ms to bind the warm device
- 0.46 ms in main-process configuration and IPC
Memory binding costs only 0.034 ms because another live sandbox retains the shared memory ublk device.
`guest_ready` is the complete `GET /health` wait while the vCPU is running. It includes guest execution, demand paging, scheduling, networking, envd processing, and occasional retry delay. It is not 22 ms of host-side polling.
A separate trace measured approximately 2.3k KVM faults and 11.60 ms of cumulative `kvm_tdp_page_fault` handler time.`envd_init` is a subsequent `POST /init` call and costs another 5.71 ms.
## Follow-up Opportunities
1. Prefetch the recorded memory working set and prefault KVM mappings before starting the vCPU using KVM_PRE_FAULT_MEMORY.
2. Cache rootfs materialization and image-open work.
3. Avoid redundant network-policy updates.
Contributor guide
Research direction
Start at the `FirecrackerSandbox::resume_from_snapshot_config` benchmark entry point and reproduce the warm-pool resume measurements in the stated Linux environment. Trace the listed stages, especially guest health and envd initialization, then use the measured breakdown to select and validate one follow-up optimization with before-and-after latency results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, rust, ubuntu
- Domain
- infrastructure, operating-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100