runsc: significantly higher startup latency on ARM64 (Graviton) vs x86_64 at scale
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
## Description
We consistently observe significantly higher container startup latency with gVisor (runsc) on ARM64 (AWS Graviton) compared to x86_64 when running at scale. This has been reproduced across dozens of benchmark iterations on Kubernetes (EKS) using bare-metal instances to eliminate hypervisor influence.
## Environment
- **gVisor runtime**: runsc (RuntimeClass `gvisor`)
- **Kubernetes**: EKS (1.31)
- **Nodes**: Bare-metal instances (no Nitro hypervisor layer) to isolate runtime behavior
- ARM64: Graviton-based bare-metal
- x86_64: Intel/AMD bare-metal
- **Workload**: `pause` container (minimal — isolates sandbox startup overhead)
- **Pod management**: `podManagementPolicy: Parallel` (all replicas created simultaneously)
## Measurement
We measure post-schedule latency: **Ready condition timestamp minus PodScheduled condition timestamp**. This excludes any time waiting for node provisioning/scheduling and captures only the time from pod bind to container readiness — i.e., the sandbox + container startup cost.
## Results
### gVisor ARM vs x86 on bare-metal (post-schedule latency, seconds)
| Pods | ARM64 | x86_64 | Delta | ARM slower by |
|---:|---:|---:|---:|---:|
| 25 | 6.6 | 6.7 | -0.1 | -1.5% |
| 50 | 11.3 | 8.2 | +3.1 | +37.8% |
| 100 | 29.1 | 17.0 | +12.1 | +71.2% |
| 200 | 35.3 | 24.6 | +10.7 | +43.5% |
### runc ARM vs x86 baseline (no gVisor) — same nodes
| Pods | ARM64 | x86_64 | Delta | ARM slower by |
|---:|---:|---:|---:|---:|
| 25 | 5.9 | 9.4 | -3.5 | -37.2% |
| 50 | 7.4 | 6.0 | +1.4 | +23.3% |
| 100 | 9.9 | 19.1 | -9.2 | -48.2% |
| 200 | 40.0 | 31.0 | +9.0 | +29.0% |
### gVisor overhead over runc (bare-metal, same node type)
| Arch | Pods | runc | gvisor | Overhead |
|---|---:|---:|---:|---:|
| arm64 | 50 | 7.4s | 11.3s | +52.7% |
| arm64 | 100 | 9.9s | 29.1s | +193.9% |
| arm64 | 200 | 40.0s | 35.3s | -11.8% |
| x86_64 | 50 | 6.0s | 8.2s | +36.7% |
| x86_64 | 100 | 19.1s | 17.0s | -11.0% |
| x86_64 | 200 | 31.0s | 24.6s | -20.6% |
## Key Observations
1. **At 25 pods**, ARM and x86 are equivalent (~6.6s). The issue emerges at scale.
2. **At 50-100 pods**, gVisor on ARM is 37-71% slower than x86 — while runc does NOT show a consistent ARM penalty (it's mixed).
3. **gVisor overhead over runc on ARM at 100 pods is ~194%**, while on x86 gVisor is actually *faster* than runc (-11%).
4. This has been consistent across dozens of runs over multiple weeks. It's not a one-off measurement artifact.
5. Using bare-metal instances eliminates the Nitro hypervisor as a confounding variable.
## Questions
1. Are there known performance gaps in the ARM64 KVM/platform backend for runsc under concurrent pod startup?
2. Is there something in the sandbox creation path (memory mapping, seccomp setup, etc.) that scales differently on ARM64?
3. Are there any tuning knobs or configuration options that might improve ARM64 performance at scale?
## Reproduction
Benchmark code: scales a StatefulSet with `podManagementPolicy: Parallel` to N replicas using a `pause` image, then measures per-pod `Ready - PodScheduled` timestamps from Kubernetes pod conditions.
Happy to provide more detailed logs, profiles, or adjust the benchmark methodology if helpful.
Contributor guide
Assessment
This issue has not been assessed yet.