google / google/gvisor

runsc: significantly higher startup latency on ARM64 (Graviton) vs x86_64 at scale

Open
#13,395 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

## Description

We consistently observe significantly higher container startup latency with gVisor (runsc) on ARM64 (AWS Graviton) compared to x86_64 when running at scale. This has been reproduced across dozens of benchmark iterations on Kubernetes (EKS) using bare-metal instances to eliminate hypervisor influence.

## Environment

- **gVisor runtime**: runsc (RuntimeClass `gvisor`)
- **Kubernetes**: EKS (1.31)
- **Nodes**: Bare-metal instances (no Nitro hypervisor layer) to isolate runtime behavior
- ARM64: Graviton-based bare-metal
- x86_64: Intel/AMD bare-metal
- **Workload**: `pause` container (minimal — isolates sandbox startup overhead)
- **Pod management**: `podManagementPolicy: Parallel` (all replicas created simultaneously)

## Measurement

We measure post-schedule latency: **Ready condition timestamp minus PodScheduled condition timestamp**. This excludes any time waiting for node provisioning/scheduling and captures only the time from pod bind to container readiness — i.e., the sandbox + container startup cost.

## Results

### gVisor ARM vs x86 on bare-metal (post-schedule latency, seconds)

| Pods | ARM64 | x86_64 | Delta | ARM slower by |
|---:|---:|---:|---:|---:|
| 25 | 6.6 | 6.7 | -0.1 | -1.5% |
| 50 | 11.3 | 8.2 | +3.1 | +37.8% |
| 100 | 29.1 | 17.0 | +12.1 | +71.2% |
| 200 | 35.3 | 24.6 | +10.7 | +43.5% |

### runc ARM vs x86 baseline (no gVisor) — same nodes

| Pods | ARM64 | x86_64 | Delta | ARM slower by |
|---:|---:|---:|---:|---:|
| 25 | 5.9 | 9.4 | -3.5 | -37.2% |
| 50 | 7.4 | 6.0 | +1.4 | +23.3% |
| 100 | 9.9 | 19.1 | -9.2 | -48.2% |
| 200 | 40.0 | 31.0 | +9.0 | +29.0% |

### gVisor overhead over runc (bare-metal, same node type)

| Arch | Pods | runc | gvisor | Overhead |
|---|---:|---:|---:|---:|
| arm64 | 50 | 7.4s | 11.3s | +52.7% |
| arm64 | 100 | 9.9s | 29.1s | +193.9% |
| arm64 | 200 | 40.0s | 35.3s | -11.8% |
| x86_64 | 50 | 6.0s | 8.2s | +36.7% |
| x86_64 | 100 | 19.1s | 17.0s | -11.0% |
| x86_64 | 200 | 31.0s | 24.6s | -20.6% |

## Key Observations

1. **At 25 pods**, ARM and x86 are equivalent (~6.6s). The issue emerges at scale.
2. **At 50-100 pods**, gVisor on ARM is 37-71% slower than x86 — while runc does NOT show a consistent ARM penalty (it's mixed).
3. **gVisor overhead over runc on ARM at 100 pods is ~194%**, while on x86 gVisor is actually *faster* than runc (-11%).
4. This has been consistent across dozens of runs over multiple weeks. It's not a one-off measurement artifact.
5. Using bare-metal instances eliminates the Nitro hypervisor as a confounding variable.

## Questions

1. Are there known performance gaps in the ARM64 KVM/platform backend for runsc under concurrent pod startup?
2. Is there something in the sandbox creation path (memory mapping, seccomp setup, etc.) that scales differently on ARM64?
3. Are there any tuning knobs or configuration options that might improve ARM64 performance at scale?

## Reproduction

Benchmark code: scales a StatefulSet with `podManagementPolicy: Parallel` to N replicas using a `pause` image, then measures per-pod `Ready - PodScheduled` timestamps from Kubernetes pod conditions.

Happy to provide more detailed logs, profiles, or adjust the benchmark methodology if helpful.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.