restore: --background cannot be requested through the shim
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
`runsc restore --background` is implemented and works, but there is no way for an
orchestrated deployment to ask for it. The flag can only be set by an out-of-tree
extension compiled into the shim.
On master (fd439eb27c) the plumbing is complete:
- `runsc restore --background` and the async page loading behind it
(`runsc/cmd/restore.go:89`, `pkg/sentry/pgalloc/save_restore.go`).
- `extension.RestoreConfig.Background` (`pkg/shim/v1/extension/extension.go:47`).
- `runsccmd.RestoreOpts.Background` emits the flag (`pkg/shim/v1/runsccmd/runsc.go:282`).
- `proc/init.go:249` forwards it, `runsc/container.go:417` and
`runsc/service.go:483` carry it.
What is missing is a caller. `RestoreConfig` arrives on `extension.RestoreRequest`,
a plain Go struct on the in-process `extension.TaskServiceExt` interface, and
`Restore` is not exposed on any wire protocol:
- containerd's Task v2 ttrpc service has no `Restore` method.
- gVisor's own `pkg/shim/v1/taskserver/task_server.proto` exposes `Checkpoint`,
`Wait`, `State` and `Version`, and no `Restore`.
So only something registered through `extension.NewExtension` /
`NewPodExtension` and compiled into the shim binary can set it. Under stock
containerd or Kubernetes, background page loading is unreachable.
I realise shim restore support is still being worked out (#3275, #11810, and
PR #13326 proposing an annotation-driven trigger). The ask is small: whatever
trigger lands, please let it carry the restore options too, not just the image
path. #13326 already carries `dev.gvisor.internal.restore.direct` alongside
`host-image-path`, so a sibling for `background` would fit naturally there. I am
happy to send that patch if the annotation route is the one you want; I did not
open a PR because the answer determines its shape and that is not my call.
## Why it is worth reaching
Measured on kind (arm64, systrap, 1 GiB working set, checkpoint evicted from the
page cache, pod cgroup read-throttled to stand in for real sandbox-node storage,
n=3). "restore blocks" is the sentry's own timeline from `Restore` start to
`MFs loaded` (eager) or `MF metadata loaded` (background):
100 MB/s eager 10856 / 10851 / 10803 ms background 566 / 535 / 476 ms
200 MB/s eager 5316 / 5346 / 5274 ms background 667 / 267 / 263 ms
unthrottled (1.8-5 GB/s) eager 419 / 347 ms background 449 / 434 ms
Time to fully warm also drops, since loading overlaps with the application
running: 0.5s + 8.0s sweep against 10.9s + 0.45s.
The unthrottled row is the caveat, and I think it is the interesting part.
Async page loading already runs concurrently with the rest of restore, and under
the shim there is an 80-400 ms window where the sandbox container has been
restored and containerd has not yet asked for the workload container. Any page
load that fits inside that window is free either way, and `--background` is
marginally behind. It only pays when the checkpoint read is slower than that gap,
which is the object-store and network-storage case rather than a warm local one.
## A question about the failure mode
With background loading a read error arrives after restore has already reported
success. Truncating pages.img mid-load, after the pod reported Running and ready:
W0826 04:41:05.503040 save_restore.go:1944] Async page loading failed: read for MemoryFile(0x661742788008) pages [[0xb810000, 0xb850000)] (total 262144 bytes) returned 0 bytes, error: EOF
W0826 04:41:05.800440 restore.go:630] Killing the sandbox after MemoryFile page loading failed: I/O error
That seems right for `runsc restore` run by hand, where the operator sees the
error. Under an orchestrator it means a sandbox that was declared healthy, and
may already be serving traffic, disappears — where an eager restore would have
failed the restore call and left the caller free to fall back to a cold start.
Is killing the sandbox the intended contract, or should a background-load failure
be recoverable? This matters more if the flag becomes reachable from Kubernetes.
Three adjacent behaviours I measured, none of which look wrong, recorded in case
they are useful:
- Deleting the checkpoint directory mid-load is harmless: the sentry holds the
fd, loading completes, no restarts.
- A checkpoint taken during the load is correct — restoring from that second
image reproduces the same memory checksum — but blocks behind the load:
`4 waiters waited 9.87s for 882917376 bytes`.
- Demand faults are prioritised but not free. At 100 MB/s a page 10% into the
working set cost 133us (the loader had already streamed past it), one at 90%
cost 999ms, against 134us on an eager restore.
## Environment
- gVisor `release-20260817.0` plus PR #13326, systrap, arm64
- kind v0.31 (Kubernetes v1.35, containerd 2.3.1) on Docker Desktop
- checkpoint written by `runsc checkpoint --leave-running`, uncompressed
(`CompressionLevelDefault` is none), so it has the separate pages file that
`--background` requires
Contributor guide
Research direction
Start with the restore plumbing in runsc/cmd/restore.go, pkg/shim/v1/extension/extension.go, pkg/shim/v1/runsccmd/runsc.go, and the trigger proposed in PR #13326. Determine how an orchestrator should pass the background option and define the behavior for asynchronous page-loading failures, then add coverage for the selected trigger and failure contract.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100