checkpoint/restore: epoll_wait never reports a restored listening socket that poll(2) and a fresh epoll registration both report readable
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
### Description
```
After checkpoint/restore, a listening TCP socket that is genuinely readable is
never reported to an epoll instance that was registered *before* the checkpoint.
The application therefore never calls accept(4) again: connections queue up in the
kernel, the port completes TCP handshakes, and no request is ever served.
Measured from inside the restored sandbox, at the same instant, on the same fd:
| probe | result |
|--------------------------------------------------------------|---------------------------|
| poll(2) on the listening fd | POLLIN (readable) |
| NEW epoll instance, same fd, EPOLLIN\|EPOLLET | reports ready (n=1) |
| NEW epoll instance, same fd, EPOLLIN (level) | reports ready (n=1) |
| epoll_wait(timeout=0) on the process's OWN pre-checkpoint epoll fds | n=0, indefinitely |
Sustained for 100+ seconds, until the container was torn down. Sample line from
the instrumented run (fd 14 is the listener; ep3/ep5 are the process's own epoll
instances, discovered via /proc/self/fd):
WEDGE t=263.6s | http9501(fd=14) rc=1 revents=0x1 errno=0
freshepoll[ET(add=0 n=1 ev=0x1) LT(add=0 n=1 ev=0x1)]
STEAL[ep3(n=0) ep5(n=0)]
accept_polls=4 accept_ok=2 task_ticks=1319 workers=2 alive_tasks=10
Pre-restore control from the same run: revents=0x0 and both fresh registrations
correctly report n=0 while nothing is queued — so the probe discriminates.
The application is a Rust binary using tokio 1.52.2 / mio 1.2.0 (mio registers
every fd EPOLLET). Its runtime stays healthy throughout: timers keep firing, tasks
keep being polled, channel wakeups work; only the accept future is never polled
again, because its waker is never invoked. A *freshly created* tokio runtime inside
the same restored sandbox (via `runsc exec`) accepts and serves normally, and a
sibling listener on a dedicated thread using blocking accept(2) keeps serving
through every occurrence. So this is specific to epoll interests that existed
before the checkpoint.
Reading the source, the pieces look like this:
* pkg/sentry/vfs/save_restore.go — epollInterest.afterLoad() does a ONE-SHOT
re-arm: it calls waiter.NotifyEvent so "the next call to
EpollInstance.ReadEvents() rechecks their readiness".
* pkg/sentry/vfs/epoll.go — EpollInstance.ReadEvents returns nil when ep.ready is
empty and never re-polls the registered set (notification-driven).
* pkg/tcpip/transport/tcp/accept.go (~:719) — deliverAccepted pushes onto
acceptQueue then calls e.waiterQueue.Notify(waiter.ReadableEvents).
Our hypothesis: the one-shot afterLoad re-arm is consumed by the runtime's first
post-restore epoll_wait, which happens *before* any connection arrives, so the
interest is legitimately found not-ready and dropped from ep.ready. From then on
readiness depends on the deliverAccepted -> waiterQueue.Notify path reaching the
restored interest — and empirically it never does. Because the connection then
sits in the accept queue forever, the socket stays permanently readable and no
further edge is ever generated.
That is where we would appreciate maintainer input: is the restored waiter.Entry
still linked into the endpoint's waiterQueue after restore? waiter.Queue's `list`
field is saved (only the mutex is state:"nosave") and
pkg/tcpip/transport/tcp/endpoint_state.go never rebuilds waiterQueue, so on paper
the linkage should survive — but the measurements say notifications do not arrive.
Adjacent observation, possibly the same root cause and easier to reason about: no
restore path in endpoint_state.go calls Notify at all. A listener whose accept
queue was NON-empty at checkpoint time would therefore come back readable with no
notification ever emitted. We did not hit that case (our accept queue was empty at
save time), but it looks like the same gap from the other direction.
Related issues (searched the tracker before filing):
* #13554 "TCP listening socket stops accepting after checkpoint/restore under
connection load" — CLOSED, fixed by f82637998 ("netstack: rebuild dispatcher
queue on restore instead of persisting it"), first released in
release-20260706.0. That report is the LOAD case, and the reporter notes: "If
you don't hammer the listening socket during checkpoint then we can access the
listening socket just fine on restore."
**Ours is the complementary case and is not fixed by that change.** We
checkpoint while the listener is IDLE with an EMPTY accept queue (poll(2)
reports revents=0x0 immediately before the checkpoint, and in one instrumented
run the listener had never accepted a single connection: accept_ok=0), and we
still reproduce on release-20260706.0 and release-20260721.0 — both of which
contain f82637998 — as well as on releases predating it (20260622 and older,
back to 20241118).
* #8926 "save/restore: test epoll bug" — CLOSED 2023, the regression test for
b/280313827 (epoll_wait *blocked* on a listening socket at save time). Our case
differs: nothing is blocked in epoll_wait at checkpoint; the interest is simply
registered and idle.
* #13404 / #13599 "hostinet: support checkpoint/restore" — CLOSED. Not applicable:
we run netstack (--network=sandbox), not hostinet.
Impact: any application using an epoll-based async runtime loses its listeners
after restore. Frequency in our setup is ~87% per restore (7/8 twice), i.e. not a
rare race. Workaround we shipped: move the accept loop to a dedicated thread using
blocking accept(2), which bypasses epoll for the listen path entirely; sockets
accepted after restore get fresh registrations and behave normally.
Things we ruled out, in case it saves you time:
* Not resource pressure. Measured the container cgroup across a wedge: zero
reclaim (pgscan/pgsteal/pgmajfault/workingset_refault all 0), peak 17% of the
memory limit, and CPU usage at 3.7% of a 10% quota — it was not starved.
* Not the application's async runtime. A fresh tokio runtime in the same sandbox
works; only pre-checkpoint registrations are affected.
* Not prior accept history. Reproduced both with accept_ok=0 (listener had never
accepted anything before the checkpoint) and accept_ok=2.
* Not the filesystem. Reproduced with a FUSE-backed overlay rootfs; a bare-runsc
A/B isolating the FUSE rootfs and an overlay-generation rotation was clean 0/5
in both arms.
* Not a recent regression. Reproduced on release-20241118.0 through
release-20260721.0, spanning both listener-restore strategies (the older
bind()+Listen(backlog) re-create and the current in-place
setEndpointState(StateListen) + fresh listenCtx).
Caveat we want to be upfront about: we could NOT minimize this. Standalone C and
Rust reproducers covering epoll-on-listener across restore, edge- vs
level-triggered registration, empty vs pending accept queue at checkpoint, an
eventfd in the same epoll set, and restore into a fresh netns with a new IP all
PASS. It only reproduces in our full runtime. Happy to run any instrumented
experiment you suggest, or to test a patch.
[guest-p72.log](https://github.com/user-attachments/files/30546262/guest-p72.log)
[epoll-probe.log](https://github.com/user-attachments/files/30546264/epoll-probe.log)
[steal-probe.log](https://github.com/user-attachments/files/30546279/steal-probe.log)
### Steps to reproduce
```
We have no minimal reproducer (see the caveat above); this is what we do, and how
to observe it.
Setup:
1. Start a container whose PID 1 tree includes a process listening on a TCP port
via an epoll-based async runtime (ours: Rust, tokio 1.52.2 / mio 1.2.0, which
registers fds EPOLLET). A second listener on a dedicated thread using blocking
accept(2) is a useful control — it keeps working.
runsc flags: --overlay2=none --allow-live-tcp-migration=false --network=sandbox
2. Health-probe the epoll-driven port until it answers, then leave it idle a few
seconds so the accept queue is empty at checkpoint time.
3. runsc checkpoint -image-path -compression flate-best-speed
4. runsc delete -force
5. Rewrite the OCI spec's network namespace to a FRESH netns with a different
guest IP (this mirrors production: restore always lands in a new slot).
6. runsc restore -detach -bundle -image-path
7. From the host, connect to the epoll-driven port every 100ms starting
immediately after restore returns.
Observed: connect() succeeds (TCP handshake completes) but no response is ever
produced. The blocking-accept control port on the same sandbox answers normally.
~87% of restores in our environment.
How to confirm it is the notification and not the app, from inside the guest:
* poll(2) the listening fd -> POLLIN
* register the same fd in a BRAND-NEW epoll instance, EPOLLET and level, with
epoll_wait(timeout=0) -> both report it ready immediately
* enumerate the process's own epoll fds (/proc/self/fd -> anon_inode:[eventpoll])
and epoll_wait(timeout=0) on them -> n=0
* count polls of the accept future -> frozen, while an unrelated timer task's tick
counter keeps increasing (runtime is alive)
```
### runsc version
```shell
runsc version release-20260721.0
spec: 1.2.1
Also reproduced on: release-20241118.0, release-20260323.0, release-20260608.0,
release-20260622.0, release-20260706.0 (prebuilt binaries from
https://storage.googleapis.com/gvisor/releases/release//x86_64/runsc)
```
### docker version (if using docker)
```shell
n/a — runsc is invoked directly by our own runtime (no docker, no containerd
shim). Docker 29.1.3 is installed on the host but is not in the path under test.
```
### uname
Linux 6.8.0-136-generic #136-Ubuntu SMP PREEMPT_DYNAMIC Wed Jul 1 21:53:05 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux Ubuntu 24.04.3 LTS, 8 vCPU, 62 GiB RAM
### kubectl (if using Kubernetes)
```shell
n/a — not Kubernetes.
```
### repo state (if built from source)
Not built from source; prebuilt release binaries (see above). Source references in this report are against release-20260721.0 / commit 9f653e577.
### runsc debug logs (if available)
```shell
Available — the runs were made with --debug --debug-log --panic-log
--debug-to-user-log. The sentry reported a clean restore ("Network stack
restored", "VFS restored", "post restore done") with no errors, and
`runsc debug --stacks` on a wedged sandbox showed a healthy sentry (tasks in
ordinary timed futex/EpollWait parks, zero semacquire). Attaching the in-guest
instrumentation logs; can provide full sentry debug logs on request.
```
Contributor guide
Research direction
Start by tracing pkg/sentry/vfs/save_restore.go and pkg/sentry/vfs/epoll.go, then inspect the waiter notification path in pkg/tcpip/transport/tcp/accept.go and pkg/tcpip/transport/tcp/endpoint_state.go. Run the documented checkpoint/restore scenario and compare restored pre-checkpoint epoll interests with fresh registrations. Done means a restored listening socket reliably wakes its existing epoll waiter when it becomes readable, including the queued-connection case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- networking, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100