firecracker-microvm / firecracker-microvm/firecracker
[Bug] vsock: 100% CPU main-loop livelock and hung connections after Resume with in-flight traffic (v1.16.x, EVENT_IDX)
- Dominant language
- Rust
- Stars
- 36.7k
- Forks
- 2.6k
- Avg merge
- 3d 3h
- Merged PRs (30d)
- 58
Description
## Describe the bug
After a `PATCH /vm {"state":"Paused"}` → `PATCH /vm {"state":"Resumed"}` cycle, if there was **in-flight bidirectional vsock traffic** at pause time, the vsock device wedges on resume:
1. The Firecracker main/event-loop thread spins at **100% CPU** in a tight `epoll_pwait` loop, and
2. All vsock connections stop making progress — existing connections hang and **new host-initiated connections are never accepted by the guest**.
The vCPUs run (guest is not paused) but block in `KVM_RUN` waiting for device interrupts that the wedged event loop never delivers, so the guest goes effectively dark (no serial output, no network).
This looks like the Pause/Resume-path analogue of #5969 / #5958, which fixed the equivalent hang **only for the snapshot LoadSnapshot path**. The Resume API path has no equivalent re-arm and additionally exhibits a busy-poll livelock. Both appear rooted in the vsock `VIRTIO_RING_F_EVENT_IDX` support added in #5872 (first shipped in v1.16.0).
## To Reproduce
Minimal shape (does not require snapshots):
1. Boot a microVM with a vsock device and a guest agent listening on a vsock port.
2. Establish steady bidirectional vsock traffic — e.g. a guest process writing to stdout streamed host→guest over one connection (host→guest RX active), while the host periodically sends requests (guest→host TX active). The key is that **both** an RX descriptor and a TX descriptor are in-flight/unconsumed at the instant of pause.
3. `PATCH /vm {"state":"Paused"}`.
4. `PATCH /vm {"state":"Resumed"}`.
5. From the host, open a **new** vsock connection to the guest agent (e.g. an `exec`-style control connection).
**Observed:** step 5's CONNECT is never accepted; the host side blocks indefinitely. Firecracker's main thread is pinned at 100% CPU. Existing connections also stop advancing.
It is intermittent across cycles — it triggers on whichever pause happens to catch a descriptor in-flight, so a workload with continuous traffic (a 1 Hz stdout writer plus periodic control RPCs) reproduces it reliably within one or two pause/resume cycles.
## Diagnostics captured on a wedged instance
Firecracker **v1.16.1**, `x86_64`, MMIO transport, two vCPUs. Captured live while the VM was hung.
**Main thread — 100% CPU, tight epoll loop.** `strace -c -p ` over 3 s:
```
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
99.86 0.491726 1 330917 epoll_pwait
0.14 0.000703 117 6 write
0.00 0.000008 2 3 read
```
~110k `epoll_pwait`/s, each returning **immediately**. Per-event trace shows a nested-epoll cycle that never clears:
```
epoll_pwait(30, [{events=EPOLLIN, data={u32=13, u64=13}}], 32, 0, ...) = 1 # inner eventpoll
epoll_pwait(9, [{events=EPOLLIN, data={u32=30, ...}}], 256, -1, ...) = 1 # outer -> inner
# repeats forever; fd 13 keeps reporting EPOLLIN and is never drained/deregistered
```
`fd 13` is an **accepted connection on the vsock backend Unix socket** (the UDS the muxer accepts host-side connections on) — i.e. pending host→guest data on a muxed vsock connection that the device muxer never consumes and never removes from the epoll set → level-triggered busy-poll.
**vCPU threads — idle, blocked in KVM_RUN.**
```
fc_vcpu 0: State S (sleeping), blocked in ioctl(KVM_RUN), ~0 CPU growth over 10s
fc_vcpu 1: State S (sleeping), blocked in ioctl(KVM_RUN), ~0 CPU growth over 10s
```
Guest is waiting for device interrupts that the livelocked event loop never injects (ping to guest dead, serial silent after resume).
**Firecracker warnings at every resume** (guest console / firecracker log):
```
[vm:fc_vcpu 0:WARN] Received a VcpuEvent::Resume message with immediate_exit enabled. immediate_exit was disabled before proceeding
[vm:fc_vcpu 1:WARN] Received a VcpuEvent::Resume message with immediate_exit enabled. immediate_exit was disabled before proceeding
[vm:main:WARN] Got a spurious notification from api thread
```
## Expected behaviour
After Resume, the vsock device should re-process any in-flight RX/TX descriptors and re-arm notifications (as the LoadSnapshot `kick()` path was fixed to do in #5958), so existing connections resume and new connections are accepted. The event loop must not busy-poll a level-triggered backend fd whose data cannot currently be delivered.
## Hypothesis (for maintainers)
Two stacked effects, both downstream of vsock EVENT_IDX (#5872):
1. **Notification state not re-armed across Resume.** With `VIRTIO_RING_F_EVENT_IDX` negotiated, a descriptor in-flight at pause leaves the used/avail-event indices in a state where, after resume, the guest's TX kick and/or the host's RX interrupt is suppressed and never retried — the same class of bug #5958 fixed for restore, but the Resume API path has no equivalent replay/re-arm hook.
2. **Muxer busy-poll under backpressure.** When host→guest RX cannot be delivered (guest not consuming because of (1)), the muxer leaves the connection's backend UDS fd registered with level-triggered `EPOLLIN` and never drains or deregisters it, pinning the main loop at 100% CPU and starving all other device servicing — which is why the whole VM (net, serial) goes dark, not just vsock.
The restore fix (#5958) replays the TX queue kick in the device's restore `kick()`. Resume appears to need the analogous re-arm, plus (or including) correct edge/level handling of the muxer connection fd so a non-drainable connection doesn't livelock the loop.
**Note on scope:** effect (2), the busy-poll, may be at least partly independent of EVENT_IDX — a level-triggered backend fd that is never drained or deregistered is a muxer event-handling issue that EVENT_IDX *exposes* (by leaving RX undeliverable) rather than *causes*. If so, disabling vsock EVENT_IDX would hide the livelock but not fix the underlying fd-handling. This is a reason to fix the re-arm + muxer handling rather than treat it purely as an EVENT_IDX regression.
## Environment
- **Firecracker:** v1.16.1 (release binary). Also expected on v1.16.0 (EVENT_IDX for vsock landed in #5872 → v1.16.0).
- **Host arch:** x86_64. vsock transport: MMIO. 2 vCPUs.
- **Setup:** Firecracker running inside a virtualized host (nested virtualization); the vsock device backs a containerd-shim (nerdbox) control channel to an in-guest agent. The bug is in Firecracker's vsock device event loop and is not specific to nesting.
- **Guest kernel / rootfs:** custom guest kernel + minimal initrd (details available on request; not believed relevant — the guest driver is stock virtio-vsock).
- **Not using snapshots** for this repro — plain PauseVM/ResumeVM API.
## Additional context
- Related: #5872 (adds `VIRTIO_RING_F_EVENT_IDX` to vsock, v1.16.0), #5958 / #5969 (restore-path vsock hang, fixed for LoadSnapshot only in v1.16.1).
- The 100% CPU main-loop symptom (distinct from the plain connection hang in #5969) may be the more actionable signal, since it's a deterministic busy-poll on a specific fd rather than a race — happy to capture additional traces (full per-event `epoll_pwait` dump, muxer state, queue indices) or provide a self-contained repro harness on request.
Contributor guide
Research direction
Start by reproducing the Pause/Resume failure with in-flight bidirectional vsock traffic and inspect the vsock device event loop, Resume path, and muxer handling of the backend Unix-socket fd. Compare the Resume behavior with the restore kick fix in #5958 and related issue #5969. Done means connections resume, new connections are accepted, descriptors are re-processed, and the main loop does not busy-poll.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100