fdbased: eventfd created for each dispatcher is never closed
- Dominant language
- Go
- Stars
- 19.3k
- Forks
- 2k
- Avg merge
- 3d 5h
- Merged PRs (30d)
- 264
Description
### Description
Every `fdbased` endpoint creates an eventfd that is never closed when the endpoint is removed from the stack. In a process that creates and removes endpoints repeatedly, the leaked fds eventually hit `RLIMIT_NOFILE` and the next endpoint fails to construct:
```
createInboundDispatcher(...) = newReadVDispatcher(777, ...) = failed to create eventfd: too many open files
```
We ran into this using `pkg/tcpip` as a library. `ls -l /proc//fd` at that point showed one `anon_inode:[eventfd]` per endpoint ever created.
It looks like this is from `newReadVDispatcher` (and the recvmmsg / packet-mmap variants) calling `stopfd.New()`, which creates the eventfd. `StopFD.Stop()` writes to that eventfd, but nothing ever closes it:
- `stopfd.StopFD` has `New` and `Stop`, no `Close`
- `fdbased.endpoint.Close()` is `{}`. `stack.LinkEndpoint` documents `Close` as "called when the endpoint is removed from a stack", and `nic.remove` does call it after `Attach(nil)`, so it's the right place but it does nothing.
- The dispatchers' `release()` free the iovec buffers and close the processor manager but don't touch the embedded `StopFD`.
`xdp.endpoint` also creates a `StopFD` in `New` and has an empty `Close()`.
When looking, I found `sharedmem` owns an eventfd too and closes it in `Wait()` after its workers have exited. As far as I can tell fdbased and xdp are the only link endpoints that allocate their own fd and never release it. I assume this hasn't come up because runsc creates its endpoints once at boot and keeps them for the sandbox's lifetime, so it's one fd per sandbox. It only bites when endpoints are created and destroyed repeatedly in one process.
Also, in `RecvMMsg` and `PacketMMap` mode it's actually two eventfds per endpoint. `createInboundDispatcher` builds a `readVDispatcher` unconditionally and then replaces it for those modes. The first one is never stored in `e.inboundDispatchers`, so no teardown can reach it, and its processor manager has already spawned its goroutines. runsc uses `RecvMMsg` by default, so every sandbox carries an orphaned eventfd and a set of orphaned processor goroutines per link fd from boot.
### Steps to reproduce
Create and remove endpoints in a loop and count eventfds in `/proc/self/fd`:
```go
// Reproduces an eventfd leak in pkg/tcpip/link/fdbased: every endpoint
// created with fdbased.New and then removed from a stack leaves its
// dispatcher's stop eventfd open.
package main
import (
"fmt"
"os"
"strings"
"golang.org/x/sys/unix"
"gvisor.dev/gvisor/pkg/tcpip/link/fdbased"
"gvisor.dev/gvisor/pkg/tcpip/network/ipv4"
"gvisor.dev/gvisor/pkg/tcpip/stack"
)
func countEventfds() int {
entries, err := os.ReadDir("/proc/self/fd")
if err != nil {
panic(err)
}
n := 0
for _, e := range entries {
target, err := os.Readlink("/proc/self/fd/" + e.Name())
if err == nil && strings.Contains(target, "anon_inode:[eventfd]") {
n++
}
}
return n
}
func cycle(mode fdbased.PacketDispatchMode) {
fds, err := unix.Socketpair(unix.AF_UNIX, unix.SOCK_SEQPACKET, 0)
if err != nil {
panic(err)
}
ep, err := fdbased.New(&fdbased.Options{
FDs: []int{fds[0]},
MTU: 1500,
PacketDispatchMode: mode,
})
if err != nil {
panic(err)
}
s := stack.New(stack.Options{
NetworkProtocols: []stack.NetworkProtocolFactory{ipv4.NewProtocol},
})
if err := s.CreateNIC(1, ep); err != nil {
panic(err)
}
// RemoveNIC -> nic.remove(closeLinkEndpoint=true) -> Attach(nil) (stops and
// joins every dispatch goroutine) -> LinkEndpoint.Close().
if err := s.RemoveNIC(1); err != nil {
panic(err)
}
s.Close()
s.Wait()
// We own fds[0] per the fdbased.New contract; close both ends ourselves.
unix.Close(fds[0])
unix.Close(fds[1])
}
func main() {
const iterations = 100
for _, m := range []struct {
name string
mode fdbased.PacketDispatchMode
}{{"Readv", fdbased.Readv}, {"RecvMMsg", fdbased.RecvMMsg}} {
before := countEventfds()
for i := 0; i < iterations; i++ {
cycle(m.mode)
}
after := countEventfds()
fmt.Printf("%-8s eventfds before=%d after=%d leaked_per_endpoint=%.1f\n",
m.name, before, after, float64(after-before)/iterations)
}
}
```
Then run it:
```
go mod init eventfdleak
go get gvisor.dev/gvisor@go
go mod tidy
docker run --rm -v "$PWD":/src -w /src -v gomodcache:/go/pkg/mod golang:1.26.3 go run .
```
On the current branch, that results in:
```
Readv eventfds before=1 after=101 leaked_per_endpoint=1.0
RecvMMsg eventfds before=101 after=301 leaked_per_endpoint=2.0
```
Instead of the count going to 1 after each `RemoveNIC`
Contributor guide
Research direction
Start with newReadVDispatcher, createInboundDispatcher, and fdbased.endpoint.Close in pkg/tcpip/link/fdbased, then compare stopfd.StopFD with the sharedmem teardown path. Check the xdp endpoint's New and Close methods and run the provided endpoint-cycle reproduction in Readv and RecvMMsg modes. Done means RemoveNIC no longer leaves eventfds or dispatcher goroutines accumulating.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, linux
- Domain
- networking, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100