google / google/gvisor

kvm: recvmsg fail to receive fds in large buffer during haproxy reload

Open
#14,497 0 comments 0 reactions 0 assignees View on GitHub
type: bug
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

### Description

A clean haproxy config demonstrates after sending SIGUSR2 to it to trigger reload (and initiate listener transfer from old to new worker) that:
- haproxy logs warning about failing to receive listeners (expected 1, got 0)
- some recvmsg via the created socketpair worked (new worker knows 1 is expected)
- but the ultimate one fails (0 received, with errno=14 "bad address")

Running with stracing, this is visible on the syscall level.

> Side-effect: the stale listeners that got sent are never read, but never GC-d either, but the old worker exited. So netstack now has two listeners, one taking but never accepting/responding SYN packets, until the listener backlog fills up (seems somaxconn capped to 1024) and then the SYNs get dropped (as reflected by netstack prom metric via export-metrics). From outside it seems SYNs are "randomly" dropped, depending on which listener did they hit (due to haproxy's SO_REUSEPORT)

### Steps to reproduce

Self-contained haproxy.cfg with docker command to run it (and then other to send signal):

```
# docker run -d --name hap --runtime=runsc-kvm -v $PWD:/usr/local/etc/haproxy:ro haproxy:3.2.22
# then reload (once or twice), until you see error:
#
# docker kill -s USR2 hap
# docker logs hap
#
# Normal state (one listener):
#
# docker exec hap sh -c 'cat /proc/net/tcp' | awk '$4=="0A"'
# 17: 00000000:1F90 00000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 18 1 0000000000000000 0 0 0 0 -1
#
# Bad state (dup listener):
#
# docker exec hap sh -c 'cat /proc/net/tcp' | awk '$4=="0A"'
# 62: 00000000:1F90 00000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 63 1 0000000000000000 0 0 0 0 -1
# 17: 00000000:1F90 00000000:0000 0A 00000000:00000000 00:00000000 00000000 99 0 18 1 0000000000000000 0 0 0 0 -1

global
maxconn 4096

defaults
mode http
timeout connect 1s
timeout client 10s
timeout server 10s

frontend fe
bind *:8080
default_backend be

backend be
server s1 192.0.2.1:8000
```

Workaround: use systrace; or don't use signal, rather do full restart

### runsc version

```shell
runsc version release-20260817.0
spec: 1.2.1
```

### docker version (if using docker)

```shell
Client:
Version: 29.1.3
API version: 1.52
Go version: go1.24.4
Git commit: 29.1.3-0ubuntu3~22.04.2
Built: Wed Apr 29 22:18:59 2026
OS/Arch: linux/amd64
Context: default

Server:
Engine:
Version: 29.1.3
API version: 1.52 (minimum version 1.44)
Go version: go1.24.4
Git commit: 29.1.3-0ubuntu3~22.04.2
Built: Wed Apr 29 22:18:59 2026
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 2.2.1
GitCommit:
runc:
Version: 1.3.3-0ubuntu1~22.04.3
GitCommit:
docker-init:
Version: 0.19.0
GitCommit:
```

### uname

Linux 5.15.0-187-generic #197-Ubuntu SMP Fri Jul 17 19:17:01 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux

### kubectl (if using Kubernetes)

```shell

```

### repo state (if built from source)

_No response_

### runsc debug logs (if available)

```shell
.boot via "--debug", "--strace", "--strace-syscalls=socketpair,sendmsg,recvmsg,execve,listen,close"

0830 00:30:31.967576 1 strace.go:561] [ 20: 20] haproxy E close(0x4 socket:[35])
I0830 00:30:31.967597 1 strace.go:599] [ 20: 20] haproxy X close(0x4 socket:[35]) = 0 (0x0) (4.84µs)
I0830 00:30:31.967610 1 strace.go:561] [ 20: 20] haproxy E close(0x6 socket:[36])
I0830 00:30:31.967644 1 strace.go:599] [ 20: 20] haproxy X close(0x6 socket:[36]) = 0 (0x0) (4.043µs)
I0830 00:30:31.967662 1 strace.go:561] [ 20: 20] haproxy E close(0x4 socket:[37])
I0830 00:30:31.967678 1 strace.go:599] [ 20: 20] haproxy X close(0x4 socket:[37]) = 0 (0x0) (2.785µs)
I0830 00:30:31.967891 1 strace.go:561] [ 20: 20] haproxy E close(0x4 /usr/lib/os-release)
I0830 00:30:31.967907 1 strace.go:599] [ 20: 20] haproxy X close(0x4 /usr/lib/os-release) = 0 (0x0) (3.057µs)
I0830 00:30:31.967947 1 strace.go:561] [ 20: 20] haproxy E close(0x4 /sys/devices/virtual/dmi/id/product_name)
I0830 00:30:31.967964 1 strace.go:599] [ 20: 20] haproxy X close(0x4 /sys/devices/virtual/dmi/id/product_name) = 0 (0x0) (2.032µs)
I0830 00:30:31.968018 1 strace.go:561] [ 20: 20] haproxy E close(0x4 /proc/cpuinfo)
I0830 00:30:31.968033 1 strace.go:599] [ 20: 20] haproxy X close(0x4 /proc/cpuinfo) = 0 (0x0) (2.146µs)
I0830 00:30:31.974503 1 strace.go:570] [ 20: 20] haproxy E socketpair(AF_UNIX, SOCK_STREAM|0x0, 0x0, 0x7fdd63b227a0)
I0830 00:30:31.974567 1 strace.go:608] [ 20: 20] haproxy X socketpair(AF_UNIX, SOCK_STREAM|0x0, 0x0, 0x7fdd63b227a0) = 0 (0x0) (25.292µs)
I0830 00:30:31.974603 1 strace.go:567] [ 20: 20] haproxy E sendmsg(0x3 socket:[6], 0x7fdd63b22690 {name=0x0, namelen=0, iovecs=0x7fdd63b22660 {base=0x7fdd63b2265e, len=2, "\x00\x00"}, control={0x7fdd63b22670 {level=SOL_SOCKET, type=SCM_RIGHTS, length=20, content: 6}}, flags=0}, 0x0)
I0830 00:30:31.974637 1 strace.go:605] [ 20: 20] haproxy X sendmsg(0x3 socket:[6], 0x7fdd63b22690 {name=0x0, namelen=0, iovecs=0x7fdd63b22660 {base=0x7fdd63b2265e, len=2}, control=0x7fdd63b22670, control_len=24, flags=0}, 0x0) = 2 (0x2) (13.441µs)
I0830 00:30:31.974668 1 strace.go:561] [ 20: 20] haproxy E close(0x6 socket:[38])
I0830 00:30:31.974685 1 strace.go:599] [ 20: 20] haproxy X close(0x6 socket:[38]) = 0 (0x0) (2.334µs)
I0830 00:30:31.974717 1 strace.go:567] [ 20: 20] haproxy E recvmsg(0x7 socket:[39], 0x7fdd63b22760 {name=0x0, namelen=0, iovecs=0x7fdd63b22740 {base=0x7fdd63b22730, len=4}, control=0x0, control_len=0, flags=0}, 0x100)
I0830 00:30:31.974784 1 strace.go:567] [ 3: 3] haproxy E recvmsg(0x4 socket:[7], 0x7f231e94a6f0 {name=0x0, namelen=0, iovecs=0x7f231e94a6c0 {base=0x7f231e94a6ba, len=2}, control=0x7f231e94a6d0, control_len=24, flags=0}, 0x0)
I0830 00:30:31.974834 1 strace.go:605] [ 3: 3] haproxy X recvmsg(0x4 socket:[7], 0x7f231e94a6f0 {name=0x0, namelen=0, iovecs=0x7f231e94a6c0 {base=0x7f231e94a6ba, len=2, "\x00\x00"}, control={0x7f231e94a6d0 {level=SOL_SOCKET, type=SCM_RIGHTS, length=20, content: 54}}, flags=0}, 0x0) = 2 (0x2) (22.806µs)
I0830 00:30:31.974865 1 strace.go:567] [ 3: 3] haproxy E recvmsg(0x4 socket:[7], 0x7f231e94a6f0 {name=0x0, namelen=0, iovecs=0x7f231e94a6c0 {base=0x7f231e94a6ba, len=2}, control=0x7f231e94a6d0, control_len=24, flags=0}, 0x0)
I0830 00:30:31.974883 1 strace.go:605] [ 3: 3] haproxy X recvmsg(0x4 socket:[7], 0x7f231e94a6f0 {name=0x0, namelen=0, iovecs=0x7f231e94a6c0 {base=0x7f231e94a6ba, len=2}, control=0x7f231e94a6d0, control_len=24, flags=0}, 0x0) = -1 errno=11 (request would block) (2.839µs)
I0830 00:30:31.974983 1 strace.go:567] [ 3: 3] haproxy E sendmsg(0x36 socket:[38], 0x7f231e94a420 {name=0x0, namelen=0, iovecs=0x7f231e94a400 {base=0x7f231e94a3fc, len=4, "\x01\x00\x00\x00"}, control={0x0 }, flags=0}, 0x0)
I0830 00:30:31.975006 1 strace.go:605] [ 3: 3] haproxy X sendmsg(0x36 socket:[38], 0x7f231e94a420 {name=0x0, namelen=0, iovecs=0x7f231e94a400 {base=0x7f231e94a3fc, len=4}, control=0x0, control_len=0, flags=0}, 0x0) = 4 (0x4) (5.846µs)
I0830 00:30:31.975095 1 strace.go:567] [ 3: 3] haproxy E sendmsg(0x36 socket:[38], 0x7f231e94a420 {name=0x0, namelen=0, iovecs=0x7f231e94a400 {base=0x7f82e84b8010, len=6, "\x00\x00\x00\x00\x00\x00"}, control={0x55666ef2cc70 {level=SOL_SOCKET, type=SCM_RIGHTS, length=20, content: 5}}, flags=0}, 0x0)
I0830 00:30:31.975107 1 strace.go:605] [ 20: 20] haproxy X recvmsg(0x7 socket:[39], 0x7fdd63b22760 {name=0x0, namelen=0, iovecs=0x7fdd63b22740 {base=0x7fdd63b22730, len=4, "\x01\x00\x00\x00"}, control={0x0 }, flags=0}, 0x100) = 4 (0x4) (363.852µs)
I0830 00:30:31.975116 1 strace.go:605] [ 3: 3] haproxy X sendmsg(0x36 socket:[38], 0x7f231e94a420 {name=0x0, namelen=0, iovecs=0x7f231e94a400 {base=0x7f82e84b8010, len=6}, control=0x55666ef2cc70, control_len=24, flags=0}, 0x0) = 6 (0x6) (4.032µs)
I0830 00:30:31.975141 1 strace.go:567] [ 20: 20] haproxy E recvmsg(0x7 socket:[39], 0x7fdd63b22760 {name=0x0, namelen=0, iovecs=0x7fdd63b22740 {base=0x555dd3b90520, len=1037736}, control=0x555dd3b8ed70, control_len=6048, flags=0}, 0x0)
I0830 00:30:31.975164 1 strace.go:605] [ 20: 20] haproxy X recvmsg(0x7 socket:[39], 0x7fdd63b22760 {name=0x0, namelen=0, iovecs=0x7fdd63b22740 {base=0x555dd3b90520, len=1037736}, control=0x555dd3b8ed70, control_len=6048, flags=0}, 0x0) = -1 errno=14 (bad address) (5.48µs)
I0830 00:30:31.975302 1 strace.go:561] [ 20: 20] haproxy E close(0x7 socket:[39])
I0830 00:30:31.975325 1 strace.go:599] [ 20: 20] haproxy X close(0x7 socket:[39]) = 0 (0x0) (8.436µs)
I0830 00:30:31.975479 1 strace.go:564] [ 20: 20] haproxy E listen(0x6 socket:[40], 0x1000)
I0830 00:30:31.975496 1 strace.go:561] [ 3: 3] haproxy E close(0x36 socket:[38])
I0830 00:30:31.975524 1 strace.go:599] [ 3: 3] haproxy X close(0x36 socket:[38]) = 0 (0x0) (5.032µs)
I0830 00:30:31.975506 1 strace.go:602] [ 20: 20] haproxy X listen(0x6 socket:[40], 0x1000) = 0 (0x0) (6.267µs)
I0830 00:30:31.975569 1 strace.go:561] [ 20: 20] haproxy E close(0x3 socket:[6])
I0830 00:30:31.975586 1 strace.go:599] [ 20: 20] haproxy X close(0x3 socket:[6]) = 0 (0x0) (2.227µs)
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with the supplied haproxy.cfg, Docker command, and runsc recvmsg/sendmsg syscall trace. Trace the runsc recvmsg path for large buffers and SCM_RIGHTS, then verify that haproxy reload receives all listener file descriptors and does not leave a duplicate stale listener.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, linux
Domain
networking, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.