cloudflare / cloudflare/quiche
tokio-quiche: server handshake fails on NICs without GSO — acceptor Retry/version-negotiation reply always sets UDP_SEGMENT cmsg (EIO)
- Dominant language
- Rust
- Stars
- 11.8k
- Forks
- 1.1k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 16
Description
## Summary
On a host whose NIC does not support GSO, a tokio-quiche **server** cannot complete any QUIC handshake. It receives the client's Initial fine, but every reply the acceptor sends (stateless Retry / version negotiation) fails with `EIO`, so the handshake never progresses and no connection is ever established. This affects IPv4 and IPv6 equally.
The crate already avoids GSO on the **client** path for exactly this reason, but the **server** acceptor path has no such guard.
## Root cause
`ConnectionAcceptor::handshake_reply` (used by `stateless_retry` and version negotiation) unconditionally calls `gso::send_to(...)` on Linux:
```rust
// tokio-quiche/src/quic/router/acceptor.rs — handshake_reply()
#[cfg(target_os = "linux")]
{
let from = with_pktinfo.then_some(incoming.local_addr);
let _ = crate::quic::io::gso::send_to(udp, to, from, send_buf, send_buf.len(), ...);
}
```
`gso::send_to` **always** attaches a `UDP_SEGMENT` cmsg (made unconditional in #2060), even for a single-segment packet:
```rust
// tokio-quiche/src/quic/io/gso.rs
cmsgs.push(ControlMessage::UdpGsoSegments(&segment_size_u16));
```
On a NIC that can't do GSO, `sendmsg()` with a `UDP_SEGMENT` cmsg returns `EIO`. So the Retry reply never leaves the box, the client keeps retransmitting its Initial, and nothing happens.
The crate is already aware that some NICs lack GSO and deliberately avoids it on the client path:
```rust
// tokio-quiche/src/quic/mod.rs
// Don't apply_max_capabilities(): some NICs don't support GSO
```
…but the server acceptor's `handshake_reply` has no equivalent guard. The `has_gso` capability only gates the io-worker's connection sends (`worker.rs::flush_buffer_to_socket` checks `self.cfg.with_gso`); it does **not** gate the acceptor's Retry/version-negotiation path. Consequently, even constructing the `QuicListener` with GSO disabled (default `SocketCapabilities`, `has_gso = false`) does not help — the acceptor still calls `gso::send_to` and still fails.
## Environment / repro
- Linux 5.15, `virtio_net`.
- The vNIC reports `scatter-gather: off [fixed]` and `tx-checksum-ip-generic: off [fixed]`, so `generic-segmentation-offload` can't be enabled (`ethtool -K eth0 gso on` → `Could not change any device features`). This feature set is negotiated by the hypervisor and is seen on some lightweight/budget cloud VPS instances.
- The capability probe is a false positive here: `setsockopt(UDP_SEGMENT, …)` **succeeds** (the kernel supports the UDP GSO API regardless of NIC), so `apply_max_capabilities()` reports `has_gso = true`. The failure only surfaces at `sendmsg()` time.
`strace` of the server while a client tries to connect — it receives:
```
recvmsg(9, {msg_name={sa_family=AF_INET6, ... "::ffff:" ...}, ...}) = 1200
```
…but every reply errors:
```
sendmsg(9, {..., msg_control=[{cmsg_len=18, cmsg_level=SOL_UDP, cmsg_type=UDP_SEGMENT}], msg_controllen=24, ...}, 0) = -1 EIO (Input/output error)
```
(Identical for IPv6 clients.)
## Impact
A server on such a NIC is completely unreachable over QUIC while address validation (stateless Retry) is enabled — which is the default. Current workarounds are limited to setting `disable_client_ip_validation` **and** building the listener without GSO, or moving to a NIC that supports GSO.
## Suggested fix
Make the acceptor's `handshake_reply` (Retry / version negotiation) respect GSO availability the same way the client path does — fall back to a plain `send_to` (no `UDP_SEGMENT` cmsg) when GSO isn't in use. Because the capability probe gives a false positive on these NICs, a fully robust fix would additionally detect `EIO` on a GSO send and disable GSO for the socket (or perform a real test-send during capability detection instead of relying on the `setsockopt` succeeding).
Contributor guide
Research direction
Start in tokio-quiche/src/quic/router/acceptor.rs at ConnectionAcceptor::handshake_reply, then compare tokio-quiche/src/quic/io/gso.rs with the client guard in tokio-quiche/src/quic/mod.rs and worker.rs::flush_buffer_to_socket. Verify that Retry and version-negotiation replies can be sent without an inappropriate UDP_SEGMENT cmsg on NICs lacking GSO, and that the server handshake succeeds in the reported environment.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, rust
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 64/100