cloudflare / cloudflare/cloudflared

2026.9.0 segfaults in quic-go QueueProbePacket (nil receiver) when running in a Docker bridge network — 2026.8.2 unaffected

Open
#1,737 0 comments 0 reactions 0 assignees View on GitHub
Priority: Normal Type: Bug
Dominant language
Go
Stars
15.6k
Forks
1.4k
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**

`cloudflared` 2026.9.0 crashes with SIGSEGV roughly every 75 seconds when running
in a Docker container on a bridge network. The panic is a nil-receiver dereference
inside the bundled quic-go fork, on the probe-timeout path.

The same container on 2026.8.2 is completely stable. Notably, a **second tunnel
running the same 2026.9.0 binary on the same machine**, as a host systemd service,
has **zero** crashes — the only meaningful difference is that it uses the host
network directly instead of the Docker bridge/NAT.

Since `QueueProbePacket` is only reached on a PTO (probe timeout), the NAT path
seems to make packet loss / timeouts frequent enough to hit a latent race that the
direct path rarely triggers.

**To Reproduce**

1. Run `cloudflare/cloudflared:2026.9.0` in a Docker container on a user-defined
bridge network (container IP `172.20.0.2`, NAT via iptables MASQUERADE).
2. Command: `tunnel --no-autoupdate --loglevel info run --token `
(remotely-managed tunnel, 3 ingress rules pointing at an internal service).
3. Let it run. It starts cleanly every time — all connectivity pre-checks PASS,
4 connections register (tpe01 / khh01, quic) — then panics after roughly a
minute and is restarted by `restart: unless-stopped`.
Observed: **145 crashes in ~6 hours**.

4. Tunnel ID: <填入或留空>
5. cloudflared config: token-based (remotely managed), no local config file.

**Expected behavior**

The tunnel stays connected without crashing, as it does on 2026.8.2.

**Environment and versions**

- OS: Ubuntu 24.04.5 LTS, kernel 6.8.0-139-generic (Docker container)
- Architecture: AMD64 — Intel Core 2 Duo P8800, 2 cores (old hardware, no AES-NI;
noting it in case goroutine timing is relevant to the race)
- Version: 2026.9.0 (built 2026-09-09-18:43 UTC), GoVersion go1.26.8
- Last known good: 2026.8.2 (built 2026-08-14-12:28 UTC)

**Logs and errors**

[signal SIGSEGV: segmentation violation code=0x1 addr=0x4c pc=0xdae18d]

goroutine 128 [running]:
github.com/quic-go/quic-go/internal/ackhandler.(*sentPacketHandler).getPacketNumberSpace(...)
/go/pkg/mod/github.com/chungthuang/quic-go@v0.45.1-0.20260529212404-a9fddf436fc4/internal/ackhandler/sent_packet_handler.go:374
github.com/quic-go/quic-go/internal/ackhandler.(*sentPacketHandler).QueueProbePacket(0x0?, 0xfc?)
/go/pkg/mod/github.com/chungthuang/quic-go@v0.45.1-0.20260529212404-a9fddf436fc4/internal/ackhandler/sent_packet_handler.go:1042 +0x12d
github.com/quic-go/quic-go.(*Conn).sendProbePacket(0x1524c553b208, 0xd8?, 0x35a126441a6)
/go/pkg/mod/github.com/chungthuang/quic-go@v0.45.1-0.20260529212404-a9fddf436fc4/connection.go:2722 +0x104
github.com/quic-go/quic-go.(*Conn).triggerSending(0x1524c553b208, 0x35a126441a6)
/go/pkg/mod/github.com/chungthuang/quic-go@v0.45.1-0.20260529212404-a9fddf436fc4/connection.go:2486 +0x16a
github.com/quic-go/quic-go.(*Conn).run(0x1524c553b208)
/go/pkg/mod/github.com/chungthuang/quic-go@v0.45.1-0.20260529212404-a9fddf436fc4/connection.go:741 +0x705
github.com/quic-go/quic-go.(*Transport).doDial.func1()
/go/pkg/mod/github.com/chungthuang/quic-go@v0.45.1-0.20260529212404-a9fddf436fc4/transport.go:326 +0x3a
created by github.com/quic-go/quic-go.(*Transport).doDial in goroutine 107
/go/pkg/mod/github.com/chungthuang/quic-go@v0.45.1-0.20260529212404-a9fddf436fc4/transport.go:325 +0x737

The `pc` is identical on every crash (`pc=0xdae18d`), with `addr` varying only
between 0x4b–0x4d — a deterministic nil dereference, not memory corruption.

**Additional context**

- **Not OOM:** `OOMKilled=false`, memory usage 17.4 MiB against a 128 MiB limit.
- **Same binary, same host, no crash:** the host-side systemd tunnel runs the same
2026.9.0 with **0** SIGSEGV over 8+ hours.
- Differences between the two processes on this machine:
- container: Docker bridge NAT (172.20.0.2), ICMP proxy **enabled**
- host: direct on the physical NIC, ICMP proxy **disabled** (ping_group_range)
- `net.core.rmem_max` / `wmem_max` are raised to 7500000 on this host, so both
processes get the full 7 MiB UDP receive buffer they ask for. The stable host
process runs with the same setting, so this does not appear to be the trigger.
- **Rolling the container back to 2026.8.2 resolves it completely:** 0 crashes in
over an hour, where the observed rate would have predicted ~48.

Contributor guide

Open the contributing guide

Research direction

Start with the bundled quic-go fork's internal/ackhandler/sent_packet_handler.go and connection.go at the QueueProbePacket and sendProbePacket stack locations shown in the panic. Compare the 2026.9.0 and 2026.8.2 dependency revisions, then reproduce the tunnel on a Docker bridge network; done means the probe-timeout path no longer produces the SIGSEGV and the container remains connected.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.