worker: one shared QUIC socket makes a single unreachable proxy pod block QUIC to all of them, permanently

Summary

The worker opens one UDP socket and reuses it for every QUIC dial, to every proxy pod, for the entire life of the process. It is created on the first dial and the only code path that releases it is HttpProxy.Close(), which runs at process exit.

Because a UDP load balancer hashes on the source port, this means every proxy pod is reached over a single balancer flow. That has two consequences.

One dead target blocks every pod

There is no isolation between hosts. If the flow is pinned to an instance that has gone away, QUIC to every proxy pod fails, including pods that are perfectly healthy.

Observed on a staging reproduction: the worker logged 3,796 timeout: no recent network activity failures in eight minutes, dialling pods that had been running and healthy for 38 minutes. The load balancer targets were registered and healthy, DNS resolved correctly, and the receiving proxies reported zero inbound QUIC connections. Packets were not reaching them.

It does not recover on its own

A load balancer flow is only re-hashed once it goes idle. While the worker keeps retrying, the flow never idles, so it is never re-placed on a healthy instance. Since the socket is only replaced at process exit, the condition persists for the life of the worker.

This is why replacing function instances has been the reliable remedy: new pods get new source ports, which produces new flows.

Proposed fix

Key the QUIC transport by host, exactly as the connection cache already is, so each proxy pod is reached from its own source port.

This gives isolation between hosts, and it makes recovery fall out of the existing lifecycle: a failed dial already removes the host from the cache, so releasing its socket at the same time means the next attempt arrives from a new source port that the balancer can place elsewhere. No timers and no new state.

Cost

One UDP socket and one reader goroutine per distinct proxy pod a worker talks to. That is single digits in staging and tens in a production cell.

Also found

validateConfig in quic-go writes defaults back into the quic.Config it is given, and every dial passed the same pointer, so concurrent dials raced on it. Confirmed by the race detector once two hosts were dialled at once.

Start by tracing the worker's shared QUIC socket lifecycle from the first dial through HttpProxy.Close(), then compare it with the existing host-keyed connection cache. Inspect validateConfig and the concurrent dial path, and use the race detector with multiple hosts. Done means hosts have isolated socket lifecycles, failed dials can recover, and concurrent configuration use is race-free.