No max age / idle timeout for pooled proxy-to-proxy HTTP/2 connections — held connections pin buffer memory until pod restart or OOM
@anthoturc is already working on this.
Since Sep 4, 2026.
- Dominant language
- Go
- Stars
- 11.5k
- Forks
- 1.4k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 53
Description
### What problem are you trying to solve?
Pooled proxy-to-proxy HTTP/2 connections currently live forever. TCP and HTTP/2 keepalives keep them healthy indefinitely, `MAX_IDLE_CONNS_PER_ENDPOINT` (default 10,000) only caps the count, the HTTP/1 connection-pool idle timeout doesn't apply to them, and the outbound discovery idle timeout only tears state down when a service receives no traffic at all. We audited the proxy's env config surface from `v2.311.0` through `v2.366.0` and found no setting that closes a pooled connection for being idle or bounds its lifetime.
Application-level timeouts don't help: our application closes its own idle connections after 5 minutes, but those only govern the app→local-proxy hop. The proxy→proxy pooled connections are owned by Linkerd and persist regardless — so the memory they pin keeps climbing until the pod is restarted manually or gets OOM-killed.
Why this matters for memory: each held server-side connection retains its receive-buffer high-water mark — bounded by the connection window (1MB default) — for its entire lifetime. On a **very high-throughput service** in our mesh (arm64 EKS), a large caller fleet holds ~300 pooled connections into each server pod. After ~300 rps load bursts we measured **~700KB retained per connection, i.e. ~200MB of proxy RSS per pod**, verified via `tcp_open_total`/`tcp_close_total` accounting on the admin endpoint. That memory was only ever released when peer pods restarted — never by idling, no matter how long.
### How should the problem be solved?
A configurable **max connection age** (and/or idle timeout) for inbound HTTP/2 server connections: after the configured age, the proxy performs the same graceful shutdown (GOAWAY) it already uses for drain — no new streams accepted, in-flight streams complete, clients reconnect transparently — with per-connection jitter to avoid synchronized shutdown storms. Similar in spirit to gRPC's `MAX_CONNECTION_AGE`.
We implemented exactly this and have a PR open: **linkerd/linkerd2-proxy#4618** (`LINKERD2_PROXY_INBOUND_SERVER_HTTP2_MAX_CONNECTION_AGE`; unset preserves current behavior; includes an integration test).
Results running it in a production-like mesh with a 15-minute age:
- Proxy RSS became a **bounded ~15-minute sawtooth instead of a monotonic plateau** — buffers released every cycle with no pod restarts (observed on a build with jemalloc enabled on aarch64 — see #15613 / linkerd2-proxy#4613 — so freed memory was actually returned to the OS).
- **Zero failed requests** across many recycle cycles; p50/p99 unchanged through recycle waves.
- Cost: one TLS handshake per connection per age period, spread over ~90s by the jitter — below measurement noise in-cluster.
### Any alternatives you've considered?
- **Scheduled pod restarts** — operational toil that masks the issue.
- **Lowering the HTTP/2 connection window** — caps the per-connection ceiling but not the lifetime; buffers still pin at high-water forever.
- **Adaptive flow control** (`..._ADAPTIVE_FLOW_CONTROL=true`) — measured strictly worse in our environment (~3× the memory plateau at lower load; the BDP estimator grows windows past the fixed default under load).
- **Application-level connection TTLs** — cannot reach the proxy-owned pooled connections, as described above.
### How would users interact with this feature?
A proxy environment variable (deliverable via `proxy.additionalEnv` or the `config.linkerd.io/proxy-additional-env` annotation), defaulting to unset/disabled so existing behavior is unchanged. Optionally a first-class Helm value/annotation later.
### Would you like to work on this feature?
Yes — implementation with tests is already open at linkerd/linkerd2-proxy#4618; happy to iterate on design (naming, defaults, idle-timeout variant, outbound-side equivalent) per maintainer feedback, and to provide graphs and metric snapshots.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.