linkerd / linkerd/linkerd2

No max age / idle timeout for pooled proxy-to-proxy HTTP/2 connections — held connections pin buffer memory until pod restart or OOM

Open
#15,623 0 comments 1 reaction 1 assignee View on GitHub

@anthoturc is already working on this.

Since Sep 4, 2026.

Dominant language
Go
Stars
11.5k
Forks
1.4k
Avg merge
3d 22h
Merged PRs (30d)
53

Description

### What problem are you trying to solve?

Pooled proxy-to-proxy HTTP/2 connections currently live forever. TCP and HTTP/2 keepalives keep them healthy indefinitely, `MAX_IDLE_CONNS_PER_ENDPOINT` (default 10,000) only caps the count, the HTTP/1 connection-pool idle timeout doesn't apply to them, and the outbound discovery idle timeout only tears state down when a service receives no traffic at all. We audited the proxy's env config surface from `v2.311.0` through `v2.366.0` and found no setting that closes a pooled connection for being idle or bounds its lifetime.

Application-level timeouts don't help: our application closes its own idle connections after 5 minutes, but those only govern the app→local-proxy hop. The proxy→proxy pooled connections are owned by Linkerd and persist regardless — so the memory they pin keeps climbing until the pod is restarted manually or gets OOM-killed.

Why this matters for memory: each held server-side connection retains its receive-buffer high-water mark — bounded by the connection window (1MB default) — for its entire lifetime. On a **very high-throughput service** in our mesh (arm64 EKS), a large caller fleet holds ~300 pooled connections into each server pod. After ~300 rps load bursts we measured **~700KB retained per connection, i.e. ~200MB of proxy RSS per pod**, verified via `tcp_open_total`/`tcp_close_total` accounting on the admin endpoint. That memory was only ever released when peer pods restarted — never by idling, no matter how long.

### How should the problem be solved?

A configurable **max connection age** (and/or idle timeout) for inbound HTTP/2 server connections: after the configured age, the proxy performs the same graceful shutdown (GOAWAY) it already uses for drain — no new streams accepted, in-flight streams complete, clients reconnect transparently — with per-connection jitter to avoid synchronized shutdown storms. Similar in spirit to gRPC's `MAX_CONNECTION_AGE`.

We implemented exactly this and have a PR open: **linkerd/linkerd2-proxy#4618** (`LINKERD2_PROXY_INBOUND_SERVER_HTTP2_MAX_CONNECTION_AGE`; unset preserves current behavior; includes an integration test).

Results running it in a production-like mesh with a 15-minute age:
- Proxy RSS became a **bounded ~15-minute sawtooth instead of a monotonic plateau** — buffers released every cycle with no pod restarts (observed on a build with jemalloc enabled on aarch64 — see #15613 / linkerd2-proxy#4613 — so freed memory was actually returned to the OS).
- **Zero failed requests** across many recycle cycles; p50/p99 unchanged through recycle waves.
- Cost: one TLS handshake per connection per age period, spread over ~90s by the jitter — below measurement noise in-cluster.

### Any alternatives you've considered?

- **Scheduled pod restarts** — operational toil that masks the issue.
- **Lowering the HTTP/2 connection window** — caps the per-connection ceiling but not the lifetime; buffers still pin at high-water forever.
- **Adaptive flow control** (`..._ADAPTIVE_FLOW_CONTROL=true`) — measured strictly worse in our environment (~3× the memory plateau at lower load; the BDP estimator grows windows past the fixed default under load).
- **Application-level connection TTLs** — cannot reach the proxy-owned pooled connections, as described above.

### How would users interact with this feature?

A proxy environment variable (deliverable via `proxy.additionalEnv` or the `config.linkerd.io/proxy-additional-env` annotation), defaulting to unset/disabled so existing behavior is unchanged. Optionally a first-class Helm value/annotation later.

### Would you like to work on this feature?

Yes — implementation with tests is already open at linkerd/linkerd2-proxy#4618; happy to iterate on design (naming, defaults, idle-timeout variant, outbound-side equivalent) per maintainer feedback, and to provide graphs and metric snapshots.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.