Outbound proxy permanently stuck on stale endpoint after silent gRPC stream failure

What is the issue?

After a pod replacement, the outbound proxy on a long-running ingress pod (HAProxy) permanently routes traffic to a dead endpoint IP. The proxy never recovers — only restarting the proxy process resolves it.

The destination controller has the correct endpoint data (linkerd diagnostics endpoints confirms this). The proxy's gRPC watch stream appears to have silently stopped delivering updates without triggering error handling or reconnection.

The proxy's discovery cache is kept alive by continuous traffic (health checks every 5s), so outboundDiscoveryCacheUnusedTimeout never triggers eviction. The gRPC stream appears healthy at the transport layer (TCP keepalive passes) but is not delivering application-level updates for specific services.

There is no feedback loop from the balancer (which sees repeated Connect failures) back to the discovery layer (which could trigger re-subscription).

How can it be reproduced?

We have not been able to reliably reproduce this on demand. The bug appears to require the proxy to run for an extended period (4+ days) before the gRPC stream silently degrades. A clean network partition does NOT trigger it — the proxy correctly reconnects.

Minimal setup that triggers it in our environment:

# Long-lived ingress pod with linkerd sidecar (DaemonSet, restarted monthly)
# Watches 200+ services via destination controller
# Receives continuous traffic (health checks every 5s per service)
#
# After 4+ days of uptime, deploy a backend pod with maxUnavailable: 100%
# The proxy misses the endpoint update for that specific service

---
# Backend deployment (the pod being replaced)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: erm
  namespace: test-ns
spec:
  replicas: 1
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: "100%"
      maxUnavailable: "100%"
  selector:
    matchLabels:
      app: erm
  template:
    metadata:
      labels:
        app: erm
      annotations:
        linkerd.io/inject: enabled
    spec:
      containers:
      - name: erm
        image: nginx:latest
        ports:
        - containerPort: 8080
---
# Service
apiVersion: v1
kind: Service
metadata:
  name: erm
  namespace: test-ns
spec:
  selector:
    app: erm
  ports:
  - port: 8080
    targetPort: 8080
---
# Server policy (deny by default, require mTLS)
apiVersion: policy.linkerd.io/v1beta1
kind: Server
metadata:
  name: erm-server-8080
  namespace: test-ns
spec:
  podSelector:
    matchLabels:
      app: erm
  port: 8080
  proxyProtocol: HTTP/1

The ingress pod in our case is a HAProxy DaemonSet with a linkerd-proxy sidecar. The proxy on this pod maintains gRPC watch streams to the destination controller for all backend services (~200+). After days of uptime, kubectl rollout restart deploy/erm -n test-ns causes the proxy to permanently route to the old (dead) pod IP.

What we tried that did NOT reproduce it:

• Network partition (NetworkPolicy blocking proxy → destination controller for 60s, then removing) — proxy correctly reconnects
• Rapid image flipping (3 deploys in quick succession) — proxy picks up new endpoint within seconds
• Restarting destination controller simultaneously with pod replacement — proxy reconnects after backoff

Conditions we believe are required:

1. Proxy running for 4+ days (long-lived pod)
2. Proxy watching many services (200+ gRPC streams multiplexed on one HTTP/2 connection to the destination controller)
3. Continuous traffic to the affected service (prevents idle cache eviction via outboundDiscoveryCacheUnusedTimeout)
4. maxUnavailable: 100% deployment strategy (no overlap between old and new pod)

Symptoms

• Response header: l5d-proxy-error: endpoint <OLD_IP>:8080: client error (Connect) → HTTP 502
• OR inbound proxy logs: client.tls=None(NoClientHello) → HTTP 403 (policy denial due to missing mTLS)
• linkerd diagnostics endpoints <service> shows the correct (new) IP — destination controller is fine
• Proxy logs show NO "Adding endpoint" message for the affected service after pod replacement
• Restarting the backend pod does NOT fix it (proxy ignores the new IP too)
• Only restarting the proxy process (kill 1 on the linkerd-proxy container) resolves it

Logs, error output, etc

Outbound proxy error (from response headers):

l5d-proxy-error: endpoint 10.20.82.182:8080: client error (Connect)

Inbound proxy denial (from pod logs):

[319.175555s] INFO inbound:server{port=8080}: linkerd_app_inbound::policy::http: Request denied server.group=policy.linkerd.io server.kind=server server.name=erm-server-8080 route.group=policy.linkerd.io route.kind=HTTPRoute route.name=erm-application-route client.tls=None(NoClientHello) client.ip=10.32.140.123

Key evidence from logs

Proxy logs (from Splunk) show the last "Adding endpoint" for the affected service was hours before the pod replacement:

2026-05-12 00:00:00.307 [29829.645441s] outbound:service{ns=my-namespace name=erm port=8080}: linkerd_pool_p2c: Adding endpoint addr=10.20.107.237:8080

After the pod replacement at 05:24 UTC, no "Adding endpoint" log appeared for this service — confirming the proxy never received the endpoint update.

Destination controller logs during the failure window show NO errors — only periodic v1 Endpoints is deprecated info messages.

Analysis

The proxy's discovery cache is kept alive by continuous traffic (health checks every 5s), so outboundDiscoveryCacheUnusedTimeout never triggers eviction. The gRPC stream appears healthy at the transport layer (TCP keepalive passes) but is not delivering application-level updates for specific services.

There is no feedback loop from the balancer (which sees repeated Connect failures) back to the discovery layer (which could trigger re-subscription). The proxy trusts its cached endpoint list until the destination controller explicitly pushes an update.

We suspect the issue may be related to HTTP/2 flow control or gRPC stream multiplexing — with 200+ watch streams on a single connection, a stall on one stream could silently block updates for specific services while others continue working.

output of linkerd check -o short

linkerd-identity
----------------
× trust anchors are using supported crypto algorithm
    Invalid trustAnchors:
	* 44869714981336368087457595129702791890 Protecht Root CA must be signed by an RSA 2048/4096 bit key, instead SHA512-RSA was used
    see https://linkerd.io/2.14/checks/#l5d-identity-trustAnchors-use-supported-crypto for hints

linkerd-viz
-----------
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
	* metrics-api-577576f989-wtgfh (proxy-edge-26.5.1)
	* prometheus-5fd569d57f-lpnnz (proxy-edge-26.5.1)
	* tap-6d5b78cddc-2kr65 (proxy-edge-26.5.1)
	* tap-injector-7fddcb967-8k7kc (proxy-edge-26.5.1)
	* web-c988656c-zkwhg (proxy-edge-26.5.1)
    see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
    metrics-api-577576f989-wtgfh running proxy-edge-26.5.1 but cli running stable-2.14.10
    see https://linkerd.io/2.14/checks/#l5d-viz-proxy-cli-version for hints

Status check results are ×

Environment

• Linkerd edge-26.1.1 (upgraded to edge-26.5.1 now and will monitor but nothing in this new version points that the issue won't come back)
• EKS, Kubernetes 1.35
• HAProxy ingress with linkerd-proxy sidecar (long-lived DaemonSet pods, ~4+ days uptime)
• ~200 - 1700 services watched per HAProxy proxy (depending on environment)

Possible solution

Any of these would prevent permanent outages:

1. Connect-failure-triggered re-resolve: After N consecutive connection failures to an endpoint, drop the cached discovery result and re-subscribe to the destination controller
2. Max-age cache eviction: In addition to idle-based eviction (outboundDiscoveryCacheUnusedTimeout), add a maxAge that forces periodic re-validation regardless of traffic
3. Application-level keepalive on destination streams: Periodic ping/pong with timeout to detect silently dead streams
4. Admin API for forced re-discovery: POST /admin/reset-discovery to allow operators to fix without process restart

Additional context

Related issues

• #11480 (fixed in edge-23.10.3, but our issue persists)
• #6842 (same symptoms, closed)
• #5852 (destination watch stream stops)
• #10590 (proxy fails to reconnect to restarted pod)

Would you like to work on fixing this bug?

no

No source files or tests are named. Start by tracing the outbound discovery cache, gRPC watch streams, and the balancer's repeated Connect failures; compare behavior during endpoint replacement and stream failure. Done should prevent a long-lived proxy from permanently routing to a stale endpoint without requiring a proxy restart.