Memory leak in xDS stream handling when unsupported Envoy versions repeatedly connect
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
## Overview
Consul server processes leak memory when unsupported (too old) Envoy proxies
repeatedly attempt xDS connections. Each rejected connection allocates resources
in `processDelta()` that are not fully cleaned up on the early-return error path,
causing unbounded memory growth over time.
This was previously reported in #9331 (Consul 1.9.0 / Envoy 1.13.0) and closed
without a fix. The same leak pattern persists in Consul 1.21.4.
## Environment
- **Consul version**: 1.21.4
- **Envoy version (connecting)**: 1.28.1 (rejected; minimum supported is 1.31.x)
- **OS**: Amazon Linux 2023, kernel 6.x (aarch64)
- **Instance type**: 8 vCPU, 16 GB RAM, zero swap
## Current Behavior
When an unsupported Envoy proxy connects, the xDS server:
1. Enters `DeltaAggregatedResources()` (`agent/xds/delta.go:63`)
2. Spawns a background goroutine to receive requests from the stream (line 69)
3. Calls `processDelta()` which allocates 5 `xDSDeltaType` handler objects with
internal maps (lines 159-171), plus child relationship maps (lines 182-189),
a `resourceMap` via `EmptyIndexedResources()` (line 145), and
`currentVersions` map (line 150)
4. On the first request, calls `DetermineSupportedProxyFeatures()` (line 232)
5. The version check fails at
`envoyextensions/xdscommon/envoy_versioning.go:49-51`:
`"Envoy 1.28.1 is too old and is not supported by Consul"`
6. `processDelta()` returns immediately with `codes.InvalidArgument` (line 234)
The early return at step 6 bypasses all cleanup that would normally occur during
a full stream lifecycle:
- **Watch cancel** (`defer watchCancel()` at line 338) is never set up because
the state machine never reaches `stateDeltaInit` (line 326-338)
- **Handler state** (5 handler objects with maps for subscriptions, resource
versions, and pending updates) relies on GC but holds references to the
`stream` object
- **Background goroutine** (lines 69-90) may remain blocked on `stream.Recv()`
after `processDelta` returns; the `reqStop` flag is set at line 98 but the
goroutine only checks it after `Recv()` returns, creating a window where it
holds references to `reqCh` and the stream
When unsupported Envoy proxies retry every ~15 seconds, this creates thousands of
leaked allocation cycles per day per Consul server.
## Expected Behavior
When an unsupported Envoy version is rejected, all resources allocated for that
stream (handler objects, maps, goroutines, channel) should be promptly cleaned up.
Repeated rejected connections from unsupported Envoy versions should not cause
memory growth on the Consul server.
## Steps to Reproduce
1. Run a Consul 1.21.4 server cluster
2. Deploy Envoy 1.28.1 sidecars (e.g., via an older consul-k8s Helm chart) that
register services with Consul and attempt xDS connections
3. Consul rejects each connection with
`"Envoy 1.28.1 is too old and is not supported by Consul"`
4. Envoy retries every ~15 seconds
5. Observe Consul server RSS memory growing continuously over hours/days
## Observed Impact
In a production environment with 5 stale Envoy 1.28.1 pods retrying every
~15 seconds (~7,200 rejected connections per day per server):
| Metric | With rejected xDS streams | Without rejected xDS streams |
| :-------------------- | :------------------------ | :--------------------------- |
| Memory after months | 8-14 GB (OOM) | ~300 MB |
| Growth rate | ~9 GB within first hour | Stable |
| Outcome | OOM kill, cluster outage | No issues |
Environments with zero Kubernetes-managed Envoy proxies (only Nomad-managed
Envoy 1.34.1) remained at ~300 MB RSS after 2-3 months of uptime, confirming
the leak is triggered specifically by the rejected xDS stream path.
## Code References (v1.21.4)
- **Version check**: `envoyextensions/xdscommon/envoy_versioning.go:49-51`
- **Stream entry point**: `agent/xds/delta.go:63-101`
(`DeltaAggregatedResources`)
- **Background goroutine**: `agent/xds/delta.go:69-90` (receiver goroutine)
- **Handler allocation**: `agent/xds/delta.go:141-189` (5 handlers + maps)
- **Early return on version failure**: `agent/xds/delta.go:232-235`
- **Watch cancel that is never reached**: `agent/xds/delta.go:326-338`
## Suggested Fix
The early-return path in `processDelta()` when the Envoy version check fails
should ensure cleanup of all allocated resources. Possible approaches:
1. **Move the version check earlier** — before allocating handler objects and
maps. Check the Envoy version in `DeltaAggregatedResources()` on the first
received request, and return the error before calling `processDelta()`.
2. **Add explicit cleanup on early return** — ensure all handler maps are
nilled out and the background goroutine is properly signaled to exit before
returning the error.
3. **Rate-limit or cache rejected versions** — if a proxy ID has been rejected
for an unsupported version, short-circuit subsequent connections from the
same source without allocating the full handler state.
## Related Issues
- #9331 — Same memory leak pattern reported in Consul 1.9.0 with Envoy 1.13.0
(closed without fix)
- #12288 — Memory leak in gRPC rebalancer (potentially related stream lifecycle
issue)
Contributor guide
Assessment
This issue has not been assessed yet.