envoyproxy / envoyproxy/gateway

memory: upstream TLS validation contexts are duplicated per-cluster

Open
#9,489 1 comment 1 reaction 0 assignees View on GitHub
kind/enhancement stale
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

## Description

When many backends validate their upstream (backend) connections against the same CA — configured via `BackendTLSPolicy` — Envoy Gateway generates a separate upstream cluster per route-rule destination, and each cluster gets its own `UpstreamTlsContext`. Envoy then builds a separate `SSL_CTX` / X509 trust store per cluster, so the CA bundle is parsed and held in memory **once per cluster**. On gateways with a large number of backend clusters that share a common (and potentially large) CA bundle, this dominates the data-plane memory footprint.

Related to https://github.com/envoyproxy/gateway/issues/5307

## Why this increases memory

Envoy materializes TLS state at the transport-socket-factory level, which is **per-cluster**, and does not share parsed TLS contexts across clusters:

- Each cluster's context construction (`ContextImpl` / `DefaultCertValidator`) parses every CA certificate (`PEM → X509 → ASN.1`) and adds it to its **own** `X509_STORE`, and loads the client cert/key into its **own** `SSL_CTX`.
- `ContextManagerImpl::createSslClientContext()` always allocates a new context — there is no content-addressed cache — so two clusters with byte-identical upstream-TLS config still build two independent contexts.
- Sharing the certificate at the SDS layer does **not** help: SDS deduplicates the raw secret *bytes* by name, but the expensive part is the **parsed** per-cluster `X509_STORE`, which is rebuilt for every cluster regardless of whether the input is identical.

Net effect: memory grows with `(number of clusters) × (size of the CA bundle)`, even when every cluster trusts the same CA.

## Example

Two backends validating against the same CA (`example-ca`). Note the clusters reference the same CA content, yet each builds its own parsed context:

```yaml
# Cluster 1 (route-a -> backend-a)
- name: httproute/ns/route-a/rule/0
type: STRICT_DNS
transport_socket_matches:
- transport_socket:
typed_config:
"@type": ...UpstreamTlsContext
common_tls_context:
tls_certificate_sds_secret_configs:
- name: envoy-gateway-system/client-cert # shared client cert
combined_validation_context:
validation_context_sds_secret_config:
name: backend-a-tls-policy/example-ca # CA = example-ca
sni: backend-a.example.com

# Cluster 2 (route-b -> backend-b)
- name: httproute/ns/route-b/rule/0
type: STRICT_DNS
transport_socket_matches:
- transport_socket:
typed_config:
"@type": ...UpstreamTlsContext
common_tls_context:
tls_certificate_sds_secret_configs:
- name: envoy-gateway-system/client-cert
combined_validation_context:
validation_context_sds_secret_config:
name: backend-b-tls-policy/example-ca # same example-ca content
sni: backend-b.example.com
```

Both clusters parse `example-ca` into their own `X509_STORE`, and each also parses the shared client cert into its own `SSL_CTX`. Because clusters are created at route-rule granularity, even multiple routes to the *same* backend produce separate clusters, each re-parsing the CA — so the number of parsed trust stores tracks the number of clusters, not the number of backends or CAs.

## Evidence

Heap profiling (`/heap_dump`, symbolized) of a production gateway shows the large majority (~90%) of the Envoy data-plane heap allocated under upstream TLS validation:

```
Ssl::ClientContextImpl -> ContextImpl -> DefaultCertValidator
-> BoringSSL PEM_X509_INFO_read / x_x509 / asn1 parsing -> malloc
```

Lowering worker `concurrency` does **not** reduce this memory, confirming the trust stores are per-cluster (shared across workers), not per-worker state.

## Impact

Deployments with many backends behind a shared corporate CA can consume multiple GB of Envoy memory at idle, scaling with `cluster count × CA bundle size`.

## Proposed fixes

- **Envoy Gateway (complementary):** deduplicate clusters by `(backend, upstream-TLS-config)` so multiple route-rules to the same backend share one cluster, and content-address the CA SDS secret. These reduce the amplification (route-rules → distinct backends, and raw byte copies), but not the per-distinct-backend parsed stores.
- **Envoy (core fix):** a content-hashed CA trust-store cache so contexts built from the same CA material share one parsed `X509_STORE` (extend the existing `CrlCache` pattern to CA certificates).

## Environment

- Envoy Gateway: v1.8.1
- Envoy: v1.38.1

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Envoy Gateway's route-rule cluster generation and Envoy's ContextManagerImpl::createSslClientContext(), the entry point named in the issue. The proposed scope is unresolved between Gateway cluster deduplication and an Envoy core trust-store cache, so first determine which project should own the change. Done requires an agreed design and evidence that identical upstream TLS material no longer creates redundant parsed stores.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend-api-design, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.