envoyproxy / envoyproxy/gateway

Flapping: TLS error: Secret is not supplied by SDS

Open
#6,619 3 comments 0 reactions 0 assignees View on GitHub
stale triage
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

*Description*:
I am testing envoy gateway in a k3s multinode setup (v1.32.6+k3s1) with two envoy gateways (internal and external running gateway:v1.4.2 and envoyproxy/envoy:distroless-v1.34.1).
The purpose of the two gateways is to implement internal and external access with different SecurityPolicy's objects applied. Therefore I have 2 HTTPRoutes which only differ in the gateway those are attached (parentRefs.name) and I have two different SecurityPolicys. The HTTPRoutes route towards the same https upstream/backend k8s service. I have one BackendTLSPolicy in place which is attached to the https backend service showing this validation (a trust to the root ca and the SAN/hostname):

```
validation:
hostname: somthing.fqdn
caCertificateRefs:
- name: clusterissuer-root-ca01
group: ''
kind: ConfigMap
```

The config map equals this setup (and contains only the root ca certificate, the signer of the upstream/backend certificate):
kubectl create configmap clusterissuer-root-ca01 --from-file=ca.crt -n namespace

The setup works fine until it begins to flap (without any change to the HTTPRoute, BackendTLSPolicy nor the clusterissuer-root-ca01 configmap or the services etc.) and both https route return with http status code 503 and error:
* upstream connect error or disconnect/reset before headers. reset reason: remote connection failure,
transport failure reason: TLS error: Secret is not supplied by SDS

If I check the generated envoy configuration I can see that the secret (ca root trust) was moved to the dynamic_warming_secrets and remains with "version_info": "uninitialized" sometimes for approx 5min but sometimes even for hours.

```
"dynamic_warming_secrets": [
{
"name": "internal/namespace-ca",
"version_info": "uninitialized",
"last_updated": "2025-07-28T16:51:15.106Z",
"secret": {
"@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
"name": "internal/namespace-ca"
}
}
]
```
and not as expected:
```
"dynamic_active_secrets": [
...
{
"name": "internal/namespace-ca",
"version_info": "086a249165bd6d0c6883a81c8fef598a5265a3af37bc657faec1c8eaa067cbbb",
"last_updated": "2025-06-18T18:02:36.577Z",
"secret": {
"@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
"name": "internal/namespace-ca",
"validation_context": {
"trusted_ca": {
"inline_bytes": "omitted_base64(pem of ca root cert)"
}
}
}
}
]
```

*Repro steps*:
Unkown so far, but the flapping occurs daily.

*Environment*:
* v1.32.6+k3s1
* gateway:v1.4.2
* envoyproxy/envoy:distroless-v1.34.1

*Logs*:
debug mode of gateway show no errors

How may I debug further or what component will take care to complete the dynamic_warming_secrets - is it gateway or envoyproxy?
/martin

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.