envoyproxy / envoyproxy/gateway

sessionPersistence is not working when using Backend mTLS with a Service FQDN

Open
#8,816 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

*Description*:
When an HTTPRoute uses `sessionPersistence` with a direct Kubernetes Service backend, Envoy Gateway routes through EndpointSlice pod endpoints. The persistence cookie is tied to the selected pod endpoint, for example `pod_ip:port`, and repeated requests stay on the same pod.

When the same Service is referenced through a `gateway.envoyproxy.io/Backend` using a Kubernetes Service DNS name, for example `tls-backend.default.svc.cluster.local`, Envoy Gateway treats the Backend endpoint as a normal FQDN endpoint. In practice this produces a cookie/backend identity based on the Service DNS/ClusterIP path, for example `cluster_ip:port`, and traffic can land on different pods across requests.

This makes session persistence behave differently for two configurations that both logically target the same Service:

1. Plain backend TLS:
- HTTPRoute backendRef points directly to `Service/tls-backend`.
- `BackendTLSPolicy` configures backend TLS.
- Session persistence uses pod endpoints.

2. Backend mTLS:
- HTTPRoute backendRef points to `Backend/tls-backend-client-cert`.
- `Backend.spec.endpoints[].fqdn.hostname` is `tls-backend.default.svc.cluster.local`.
- Backend configures client certificate and CA references.
- Session persistence uses the Service DNS/ClusterIP path and is not sticky to a pod.

*Repro Steps*:

1. Install Envoy Gateway with the Backend API enabled.

2. Deploy a backend Service with at least two ready pods. The Service should be reachable as `tls-backend.default.svc.cluster.local` and expose port `443`.

3. Configure HTTPRoute session persistence against the Service directly:

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: sticky-service
namespace: default
spec:
parentRefs:
- name: eg
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: tls-backend
port: 443
sessionPersistence:
sessionName: Session-A
type: Cookie
absoluteTimeout: 10s
cookieConfig:
lifetimeType: Permanent
```

4. Configure backend TLS for the Service with `BackendTLSPolicy` as usual.

5. Send a request, capture the persistence cookie, then send several more requests with that cookie.

Expected and observed for this case:

- Requests stay on the same pod.
- The persistence cookie/backend identity corresponds to a pod endpoint, for example `pod_ip:port`.

6. Replace the direct Service backendRef with a `Backend` backendRef and define the Backend with an FQDN endpoint pointing at the same Service DNS name:

```yaml
apiVersion: gateway.envoyproxy.io/v1alpha1
kind: Backend
metadata:
name: tls-backend-client-cert
namespace: default
spec:
endpoints:
- fqdn:
hostname: tls-backend.default.svc.cluster.local
port: 443
tls:
clientCertificateRef:
kind: Secret
name: example-client-cert
caCertificateRefs:
- group: ""
kind: ConfigMap
name: example-ca
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: sticky-backend
namespace: default
spec:
parentRefs:
- name: eg
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- group: gateway.envoyproxy.io
kind: Backend
name: tls-backend-client-cert
sessionPersistence:
sessionName: Session-A
type: Cookie
absoluteTimeout: 10s
cookieConfig:
lifetimeType: Permanent
```

7. Send the same request sequence with the persistence cookie.

Actual result:

- The cookie/backend identity is based on the Service DNS/ClusterIP path, for example `cluster_ip:port`.
- Requests can hit different backend pods even when the same persistence cookie is sent.

Expected result:

- Because the Backend FQDN names an in-cluster Service, Envoy Gateway should produce endpoint-routed configuration using the Service's EndpointSlices, preserving mTLS settings from the Backend.
- The sticky session behavior should match the direct Service + BackendTLSPolicy case.

*Environment*:

- Envoy Gateway version: `v1.7.1`
- Envoy version: `v1.37.1`
- Kubernetes version/distribution: `v1.34.5`
- Backend API enabled: `yes`

*Logs*:

No specific logs are generated.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.