envoyproxy / envoyproxy/gateway

listener hostname isolation not applied to nested wildcard listeners on a shared port

Open
#9,895 8 comments 1 reaction 1 assignee Claimed by @omkar619-dev View on GitHub
kind/bug
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

Hi Envoy Gateway maintainers — thank you for the project, and in particular for the quick turnaround on the v1.9.1 security patch. We upgraded a fleet of clusters to it the day it shipped, and it went smoothly on most of them. On one cluster we hit what looks like a regression in listener hostname isolation, and I wanted to write it up carefully in case it is useful.

I have confirmed against `SECURITY.md` and the threat model that this is not a crash and not attacker-triggerable, so I am filing it publicly rather than emailing the security list. Happy to move it if you would prefer.

*Description*:

Since v1.9.1, an HTTPRoute appears to attach to **every** listener on a port whose hostname loosely matches, rather than only the most specific one.

With `*.dev.example.com` and `*.example.com` both on port 80, a route for `test.dev.example.com` attaches to **both** (`attachedRoutes` 1/1; expected 1/0). Because both listeners share a single `RouteConfiguration`, the domain is emitted twice and Envoy rejects the whole config with `Only unique values for domains are permitted`. Every route on that port is then lost.

On HTTPS the symptom is quieter: the nested listener's `RouteConfiguration` is never emitted at all. TLS completes, the correct certificate is served, and every request returns 404 — with nothing logged. In both cases all listeners report `Programmed: True`.

Expected behaviour, per the Gateway API v1.6.1 `Listener.hostname` documentation:

> inbound request hostnames MUST match from the most specific to least specific Hostname values to choose the correct Listener and its associated set of Routes.
>
> Additionally, if there are multiple wildcard entries, more specific wildcard entries must be processed before less specific wildcard entries. For example, `"*.foo.example.com"` takes precedence over `"*.example.com"`.

v1.9.0 handles the identical Gateway correctly. We verified this on a second cluster with the same listener topology still running v1.9.0: all route configs delivered, no duplicate domains, and no xDS rejections in the controller's entire log. The affected configuration on the broken cluster had been stable for roughly eight weeks across 1.8.2, 1.8.3 and 1.9.0 — including a proxy pod restart the day before — and broke within seconds of the v1.9.1 rollout.

One thing that may be worth highlighting: **the failure is latent.** Envoy keeps its last known good config, so a cluster can look completely healthy for hours after upgrading. The damage only lands when a proxy pod restarts — which a rolling upgrade itself triggers. That made it easy to mistake for an unrelated problem.

*Repro steps*:

A Gateway with two HTTP listeners on the same port, where one hostname is nested inside the other:

```yaml
listeners:
- name: http-dev
port: 80
protocol: HTTP
hostname: "*.dev.example.com"
- name: http-apex
port: 80
protocol: HTTP
hostname: "*.example.com" # less specific, same port
```

An HTTPRoute with no `sectionName`, so attachment is decided by hostname match:

```yaml
spec:
parentRefs: [{ name: my-gateway }]
hostnames: ["test.dev.example.com"]
```

Observed:

```
v1.9.0: http-dev attachedRoutes=1, http-apex attachedRoutes=0 -> config accepted, route served
v1.9.1: http-dev attachedRoutes=1, http-apex attachedRoutes=1 -> config rejected
```

The same shape on HTTPS listeners (`*.dev.example.com` and `*.example.com` on port 443) produces the silent variant instead, where the nested listener's `RouteConfiguration` is simply absent.

A reliable way to detect it on a live cluster: compare each listener's `rds.route_config_name` in `/config_dump` against the names actually present in `RoutesConfigDump`. A referenced-but-absent config is the silent 404 case. A second tell is an inflated `attachedRoutes` — a listener claiming routes that no HTTPRoute lists it as a parent for.

*Environment*:

```
Envoy Gateway: v1.9.1 (v1.9.0 unaffected)
Envoy Proxy: distroless-v1.39.1
Gateway API: v1.6.1, standard channel
Kubernetes: v1.36.3+k3s1
Provider: Kubernetes
Install: gateway-helm chart 1.9.1
```

*Logs*:

Envoy Gateway controller:

```
error xds cache/snapshotcache.go:396 Envoy rejected the last update for type
type.googleapis.com/envoy.config.route.v3.RouteConfiguration on node
envoy-my-gateway- with code 13 and message Only unique values for domains
are permitted. Duplicate entry of domain test.dev.example.com in route
my-gateway/http; the proxy is still serving its last known good config and a
restarting proxy will fail to load this config
```

Envoy proxy (data plane):

```
[warning][config] delta_subscription_state.cc:282 delta config for
type.googleapis.com/envoy.config.route.v3.RouteConfiguration rejected:
Only unique values for domains are permitted. Duplicate entry of domain
test.dev.example.com in route my-gateway/http

[warning][config] grpc_subscription_impl.cc:138 gRPC config for
type.googleapis.com/envoy.config.route.v3.RouteConfiguration rejected:
(same message)
```

Access logs for unaffected hosts continue normally (200/302). The HTTPS variant produces no log line at all — requests simply 404 with no route matched.

*Additional context*:

I was not able to isolate the change responsible, and I would rather say so than guess. `computeHosts()` in `internal/gatewayapi/helpers.go` differs between the two tags by exactly one added guard, from #9768:

```go
if listener.hostnameConflictLoser {
continue
}
```

However, that flag is only set in `validateConflictedHostnameListeners()` when two listeners share an **identical** hostname. Our listeners have different hostnames and carry no `Conflicted` condition, so on the face of it that guard should be a no-op in this scenario — which is why I am not confident it is the cause. For completeness, the other files changed under `internal/gatewayapi/` between the tags are `translator.go` (+99/-7), `securitypolicy.go` (+127/-8), `listener.go` (+34/-9, access-log related), `globalresources.go`, `resource.go` and `runner.go`.

*Workaround*:

Replacing the less-specific wildcard with an exact hostname, so that nothing is nested, resolves both symptoms. We verified this on the affected cluster: `attachedRoutes` dropped from 5 to 1, the missing `RouteConfiguration` reappeared, and the affected host returned 200 instead of 404. A listener hostname may be narrower than the certificate it presents, so TLS coverage is unaffected.

This does not generalise, unfortunately — a genuine multi-tenant wildcard such as `*.pages.example.com` cannot be enumerated as exact hostnames. For those cases the options seem to be staying on v1.9.0 or splitting the nested zone onto its own Gateway.

Thank you for taking a look, and please let me know if there is any additional detail I can gather from the affected cluster — it is a non-production environment, so I am happy to experiment on it or try a patched build if that would help.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.