envoyproxy / envoyproxy/gateway

Dot in HTTPRoute name produces duplicate envoy_cluster_total_match_count Prometheus series

Open
#9,576 6 comments 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

*Description*:

When an `HTTPRoute`'s name contains a dot (a valid Kubernetes name, e.g. `echo.a.example.com`),
the Envoy proxy emits `envoy_cluster_total_match_count` on `/stats/prometheus` with a **truncated
`envoy_cluster_name` label** (cut at the first dot). When two clusters share the same
before-the-first-dot prefix — two HTTPRoutes whose names differ only after a dot, or one dotted
HTTPRoute with several rules — **the same series (identical metric name and labels) is emitted
once per cluster in a single scrape response**:

```
envoy_cluster_total_match_count{socket_match_name="default",envoy_cluster_name="httproute/default/echo"} 6
envoy_cluster_total_match_count{socket_match_name="default",envoy_cluster_name="httproute/default/echo"} 1
```

Once the two values diverge (they are distinct counters), Prometheus rejects the samples —
*"Error on ingesting samples with different value but same timestamp"* — incrementing
`prometheus_target_scrapes_sample_duplicate_timestamp_total` and firing the kube-prometheus-stack
`PrometheusDuplicateTimestamps` alert.

**Expected:** each `{__name__, labels}` series appears at most once per response, and
`envoy_cluster_name` carries the full cluster name.

See "Possible root cause" at the bottom.

*Repro steps*:

Stock Envoy Gateway v1.8.2 on kind — no EnvoyProxy customization:

```bash
kind create cluster --name eg-dup
helm install eg oci://docker.io/envoyproxy/gateway-helm --version v1.8.2 \
-n envoy-gateway-system --create-namespace
kubectl wait --timeout=300s -n envoy-gateway-system deployment/envoy-gateway --for=condition=Available
kubectl apply -f repro.yaml # manifests below
kubectl wait --timeout=120s -n default deployment/echo --for=condition=Available

POD=$(kubectl -n envoy-gateway-system get pods -l gateway.envoyproxy.io/owning-gateway-name=eg -o name | head -1)
kubectl -n envoy-gateway-system port-forward "$POD" 19001:19001 19000:19000 &
sleep 2
```

Output (deterministic, no traffic needed for the duplication itself).
The two clusters hold two distinct counters:

```
curl -s 'localhost:19000/stats?filter=total_match_count' | grep httproute

cluster.httproute/default/echo.a.example.com/rule/0.default.total_match_count: 1
cluster.httproute/default/echo.b.example.com/rule/0.default.total_match_count: 1
```

but on `/stats/prometheus` both render as the same series, emitted twice:

```
curl -s localhost:19001/stats/prometheus | grep 'envoy_cluster_total_match_count{' | grep httproute

envoy_cluster_total_match_count{socket_match_name="default",envoy_cluster_name="httproute/default/echo"} 1
envoy_cluster_total_match_count{socket_match_name="default",envoy_cluster_name="httproute/default/echo"} 1
```

Being distinct counters, their values diverge as soon as the clusters open different numbers of
upstream connections — at which point Prometheus starts rejecting the samples (see Logs).

`repro.yaml` — one Gateway, one echo backend, and **two HTTPRoutes whose names contain dots and
share the prefix before the first dot**:

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: eg
spec:
controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: eg
namespace: default
spec:
gatewayClassName: eg
listeners:
- name: http
protocol: HTTP
port: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: echo
namespace: default
spec:
replicas: 1
selector:
matchLabels: {app: echo}
template:
metadata:
labels: {app: echo}
spec:
containers:
- name: echo
image: gcr.io/k8s-staging-gateway-api/echo-basic:v20231214-v1.0.0-140-gf544a46e
ports: [{containerPort: 3000}]
---
apiVersion: v1
kind: Service
metadata:
name: echo
namespace: default
spec:
selector: {app: echo}
ports: [{port: 3000, targetPort: 3000}]
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: echo.a.example.com
namespace: default
spec:
parentRefs: [{name: eg}]
hostnames: ["echo-a.example.com"]
rules:
- backendRefs: [{name: echo, port: 3000}]
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: echo.b.example.com
namespace: default
spec:
parentRefs: [{name: eg}]
hostnames: ["echo-b.example.com"]
rules:
- backendRefs: [{name: echo, port: 3000}]
```

We originally hit this in production: our controller names HTTPRoutes after service hostnames
(dots included), and every affected proxy showed the duplicated series for each such cluster,
with intermittent Prometheus scrape rejections under traffic.

*Environment*:

- Envoy Gateway v1.8.2
- Envoy `0ebfcfe5b0484b89ca85b761da9e05ce75dbda8d/1.38.3/Clean/RELEASE/BoringSSL`
- Kubernetes: kind (repro); EKS 1.35 (production)
- Prometheus via kube-prometheus-stack, scraping `/stats/prometheus` on `:19001`

*Logs*:

Prometheus scrape rejection when the duplicate copies diverge under traffic (production):

```
level=WARN source=scrape_append_v2.go:402 msg="Error on ingesting samples with different value but same timestamp" component="scrape manager" scrape_pool=serviceMonitor/envoy-gateway-system/envoy-gateway-proxy/0 target=http://:19001/stats/prometheus num_dropped=4
```

*Possible root cause*:

The bootstrap template
([`internal/xds/bootstrap/bootstrap.yaml.tpl`](https://github.com/envoyproxy/gateway/blob/main/internal/xds/bootstrap/bootstrap.yaml.tpl))
ships this stats tag:

```yaml
- regex: "^cluster(\\..+\\.(.+))\\.total_match_count$"
tag_name: socket_match_name
```

For a raw stat `cluster.httproute/default/echo.a.example.com/rule/0.default.total_match_count`,
this greedy regex removes the **entire** middle (`.httproute/default/echo.a.example.com/rule/0.default`)
from the tag-extracted name, while Envoy's default `envoy_cluster_name` extractor takes the
cluster name only **up to the first dot** (`httproute/default/echo`). Everything after the first
dot — the part that distinguishes the clusters — is discarded from both the name and the labels,
so distinct clusters produce identical series:

```
# raw admin /stats — two distinct counters:
cluster.httproute/default/echo.a.example.com/rule/0.default.total_match_count: 6
cluster.httproute/default/echo.b.example.com/rule/0.default.total_match_count: 1
```

Envoy Gateway puts the user-controlled HTTPRoute name verbatim into the cluster name
(`httproute///rule/`), so any dotted route name triggers this. Only
`total_match_count` collides into exact duplicates (it is the only cluster stat whose tag regex
strips the whole middle); other stats of dotted routes instead leak the post-dot remainder into
the metric name (e.g. `envoy_cluster_a.example.com/rule/0_upstream_rq_total`).

> Prepared with the assistance of an AI agent; the reporter has reviewed and understands the contents.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with internal/xds/bootstrap/bootstrap.yaml.tpl and inspect the stats tag configuration, then reproduce the issue using the supplied repro.yaml and /stats/prometheus endpoint. Compare the raw /stats cluster names with the rendered Prometheus series. Done means dotted HTTPRoute clusters retain distinct full names and no duplicate metric series are emitted.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, prometheus
Domain
backend, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.