envoyproxy / envoyproxy/gateway

Merge Gateway Prometheus Virtual Host metrics random assigned wrong with multiple listeners

Open
#3,305 1 comment 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

*Description*:
Prometheus Virtual Host metrics are being assigned wrongly having cases where the wrong host is being attached to the wrong vhost.

I am not sure what the correct behaviour is in some cases we just get metrics like `envoy_vhost_vcluster_upstream_rq_retry` and in the current case we get `vhost..vcluster.` which is what the envoy docs say but with the wrong virtual host name attached to the wrong virtual host cluster.

A. The envoy_vhost_vcluster_upstream_rq_retry metrics that get emitted do have labels for `envoy_virtual_cluster` and `envoy_virtual_host`

B. `vhost..vcluster.` only have the `envoy_virtual_host` label attached and no cluster label.

I hope A is the correct way as its easier to build dashboards from.

*Repro steps*:

Take a example :

```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: foo
spec:
gatewayClassName: merge-gateway
listeners:
- name: foo.com
protocol: HTTP
hostname: foo.com
port: 80
allowedRoutes:
namespaces:
from: Same
- name: bar.com
protocol: HTTP
hostname: bar.com
port: 80
allowedRoutes:
namespaces:
from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: foobar-combined
spec:
parentRefs:
- name: foo
group: gateway.networking.k8s.io
kind: Gateway
hostnames:
- foo.com
- foobar.com
rules:
- backendRefs:
- kind: Service
group: ''
name: my-svc
port: 80
weight: 1
matches:
- path:
type: PathPrefix
value: /
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: a
spec:
gatewayClassName: merge-gateway
listeners:
- name: a.foo.com
protocol: HTTP
hostname: foo.com
port: 80
allowedRoutes:
namespaces:
from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: a
spec:
parentRefs:
- name: foo
group: gateway.networking.k8s.io
kind: Gateway
hostnames:
- a.foo.com
rules:
- backendRefs:
- kind: Service
group: ''
name: my-svc
port: 80
weight: 1
matches:
- path:
type: PathPrefix
value: /
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: ab
spec:
gatewayClassName: merge-gateway
listeners:
- name: ab.foo.com
protocol: HTTP
hostname: ab.foo.com
port: 80
allowedRoutes:
namespaces:
from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: foobar-combined
spec:
parentRefs:
- name: ab
group: gateway.networking.k8s.io
kind: Gateway
hostnames:
- ab.foo.com
rules:
- backendRefs:
- kind: Service
group: ''
name: my-svc
port: 80
weight: 1
matches:
- path:
type: PathPrefix
value: /
```

Its hard to pinpoint what's going on and seems to be random in how it picks for example I got ```envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry{envoy_virtual_host="api-gateway/ab/ab"} 0```
for the metric name.

Metrics dump
```
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry counter
envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry_limit_exceeded counter
envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry_limit_exceeded{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry_overflow counter
envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry_overflow{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry_success counter
envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry_success{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_timeout counter
envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_timeout{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_total counter
envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_total{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry counter
envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry_limit_exceeded counter
envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry_limit_exceeded{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry_overflow counter
envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry_overflow{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry_success counter
envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry_success{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_timeout counter
envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_timeout{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_total counter
envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_total{envoy_virtual_host="api-gateway/ab/ab"} 0
```

it looks like its merging the virtual clusters.

*Environment*:
k8s 1.29.0
envoy gateway 1.0.1 Merge gateways

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.