envoyproxy / envoyproxy/gateway
Merge Gateway Prometheus Virtual Host metrics random assigned wrong with multiple listeners
- Lingua principale
- Go
- Stelle
- 3k
- Fork
- 864
- Merge medio
- 2g 2h
- PR unite (30g)
- 140
Descrizione
*Description*:
Prometheus Virtual Host metrics are being assigned wrongly having cases where the wrong host is being attached to the wrong vhost.
I am not sure what the correct behaviour is in some cases we just get metrics like `envoy_vhost_vcluster_upstream_rq_retry` and in the current case we get `vhost..vcluster.` which is what the envoy docs say but with the wrong virtual host name attached to the wrong virtual host cluster.
A. The envoy_vhost_vcluster_upstream_rq_retry metrics that get emitted do have labels for `envoy_virtual_cluster` and `envoy_virtual_host`
B. `vhost..vcluster.` only have the `envoy_virtual_host` label attached and no cluster label.
I hope A is the correct way as its easier to build dashboards from.
*Repro steps*:
Take a example :
```yaml
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: foo
spec:
gatewayClassName: merge-gateway
listeners:
- name: foo.com
protocol: HTTP
hostname: foo.com
port: 80
allowedRoutes:
namespaces:
from: Same
- name: bar.com
protocol: HTTP
hostname: bar.com
port: 80
allowedRoutes:
namespaces:
from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: foobar-combined
spec:
parentRefs:
- name: foo
group: gateway.networking.k8s.io
kind: Gateway
hostnames:
- foo.com
- foobar.com
rules:
- backendRefs:
- kind: Service
group: ''
name: my-svc
port: 80
weight: 1
matches:
- path:
type: PathPrefix
value: /
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: a
spec:
gatewayClassName: merge-gateway
listeners:
- name: a.foo.com
protocol: HTTP
hostname: foo.com
port: 80
allowedRoutes:
namespaces:
from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: a
spec:
parentRefs:
- name: foo
group: gateway.networking.k8s.io
kind: Gateway
hostnames:
- a.foo.com
rules:
- backendRefs:
- kind: Service
group: ''
name: my-svc
port: 80
weight: 1
matches:
- path:
type: PathPrefix
value: /
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: ab
spec:
gatewayClassName: merge-gateway
listeners:
- name: ab.foo.com
protocol: HTTP
hostname: ab.foo.com
port: 80
allowedRoutes:
namespaces:
from: Same
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: foobar-combined
spec:
parentRefs:
- name: ab
group: gateway.networking.k8s.io
kind: Gateway
hostnames:
- ab.foo.com
rules:
- backendRefs:
- kind: Service
group: ''
name: my-svc
port: 80
weight: 1
matches:
- path:
type: PathPrefix
value: /
```
Its hard to pinpoint what's going on and seems to be random in how it picks for example I got ```envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry{envoy_virtual_host="api-gateway/ab/ab"} 0```
for the metric name.
Metrics dump
```
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry counter
envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry_limit_exceeded counter
envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry_limit_exceeded{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry_overflow counter
envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry_overflow{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry_success counter
envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_retry_success{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_timeout counter
envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_timeout{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_total counter
envoy_vhost_foo_com_ab_foo_com_vcluster_ab_foo_com_upstream_rq_total{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry counter
envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry_limit_exceeded counter
envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry_limit_exceeded{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry_overflow counter
envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry_overflow{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry_success counter
envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_retry_success{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_timeout counter
envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_timeout{envoy_virtual_host="api-gateway/ab/ab"} 0
# TYPE envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_total counter
envoy_vhost_foo_com_ab_foo_com_vcluster_other_upstream_rq_total{envoy_virtual_host="api-gateway/ab/ab"} 0
```
it looks like its merging the virtual clusters.
*Environment*:
k8s 1.29.0
envoy gateway 1.0.1 Merge gateways
Guida per i contributori
Nessuna guida per i contributori indicizzata per questo repository
Valutazione
Questa issue non è ancora stata valutata.