envoyproxy / envoyproxy/gateway

feat: Metadata - add backend identity to lb_endpoints for use in access logs

Open
#9,408 1 comment 0 reactions 0 assignees View on GitHub
stale triage
Dominant language
Go
Stars
3k
Forks
864
Avg merge
2d 2h
Merged PRs (30d)
140

Description

### Description

Envoy Gateway already attaches filter_metadata to routes and clusters to support backend attribution in observability tooling (introduced in #3602). Extending this same mechanism to per-endpoint (lb_endpoint) metadata would let us identify the specific upstream Kubernetes Service an endpoint belongs to, improving how precisely we can attribute individual requests to a backend.

At present we are using the following access log configuration:

```yaml
namespace_extracted: "%CEL(xds.route_metadata.filter_metadata['envoy-gateway']['resources'][0]['namespace'])%"
upstream_service: "%CEL(xds.cluster_metadata.filter_metadata['envoy-gateway']['resources'][0]['name'])%"
```

However, this is unreliable for identifying the actual backend that served a request due to:

- **Weighted/multiple backendRefs**: an HTTPRoute rule can reference several backendRefs
in one cluster. Cluster metadata only exposes `resources[0]`, i.e. always the *first*
configured backend — not necessarily the one the load balancer picked for a given request.
- **Cross-namespace backendRefs**: Gateway API explicitly allows a backendRef to live in a
different namespace than the HTTPRoute (via ReferenceGrant). Our current namespace
extraction pulls it from route metadata, which reflects the HTTPRoute's namespace, not
the backend's — so cross-namespace routing produces an incorrect namespace in the log.

Dumping the load balancer endpoints for one httproute on a dev cluster:

```
$ curl -s 'http://localhost:19000/config_dump?resource=dynamic_endpoint_configs&include_eds' \
| jq '.configs[] | select(.endpoint_config.cluster_name | contains("cms-api-gateway"))
| .endpoint_config.endpoints[].lb_endpoints[] | {address: .endpoint.address, metadata}'
{
"address": {
"socket_address": {
"address": "10.110.73.48",
"port_value": 8080
}
},
"metadata": null
}
{
"address": {
"socket_address": {
"address": "10.110.35.180",
"port_value": 8080
}
},
"metadata": null
}
...
```

We can see that envoy's data plane and xDS schema (`envoy.config.endpoint.v3.LbEndpoint.metadata`) already
support this, it's just never populated by Envoy Gateway's translator

### Proposal

Populate the existing filter_metadata field on each lb_endpoint with a backend struct carrying the identity of the corresponding backendRef:

```yaml
metadata:
filterMetadata:
envoy-gateway:
backend:
kind: "Service"
name: "my-service"
namespace: "default"
section: "8080"
```

This makes the fields addressable in access log format strings, e.g:
```
%METADATA(UPSTREAM_HOST:envoy-gateway:backend:name)%
%METADATA(UPSTREAM_HOST:envoy-gateway:backend:namespace)%
%METADATA(UPSTREAM_HOST:envoy-gateway:backend:section)%
```

The backend key name aligns with Gateway API's backendRef terminology and is consistent with how EG names things internally.

### Relevant Links:
* [https://gateway.envoyproxy.io/community/design/metadata/](https://gateway.envoyproxy.io/community/design/metadata/)
* [https://gateway.envoyproxy.io/community/design/backend/](https://gateway.envoyproxy.io/community/design/backend/)
* [https://github.com/envoyproxy/gateway/pull/3602](https://github.com/envoyproxy/gateway/pull/3602)
* [https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage](https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating Envoy Gateway’s translator that builds lb_endpoints, then trace how each endpoint maps to a backendRef and review the existing filter_metadata handling introduced in #3602. Populate each endpoint with the proposed envoy-gateway/backend identity and verify it works for weighted and cross-namespace backends.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.