Canary Analysis with Argo Rollouts and Envoy Proxy Metrics

*Description*:
As part of our team strategy to migrate off of the recently retired NGinx Ingress, we are moving to use Envoy Gateway and integrating it with our Argo Rollouts based Canary releases using Argo's [gateway-api-plugin](https://rollouts-plugin-trafficrouter-gatewayapi.readthedocs.io/en/latest/quick-start/). One of the requirements of the plugins is to configure two `backendRefs` so that Argo can adjust weights as the canary rolls out. As part of our canary analysis, we check for the success rate and latency of the canary as compared to the stable rollout. With ingress, we were able to filter by the k8s Service endpoint. With envoy gateway/proxy, however, it looks like the service filter on metrics like `envoy_cluster_upstream_rq_total` refers to the HTTPRoute rather than the k8s service, which in the gateway-api-plugin case will always be the same. It also seems like we cannot filter by a specific `backendRef.` So we have a few of questions:

* Is our observation correct, that the service label on metrics like `envoy_cluster_upstream_rq_total` will always refer to the HTTPRoute rather than a k8s Service?
* If the service label cannot then be used to filter by the k8s service, what would be the most reliable way to filter by backendRef?
* Alternative, we did see that there is some kind of canary feature, where upstream requests containing a `x-envoy-upstream-canary` header will have separate metrics. Would it make sense to use this feature for this use case were we could add a HTTPRoute response rewriter that adds the header?