Observability: Gateway client requests clash with WebClient request metrics

Describe the bug
After upgrading to spring boot 3 with gateway 4.0.6 we've noticed that our metrics for the webclient are not available anymore in prometheus

With the use of micrometer observability an introduction was done to provide the http.client.request metrics for the gateway client.
This clashes with the default http.client.request provided by the DefaultClientRequestObservationConvention from spring-web.
This was introduced in: https://github.com/spring-cloud/spring-cloud-gateway/pull/2715/files

The prometheus lowCardinalityValues defined in GatewayDocumentedObservation differ from the ones defined in the ClientHttpObservationDocumentation. GatewayDocumentedObservation has for example http.method, http.status_code and the ClientHttpObservationDocumentation has method, uri, status

This causes the metric http.client.requests be reported with different labels, ones for the Gateway WebClient metrics and one for the Spring Web configured one.

Prometheus unfortunately ignores metrics if a metric is encountered with different labels than the initially scraped ones, for context:

• https://github.com/prometheus/client_java/issues/696
• https://github.com/micrometer-metrics/micrometer/issues/877

Ideally spring cloud client request metrics should have the prefix spring.cloud.gateway for the DefaultGatewayObservationConvention.

Its not easy to override this property either since the values are reference by the singleton instance. Only two approaches currently are: shadow the ObservedRequestHttpHeadersFilter and the DefaultGatewayObservationConvention or disable observability which is not desirable.

EDIT

Looks like overriding the bean DefaultGatewayObservationConvention and overriding the name works as a workaround:

@Component
public class CloudGatewayPrefixedGatewayObservationConvention extends DefaultGatewayObservationConvention {

    @Override
    @NonNull
    public String getName() {
        return "spring.cloud.gateway.http.client.requests";
    }
}

Start by reading DefaultGatewayObservationConvention and GatewayDocumentedObservation, then compare their metric name and low-cardinality values with DefaultClientRequestObservationConvention and ClientHttpObservationDocumentation. Check how ObservedRequestHttpHeadersFilter uses these conventions. Done means gateway and WebClient metrics can coexist in Prometheus without conflicting names or label sets, while retaining observability.