Feign metrics are incompatible with RestTemplate metrics when using Prometheus

Describe the bug
When a spring boot/cloud project has observations, metrics and prometheus enabled with Feign, the http_client_requests_active metric registered by Feign is not compatible with the one registered by notably RestTemplate (maybe others). This is a problem as soon as you have for exemplle a DiscoveryClient like EurekaDiscoveryClient that uses RestTemplates under the hood.

This results in the following warning when first calling a method of the FeignClient :

The meter (MeterId{name='http.client.requests.active', tags=[tag(clientName=com.example.demo.TestFeignClient),tag(http.method=GET),tag(http.status_code=CLIENT_ERROR),tag(http.url=/outside/test)]}) registration has failed: Prometheus requires that all meters with the same name have the same set of tag keys. There is already an existing meter named 'http_client_requests_active_seconds' containing tag keys [client_name, exception, method, outcome, status, uri]. The meter you are attempting to register has keys [clientName, http_method, http_status_code, http_url]. Note that subsequent logs will be logged at debug level.

Consequently, et more importantly, the feign metrics are not exported by prometheus.

The metrics are effectively not compatible :

RestTemplate http_client_requests_active_seconds_count :
http_client_requests_active_seconds_count{client_name="localhost",exception="none",method="GET",outcome="UNKNOWN",status="CLIENT_ERROR",uri="/test"} 0
http_client_requests_active_seconds_sum{client_name="localhost",exception="none",method="GET",outcome="UNKNOWN",status="CLIENT_ERROR",uri="/test"} 0.0

FeignClient http_client_requests_active_seconds_count : 
http_client_requests_active_seconds_count{clientName="com.example.demo.TestFeignClient",http_method="GET",http_status_code="CLIENT_ERROR",http_url="/outside/test"} 0
http_client_requests_active_seconds_sum{clientName="com.example.demo.TestFeignClient",http_method="GET",http_status_code="CLIENT_ERROR",http_url="/outside/test"} 0.0

The FeignClient metric tags should probly look like this instead :

http_client_requests_active_seconds_count{client_name="com.example.demo.TestFeignClient",method="GET",status="CLIENT_ERROR",uri="/outside/test"} 0
http_client_requests_active_seconds_sum{client_name="com.example.demo.TestFeignClient",method="GET",status="CLIENT_ERROR",uri="/outside/test"} 0.0

Sample
I have a simple repro here

Just run the tests and they will fail. If you replace the restTemplateBuilder.build() by new RestTemplate() the test passes. Demonstrating that the problem effectively comes from the restTemplate being observerd.

Below the classes in the project :

@SpringBootApplication
@RestController
@EnableFeignClients
@RequiredArgsConstructor
public class TestApplication {

    private final TestFeignClient testFeignClient;

    @GetMapping("/test")
    public String test() {
        return testFeignClient.test();
    }

    public static void main(String[] args) {
        SpringApplication.run(TestApplication.class, args);
    }
}

@FeignClient(value = "TestFeignClient", name = "TestFeignClient", url = "http://localhost:9004")
public interface TestFeignClient {

    @GetMapping("/outside/test")
    String test();

}

@SpringBootTest(classes = TestApplication.class, webEnvironment = WebEnvironment.DEFINED_PORT, properties = {
        "spring.application.name=demo",
        "server.port=18080",
        "management.tracing.enabled=true",
        "management.prometheus.metrics.export.enabled=true",
        "management.endpoints.web.exposure.include=prometheus"
})
@WireMockTest(httpPort = 9004)
class DemoTestApplicationTests {

    @Autowired
    private RestTemplateBuilder restTemplateBuilder;

    @Test
    void repro() {
        WireMock.stubFor(get(urlPathEqualTo("/outside/test")) //
                .willReturn(aResponse() //
                        .withStatus(200) //
                        .withHeader(HttpHeaders.CONTENT_TYPE, MediaType.APPLICATION_JSON_VALUE) //
                        .withBody("hello world")));

        RestTemplate restTemplate = restTemplateBuilder.build(); // replace by "new RestTemplate()" for test to pass
        String response = restTemplate.getForObject("http://localhost:18080/test", String.class);
        assertAll(
                () -> assertThat(response).isEqualTo("hello world"),
                () -> assertThat(logAppender.list)
                        .extracting(ILoggingEvent::getMessage)
                        .noneMatch(msg -> msg.contains("registration has failed")),
                () -> assertThat(restTemplate.getForObject("http://localhost:18080/actuator/prometheus", String.class))
                        .contains("http_client_requests_active_seconds_count{clientName=\"com.example.demo.TestFeignClient")
        );
    }

    // setup of log appender to capture logs

    private ListAppender<ILoggingEvent> logAppender;

    private Logger logger;

    @BeforeEach
    void setUp() {
        logger = (Logger) LoggerFactory.getLogger(io.micrometer.prometheusmetrics.PrometheusMeterRegistry.class);
        logAppender = new ListAppender<>();
        logAppender.start();
        logger.addAppender(logAppender);
    }

    @AfterEach
    void tearDown() {
        logger.detachAppender(logAppender);
    }

}

Start with the linked reproduction's DemoTestApplicationTests and inspect how Feign and RestTemplate metrics are registered under the Prometheus configuration. The fix is complete when the Feign metric uses compatible tag keys and the test produces no registration warning while exposing the metric through the actuator Prometheus endpoint.