apache / apache/dubbo

[Bug] Dubbo ThreadPoolRejectMetric & ErrorCodeMetric cannot be exported to Prometheus

Open
#16,148 1 comment 0 reactions 0 assignees View on GitHub
help wanted
Dominant language
Java
Stars
41.6k
Forks
26.4k
Avg merge
15h 13m
Merged PRs (30d)
4

Description

### Pre-check

- [x] I am sure that all the content I provide is in English.

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/dubbo/issues?q=is%3Aissue) and found no similar issues.

### Apache Dubbo Component

Java SDK (apache/dubbo)

### Dubbo Version

Dubbo 3.3.6
SpringBoot 3.4.13
micrometer-registry-prometheus 1.14.14

### Steps to reproduce this issue

### Description

Subclasses of `MetricsNameCountSampler`, including `ThreadRejectMetricsCountSampler` and `ErrorCodeSampler`, do not support exporting and registering metrics that are newly added during runtime to Prometheus.

Currently, Dubbo's `MetricsReporter` periodically exports metrics by polling `resetIfSamplesChanged`. When new metrics are dynamically generated, they are usually re-registered to the `CompositeMeterRegistry`.

For the Dubbo Provider's ThreadPoolExhaustedListener and error codes, the metric data is generally empty during the application startup phase.

After the application runs for a while, corresponding events are generated, and `MetricsCountSampler.inc` is called to record the relevant metrics.

However, when these new metrics are added, the `samplesChanged` field in `MetricsNameCountSampler` is not updated, which fails to trigger the re-registration of the metrics to the PrometheusRegistry.

### Steps to reproduce are as follows:
1. Create a Dubbo 3.3.6 project.
2. Adjust the configuration:
```yaml
dubbo.metrics.enable-threadpool: true
dubbo.metrics.use-global-registry: true
dubbo.protocol.threads: 2
```
3. Implement a Dubbo Service using the Triple protocol, with an internal sleep of 60 seconds.
4. After the Dubbo Provider starts successfully, **wait for 3 minutes**, then make an HTTP call to the Dubbo Service to trigger thread pool exhaustion.
5. Access the Spring Boot management endpoint `/actuator/prometheus` and search for the metric: `dubbo_thread_pool_reject_thread_count` , but it cannot be found

### What you expected to happen

Expected result:
Dynamically added metrics in MetricsNameCountSampler can be correctly exported to the PrometheusRegistry , such as dubbo_thread_pool_reject_thread_count and dubbo_error_code_count_total.

### Anything else

_No response_

### Do you have a (mini) reproduction demo?

- [ ] Yes, I have a minimal reproduction demo to help resolve this issue more effectively!

### Are you willing to submit a pull request to fix on your own?

- [ ] Yes I am willing to submit a pull request on my own!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.