[PIP-214][broker]Add broker level metrics statistics and expose to prometheus
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
Currently, pulsar does not statistics broker level metrics, and all relevant metrics are 0 by default.
When the number of topic partitions reaches more than 100000 or even millions, and the topic level metrics are exposed (exposeTopicLevelMetricsInPrometheus=true), if you want to query the metrics of the broker dimension, you need to summarize all topics, and the performance becomes very poor, or even the results cannot be queried from the promethus. Common broker level metrics include:
```
pulsar_topics_count
pulsar_subscriptions_count
pulsar_producers_count
pulsar_consumers_count
pulsar_rate_in
pulsar_rate_out
pulsar_throughput_in
pulsar_throughput_out
pulsar_storage_size
pulsar_storage_logical_size
pulsar_storage_write_rate
pulsar_storage_read_rate
pulsar_msg_backlog
```
We need to statistics the metrics of the broker dimension and expose them to prometheus to improve the performance of the monitoring of the broker dimension. Modify the original metrics name as follows:
```
pulsar_broker_topics_count
pulsar_broker_subscriptions_count
pulsar_broker_producers_count
pulsar_broker_consumers_count
pulsar_broker_rate_in
pulsar_broker_rate_out
pulsar_broker_throughput_in
pulsar_broker_throughput_out
pulsar_broker_storage_size
pulsar_broker_storage_logical_size
pulsar_broker_storage_write_rate
pulsar_broker_storage_read_rate
pulsar_broker_msg_backlog
```
### Solution
_No response_
### Alternatives
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
PR:https://github.com/apache/pulsar/pull/18116
Contributor guide
Research direction
The issue names no files or tests and links PR 18116, which should be reviewed first. Trace the existing broker-level Prometheus metrics path and compare it with the listed broker metric names; done means those broker statistics are exposed with the requested names and monitoring performance improves.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, prometheus
- Domain
- backend, observability-sre
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100