apache / apache/pulsar

No way to disambiguate between namespace and topic Prometheus metrics

Open
#13,924 5 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

**Describe the bug**

- https://pulsar.apache.org/docs/en/next/reference-metrics/#namespace-metrics
- https://pulsar.apache.org/docs/en/next/reference-metrics/#topic-metrics

The following metrics are duplicated:

```
pulsar_subscriptions_count
pulsar_producers_count
pulsar_consumers_count
pulsar_rate_in
pulsar_rate_out
pulsar_throughput_in
pulsar_throughput_out
pulsar_storage_size
pulsar_storage_logical_size
pulsar_storage_backlog_size
pulsar_storage_offloaded_size
pulsar_storage_write_rate
pulsar_storage_read_rate
pulsar_subscription_delayed
pulsar_storage_write_latency_le_*
pulsar_entry_size_le_*
pulsar_replication_rate_in
pulsar_replication_rate_out
pulsar_replication_throughput_in
pulsar_replication_throughput_out
pulsar_replication_backlog
```

**Expected behavior**

There should be no duplicate metrics

**To Reproduce**

Steps to reproduce the behavior:

```
curl -L http://localhost:8080/metrics
```

```yaml
version: '3'

services:
pulsar:
container_name: pulsar
image: apachepulsar/pulsar:2.9.1
command:
- bash
- -c
- >
bin/apply-config-from-env-with-prefix.py BOOKKEEPER_ conf/bookkeeper.conf &&
bin/apply-config-from-env-with-prefix.py BROKER_ conf/broker.conf &&
bin/apply-config-from-env-with-prefix.py STANDALONE_ conf/standalone.conf &&
exec bin/pulsar standalone
ports:
- '6650:6650'
- '8080:8080'
environment:
- BOOKKEEPER_enableStatistics=true
- BOOKKEEPER_prometheusStatsHttpPort=8080
- BROKER_exposeTopicLevelMetricsInPrometheus=true
- BROKER_exposeConsumerLevelMetricsInPrometheus=true
- BROKER_exposeProducerLevelMetricsInPrometheus=true
- BROKER_exposeManagedLedgerMetricsInPrometheus=true
- BROKER_exposeManagedCursorMetricsInPrometheus=true
- BROKER_exposePublisherStats=true
- BROKER_exposePreciseBacklogInPrometheus=true
- BROKER_splitTopicAndPartitionLabelInPrometheus=true
- STANDALONE_exposeTopicLevelMetricsInPrometheus=true
- STANDALONE_exposeConsumerLevelMetricsInPrometheus=true
- STANDALONE_exposeProducerLevelMetricsInPrometheus=true
- STANDALONE_exposeManagedLedgerMetricsInPrometheus=true
- STANDALONE_exposeManagedCursorMetricsInPrometheus=true
- STANDALONE_exposePublisherStats=true
- STANDALONE_exposePreciseBacklogInPrometheus=true
- STANDALONE_splitTopicAndPartitionLabelInPrometheus=true
```

**Desktop (please complete the following information):**

- OS: Windows 10

Contributor guide

Open the contributing guide

Research direction

Reproduce the duplicate output with the provided Docker configuration and curl request, then compare the namespace and topic metrics described in the reference-metrics documentation. Trace the metric-generation entry points for those two scopes and determine how they can be distinguished without duplicate names. Done means the exposed metrics are unambiguous and the reference documentation reflects the result.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, prometheus
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.