apache / apache/pulsar

A newly created topic metrics do not survive after broker restarts

Open
#10,857 5 comments 0 reactions 0 assignees View on GitHub
lifecycle/stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 22h
Merged PRs (30d)
142

Description

**Describe the bug**
A newly created topic's Prometheus metrics does not survive broker restarts. The topic is being created by pulsar_admin with no traffic nor subscription. `pulsar_topics_count` is not accurate after the broker restarts.

**To Reproduce**
1. Create a persistent topic with pulsar-admin
2. Verify the topic's prometheus metrics exist under pulsar_in_bytes_total, pulsar_consumers_count, and a few others.
3. `pulsar_topics_count` is also incremented by 1
4. Restart the broker
5. After the broker has restarted, all metrics can be queried successfully.
6. However, the newly created topic's metrics are missing from the prometheus query. `pulsar_topics_count` is also reset to the number before the topic was created.

**Expected behavior**
The new created topic metrics survive the broker restarts.

**Screenshots**
n/a

**Desktop (please complete the following information):**
Tested on 2.7.2

**Additional context**
Add any other context about the problem here.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue with pulsar-admin: create a persistent topic, query pulsar_in_bytes_total, pulsar_consumers_count, and pulsar_topics_count, then restart the broker and repeat the queries. Compare the metrics before and after the restart; done means the newly created topic remains represented and pulsar_topics_count stays incremented.

Written by the indexing model from the issue text.

Assessment

Tech stack
prometheus
Domain
distributed-systems, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.