[Bug] Duplicate metrics with same timestamp cause Prometheus ingestion errors
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before reporting
- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Read release policy
- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.
### User environment
Broker version: 4.0.4
Broker Operating system and hardware type: Linux pulsar-preprod-broker-0 6.1.134-152.225.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Wed May 7 09:10:59 UTC 2025 x86_64 GNU/Linux
Broker Java version: openjdk version "21.0.6" 2025-01-21 LTS
OpenJDK Runtime Environment Corretto-21.0.6.7.1 (build 21.0.6+7-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.6.7.1 (build 21.0.6+7-LTS, mixed mode)
### Issue Description
#### What happened?
We are using Prometheus to scrape Pulsar broker metrics (via `/metrics` on port 8080). Prometheus regularly logs the following warning:
```
level=WARN msg="Error on ingesting samples with different value but same timestamp"
```
Upon inspecting the `/metrics` endpoint, we noticed that some metrics (e.g., `pulsar_out_bytes_total`) appear multiple times with the **same timestamp and identical labels**, but with **different values**. This causes Prometheus to drop those samples.
Example output from `/metrics`:
```
pulsar_out_bytes_total{cluster="pulsar-prod",namespace="N",topic="T",partition="-1",subscription="S",consumer_name="",consumer_id="10"} 26067
pulsar_out_bytes_total{cluster="pulsar-prod",namespace="N",topic="T",partition="-1",subscription="S",consumer_name="",consumer_id="10"} 34955
```
#### What were you trying to do?
Collect Pulsar broker metrics in Prometheus for observability.
#### What did you expect to happen?
Each metric should appear only **once** per timestamp and label set, as required by Prometheus exposition format.
#### What actually happened instead?
Some metrics appear multiple times in the same scrape with the same labels and timestamp, but different values, causing ingestion errors in Prometheus and dropped samples.
#### Why do you believe this is a bug?
The Prometheus exposition format does **not support multiple values** for the same metric (same name + labels) and timestamp. This leads to errors in Prometheus and loss of observability data. This suggests that the broker metrics exporter is generating duplicated time series entries incorrectly.
### Error messages
```text
time=2025-05-20T13:14:34.901Z level=WARN source=scrape.go:1884 msg="Error on ingesting samples with different value but same timestamp" component="scrape manager" scrape_pool=podMonitor/pulsar-prod/pulsar-prod-broker/0 target=http://10.1.28.2:8080/metrics num_dropped=5832
```
### Reproducing the issue
1. Deploy Pulsar in a Kubernetes cluster with Prometheus scraping the `/metrics` endpoint on port 8080 (default broker metrics).
2. Ensure Prometheus is configured with `podMonitor` (or similar) to scrape Pulsar brokers.
3. Observe repeated logs in Prometheus:
```
Error on ingesting samples with different value but same timestamp
```
4. Manually request `/metrics` from a broker, and observe duplicate lines for some metrics with same labels and different values.
We reproduced this issue on:
- Pulsar 4.0.4 (broker in Kubernetes)
- Prometheus 3.3.0
We haven’t tried to reproduce this with a standalone broker, but the issue appears to be with the way metrics are emitted when multiple consumers or threads update the same metric concurrently.
### Additional information
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start at the Pulsar broker `/metrics` endpoint and the metrics emission path, reproducing the duplicate `pulsar_out_bytes_total` lines under concurrent consumer or thread updates. Done means each metric has one value per timestamp and label set, and Prometheus no longer reports ingestion errors during the documented Kubernetes scrape scenario.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, kubernetes, prometheus
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100