apache / apache/pulsar

[Bug] Duplicate metrics with same timestamp cause Prometheus ingestion errors

Open
#24,322 2 comments 0 reactions 0 assignees View on GitHub
type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before reporting

- [x] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Read release policy

- [x] I understand that [unsupported versions](https://pulsar.apache.org/contribute/release-policy/#supported-versions) don't get bug fixes. I will attempt to reproduce the issue on a supported version of Pulsar client and Pulsar broker.

### User environment

Broker version: 4.0.4
Broker Operating system and hardware type: Linux pulsar-preprod-broker-0 6.1.134-152.225.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Wed May 7 09:10:59 UTC 2025 x86_64 GNU/Linux
Broker Java version: openjdk version "21.0.6" 2025-01-21 LTS
OpenJDK Runtime Environment Corretto-21.0.6.7.1 (build 21.0.6+7-LTS)
OpenJDK 64-Bit Server VM Corretto-21.0.6.7.1 (build 21.0.6+7-LTS, mixed mode)

### Issue Description

#### What happened?

We are using Prometheus to scrape Pulsar broker metrics (via `/metrics` on port 8080). Prometheus regularly logs the following warning:

```
level=WARN msg="Error on ingesting samples with different value but same timestamp"
```

Upon inspecting the `/metrics` endpoint, we noticed that some metrics (e.g., `pulsar_out_bytes_total`) appear multiple times with the **same timestamp and identical labels**, but with **different values**. This causes Prometheus to drop those samples.

Example output from `/metrics`:
```
pulsar_out_bytes_total{cluster="pulsar-prod",namespace="N",topic="T",partition="-1",subscription="S",consumer_name="",consumer_id="10"} 26067
pulsar_out_bytes_total{cluster="pulsar-prod",namespace="N",topic="T",partition="-1",subscription="S",consumer_name="",consumer_id="10"} 34955
```

#### What were you trying to do?

Collect Pulsar broker metrics in Prometheus for observability.

#### What did you expect to happen?

Each metric should appear only **once** per timestamp and label set, as required by Prometheus exposition format.

#### What actually happened instead?

Some metrics appear multiple times in the same scrape with the same labels and timestamp, but different values, causing ingestion errors in Prometheus and dropped samples.

#### Why do you believe this is a bug?

The Prometheus exposition format does **not support multiple values** for the same metric (same name + labels) and timestamp. This leads to errors in Prometheus and loss of observability data. This suggests that the broker metrics exporter is generating duplicated time series entries incorrectly.

### Error messages

```text
time=2025-05-20T13:14:34.901Z level=WARN source=scrape.go:1884 msg="Error on ingesting samples with different value but same timestamp" component="scrape manager" scrape_pool=podMonitor/pulsar-prod/pulsar-prod-broker/0 target=http://10.1.28.2:8080/metrics num_dropped=5832
```

### Reproducing the issue

1. Deploy Pulsar in a Kubernetes cluster with Prometheus scraping the `/metrics` endpoint on port 8080 (default broker metrics).
2. Ensure Prometheus is configured with `podMonitor` (or similar) to scrape Pulsar brokers.
3. Observe repeated logs in Prometheus:
```
Error on ingesting samples with different value but same timestamp
```
4. Manually request `/metrics` from a broker, and observe duplicate lines for some metrics with same labels and different values.

We reproduced this issue on:
- Pulsar 4.0.4 (broker in Kubernetes)
- Prometheus 3.3.0

We haven’t tried to reproduce this with a standalone broker, but the issue appears to be with the way metrics are emitted when multiple consumers or threads update the same metric concurrently.

### Additional information

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start at the Pulsar broker `/metrics` endpoint and the metrics emission path, reproducing the duplicate `pulsar_out_bytes_total` lines under concurrent consumer or thread updates. Done means each metric has one value per timestamp and label set, and Prometheus no longer reports ingestion errors during the documented Kubernetes scrape scenario.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes, prometheus
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.