apache / apache/pulsar

[Bug] Promethus metric duplication in case of using functionsWorker

Open
#20,448 2 comments 0 reactions 0 assignees View on GitHub
Stale type/bug
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Version

Ubuntu 20.04
Pulsar 2.10.4

### Minimal reproduce step

Setting up Debezium source via example doc
https://pulsar.apache.org/docs/2.10.x/io-cdc-debezium/#example-of-postgresql

- enable `functionsWorkerEnabled` in `broker.conf`
- instead of `localrun` use `pulsar-admin source create`
- check metrics duplication `curl http://127.0.0.1:8080/metrics/ | grep jvm_classes_loaded` or `curl http://127.0.0.1:8080/metrics/ | grep process_cpu_seconds_total`

### What did you expect to see?

Prometheus metrics from pulsar server are valid w/o duplication

### What did you see instead?

```
error reading metrics for "http://127.0.0.1:8080/metrics/": reading text format failed: text format parsing error in line 328:
second TYPE line for metric name "process_cpu_seconds_total", or TYPE reported after samples

error reading metrics for "http://127.0.0.1:8080/metrics/": reading text format failed: text format parsing error in line 331:
second TYPE line for metric name "jvm_classes_loaded", or TYPE reported after samples
```

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with the Debezium PostgreSQL example and the functionsWorkerEnabled setting in broker.conf, then reproduce using pulsar-admin source create rather than localrun. Inspect the /metrics/ endpoint with the provided curl commands and compare the duplicate process_cpu_seconds_total and jvm_classes_loaded output. Done means Prometheus accepts the metrics without duplicate TYPE declarations.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, postgresql, prometheus
Domain
backend, observability, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.