Make OpenTelemetry usable for Pulsar monitoring by creating example Grafana dashboards
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
The current Apache Pulsar Helmchart loads dashboards from https://github.com/lhotari/pulsar-grafana-dashboards. That's a project that I have created by pulling in dashboards from various sources and cleaned them up to make them usable. These dashboards should be converted to be used together with Pulsar OpenTelemetry metrics. The revisited dashboards could be stored directly in the Apache Pulsar Helmchart project for further maintenance.
Apache Pulsar Helmchart currently uses VictoriaMetrics which [natively supports OTLP ingestion](https://docs.victoriametrics.com/victoriametrics/integrations/opentelemetry/). Pulsar and BookKeeper should be configured to push OpenTelemetry metrics directly to the OTLP metrics endpoint.
Besides making the changes, OpenTelemetry should be documented in the Pulsar documentation as the preferred option for metrics. Obviously many customers will continue to use Prometheus.
It would be useful if we could get rid of the duplicate code in Pulsar to expose both Prometheus metrics and OpenTelemetry metrics. However, exposing Prometheus metrics in a different way could be a performance challenge since in large deployments, the uncompressed size of metrics could exceed 500MB. The current /metrics endpoint in Pulsar is heavily optimized for high volumes of metrics.
The metric names would break also when using OpenTelemetry Prometheus exporter, so it would be a major breaking change to remove the existing native Prometheus metrics support in Pulsar.
Contributor guide
Research direction
Start by reviewing how the Apache Pulsar Helmchart currently loads dashboards from the linked pulsar-grafana-dashboards project, then read the VictoriaMetrics OTLP metrics integration. The work is done when revised dashboards use Pulsar OpenTelemetry metrics, Pulsar and BookKeeper push to the OTLP endpoint, and Pulsar documentation identifies OpenTelemetry as the preferred metrics option while covering Prometheus compatibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grafana, helm, prometheus
- Domain
- devops, distributed-systems, documentation, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100