apache / apache/pulsar

Make OpenTelemetry usable for Pulsar monitoring by creating example Grafana dashboards

Open
#25,661 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

The current Apache Pulsar Helmchart loads dashboards from https://github.com/lhotari/pulsar-grafana-dashboards. That's a project that I have created by pulling in dashboards from various sources and cleaned them up to make them usable. These dashboards should be converted to be used together with Pulsar OpenTelemetry metrics. The revisited dashboards could be stored directly in the Apache Pulsar Helmchart project for further maintenance.

Apache Pulsar Helmchart currently uses VictoriaMetrics which [natively supports OTLP ingestion](https://docs.victoriametrics.com/victoriametrics/integrations/opentelemetry/). Pulsar and BookKeeper should be configured to push OpenTelemetry metrics directly to the OTLP metrics endpoint.

Besides making the changes, OpenTelemetry should be documented in the Pulsar documentation as the preferred option for metrics. Obviously many customers will continue to use Prometheus.

It would be useful if we could get rid of the duplicate code in Pulsar to expose both Prometheus metrics and OpenTelemetry metrics. However, exposing Prometheus metrics in a different way could be a performance challenge since in large deployments, the uncompressed size of metrics could exceed 500MB. The current /metrics endpoint in Pulsar is heavily optimized for high volumes of metrics.

The metric names would break also when using OpenTelemetry Prometheus exporter, so it would be a major breaking change to remove the existing native Prometheus metrics support in Pulsar.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing how the Apache Pulsar Helmchart currently loads dashboards from the linked pulsar-grafana-dashboards project, then read the VictoriaMetrics OTLP metrics integration. The work is done when revised dashboards use Pulsar OpenTelemetry metrics, Pulsar and BookKeeper push to the OTLP endpoint, and Pulsar documentation identifies OpenTelemetry as the preferred metrics option while covering Prometheus compatibility.

Written by the indexing model from the issue text.

Assessment

Tech stack
grafana, helm, prometheus
Domain
devops, distributed-systems, documentation, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.