apache / apache/pulsar

[feature] Add Metrics to understand Load Balancing Decisions

Open
#17,860 1 comment 2 reactions 0 assignees View on GitHub
Stale
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

When pushing pulsar clusters to see maximum performance it is important to understand which metric is causing bundle splits and unloading. The metrics shared only show that the split and/or unload occurred.
1. [bundle unload metrics](../blob/master/site2/docs/reference-metrics.md#bundleunloading-metrics)
2. [bundle split metrics](../blob/master/site2/docs/reference-metrics.md#bundlesplit-metrics)

The logs give values, but it is hard to reason about these or monitor them.

### Solution

Add reason counts for load balancing: bundle splits, and unloads.

1. Bandwidth
2. CPU
3. Direct Memory
4. Memory
5. Admin

### Alternatives

Make the log files be much clearer about which metric has triggered a bundle split or unload.

### Anything else?

This is related to #16691

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start by reading the bundle unloading and bundle split metrics sections in site2/docs/reference-metrics.md, then review related issue #16691 for context. Identify the load-balancing paths that decide between bandwidth, CPU, direct memory, memory, and admin triggers. Done means reason-specific counts are exposed for bundle splits and unloads and documented with the existing metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, observability-sre
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.