apache / apache/pulsar

how to avoid frequent invalid balance

Open
#21,722 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

load balance config :
```
loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder
loadBalancerHistoryResourcePercentage=0.9
loadBalancerBandwithInResourceWeight=1.0
loadBalancerBandwithOutResourceWeight=1.0
loadBalancerCPUResourceWeight=0.0
loadBalancerMemoryResourceWeight=0.0
loadBalancerDirectMemoryResourceWeight=0.0
```
We configure load balancing based on network traffic, but when the shedder triggers a broker to unload the bundle, the bundle is allocated back to the original broker within a short period of time. This ineffective load balancing will cause traffic fluctuations. some log like:

> 2023-12-11 12:02:41.743 INFO org.apache.pulsar.broker.namespace.OwnedBundle handleUnloadRequest 192.168.31.1 Disabling ownership: chenjianbo-tenant/chenjianbo-ns/0x6444443e_0x66666660
> 2023-12-11 12:02:41.855 INFO org.apache.pulsar.broker.namespace.OwnershipCache lambda$tryAcquiringOwnership$1 192.168.31.1 Successfully acquired ownership of OwnedBundle(bundle=chenjianbo-tenant/chenjianbo-ns/0x6444443e_0x66666660, isActive=1)

### Solution

During debugging, we found that the `LoadManagerShared.removeMostServicingBrokersForNamespace` method filtered out all other brokers due to the number of bundles, leaving only the broker with the unloaded bundle. we test that removing this filtering method is a good strategy, because If a broker with a smaller number of bundles unloads a bundle, the broker will be assigned back to the bundle, causing frequent balancing

```
LoadManagerShared.removeMostServicingBrokersForNamespace(serviceUnit.toString(), brokerCandidateCache,
brokerToNamespaceToBundleRange);
```

Why is this strategy needed? Can we add a switch to configure it to be turned off?
### Alternatives

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start with LoadManagerShared.removeMostServicingBrokersForNamespace and the listed load-balancer configuration properties. Reproduce the unload and immediate reacquisition sequence using the log pattern in the issue, then determine whether the filtering needs configurable behavior. Done means the intended behavior and configuration option are validated against the frequent rebalancing case.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.