how to avoid frequent invalid balance
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
load balance config :
```
loadBalancerLoadSheddingStrategy=org.apache.pulsar.broker.loadbalance.impl.ThresholdShedder
loadBalancerHistoryResourcePercentage=0.9
loadBalancerBandwithInResourceWeight=1.0
loadBalancerBandwithOutResourceWeight=1.0
loadBalancerCPUResourceWeight=0.0
loadBalancerMemoryResourceWeight=0.0
loadBalancerDirectMemoryResourceWeight=0.0
```
We configure load balancing based on network traffic, but when the shedder triggers a broker to unload the bundle, the bundle is allocated back to the original broker within a short period of time. This ineffective load balancing will cause traffic fluctuations. some log like:
> 2023-12-11 12:02:41.743 INFO org.apache.pulsar.broker.namespace.OwnedBundle handleUnloadRequest 192.168.31.1 Disabling ownership: chenjianbo-tenant/chenjianbo-ns/0x6444443e_0x66666660
> 2023-12-11 12:02:41.855 INFO org.apache.pulsar.broker.namespace.OwnershipCache lambda$tryAcquiringOwnership$1 192.168.31.1 Successfully acquired ownership of OwnedBundle(bundle=chenjianbo-tenant/chenjianbo-ns/0x6444443e_0x66666660, isActive=1)
### Solution
During debugging, we found that the `LoadManagerShared.removeMostServicingBrokersForNamespace` method filtered out all other brokers due to the number of bundles, leaving only the broker with the unloaded bundle. we test that removing this filtering method is a good strategy, because If a broker with a smaller number of bundles unloads a bundle, the broker will be assigned back to the bundle, causing frequent balancing
```
LoadManagerShared.removeMostServicingBrokersForNamespace(serviceUnit.toString(), brokerCandidateCache,
brokerToNamespaceToBundleRange);
```
Why is this strategy needed? Can we add a switch to configure it to be turned off?
### Alternatives
_No response_
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Contributor guide
Research direction
Start with LoadManagerShared.removeMostServicingBrokersForNamespace and the listed load-balancer configuration properties. Reproduce the unload and immediate reacquisition sequence using the log pattern in the issue, then determine whether the filtering needs configurable behavior. Done means the intended behavior and configuration option are validated against the frequent rebalancing case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100