Unload Rate Limiting during Graceful Shutdown
- Dominant language
- Java
- Stars
- 15.3k
- Forks
- 3.8k
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 160
Description
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
When Pulsar undergoes a graceful shutdown, it unloads all topics, but there is no limit on the unloading rate. As a result, during a rolling upgrade, the broker cluster might generate a large volume of lookup requests in a short period, which could affect the service of the broker. Implementing a limit on the unload rate would make the upgrade process smoother.
### Solution
This issue has been previously raised in a pull request https://github.com/apache/pulsar/pull/14114 and a solution was proposed by adding an unload interface with concurrency control for admin CLI.
However, the current broker graceful shutdown doesn't use this interface. Hence, my proposal is pretty straightforward:
1. Add a **dynamic** config named `brokerShutdownMaxBundleUnloadPerMinute` to the Pulsar broker to control the concurrency of unloading.
2. Modify [this code](https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java#L557) to use [this function](https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L790-L791) with the concurrency parameter for unloading.
### Alternatives
_No response_
### Anything else?
mail list: https://lists.apache.org/thread/7f5f9gn7qq86hstrkw5oz51g0rxrqq37
### Are you willing to submit a PR?
- [X] I'm willing to submit a PR!
Contributor guide
Research direction
Start at pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java near line 557 and pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java near lines 790-791, then trace the broker graceful-shutdown configuration handling. Done means graceful shutdown uses the dynamic brokerShutdownMaxBundleUnloadPerMinute setting to limit bundle unloading.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100