apache / apache/pulsar

Unload Rate Limiting during Graceful Shutdown

Open
#20,753 1 comment 0 reactions 0 assignees View on GitHub
Stale type/enhancement
Dominant language
Java
Stars
15.3k
Forks
3.8k
Avg merge
1d 14h
Merged PRs (30d)
160

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.

### Motivation

When Pulsar undergoes a graceful shutdown, it unloads all topics, but there is no limit on the unloading rate. As a result, during a rolling upgrade, the broker cluster might generate a large volume of lookup requests in a short period, which could affect the service of the broker. Implementing a limit on the unload rate would make the upgrade process smoother.

### Solution

This issue has been previously raised in a pull request https://github.com/apache/pulsar/pull/14114 and a solution was proposed by adding an unload interface with concurrency control for admin CLI.

However, the current broker graceful shutdown doesn't use this interface. Hence, my proposal is pretty straightforward:

1. Add a **dynamic** config named `brokerShutdownMaxBundleUnloadPerMinute` to the Pulsar broker to control the concurrency of unloading.

2. Modify [this code](https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java#L557) to use [this function](https://github.com/apache/pulsar/blob/7636e8989f4d3fc24fce69a356d54e1c550945ed/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L790-L791) with the concurrency parameter for unloading.

### Alternatives

_No response_

### Anything else?

mail list: https://lists.apache.org/thread/7f5f9gn7qq86hstrkw5oz51g0rxrqq37

### Are you willing to submit a PR?

- [X] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Research direction

Start at pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java near line 557 and pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java near lines 790-791, then trace the broker graceful-shutdown configuration handling. Done means graceful shutdown uses the dynamic brokerShutdownMaxBundleUnloadPerMinute setting to limit bundle unloading.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.