Support rate-limited session disconnection for graceful node draining
- Dominant language
- Java
- Stars
- 792
- Forks
- 95
- Avg merge
- 10h 45m
- Merged PRs (30d)
- 4
Description
**Is your feature request related to a problem? Please describe.**
When performing rolling updates on large BifroMQ clusters, operators need to drain connections from a node before taking it offline. The only available mechanism is DELETE /kill, which disconnects all matching sessions simultaneously with no rate control. On a node with hundreds of thousands of
connections, this causes a sudden reconnection storm: all devices receive TCP FIN at the same moment and immediately retry, creating a spike that can overload the remaining nodes and trigger cascading failures.
The codebase already acknowledges this gap with an explicit TODO comment at `bifromq-session-dict/bifromq-session-dict-server/src/main/java/.../SessionDictService.java:102:`
```
// TODO: support disconnect a constant rate
sessionRegistrations.forEach(reg -> reg.stop(request.getKiller(), request.getServerRedirection()));
```
**Describe the solution you'd like**
Implement the TODO: add an optional rate field to KillAllRequest that limits how many sessions are disconnected per second. A value of 0 preserves the current behavior (no rate limit) for backwards compatibility.
Proposed changes across 4 files:
SessionDictService.proto — add field to KillAllRequest:
` uint32 rate = 6; // disconnect rate per second, 0 = unlimited (default)`
SessionDictService.java — replace the forEach with a rate-limited loop:
```
// replace the TODO line with:
RateLimiter limiter = (request.getRate() > 0)
? RateLimiter.create(request.getRate()) : null;
for (ISessionRegistry.SessionRegistration reg : sessionRegistrations) {
if (limiter != null) limiter.acquire();
reg.stop(request.getKiller(), request.getServerRedirection());
}
```
ISessionDictClient.java — add rate parameter with a default method to keep backwards compatibility.
KillHandler.java — read an optional rate header and pass it through to the client call.
The resulting API call would look like:
```
# Drain a node at 500 disconnections/second
curl -X DELETE http://:8091/kill \
-H "tenant_id: " \
-H "rate: 500"
```
Since a rate-limited kill may run for a long time (e.g. 1,000,000 connections / 500 per second = 2000 seconds), the implementation should either increase the gRPC timeout accordingly or execute asynchronously and return 202 Accepted immediately.
**Describe alternatives you've considered**
- External script calling DELETE /kill per client-id one by one: Requires a separate API to list all sessions on a node, which does not currently exist. Also introduces network round-trip overhead for every individual disconnect.
- NLB weight-based draining without forced disconnect: New connections stop arriving, but existing connections stay until the device disconnects naturally. This is too slow for planned maintenance windows.
- Abrupt killAll (current behavior): Works but causes a reconnection storm. Acceptable only for small deployments or off-peak windows.
**Additional context**
The RateLimiter from Guava is already a transitive dependency in the project, so no new dependency is needed. The change is entirely backward-compatible: omitting the rate header gives identical behavior to today.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the TODO in bifromq-session-dict/bifromq-session-dict-server/.../SessionDictService.java and trace KillAllRequest through SessionDictService.proto, ISessionDictClient.java, and KillHandler.java. Review the existing DELETE /kill flow first, then determine how long-running requests should be handled. Done means an omitted or zero rate preserves current behavior, a positive rate reaches session disconnections, and the API behavior for long drains is defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java
- Domain
- api, backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100