apache / apache/bifromq

Support rate-limited session disconnection for graceful node draining

Open
#254 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
792
Forks
95
Avg merge
10h 45m
Merged PRs (30d)
4

Description

**Is your feature request related to a problem? Please describe.**

When performing rolling updates on large BifroMQ clusters, operators need to drain connections from a node before taking it offline. The only available mechanism is DELETE /kill, which disconnects all matching sessions simultaneously with no rate control. On a node with hundreds of thousands of
connections, this causes a sudden reconnection storm: all devices receive TCP FIN at the same moment and immediately retry, creating a spike that can overload the remaining nodes and trigger cascading failures.

The codebase already acknowledges this gap with an explicit TODO comment at `bifromq-session-dict/bifromq-session-dict-server/src/main/java/.../SessionDictService.java:102:`

```
// TODO: support disconnect a constant rate
sessionRegistrations.forEach(reg -> reg.stop(request.getKiller(), request.getServerRedirection()));
```

**Describe the solution you'd like**

Implement the TODO: add an optional rate field to KillAllRequest that limits how many sessions are disconnected per second. A value of 0 preserves the current behavior (no rate limit) for backwards compatibility.

Proposed changes across 4 files:

SessionDictService.proto — add field to KillAllRequest:
` uint32 rate = 6; // disconnect rate per second, 0 = unlimited (default)`

SessionDictService.java — replace the forEach with a rate-limited loop:
```
// replace the TODO line with:
RateLimiter limiter = (request.getRate() > 0)
? RateLimiter.create(request.getRate()) : null;
for (ISessionRegistry.SessionRegistration reg : sessionRegistrations) {
if (limiter != null) limiter.acquire();
reg.stop(request.getKiller(), request.getServerRedirection());
}
```

ISessionDictClient.java — add rate parameter with a default method to keep backwards compatibility.

KillHandler.java — read an optional rate header and pass it through to the client call.

The resulting API call would look like:
```

# Drain a node at 500 disconnections/second
curl -X DELETE http://:8091/kill \
-H "tenant_id: " \
-H "rate: 500"
```

Since a rate-limited kill may run for a long time (e.g. 1,000,000 connections / 500 per second = 2000 seconds), the implementation should either increase the gRPC timeout accordingly or execute asynchronously and return 202 Accepted immediately.

**Describe alternatives you've considered**

- External script calling DELETE /kill per client-id one by one: Requires a separate API to list all sessions on a node, which does not currently exist. Also introduces network round-trip overhead for every individual disconnect.
- NLB weight-based draining without forced disconnect: New connections stop arriving, but existing connections stay until the device disconnects naturally. This is too slow for planned maintenance windows.
- Abrupt killAll (current behavior): Works but causes a reconnection storm. Acceptable only for small deployments or off-peak windows.

**Additional context**

The RateLimiter from Guava is already a transitive dependency in the project, so no new dependency is needed. The change is entirely backward-compatible: omitting the rate header gives identical behavior to today.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the TODO in bifromq-session-dict/bifromq-session-dict-server/.../SessionDictService.java and trace KillAllRequest through SessionDictService.proto, ISessionDictClient.java, and KillHandler.java. Review the existing DELETE /kill flow first, then determine how long-running requests should be handled. Done means an omitted or zero rate preserves current behavior, a positive rate reaches session disconnections, and the API behavior for long drains is defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
api, backend-api-design, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.