Reorganize event bus consumer groups with periodic `NoopEvent` firing
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
### The current event bus
Currently there are four consumer groups in the event bus:
- `manager`
- `agent`
- `storage-proxy`
- `appwsproxy`
All producers and dispatchers share _a single global key_ for all events regardless of the target consumer group:
```mermaid
flowchart LR
classDef RedisKey fill:#fcc5,stroke:#f33
P["Producer(to:*): XADD"] --> K{{events}}
subgraph "EventDispatcher in Manager(s)"
Cm["Consumer(manager): XREADGROUP"]
Sm["Subscriber: XREAD"]
end
subgraph "EventDispatcher in Agent(s)"
Ca["Consumer(agent): XREADGROUP"]
Sa["Subscriber: XREAD"]
end
K --> Cm
K --> Sm
K --> Ca
K --> Sa
K --> More[...]
class K RedisKey
style More fill:transparent,stroke:transparent
```
### Proposal 1: Reorganize the event bus
First, let's split the Redis key for each consumer group to improve isolation between different consumer groups.
```mermaid
flowchart LR
classDef RedisKey fill:#fcc5,stroke:#f33
subgraph "EventDispatcher in Manager(s)"
Cm["Consumer(manager): XREADGROUP"]
Sm["Subscriber: XREAD"]
end
subgraph "EventDispatcher in Agent(s)"
Ca["Consumer(agent): XREADGROUP"]
Sa["Subscriber: XREAD"]
end
Pm["Producer(to:manager): XADD"] --> Km{{events.manager}}
Km --> Cm
Km --> Sm
Pa["Producer(to:agent): XADD"] --> Ka{{events.agent}}
Ka --> Ca
Ka --> Sa
class Ka RedisKey
class Km RedisKey
More["⋮"]
style More fill:transparent,stroke:transparent
```
### Proposal 2: Introduce periodic noop events
In some production setups, we are experiencing network connection drops in long-running Redis client connections that issues blocking commands like `XREAD` and `XREADGROUP`. To prevent that, we have a TCP keepalive option (lablup/backend.ai-common#88, lablup/backend.ai-common#96), but it would be also good to have explicit "ping" or "noop" message deliveries to keep the channel alive.
Let's fire `NoopEvent` to each consumer group:
- Make the interval for each consumer group configurable separately via etcd (shared config).
- The manager processes are the sender.\* Use `common.lock.FileLock` to send only one message for each timer interval **per manager node**.
- It is fine to have two or three more duplicate noop messages in HA setup.
JIRA Issue: BA-89
Contributor guide
Assessment
This issue has not been assessed yet.