lablup / lablup/backend.ai

Reorganize event bus consumer groups with periodic `NoopEvent` firing

Open
#2,755 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

### The current event bus

Currently there are four consumer groups in the event bus:

- `manager`
- `agent`
- `storage-proxy`
- `appwsproxy`
All producers and dispatchers share _a single global key_ for all events regardless of the target consumer group:

```mermaid
flowchart LR
classDef RedisKey fill:#fcc5,stroke:#f33

P["Producer(to:*): XADD"] --> K{{events}}

subgraph "EventDispatcher in Manager(s)"
Cm["Consumer(manager): XREADGROUP"]
Sm["Subscriber: XREAD"]
end
subgraph "EventDispatcher in Agent(s)"
Ca["Consumer(agent): XREADGROUP"]
Sa["Subscriber: XREAD"]
end

K --> Cm
K --> Sm
K --> Ca
K --> Sa
K --> More[...]
class K RedisKey
style More fill:transparent,stroke:transparent
```

### Proposal 1: Reorganize the event bus

First, let's split the Redis key for each consumer group to improve isolation between different consumer groups.

```mermaid
flowchart LR
classDef RedisKey fill:#fcc5,stroke:#f33

subgraph "EventDispatcher in Manager(s)"
Cm["Consumer(manager): XREADGROUP"]
Sm["Subscriber: XREAD"]
end
subgraph "EventDispatcher in Agent(s)"
Ca["Consumer(agent): XREADGROUP"]
Sa["Subscriber: XREAD"]
end

Pm["Producer(to:manager): XADD"] --> Km{{events.manager}}
Km --> Cm
Km --> Sm

Pa["Producer(to:agent): XADD"] --> Ka{{events.agent}}
Ka --> Ca
Ka --> Sa

class Ka RedisKey
class Km RedisKey

More["⋮"]
style More fill:transparent,stroke:transparent
```

### Proposal 2: Introduce periodic noop events

In some production setups, we are experiencing network connection drops in long-running Redis client connections that issues blocking commands like `XREAD` and `XREADGROUP`. To prevent that, we have a TCP keepalive option (lablup/backend.ai-common#88, lablup/backend.ai-common#96), but it would be also good to have explicit "ping" or "noop" message deliveries to keep the channel alive.

Let's fire `NoopEvent` to each consumer group:

- Make the interval for each consumer group configurable separately via etcd (shared config).
- The manager processes are the sender.\* Use `common.lock.FileLock` to send only one message for each timer interval **per manager node**.
- It is fine to have two or three more duplicate noop messages in HA setup.

JIRA Issue: BA-89

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.