lablup / lablup/backend.ai

Leader election-based distributed timer and image rescan rate limiting

Open
#415 1 comment 0 reactions 1 assignee Assigned to @jopemachine View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Recently we have replaced our distributed locks and global timers to use etcd's concurrency API to guarantee active-active HA.

However, there are still edge cases that require a global coordination of all manager processes, such as rate-limited container registry access (e.g., Docker Hub with anonymous user). Since there are many manager processes that receives the API requests in a load-balanced fashion, it is difficult to share the rate-limit states between different manager processes. This is why lablup/backend.ai-manager#501 is on hold.

Let's localize such globally coordinated states to a single manager process, or a leader.To keep high availability, we should perform periodic checks on the liveness the leader and re-elect it, and fortunately etcd provides the facilities to implement this.

- [x] manager: Implement leader election of manager processes with periodic leader status checks.
- [x] manager: Rewrite global timer to run on the leader manager process. (When a new leader is elected, the new one should start global timers while the old one should stop, of course when the old one is still alive.)
- [ ] manager: Add a generic "leader task" message queue based on Redis stream to reroute API requests accepted by arbitrary manager processes that should be exclusively processed by the leader
- [ ] manager: Rewrite lablup/backend.ai-manager#501 to use a local `aiolimiter` state to implement its own rate-limiting to the container registries. Use the leader task queue to trigger the rescan task.

JIRA Issue: BA-269

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.