lablup / lablup/backend.ai

Make AppProxy coordinator stateless by removing its own DB

Open
#13,326 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
15h 13m
Merged PRs (30d)
368

Description

Motivation

The AppProxy coordinator keeps its own PostgreSQL database while Traefik reads the same routing state from etcd. For circuits the duplication is total: the whole circuit record is serialized into the etcd middleware payload (models/circuit.py:469-471) on top of the router and service subtrees. Keeping the two stores in sync is what surfaced as BA-5499 and again as BA-7121, where a stale in-memory snapshot overwrites newer etcd state. Neither the per-circuit asyncio lock nor the 30s reconcile loop structurally prevents it, because both leave the DB read outside the critical section and the coordinator is designed to run as multiple instances.

Objective

Remove every DB access from the coordinator and make it a stateless applier. Ownership splits into the manager DB (source of truth) and etcd (shared state). Components integrate through etcd and Valkey only; the coordinator never queries the manager.

State relocation
- endpoints (health check config): the manager already owns the original; the coordinator keeps only what it renders into etcd.
- circuits: etcd.
- workers and worker_app_filters: etcd with leases, replacing the 15s LOST-detection polling.
- tokens: JWT or Valkey TTL.
- occupied_slots: derived from the circuit count in etcd, so the counter and its drift reconciler disappear.

Convergence

Convergence responsibility moves from the coordinator to the manager. The coordinator-side reconciler is deleted, since a stateless applier has no desired state to reconcile against. The manager pushes declarative per-endpoint desired state at create and terminate points, and runs a low-frequency sweep to catch lost calls, orphaned keys, and etcd restores.

Design is tracked by BEP-1073. Implementation issues are defined after the BEP settles.

JIRA Issue: BA-7123

Contributor guide

Open the contributing guide

Research direction

Start by reading BEP-1073 and the circuit serialization references in models/circuit.py:469-471. The implementation issues and completion criteria are explicitly deferred until the BEP settles, so there is no defined entry point or test target yet.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, databases, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.