Implement SD-based client pool with cache and self-preservation
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Objective
Implement a service catalog cache and SD-based client pool that provides resilient inter-service communication with graceful degradation.
## Scope
### 1. ServiceCatalogCache
- In-memory cache of `service_catalog` data
- Full fetch on startup from Manager API or DB (for Manager itself)
- Periodic delta sync (every 30-60s)
- **Self-preservation mode** (Eureka pattern): on backend failure, freeze last known state instead of evicting services
- Staleness indicator on cached data
- Optional disk snapshot for restart recovery
### 2. SDClientPool
- Built on top of `ServiceCatalogCache`
- Select endpoints by `(service_group, role, scope)`
- Manage HTTP (`aiohttp.ClientSession`) and RPC (`PeerInvoker`) connections
- Auto-sync with cache changes: add new services, remove deregistered ones
- Load balancing: round-robin or least-connections within a service group
- Existing connections survive cache sync (data plane / control plane separation)
### 3. Integration Points
- Replace hardcoded storage proxy addresses in Manager with `SDClientPool("storage-proxy", role="manager-api", scope="cluster")`
- Replace `AgentClientPool` address resolution with SD-based lookup (keep ZMQ connection logic)
- Replace AppProxy client address lookup with SD-based resolution
## Acceptance Criteria
- [ ] `ServiceCatalogCache` caches and syncs service catalog
- [ ] Self-preservation mode activates on backend failure, freezes state
- [ ] `SDClientPool` selects endpoints by role/scope
- [ ] Existing connections maintained during cache sync
- [ ] Tests for cache sync, self-preservation, and pool lifecycle
JIRA Issue: BA-4318
Contributor guide
Assessment
This issue has not been assessed yet.