Unified Service Discovery with DB-backed Service Catalog
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Goal
Replace the current fragmented service discovery mechanisms (Redis hash, etcd, event-based heartbeat) with a unified, DB-backed service catalog that provides:
1. **Consistent registration** — All components (Agent, Manager, Storage Proxy, AppProxy, Webserver) register via Redis events
1. **Queryable catalog** — DB-backed `service_catalog` and `service_catalog_endpoint` tables, exposed via GraphQL for user-facing service status views
1. **Multi-endpoint support** — Each service registers multiple endpoints with `role` (api, rpc, metrics, etc.) and `scope` (host, container, cluster, public) dimensions
1. **Config-driven endpoints** — Fully explicit `[service-discovery.endpoints]` TOML config section with CLI migration tool for existing deployments
1. **Resilient client pools** — SD-based client pools with in-memory cache and self-preservation mode (Eureka/Consul patterns)
## Current State
|Component|Registration Method|Backend|
|---|---|---|
|Agent|Heartbeat event|Event → DB|
|Manager|ServiceDiscoveryLoop|ETCD / Redis|
|Storage Proxy|ServiceDiscoveryLoop|ETCD / Redis|
|AppProxy|ServiceDiscoveryLoop|Redis only|
|Webserver|ServiceDiscoveryLoop|ETCD / Redis|
## Target State
All components → `ServiceRegisteredEvent` (Redis Stream) → Manager consumes → DB catalog → GraphQL query
## Phases
- Phase 1: Common foundation (config models, event types, DB models)
- Phase 2: CLI migration tool (`config migrate-sd`)
- Phase 3: Runtime integration (event publishing, Manager handler, GraphQL API)
- Phase 4: SD-based client pool (ServiceCatalogCache + SDClientPool)
## Key Design Decisions
- **Redis events as transport**: Leverages existing event infrastructure (Redis Stream DB 4), no new config needed for service-to-service communication
- **DB as authoritative store**: Enables SQL queries, GraphQL exposure, audit trails
- **role + scope endpoint model**: Supports diverse network topologies (host, container, k8s cluster, public LB)
- **Config-driven (Option B)**: Explicit `[service-discovery.endpoints]` section; CLI migration tool generates from existing config fields
- **Self-preservation on backend failure**: Freeze last known state instead of deregistering services (inspired by Netflix Eureka)
JIRA Issue: BA-4313
Contributor guide
Assessment
This issue has not been assessed yet.