Implement Service Discovery for App-proxy Worker Registration
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Overview
Refactor the app-proxy worker registration mechanism to use service discovery instead of direct registration with the coordinator. This will improve system resilience by removing the hard dependency on successful coordinator registration for worker startup.
## Current State
- Workers directly register with the coordinator via `register_worker()` function
- Worker startup fails if registration with coordinator fails
- Coordinator maintains worker registry in database
- Tight coupling between worker startup and coordinator availability
## Target State
- Workers self-register via service discovery (Redis/Etcd)
- Workers can start independently of coordinator availability
- Coordinator discovers workers through service discovery
- Loose coupling allows more resilient system operation
## Implementation Approach
1. **Worker Side Changes**
- Implement service discovery registration in worker startup
- Remove hard dependency on coordinator registration
- Maintain backward compatibility during transition
1. **Coordinator Side Changes**
- Implement worker discovery via service discovery
- Sync discovered workers with database
- Handle dynamic worker join/leave events
## Benefits
- Improved fault tolerance - workers can start even if coordinator is temporarily unavailable
- Dynamic discovery - automatic detection of new workers
- Simplified worker lifecycle management
- Better support for horizontal scaling
## Acceptance Criteria
- [ ] Workers can start and register themselves via service discovery
- [ ] Coordinator can discover all registered workers via service discovery
- [ ] System maintains backward compatibility during transition period
- [ ] All existing functionality continues to work with new discovery mechanism
- [ ] Documentation updated with new architecture
JIRA Issue: BA-2380
Contributor guide
Assessment
This issue has not been assessed yet.