Worker self-registration via service discovery
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Story
As an app-proxy worker, I want to register myself via service discovery so that I can start independently of coordinator availability.
## Background
Currently, workers fail to start if they cannot register with the coordinator. This creates a tight coupling that reduces system resilience. By implementing self-registration via service discovery, workers can start and operate even when the coordinator is temporarily unavailable.
## Related Epic
BA-2380: Implement Service Discovery for App-proxy Worker Registration
## Technical Details
### Changes in `worker/server.py`
1. Modify `worker_registration_ctx` to:
- Create ServiceMetadata for the worker with all necessary information
- Register with service discovery instead of directly with coordinator
- Handle registration failures gracefully (log warning but continue startup)
- Maintain heartbeat with service discovery
1. ServiceMetadata should include:
- Worker ID (UUID)
- Display name (authority)
- Service group ("appproxy-worker")
- Endpoint information (hostname, API port, TLS settings)
- Frontend mode and protocol
- Port range or wildcard domain configuration
- Accepted traffic types and app filters
1. Backward compatibility:
- Keep existing coordinator registration as optional/fallback
- Add configuration flag to enable service discovery registration
- Gradual migration path
### Service Discovery Integration
- Use existing `ServiceDiscoveryLoop` from `ai.backend.common.service_discovery`
- Configure appropriate heartbeat interval (default 60s)
- Handle both Redis and Etcd service discovery backends
## Implementation Steps
1. Create worker ServiceMetadata model extending base ServiceMetadata
1. Implement service discovery registration in worker startup
1. Replace hard failure on coordinator registration with soft warning
1. Add heartbeat mechanism via ServiceDiscoveryLoop
1. Test with both Redis and Etcd backends
1. Update worker configuration schema
## Acceptance Criteria
- [ ] Worker creates appropriate ServiceMetadata with all required information
- [ ] Worker successfully registers with service discovery on startup
- [ ] Worker continues to run even if coordinator registration fails
- [ ] Worker maintains heartbeat with service discovery
- [ ] Worker unregisters from service discovery on shutdown
- [ ] Both Redis and Etcd backends are supported
- [ ] Existing coordinator registration can be disabled via configuration
- [ ] Logs clearly indicate registration status and any failures
## Dependencies
- Epic: BA-2380
- Requires: `ai.backend.common.service_discovery` module
## Testing
- Unit tests for ServiceMetadata creation
- Integration tests for service discovery registration
- Failure scenario testing (coordinator unavailable)
- Heartbeat and cleanup testing
JIRA Issue: BA-2381
Contributor guide
Assessment
This issue has not been assessed yet.