lablup / lablup/backend.ai

Worker self-registration via service discovery

Open
#5,884 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Story

As an app-proxy worker, I want to register myself via service discovery so that I can start independently of coordinator availability.

## Background

Currently, workers fail to start if they cannot register with the coordinator. This creates a tight coupling that reduces system resilience. By implementing self-registration via service discovery, workers can start and operate even when the coordinator is temporarily unavailable.

## Related Epic

BA-2380: Implement Service Discovery for App-proxy Worker Registration

## Technical Details

### Changes in `worker/server.py`

1. Modify `worker_registration_ctx` to:
- Create ServiceMetadata for the worker with all necessary information
- Register with service discovery instead of directly with coordinator
- Handle registration failures gracefully (log warning but continue startup)
- Maintain heartbeat with service discovery
1. ServiceMetadata should include:
- Worker ID (UUID)
- Display name (authority)
- Service group ("appproxy-worker")
- Endpoint information (hostname, API port, TLS settings)
- Frontend mode and protocol
- Port range or wildcard domain configuration
- Accepted traffic types and app filters
1. Backward compatibility:
- Keep existing coordinator registration as optional/fallback
- Add configuration flag to enable service discovery registration
- Gradual migration path

### Service Discovery Integration

- Use existing `ServiceDiscoveryLoop` from `ai.backend.common.service_discovery`
- Configure appropriate heartbeat interval (default 60s)
- Handle both Redis and Etcd service discovery backends

## Implementation Steps

1. Create worker ServiceMetadata model extending base ServiceMetadata
1. Implement service discovery registration in worker startup
1. Replace hard failure on coordinator registration with soft warning
1. Add heartbeat mechanism via ServiceDiscoveryLoop
1. Test with both Redis and Etcd backends
1. Update worker configuration schema

## Acceptance Criteria

- [ ] Worker creates appropriate ServiceMetadata with all required information
- [ ] Worker successfully registers with service discovery on startup
- [ ] Worker continues to run even if coordinator registration fails
- [ ] Worker maintains heartbeat with service discovery
- [ ] Worker unregisters from service discovery on shutdown
- [ ] Both Redis and Etcd backends are supported
- [ ] Existing coordinator registration can be disabled via configuration
- [ ] Logs clearly indicate registration status and any failures

## Dependencies

- Epic: BA-2380
- Requires: `ai.backend.common.service_discovery` module

## Testing

- Unit tests for ServiceMetadata creation
- Integration tests for service discovery registration
- Failure scenario testing (coordinator unavailable)
- Heartbeat and cleanup testing

JIRA Issue: BA-2381

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.