lablup / lablup/backend.ai

Integrate exporter health checks with service discovery framework

Open
#6,025 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Story

As a system administrator, I want the watcher to automatically perform health checks on registered exporters through the service discovery framework, so that I can ensure metrics collection reliability and automatic recovery.

## Description

Integrate the registered exporters with the existing service discovery framework in `src/ai/backend/common/service_discovery/` to enable automatic health monitoring, discovery, and registration of exporter endpoints.

## Acceptance Criteria

- [ ] Exporters are automatically registered with service discovery on startup
- [ ] Periodic health checks are performed for all registered exporters
- [ ] Unhealthy exporters are automatically detected and marked
- [ ] Failed exporters trigger retry logic with configurable backoff
- [ ] Service discovery provides endpoint information for metrics collection
- [ ] Integration with existing monitoring and alerting systems
- [ ] Graceful handling of transient failures

## Technical Details

### Service Discovery Integration

```python
# Integration with existing service discovery
class ExporterServiceDiscovery:
async def register_exporter(
self,
exporter_id: str,
exporter_type: str,
endpoint: str,
metadata: dict
) -> None:
"""Register exporter with service discovery"""

async def health_check(self, exporter_id: str) -> bool:
"""Perform health check on exporter"""

async def discover_exporters(self) -> list[ExporterInfo]:
"""Discover all available exporters"""

async def deregister_exporter(self, exporter_id: str) -> None:
"""Remove exporter from service discovery"""
```

### Health Check Implementation

- **Endpoint Health**: HTTP/HTTPS endpoint availability check
- **Metrics Validation**: Verify exporter is producing valid metrics
- **Resource Monitoring**: Check CPU/memory usage of exporter process
- **Response Time**: Monitor latency of metrics endpoint

### Health Check Configuration

```yaml
health_check:
interval: 30s
timeout: 5s
retries: 3
backoff_multiplier: 2
max_backoff: 300s
failure_threshold: 3 # failures before marking unhealthy
recovery_threshold: 2 # successes before marking healthy
```

### Recovery Strategies

1. **Automatic Restart**: Restart failed exporters
1. **Circuit Breaker**: Temporarily disable failing exporters
1. **Fallback**: Use alternative exporters if available
1. **Alert Generation**: Notify administrators of persistent failures

## Dependencies

- Existing service discovery framework (`src/ai/backend/common/service_discovery/`)
- Health check libraries (aiohttp for HTTP checks)
- Monitoring/alerting system integration

## Testing Requirements

- Unit tests for health check logic
- Integration tests with service discovery
- Test failure detection and recovery
- Test retry logic and backoff
- Simulate various failure scenarios
- Performance tests for health check overhead

## Documentation

- Integration guide with service discovery
- Health check configuration reference
- Troubleshooting guide for common issues
- Monitoring and alerting setup guide

## Implementation Notes

- Leverage existing service discovery patterns in the codebase
- Ensure compatibility with current watcher architecture
- Minimize performance impact of health checks
- Consider using asyncio for concurrent health checks

JIRA Issue: BA-2502

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.