Integrate exporter health checks with service discovery framework
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Story
As a system administrator, I want the watcher to automatically perform health checks on registered exporters through the service discovery framework, so that I can ensure metrics collection reliability and automatic recovery.
## Description
Integrate the registered exporters with the existing service discovery framework in `src/ai/backend/common/service_discovery/` to enable automatic health monitoring, discovery, and registration of exporter endpoints.
## Acceptance Criteria
- [ ] Exporters are automatically registered with service discovery on startup
- [ ] Periodic health checks are performed for all registered exporters
- [ ] Unhealthy exporters are automatically detected and marked
- [ ] Failed exporters trigger retry logic with configurable backoff
- [ ] Service discovery provides endpoint information for metrics collection
- [ ] Integration with existing monitoring and alerting systems
- [ ] Graceful handling of transient failures
## Technical Details
### Service Discovery Integration
```python
# Integration with existing service discovery
class ExporterServiceDiscovery:
async def register_exporter(
self,
exporter_id: str,
exporter_type: str,
endpoint: str,
metadata: dict
) -> None:
"""Register exporter with service discovery"""
async def health_check(self, exporter_id: str) -> bool:
"""Perform health check on exporter"""
async def discover_exporters(self) -> list[ExporterInfo]:
"""Discover all available exporters"""
async def deregister_exporter(self, exporter_id: str) -> None:
"""Remove exporter from service discovery"""
```
### Health Check Implementation
- **Endpoint Health**: HTTP/HTTPS endpoint availability check
- **Metrics Validation**: Verify exporter is producing valid metrics
- **Resource Monitoring**: Check CPU/memory usage of exporter process
- **Response Time**: Monitor latency of metrics endpoint
### Health Check Configuration
```yaml
health_check:
interval: 30s
timeout: 5s
retries: 3
backoff_multiplier: 2
max_backoff: 300s
failure_threshold: 3 # failures before marking unhealthy
recovery_threshold: 2 # successes before marking healthy
```
### Recovery Strategies
1. **Automatic Restart**: Restart failed exporters
1. **Circuit Breaker**: Temporarily disable failing exporters
1. **Fallback**: Use alternative exporters if available
1. **Alert Generation**: Notify administrators of persistent failures
## Dependencies
- Existing service discovery framework (`src/ai/backend/common/service_discovery/`)
- Health check libraries (aiohttp for HTTP checks)
- Monitoring/alerting system integration
## Testing Requirements
- Unit tests for health check logic
- Integration tests with service discovery
- Test failure detection and recovery
- Test retry logic and backoff
- Simulate various failure scenarios
- Performance tests for health check overhead
## Documentation
- Integration guide with service discovery
- Health check configuration reference
- Troubleshooting guide for common issues
- Monitoring and alerting setup guide
## Implementation Notes
- Leverage existing service discovery patterns in the codebase
- Ensure compatibility with current watcher architecture
- Minimize performance impact of health checks
- Consider using asyncio for concurrent health checks
JIRA Issue: BA-2502
Contributor guide
Assessment
This issue has not been assessed yet.