Integrate metrics exporters with service discovery and watcher system
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Overview
Integrate node-level and specific metrics collection into Backend.AI's internal metrics system through Prometheus exporters, leveraging the service discovery framework and watcher system.
## Background
We need to collect various node-level metrics and specialized metrics (especially for GPU environments) through Prometheus exporters and integrate them into our internal metrics system. This integration should be seamlessly managed through the watcher system with proper service discovery.
## Goals
1. Enable dynamic registration of metrics exporters through the watcher system
1. Provide configuration-based exporter management
1. Implement health check capabilities through service discovery
## Scope
### Phase 1 - Core Integration
- **Exporter Registration System**
- Enable watcher to register specific metrics exporters (DCGM exporter, node exporter, etc.)
- Support fixture-based injection for exporter configuration
- Design extensible architecture for future exporter additions
- **Configuration Management**
- Implement configuration schema for exporter selection
- Support runtime configuration updates
- Provide sensible defaults for common exporters
- **Service Discovery Integration**
- Implement health check mechanisms for registered exporters
- Integrate with existing service discovery framework in `src/ai/backend/common/service_discovery/`
- Support automatic discovery and registration of exporter endpoints
## Technical Requirements
### Exporter Registration
- Support for GPU-specific exporters (DCGM exporter for NVIDIA GPUs)
- Support for general system exporters (node exporter for system metrics)
- Flexible fixture/plugin system for custom exporters
- Registry pattern for managing multiple exporters
### Configuration Schema
```yaml
watcher:
exporters:
- type: dcgm
enabled: true
port: 9400
health_check_interval: 30s
- type: node
enabled: true
port: 9100
health_check_interval: 30s
```
### Health Check Implementation
- Periodic health checks for all registered exporters
- Automatic deregistration of unhealthy exporters
- Retry logic with configurable backoff
- Integration with existing monitoring and alerting
## Success Criteria
- [ ] Watcher can dynamically register and manage multiple exporters
- [ ] Configuration-driven exporter selection is functional
- [ ] Health checks are performed reliably through service discovery
- [ ] Metrics are successfully collected and integrated into Backend.AI metrics system
- [ ] Documentation and examples are provided for common use cases
## Dependencies
- Existing service discovery framework
- Watcher system
- Prometheus client libraries
- Target exporters (DCGM, node exporter, etc.)
## Future Considerations
- Support for custom exporter development
- Automated exporter deployment and lifecycle management
- Integration with Kubernetes operators for cloud-native deployments
- Performance optimization for high-frequency metrics collection
JIRA Issue: BA-2499
Contributor guide
Assessment
This issue has not been assessed yet.