lablup / lablup/backend.ai

Integrate metrics exporters with service discovery and watcher system

Open
#6,022 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Overview

Integrate node-level and specific metrics collection into Backend.AI's internal metrics system through Prometheus exporters, leveraging the service discovery framework and watcher system.

## Background

We need to collect various node-level metrics and specialized metrics (especially for GPU environments) through Prometheus exporters and integrate them into our internal metrics system. This integration should be seamlessly managed through the watcher system with proper service discovery.

## Goals

1. Enable dynamic registration of metrics exporters through the watcher system
1. Provide configuration-based exporter management
1. Implement health check capabilities through service discovery

## Scope

### Phase 1 - Core Integration

- **Exporter Registration System**
- Enable watcher to register specific metrics exporters (DCGM exporter, node exporter, etc.)
- Support fixture-based injection for exporter configuration
- Design extensible architecture for future exporter additions
- **Configuration Management**
- Implement configuration schema for exporter selection
- Support runtime configuration updates
- Provide sensible defaults for common exporters
- **Service Discovery Integration**
- Implement health check mechanisms for registered exporters
- Integrate with existing service discovery framework in `src/ai/backend/common/service_discovery/`
- Support automatic discovery and registration of exporter endpoints

## Technical Requirements

### Exporter Registration

- Support for GPU-specific exporters (DCGM exporter for NVIDIA GPUs)
- Support for general system exporters (node exporter for system metrics)
- Flexible fixture/plugin system for custom exporters
- Registry pattern for managing multiple exporters

### Configuration Schema

```yaml
watcher:
exporters:
- type: dcgm
enabled: true
port: 9400
health_check_interval: 30s
- type: node
enabled: true
port: 9100
health_check_interval: 30s
```

### Health Check Implementation

- Periodic health checks for all registered exporters
- Automatic deregistration of unhealthy exporters
- Retry logic with configurable backoff
- Integration with existing monitoring and alerting

## Success Criteria

- [ ] Watcher can dynamically register and manage multiple exporters
- [ ] Configuration-driven exporter selection is functional
- [ ] Health checks are performed reliably through service discovery
- [ ] Metrics are successfully collected and integrated into Backend.AI metrics system
- [ ] Documentation and examples are provided for common use cases

## Dependencies

- Existing service discovery framework
- Watcher system
- Prometheus client libraries
- Target exporters (DCGM, node exporter, etc.)

## Future Considerations

- Support for custom exporter development
- Automated exporter deployment and lifecycle management
- Integration with Kubernetes operators for cloud-native deployments
- Performance optimization for high-frequency metrics collection

JIRA Issue: BA-2499

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.