lablup / lablup/backend.ai

Add configuration management for exporter selection in watcher

Open
#6,024 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Story

As a system administrator, I want to configure which metrics exporters are enabled through configuration files, so that I can easily manage and customize metrics collection per environment.

## Description

Implement a configuration system that allows users to specify which exporters should be activated when the watcher starts, including their specific settings and parameters.

## Acceptance Criteria

- [ ] Configuration schema supports multiple exporter definitions
- [ ] Each exporter can be enabled/disabled via configuration
- [ ] Support for exporter-specific parameters (port, interval, etc.)
- [ ] Configuration validation on startup
- [ ] Support for runtime configuration updates (hot-reload)
- [ ] Default configurations for common exporters
- [ ] Environment variable overrides for configuration values

## Technical Details

### Configuration Schema

```yaml
watcher:
exporters:
dcgm:
enabled: true
port: 9400
health_check_interval: 30s
gpu_devices: "all" # or specific GPU IDs
metrics_groups:
- "GPU_UTIL"
- "MEMORY"
- "POWER"

node:
enabled: true
port: 9100
health_check_interval: 30s
collectors:
- "cpu"
- "memory"
- "disk"
- "network"

custom:
- name: "app_metrics"
enabled: false
module: "ai.backend.watcher.exporters.custom.app"
config:
key: "value"
```

### Configuration Loading

- Use Pydantic models for configuration validation
- Support YAML and environment variable inputs
- Implement configuration merging (defaults -> file -> env)
- Validate exporter availability before enabling

### Hot-reload Support

- File watcher for configuration changes
- Graceful reload without dropping metrics
- Validation before applying new configuration
- Rollback on invalid configuration

## Dependencies

- Pydantic for configuration models
- YAML parser
- File system watcher for hot-reload

## Testing Requirements

- Unit tests for configuration parsing
- Validation tests for invalid configurations
- Integration tests for configuration updates
- Test environment variable overrides
- Test hot-reload functionality

## Documentation

- Configuration reference guide
- Example configurations for different scenarios
- Environment variable documentation
- Migration guide from static to dynamic configuration

JIRA Issue: BA-2501

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.