lablup / lablup/backend.ai

Automatic Prometheus Registration for Traefik

Open
#8,251 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
670
Forks
183
Avg merge
17h 7m
Merged PRs (30d)
358

Description

## Summary

Enable comprehensive network observability for the AppProxy-Traefik-Container traffic path by automatically registering metrics endpoints with Prometheus service discovery.

## Background

Currently, when using Traefik as the backend for AppProxy:

1. **Traefik's own metrics are not collected** - Traefik exposes a `/metrics` endpoint with valuable
proxy performance data (request counts, latencies, connection states), but this endpoint is not
registered with the Prometheus service discovery.
1. **Container metrics are not automatically registered** - When Circuits and Routes are created via
the Traefik backend path, the container's `/metrics` endpoints are not registered with service
discovery, preventing Prometheus from scraping them.

## Current Architecture

Manager → AppProxy Coordinator → etcd → Traefik → Container

╳ Missing: Service Discovery registration

Service Discovery ← Prometheus HTTP SD

## Goals

1. **Traefik Metrics Collection**: Register Traefik's `/metrics` endpoint with service discovery so
Prometheus can scrape proxy-level metrics (request throughput, latency, error rates, active
connections).
1. **Automatic Route Metrics Registration**: When Circuits/Routes are created or updated via the
Traefik backend, automatically register container metrics endpoints with service discovery.
1. **Metrics Correlation**: Include Circuit ID, Session ID, and Worker ID as Prometheus labels to
enable correlation between AppProxy routing decisions and container-level metrics.

## Scope

### In Scope

- Register AppProxy Coordinator/Worker metrics endpoints with service discovery
- Register Traefik metrics endpoint with service discovery when Traefik backend is enabled
- Modify `CircuitManager` to register/deregister Route metrics endpoints on Circuit lifecycle events
- Add appropriate labels (circuit_id, session_id, worker_authority) for metrics correlation

### Out of Scope

- Changes to Prometheus configuration (already supports HTTP SD)
- Changes to Traefik configuration (already exposes /metrics)
- Grafana dashboard creation (separate task)

## Acceptance Criteria

- [ ] When AppProxy Coordinator starts, it registers its `/metrics` endpoint with service discovery
- [ ] When AppProxy Worker starts with Traefik enabled, it registers Traefik's `/metrics` endpoint
with service discovery
- [ ] When a Circuit is created via Traefik backend, all healthy Route metrics endpoints are
registered with service discovery
- [ ] When a Circuit's Routes are updated, service discovery registrations are updated accordingly
- [ ] When a Circuit is deleted, its Route metrics endpoints are deregistered from service discovery
- [ ] Prometheus can discover and scrape all registered endpoints via HTTP SD
- [ ] Metrics include labels for circuit_id, session_id, and worker_authority

## Technical Notes

- Leverage existing `ServiceDiscovery` interface (`src/ai/backend/common/service_discovery/`)
- Use `ModelServiceMetadata` for Route metrics registration (already used by Manager's
RouteExecutor)
- Reference Manager's `sync_service_discovery()` implementation in
`src/ai/backend/manager/sokovan/deployment/route/executor.py`

## Related Components

- AppProxy Coordinator: `src/ai/backend/appproxy/coordinator/`
- AppProxy Worker: `src/ai/backend/appproxy/worker/`
- Service Discovery: `src/ai/backend/common/service_discovery/`
- Prometheus HTTP SD endpoint: `GET /metrics/service_discovery` (Manager)

JIRA Issue: BA-4038

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.