kubeflow / kubeflow/mcp-server

feat(core): OpenTelemetry metrics, Prometheus scrape, and HTTP RED

Open
#89 4 comments 0 reactions 1 assignee Assigned to @abhijeet-dhumal View on GitHub
area/core area/engprod
Dominant language
Python
Stars
44
Forks
54
Avg merge
2d 18h
Merged PRs (30d)
29

Description

### Description

#18 / #21 added OTel **tracing** for tool calls. #87 / #88 add HTTP `/health` and `/ready` for probes.

We still lack the rest of the Phase-2 observability path from the roadmap:

- Prometheus-compatible metrics (tool counters/histograms, readiness gauge)
- `GET /metrics` scrape endpoint
- HTTP RED (rate / errors / duration) for the HTTP edge
- Optional OTLP metrics export alongside traces
- Resilience signals (rate-limit / circuit-breaker) as metrics

Platform teams can follow traces today, but cannot scrape SLO-style series or alert without parsing logs.

### Proposal

1. Optional `.[otel]` extra — OpenTelemetry SDK + OTLP exporters + Prometheus metric reader/exporter; no-op when the extra is not installed.
2. 'GET /metrics' unauthenticated scrape route on the same HTTP edge as probes (document NetworkPolicy guidance).
3. Instruments (stable names; Prometheus export via OTel to Prom):
- tool call count + duration ('tool.name', success)
- server ready gauge (aligned with '/ready')
- HTTP RED for edge routes
- resilience events (rate limited / circuit open)
4. Docker : install `.[otel]` in the published image so scrape works out of the box.
6. Docs : README + ARCHITECTURE: what is scraped vs OTLP-exported;

### Out of scope

- W3C TraceContext continuation into Kubeflow SDK child spans (SDK-side; track on kubeflow/sdk)
- Langfuse / MLflow exporters
- Helm chart / ServiceMonitor CRDs (follow-up once scrape works)
- Multi-process metric aggregation

### Done when

- [ ] `GET /metrics` returns Prometheus text when `.[otel]` is installed
- [ ] Tool calls and HTTP edge emit the documented series (unit + golden name contract)
- [ ] OTLP metrics export works when an endpoint is configured
- [ ] Image with `.[otel]` scrapes cleanly in a local Docker run
- [ ] README / ARCHITECTURE document scrape + OTLP setup
- [ ] Behavior without `.[otel]` remains safe (no hard crash; clear `/metrics` response)

### Relates to

- Follows #18 / #21 (tracing)
- Follows #87 / #88 (probes; metrics explicitly deferred there)
- ROADMAP Phase 2 observability

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.