ADORSYS-GIS / ADORSYS-GIS/ai-helm

[Ticket]: Traces-to-logs mapping dashboard

Abierto
#674 0 comentarios 0 reacciones 0 asignados Ver en GitHub
loki observability tempo ticket
Lenguaje dominante
Go Template
Estrellas
3
Forks
1
Merge medio
19 h 24 min
PR fusionados (30 d)
80

Descripción

### Type
Feature

### Summary
We need a dashboard (or documented Grafana workflow) that maps Tempo spans to Loki logs, so we can identify with certainty what transpired during a given exchange through the AI Gateway. Expected result: a usable, documented path from a log line or user/request identifier to the relevant trace(s), and back.

### Intent
Completes the trace/log correlation research already done this sprint (core-gateway trace-log correlation work, ai-helm PRs #630/#632) — this ticket is specifically about landing on the best way to present that as a dashboard/workflow, not re-deriving the underlying wiring.

### Source of truth (links)
This sprint's trace/log correlation investigation — see ai-helm PR #630 ("trace-log correlation + scoped AI Gateway tracing") and PR #632 ("trace all gateway traffic — per-route sampling doesn't exist"), and the live findings during that investigation: Envoy's own "ingress" trace (with ext_authz/ext_proc/router spans) is what the access log's `traceparent` field actually resolves to; the AI Gateway extproc's own `ChatCompletion` span (with prompt/response content) is a separate, disconnected trace due to a propagation gap between Envoy's native tracing and the ext_proc protocol (not yet filed upstream as of this writing).

### Current Behavior
Log→trace correlation works (Loki `traceparent` field → Tempo derived link) and reaches Envoy's hop-level "ingress" trace (auth check, ext_proc round-trip, upstream call). The content-bearing `ChatCompletion` trace (actual prompt/response) is a separate trace_id, locatable only by matching timestamp/duration manually.

### Expected Behavior
A dashboard or documented workflow that takes a user/request of interest and reliably surfaces both: (a) the hop-level trace (auth, timing per leg), and (b) the content-level trace (prompt/response), without requiring the manual timestamp-matching workaround.

### Acceptance Criteria
- [ ] Given a specific log line (or a request from the "chats by user" dashboard), when following the documented path, then both the hop-level and content-level traces for that request are found with certainty (not by guessing off timestamps).
- [ ] Given the two-trace reality is not fully fixable without an upstream fix (see Technical Context), then the dashboard/workflow explicitly documents this limitation rather than silently implying full correlation.
- [ ] Verification evidence is provided (see below).

### Out of Scope
Fixing the upstream propagation gap in `envoyproxy/ai-gateway` itself (separate, cross-project effort — may need its own ticket/upstream issue if pursued).

### Technical Context
Relevant chart: `charts/core-gateway` (EnvoyProxy tracing config, access log format). Grafana datasources: `environments/prod/values/grafana.yaml`-equivalent in `ai-helm-values` (Tempo `tracesToLogsV2`, Loki `derivedFields`). See this sprint's session for the full live-cluster diagnosis (xDS route/listener dumps, `cluster.tracing` stats) if picking this up.

### Risks
The two-trace split may not be fully closeable without patching the upstream `envoyproxy/ai-gateway` extproc — scope the dashboard/workflow to work *with* that limitation rather than assuming it'll be fixed first.

### Test Plan
TBD by ticket owner — manual walkthrough from a real user's log line to both traces; document the steps.

### Verification evidence
N/A — not yet implemented. To be filled in by the ticket owner on completion.

### Human accountable owner
TBD — to be assigned during sprint planning.

### AI Usage Declaration
Drafting the ticket, Understanding code, Reviewing the diff

### Human verification completed
(unchecked — pending assignment and human review)

Guía de contribución

Abrir la guía de contribución

Línea de trabajo

Start with the referenced chart `charts/core-gateway` and the Grafana datasource values in the `ai-helm-values` equivalent of `environments/prod/values/grafana.yaml`, especially Tempo `tracesToLogsV2` and Loki `derivedFields`. Read PRs #630 and #632 for the trace/log correlation context. Done means a documented Grafana dashboard or workflow can reliably go from a log line or user/request to both trace types and states the known two-trace limitation.

Escrito por el modelo de indexación a partir del texto del issue.

Evaluación

Stack tecnológico
grafana, helm, kubernetes
Área
devops, observability
Tipo de issue
Nueva funcionalidad
Dificultad
4/5
Tiempo estimado
3-5 días
Estado de actividad
Tranquilo
Claridad
Bastante claro
Aptitud para principiantes
42/100

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.