ADORSYS-GIS / ADORSYS-GIS/ai-helm
[Ticket]: Traces-to-logs mapping dashboard
- Dominant language
- Go Template
- Stars
- 3
- Forks
- 1
- Avg merge
- 19h 24m
- Merged PRs (30d)
- 80
Description
### Type
Feature
### Summary
We need a dashboard (or documented Grafana workflow) that maps Tempo spans to Loki logs, so we can identify with certainty what transpired during a given exchange through the AI Gateway. Expected result: a usable, documented path from a log line or user/request identifier to the relevant trace(s), and back.
### Intent
Completes the trace/log correlation research already done this sprint (core-gateway trace-log correlation work, ai-helm PRs #630/#632) — this ticket is specifically about landing on the best way to present that as a dashboard/workflow, not re-deriving the underlying wiring.
### Source of truth (links)
This sprint's trace/log correlation investigation — see ai-helm PR #630 ("trace-log correlation + scoped AI Gateway tracing") and PR #632 ("trace all gateway traffic — per-route sampling doesn't exist"), and the live findings during that investigation: Envoy's own "ingress" trace (with ext_authz/ext_proc/router spans) is what the access log's `traceparent` field actually resolves to; the AI Gateway extproc's own `ChatCompletion` span (with prompt/response content) is a separate, disconnected trace due to a propagation gap between Envoy's native tracing and the ext_proc protocol (not yet filed upstream as of this writing).
### Current Behavior
Log→trace correlation works (Loki `traceparent` field → Tempo derived link) and reaches Envoy's hop-level "ingress" trace (auth check, ext_proc round-trip, upstream call). The content-bearing `ChatCompletion` trace (actual prompt/response) is a separate trace_id, locatable only by matching timestamp/duration manually.
### Expected Behavior
A dashboard or documented workflow that takes a user/request of interest and reliably surfaces both: (a) the hop-level trace (auth, timing per leg), and (b) the content-level trace (prompt/response), without requiring the manual timestamp-matching workaround.
### Acceptance Criteria
- [ ] Given a specific log line (or a request from the "chats by user" dashboard), when following the documented path, then both the hop-level and content-level traces for that request are found with certainty (not by guessing off timestamps).
- [ ] Given the two-trace reality is not fully fixable without an upstream fix (see Technical Context), then the dashboard/workflow explicitly documents this limitation rather than silently implying full correlation.
- [ ] Verification evidence is provided (see below).
### Out of Scope
Fixing the upstream propagation gap in `envoyproxy/ai-gateway` itself (separate, cross-project effort — may need its own ticket/upstream issue if pursued).
### Technical Context
Relevant chart: `charts/core-gateway` (EnvoyProxy tracing config, access log format). Grafana datasources: `environments/prod/values/grafana.yaml`-equivalent in `ai-helm-values` (Tempo `tracesToLogsV2`, Loki `derivedFields`). See this sprint's session for the full live-cluster diagnosis (xDS route/listener dumps, `cluster.tracing` stats) if picking this up.
### Risks
The two-trace split may not be fully closeable without patching the upstream `envoyproxy/ai-gateway` extproc — scope the dashboard/workflow to work *with* that limitation rather than assuming it'll be fixed first.
### Test Plan
TBD by ticket owner — manual walkthrough from a real user's log line to both traces; document the steps.
### Verification evidence
N/A — not yet implemented. To be filled in by the ticket owner on completion.
### Human accountable owner
TBD — to be assigned during sprint planning.
### AI Usage Declaration
Drafting the ticket, Understanding code, Reviewing the diff
### Human verification completed
(unchecked — pending assignment and human review)
Contributor guide
Assessment
This issue has not been assessed yet.