ADORSYS-GIS / ADORSYS-GIS/ai-helm

[Ticket]: Traces-to-logs mapping dashboard

Open
#674 0 comments 0 reactions 0 assignees View on GitHub
loki observability tempo ticket
Dominant language
Go Template
Stars
3
Forks
1
Avg merge
19h 24m
Merged PRs (30d)
80

Description

### Type
Feature

### Summary
We need a dashboard (or documented Grafana workflow) that maps Tempo spans to Loki logs, so we can identify with certainty what transpired during a given exchange through the AI Gateway. Expected result: a usable, documented path from a log line or user/request identifier to the relevant trace(s), and back.

### Intent
Completes the trace/log correlation research already done this sprint (core-gateway trace-log correlation work, ai-helm PRs #630/#632) — this ticket is specifically about landing on the best way to present that as a dashboard/workflow, not re-deriving the underlying wiring.

### Source of truth (links)
This sprint's trace/log correlation investigation — see ai-helm PR #630 ("trace-log correlation + scoped AI Gateway tracing") and PR #632 ("trace all gateway traffic — per-route sampling doesn't exist"), and the live findings during that investigation: Envoy's own "ingress" trace (with ext_authz/ext_proc/router spans) is what the access log's `traceparent` field actually resolves to; the AI Gateway extproc's own `ChatCompletion` span (with prompt/response content) is a separate, disconnected trace due to a propagation gap between Envoy's native tracing and the ext_proc protocol (not yet filed upstream as of this writing).

### Current Behavior
Log→trace correlation works (Loki `traceparent` field → Tempo derived link) and reaches Envoy's hop-level "ingress" trace (auth check, ext_proc round-trip, upstream call). The content-bearing `ChatCompletion` trace (actual prompt/response) is a separate trace_id, locatable only by matching timestamp/duration manually.

### Expected Behavior
A dashboard or documented workflow that takes a user/request of interest and reliably surfaces both: (a) the hop-level trace (auth, timing per leg), and (b) the content-level trace (prompt/response), without requiring the manual timestamp-matching workaround.

### Acceptance Criteria
- [ ] Given a specific log line (or a request from the "chats by user" dashboard), when following the documented path, then both the hop-level and content-level traces for that request are found with certainty (not by guessing off timestamps).
- [ ] Given the two-trace reality is not fully fixable without an upstream fix (see Technical Context), then the dashboard/workflow explicitly documents this limitation rather than silently implying full correlation.
- [ ] Verification evidence is provided (see below).

### Out of Scope
Fixing the upstream propagation gap in `envoyproxy/ai-gateway` itself (separate, cross-project effort — may need its own ticket/upstream issue if pursued).

### Technical Context
Relevant chart: `charts/core-gateway` (EnvoyProxy tracing config, access log format). Grafana datasources: `environments/prod/values/grafana.yaml`-equivalent in `ai-helm-values` (Tempo `tracesToLogsV2`, Loki `derivedFields`). See this sprint's session for the full live-cluster diagnosis (xDS route/listener dumps, `cluster.tracing` stats) if picking this up.

### Risks
The two-trace split may not be fully closeable without patching the upstream `envoyproxy/ai-gateway` extproc — scope the dashboard/workflow to work *with* that limitation rather than assuming it'll be fixed first.

### Test Plan
TBD by ticket owner — manual walkthrough from a real user's log line to both traces; document the steps.

### Verification evidence
N/A — not yet implemented. To be filled in by the ticket owner on completion.

### Human accountable owner
TBD — to be assigned during sprint planning.

### AI Usage Declaration
Drafting the ticket, Understanding code, Reviewing the diff

### Human verification completed
(unchecked — pending assignment and human review)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.