agent-substrate / agent-substrate/substrate

ateapi: postgres store has no trace instrumentation - DB time is invisible in every trace

Aperta
#1,455 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
area/api-machinery area/observability bug-bash kind/bug prio/P1
Lingua principale
Go
Stelle
1.8k
Fork
316
Merge medio
2g 43m
PR unite (30g)
287

Descrizione

## Summary

The ateapi postgres store (`cmd/ateapi/internal/store/atepg/`) creates its `pgxpool` without any tracer hook, so no span is ever emitted for database work. Every RPC trace ends at the gRPC server span produced by the `otelgrpc` stats handler; the time spent in PostgreSQL is indistinguishable from handler/auth overhead.

## Observed

On a live GKE install with tracing enabled end to end:

- `kubectl ate get actor -a --trace` produces a trace with **exactly one span**: `ateapi.Control/GetActor` (~97 ms in our run). For a single-row read that is slow enough to want a breakdown, and the trace offers none.
- This is not specific to reads: rich mutation traces (25-span suspend, 33-span resume) contain `step.*`, atelet, ateom, and GCS client spans — but zero postgres spans, despite every one of those flows doing multiple DB round-trips (actor load, state transitions, outbox).

## Root cause

- `cmd/ateapi/internal/store/atepg/atepg.go` has no OTel/tracing references at all; the pool is built from `pgxpool.ParseConfig` with `ConnConfig.Tracer` unset.
- The only vendored OTel instrumentation libraries are `otelgrpc` and `net/http` — there is no pgx instrumentation in the dependency tree.

## Suggested fix

Attach a tracer via pgx v5's native hook: set `config.ConnConfig.Tracer` on the pool config, either with [`otelpgx`](https://github.com/exaring/otelpgx) or a small in-tree `pgx.QueryTracer` that opens spans from the request context. Spans should parent under the active RPC span so `GetActor` shows `db.query` children with statement names.

## Related test-documentation nit

A bug-bash test we ran expects `kubectl ate get actor --trace` to also capture a "worker handshake". That expectation is wrong by design: `GetActor` is a pure store read (`cmd/ateapi/internal/controlapi/actor.go` → `s.store.GetActor`), and worker spans (`step.AssignWorker`, `atelet.AteomHerder/*`) only appear on mutation traces. Worth correcting wherever that test recipe lives, but the actionable gap in this issue is the missing DB spans.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.