agent-substrate / agent-substrate/substrate
ateapi: postgres store has no trace instrumentation - DB time is invisible in every trace
- Linguagem predominante
- Go
- Estrelas
- 1.8k
- Forks
- 316
- Merge médio
- 2d 43min
- PRs com merge (30d)
- 287
Descrição
## Summary
The ateapi postgres store (`cmd/ateapi/internal/store/atepg/`) creates its `pgxpool` without any tracer hook, so no span is ever emitted for database work. Every RPC trace ends at the gRPC server span produced by the `otelgrpc` stats handler; the time spent in PostgreSQL is indistinguishable from handler/auth overhead.
## Observed
On a live GKE install with tracing enabled end to end:
- `kubectl ate get actor -a --trace` produces a trace with **exactly one span**: `ateapi.Control/GetActor` (~97 ms in our run). For a single-row read that is slow enough to want a breakdown, and the trace offers none.
- This is not specific to reads: rich mutation traces (25-span suspend, 33-span resume) contain `step.*`, atelet, ateom, and GCS client spans — but zero postgres spans, despite every one of those flows doing multiple DB round-trips (actor load, state transitions, outbox).
## Root cause
- `cmd/ateapi/internal/store/atepg/atepg.go` has no OTel/tracing references at all; the pool is built from `pgxpool.ParseConfig` with `ConnConfig.Tracer` unset.
- The only vendored OTel instrumentation libraries are `otelgrpc` and `net/http` — there is no pgx instrumentation in the dependency tree.
## Suggested fix
Attach a tracer via pgx v5's native hook: set `config.ConnConfig.Tracer` on the pool config, either with [`otelpgx`](https://github.com/exaring/otelpgx) or a small in-tree `pgx.QueryTracer` that opens spans from the request context. Spans should parent under the active RPC span so `GetActor` shows `db.query` children with statement names.
## Related test-documentation nit
A bug-bash test we ran expects `kubectl ate get actor --trace` to also capture a "worker handshake". That expectation is wrong by design: `GetActor` is a pure store read (`cmd/ateapi/internal/controlapi/actor.go` → `s.store.GetActor`), and worker spans (`step.AssignWorker`, `atelet.AteomHerder/*`) only appear on mutation traces. Worth correcting wherever that test recipe lives, but the actionable gap in this issue is the missing DB spans.
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.