agent-substrate / agent-substrate/substrate

ateapi: postgres store has no trace instrumentation - DB time is invisible in every trace

未關閉
#1,455 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/api-machinery area/observability bug-bash kind/bug prio/P1
主要語言
Go
星號
1.8k
分支
316
平均合併
2 天 43 分鐘
30 天內合併 PR
287

描述

## Summary

The ateapi postgres store (`cmd/ateapi/internal/store/atepg/`) creates its `pgxpool` without any tracer hook, so no span is ever emitted for database work. Every RPC trace ends at the gRPC server span produced by the `otelgrpc` stats handler; the time spent in PostgreSQL is indistinguishable from handler/auth overhead.

## Observed

On a live GKE install with tracing enabled end to end:

- `kubectl ate get actor -a --trace` produces a trace with **exactly one span**: `ateapi.Control/GetActor` (~97 ms in our run). For a single-row read that is slow enough to want a breakdown, and the trace offers none.
- This is not specific to reads: rich mutation traces (25-span suspend, 33-span resume) contain `step.*`, atelet, ateom, and GCS client spans — but zero postgres spans, despite every one of those flows doing multiple DB round-trips (actor load, state transitions, outbox).

## Root cause

- `cmd/ateapi/internal/store/atepg/atepg.go` has no OTel/tracing references at all; the pool is built from `pgxpool.ParseConfig` with `ConnConfig.Tracer` unset.
- The only vendored OTel instrumentation libraries are `otelgrpc` and `net/http` — there is no pgx instrumentation in the dependency tree.

## Suggested fix

Attach a tracer via pgx v5's native hook: set `config.ConnConfig.Tracer` on the pool config, either with [`otelpgx`](https://github.com/exaring/otelpgx) or a small in-tree `pgx.QueryTracer` that opens spans from the request context. Spans should parent under the active RPC span so `GetActor` shows `db.query` children with statement names.

## Related test-documentation nit

A bug-bash test we ran expects `kubectl ate get actor --trace` to also capture a "worker handshake". That expectation is wrong by design: `GetActor` is a pure store read (`cmd/ateapi/internal/controlapi/actor.go` → `s.store.GetActor`), and worker spans (`step.AssignWorker`, `atelet.AteomHerder/*`) only appear on mutation traces. Worth correcting wherever that test recipe lives, but the actionable gap in this issue is the missing DB spans.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。