agent-substrate / agent-substrate/substrate

ateapi: postgres store has no trace instrumentation - DB time is invisible in every trace

オープン
#1,455 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
area/api-machinery area/observability bug-bash kind/bug prio/P1
主要言語
Go
スター
1.8k
フォーク
316
平均マージ
2日 43分
マージ済み PR(30日)
287

説明

## Summary

The ateapi postgres store (`cmd/ateapi/internal/store/atepg/`) creates its `pgxpool` without any tracer hook, so no span is ever emitted for database work. Every RPC trace ends at the gRPC server span produced by the `otelgrpc` stats handler; the time spent in PostgreSQL is indistinguishable from handler/auth overhead.

## Observed

On a live GKE install with tracing enabled end to end:

- `kubectl ate get actor -a --trace` produces a trace with **exactly one span**: `ateapi.Control/GetActor` (~97 ms in our run). For a single-row read that is slow enough to want a breakdown, and the trace offers none.
- This is not specific to reads: rich mutation traces (25-span suspend, 33-span resume) contain `step.*`, atelet, ateom, and GCS client spans — but zero postgres spans, despite every one of those flows doing multiple DB round-trips (actor load, state transitions, outbox).

## Root cause

- `cmd/ateapi/internal/store/atepg/atepg.go` has no OTel/tracing references at all; the pool is built from `pgxpool.ParseConfig` with `ConnConfig.Tracer` unset.
- The only vendored OTel instrumentation libraries are `otelgrpc` and `net/http` — there is no pgx instrumentation in the dependency tree.

## Suggested fix

Attach a tracer via pgx v5's native hook: set `config.ConnConfig.Tracer` on the pool config, either with [`otelpgx`](https://github.com/exaring/otelpgx) or a small in-tree `pgx.QueryTracer` that opens spans from the request context. Spans should parent under the active RPC span so `GetActor` shows `db.query` children with statement names.

## Related test-documentation nit

A bug-bash test we ran expects `kubectl ate get actor --trace` to also capture a "worker handshake". That expectation is wrong by design: `GetActor` is a pure store read (`cmd/ateapi/internal/controlapi/actor.go` → `s.store.GetActor`), and worker spans (`step.AssignWorker`, `atelet.AteomHerder/*`) only appear on mutation traces. Worth correcting wherever that test recipe lives, but the actionable gap in this issue is the missing DB spans.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。