agent-substrate / agent-substrate/substrate

ate-api-server: add /statusz (feature parity with atenet router, KEP-4827 convention)

オープン
#1,458 コメント 1 件 リアクション 0 件 担当者 0 名 GitHub で見る
area/api-machinery area/observability bug-bash kind/feature prio/P2
主要言語
Go
スター
1.9k
フォーク
317
平均マージ
1日 23時間
マージ済み PR(30日)
289

説明

## Summary

Feature request: give `ate-api-server` a `/statusz` endpoint following the pattern the atenet router already ships (`--status-port`, default 4040, see `cmd/atenet/internal/router/status.go`). Today the two control-plane components are inconsistent: the router serves a rich HTML status dashboard, while ateapi has nothing on 4040 (connection refused) and its only HTTP port (9090) serves just `/metrics` and `/healthz`.

This is also the direction upstream Kubernetes is taking for its own control-plane components via [KEP-4827 component statusz](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/4827-component-statusz) (with `/flagz` in KEP-4828), so following it here means adopting an emerging convention rather than inventing one.

## Motivation — three real debugging sessions that needed it

During a bug bash on a live GKE install (substrate rev 0d85e8572161), every ateapi-side incident required log spelunking that a status page would have shortcut:

1. **Wedged actors after template deletion.** Deleting an ActorTemplate with live actors left them unable to suspend ("actor template not found") or delete (`FailedPrecondition`). ateapi's own metrics counted these failures (`ate_actor_lifecycle_operation_duration_seconds{error_type="FailedPrecondition"}`), but there was no place to *see* recent failing RPCs. The router's "Recent 100 Processing Inquiries" table is exactly the right pattern; ateapi needs the control-plane equivalent.
2. **Silent golden-snapshot failure.** A template with an unpullable image (MANIFEST_UNKNOWN) retried `AteomHerder/Run` every ~20s for 45 minutes while `kubectl ate get actor-template` showed an empty ERROR column and `status: {}`. A statusz section for in-flight template/golden work would have surfaced the retry loop and its last error immediately.
3. **Postgres invisibility.** DB behavior is currently invisible in traces (#1455) *and* there is no page showing pool utilization, connectivity, or outbox depth — the two gaps compound.

The router page also proved its worth in the same bash: an operator flag change (`--route-timeout=180s`) was immediately visible under "Global Arguments & Configuration".

## Suggested content

- Build revision, uptime, resolved flags (KEP-4827/4828 style)
- PostgreSQL: pool stats (in-use/idle/max), connectivity health check with ok/err counters + last-known-good, outbox depth/lag
- Worker cache: known workers per pool, eligible-worker counts
- Recent N control-plane RPCs with method, caller identity, status, latency (router-table analogue), or at minimum recent non-OK RPCs
- Golden-snapshot / template work in flight, with last error per template
- Drain state (`--drain-delay` etc. are already flags)

## Two cautions

1. **Redaction**: the router page dumps full `os.Args` and resolved flags verbatim. ateapi's flags include `--postgres-connection-string` (password-free on GKE installs thanks to mTLS cert auth, but other deployments may embed credentials in it). Connection strings and env-sourced values must be redacted on the page.
2. **Separate port**: like the router, serve it on a dedicated `--status-port`, cluster-internal only — never on the 443 gRPC listener.

## Related

- #1457 — the router's existing statusz has an RBAC bug in its Service IP panel; worth keeping in mind so a shared status-page framework (extracting the router's implementation into an internal package) doesn't inherit it.
- #1455 — postgres trace instrumentation gap; the DB section above is the status-page half of the same visibility problem.

コントリビューションガイド

コントリビューションガイドを開く

調査の方向性

Start in `cmd/atenet/internal/router/status.go` to mirror how the router status page is structured, then locate the ate-api-server HTTP setup that currently exposes `/metrics` and `/healthz`. Add a dedicated `--status-port` listener and wire `/statusz` there, reusing existing flag/config handling. Inspect the relevant config, DB, worker, and RPC paths to populate status data while redacting sensitive connection-string or env-derived values. Done when the new endpoint responds with status information and `9090` behavior remains focused on existing health/metrics routes.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
go
領域
backend
issue の種類
機能追加
難易度
4/5
見積もり時間
3〜5日
活発さ
活発
明瞭さ
おおむね明確
初心者へのやさしさ
58/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。