agent-substrate / agent-substrate/substrate

Initial set of platform metrics around lifecycle, scheduling, and fleet-saturation

未關閉
#433 7 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
area/observability kind/feature prio/P0
主要語言
Go
星號
1.8k
分支
316
平均合併
2 天 43 分鐘
30 天內合併 PR
287

描述

Opening this issue to track the foundational work around performance/scaling-oriented platform metrics that came up in #174 and in the load-testing threads (#168, #176, #98, #44). I proposing a high-level plan here first, but if we get too deep in the details (exact attributes, buckets, etc.), happy to open a Google Doc so we can discuss it there as well.

## Scope
When we load test substrate today, we cannot really answer core operational questions with today's telemetry, e.g. is the fleet healthy under load, and where is the bottleneck?
We have three metrics total (`rpc.server.call.duration`, `atenet.router.route.duration`, `atelet.snapshot.size`), but nothing for lifecycle operations, pool saturation, or scheduling outcomes, and some components (ateom, atecontroller) emit no metrics at all. I think we have a good foundation in #174, so I am cleaning that up and using this to track the implementation.

1. otlp push instead of pull
2. Cardinality-wise actor, atespace, and session atrs go on traces/logs/events only, never on metric labels. Template, worker pool, sandbox class, outcome/phase, and cluster are the metric dimensions. We can add exemplars on the duration histograms, so a latency spike can be tracked back to a sampled actor trace. This can be a follow-up.
3. Actor level analytics will be covered by events (named LogRecords in OTel). This came up in #174's "are there vital metrics we should push out to logs". It needs the cgroup work first (#161, blocked on #288), so just mentioning this here for context for now

## Initial set of metrics

ateapi emits the lifecycle/scheduler/fleet signals, atelet emits the restore/checkpoint phase information.

| Metric | Type | Use-case |
|---|---|---|
| `ate.actor.lifecycle.operation.duration` (`operation.name`, `error.type`, template, pool, sandbox class, snapshot kind) | histogram | activation-latency, suspend/resume rates and error rates |
| `ate.actors` (`status`, template) | updowncounter | fleet density, actor-to-worker multiplexing ratio |
| `ate.workerpool.workers` (pool, `state`, sandbox class) | observable updowncounter | pool saturation (see #198) |
| `ate.scheduler.assignment.duration` (pool, `outcome`, `error.type`) | histogram | "no free workers" rate (see #27, #398) |
| `ate.actor.restore.duration` / `ate.actor.checkpoint.duration` (`phase`, template, sandbox class, snapshot kind) | histograms | cold-start bottleneck, see related #166/#228 |

cc @alexvanboxel @juli4n, @zoez7, @EItanya

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。