agent-substrate / agent-substrate/substrate

Actor instrumented metrics might merge and inflate

未關閉
#761 6 則留言 0 個 reaction 已指派 1 人 已被 @git286 認領 在 GitHub 檢視
area/observability kind/bug
主要語言
Go
星號
1.8k
分支
316
平均合併
2 天 43 分鐘
30 天內合併 PR
287

描述

Any workload that exports OTel metrics loses per-actor identity when it moves to Substrate. Counters come out wrong, and nothing errors.

Two things cause it.

**Everything computed at startup is shared.** The golden snapshot is taken after init, so an OTel `Resource` built in `main()` is identical for every actor of the template. The usual escapes don't work either: `os.Hostname()` is hardcoded to `runsc` for every sandbox (`cmd/atelet/oci.go:271`), and ActorTemplate doesn't support `valueFrom.fieldRef`, so you can't inject a pod name (`docs/api-guide.md:101`).

**Collector enrichment attributes to the worker pod.** Actor egress is masqueraded behind the pod IP in both ateom runtimes (`cmd/ateom-microvm/net.go:22`), so `k8sattributes` stamps the *host worker's* `k8s.pod.name` / `k8s.node.name`. That merges every co-located actor into one series, and changes an actor's series identity every time it migrates. Every actor also shares one constant interior IP (`169.254.17.2`), so no IP-based processor can tell them apart even in principle.

Note that push doesn't get us out of this, it's what gets us into it. Actors can't be scraped, and the pull model is what hands you an `instance` label for free, derived from the target address rather than from anything the workload says. With OTLP push, identity has to come from the payload, and the payload is the part that's frozen.

The result is one series with many writers. OTLP counters default to cumulative, so the merged series goes up and down, `rate()`/`increase()` treat every decrease as a reset, and totals inflate. Gauges show a random actor. The dashboards look plausible, which is the worst part and the pod/node labels make it read like ordinary pod churn, so you debug the wrong thing.

Adding `actor_uid` as a metric label isn't the fix; it's unbounded and churns on every actor create.

Things I think we need to decide:

- what identifies an actor's metrics on the wire vs. what survives into storage
- whether Substrate stamps it out-of-process (the identity dir seems like the natural place, since it's outside the snapshot) or we leave it to actor authors
- interim guidance for actor authors either way

Related: #503 (telemetry continuity), #450 (lifecycle hooks). PR #750's demo hits this: its `actorName()` returns `runsc` for every actor.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。