agent-substrate / agent-substrate/substrate
Actor instrumented metrics might merge and inflate
- Vorherrschende Sprache
- Go
- Sterne
- 1.8k
- Forks
- 316
- Ø Merge
- 2 T. 43 Min.
- Gemergte PRs (30 T.)
- 287
Beschreibung
Any workload that exports OTel metrics loses per-actor identity when it moves to Substrate. Counters come out wrong, and nothing errors.
Two things cause it.
**Everything computed at startup is shared.** The golden snapshot is taken after init, so an OTel `Resource` built in `main()` is identical for every actor of the template. The usual escapes don't work either: `os.Hostname()` is hardcoded to `runsc` for every sandbox (`cmd/atelet/oci.go:271`), and ActorTemplate doesn't support `valueFrom.fieldRef`, so you can't inject a pod name (`docs/api-guide.md:101`).
**Collector enrichment attributes to the worker pod.** Actor egress is masqueraded behind the pod IP in both ateom runtimes (`cmd/ateom-microvm/net.go:22`), so `k8sattributes` stamps the *host worker's* `k8s.pod.name` / `k8s.node.name`. That merges every co-located actor into one series, and changes an actor's series identity every time it migrates. Every actor also shares one constant interior IP (`169.254.17.2`), so no IP-based processor can tell them apart even in principle.
Note that push doesn't get us out of this, it's what gets us into it. Actors can't be scraped, and the pull model is what hands you an `instance` label for free, derived from the target address rather than from anything the workload says. With OTLP push, identity has to come from the payload, and the payload is the part that's frozen.
The result is one series with many writers. OTLP counters default to cumulative, so the merged series goes up and down, `rate()`/`increase()` treat every decrease as a reset, and totals inflate. Gauges show a random actor. The dashboards look plausible, which is the worst part and the pod/node labels make it read like ordinary pod churn, so you debug the wrong thing.
Adding `actor_uid` as a metric label isn't the fix; it's unbounded and churns on every actor create.
Things I think we need to decide:
- what identifies an actor's metrics on the wire vs. what survives into storage
- whether Substrate stamps it out-of-process (the identity dir seems like the natural place, since it's outside the snapshot) or we leave it to actor authors
- interim guidance for actor authors either way
Related: #503 (telemetry continuity), #450 (lifecycle hooks). PR #750's demo hits this: its `actorName()` returns `runsc` for every actor.
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.