agent-substrate / agent-substrate/substrate

Actor instrumented metrics might merge and inflate

Open
#761 6 comments 0 reactions 1 assignee Claimed by @git286 View on GitHub
area/observability kind/bug
Dominant language
Go
Stars
1.8k
Forks
316
Avg merge
2d 43m
Merged PRs (30d)
287

Description

Any workload that exports OTel metrics loses per-actor identity when it moves to Substrate. Counters come out wrong, and nothing errors.

Two things cause it.

**Everything computed at startup is shared.** The golden snapshot is taken after init, so an OTel `Resource` built in `main()` is identical for every actor of the template. The usual escapes don't work either: `os.Hostname()` is hardcoded to `runsc` for every sandbox (`cmd/atelet/oci.go:271`), and ActorTemplate doesn't support `valueFrom.fieldRef`, so you can't inject a pod name (`docs/api-guide.md:101`).

**Collector enrichment attributes to the worker pod.** Actor egress is masqueraded behind the pod IP in both ateom runtimes (`cmd/ateom-microvm/net.go:22`), so `k8sattributes` stamps the *host worker's* `k8s.pod.name` / `k8s.node.name`. That merges every co-located actor into one series, and changes an actor's series identity every time it migrates. Every actor also shares one constant interior IP (`169.254.17.2`), so no IP-based processor can tell them apart even in principle.

Note that push doesn't get us out of this, it's what gets us into it. Actors can't be scraped, and the pull model is what hands you an `instance` label for free, derived from the target address rather than from anything the workload says. With OTLP push, identity has to come from the payload, and the payload is the part that's frozen.

The result is one series with many writers. OTLP counters default to cumulative, so the merged series goes up and down, `rate()`/`increase()` treat every decrease as a reset, and totals inflate. Gauges show a random actor. The dashboards look plausible, which is the worst part and the pod/node labels make it read like ordinary pod churn, so you debug the wrong thing.

Adding `actor_uid` as a metric label isn't the fix; it's unbounded and churns on every actor create.

Things I think we need to decide:

- what identifies an actor's metrics on the wire vs. what survives into storage
- whether Substrate stamps it out-of-process (the identity dir seems like the natural place, since it's outside the snapshot) or we leave it to actor authors
- interim guidance for actor authors either way

Related: #503 (telemetry continuity), #450 (lifecycle hooks). PR #750's demo hits this: its `actorName()` returns `runsc` for every actor.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.