agent-substrate / agent-substrate/substrate
[Design] ate.actor.activation
- Dominant language
- Go
- Stars
- 1.8k
- Forks
- 316
- Avg merge
- 2d 43m
- Merged PRs (30d)
- 287
Description
**Living design doc for easier collaboration, please use this for comments**: https://docs.google.com/document/d/1tpUZNqH-xaSXzMVJf95CpWtKbIBD-mH93nqlDcEBOe8/edit?tab=t.0
It requires to be in: ate-dev@googlegroups.com
## Problem
An actor occupies many workers throughout its life. It runs on worker A for an hour, suspends, resumes on worker B three days later, crashes, comes back. Each time it occupies a worker we emit spans and logs, but we have nothing to refer to these iperiods, or "residencies" by. So for example "show me everything this actor did on Tuesday afternoon" is not a query we can run today.
Proposal: create an id per "residency", call it activation, and set it on the signals where high cardinality is allowed, so spans and logs.
## High level mechanism
ateapi creates a ULID/UUID (having it sortable by creation timestamp is nice) when it commits an actor into RUNNING, stores it on the Actor record, and it propagates trough the restore path (ateapi -> atelet -> ateom), for which we already have all the plumbing in place.
Where it goes
- spans: lifecycle operations plus the router resume span
- logs: through ateattr, next to the actor identity keys we set in #1035
- usage events later (#550 / #896): a residency is the natural scope for "this cost X".
- NOT on metrics
- entities later, if we want to adopt them. its own type, activation_of the actor, runs_on the pod, see https://opentelemetry.io/docs/specs/otel/entities/entity-events/
We should consider storing the previous residency's suspend span context on the Actor record, and link each resume span back to it.
## Benefits
- the actor timeline becomes one query instead of lots of joins across pod names and time windows
- usage and cost can be split into residency beyond actor and request
- we can make crash loops become visible as incarnation rate is increasing
- for #853, it seems to be the proper writer identity for the relayed metric stream. One incarnation is exactly one writer by design, which is a cleaner than a worker pod
## Open questions
1. What ends an incarnation apart from SUSPEND? What about PAUSE, where the actor may come back to the same worker? Is an incarnation one running interval or one placement? This will affect the design.
2. Proto field or telemetry-internal? On the proto it's more useful and clients can correlate against it, but it's an API change, and we have multiple changes in-flgiht for the API already
cc. @git286, @juli4n, @maxsmythe
Contributor guide
Assessment
This issue has not been assessed yet.