agent-substrate / agent-substrate/substrate
atelet state persistence: recover per-actor state across restarts
- Langage dominant
- Go
- Étoiles
- 1.8k
- Forks
- 316
- Merge moyen
- 2 j 43 min
- PR mergées (30 j)
- 287
Description
## Problem
atelet keeps per-actor state only in memory. The first concrete case is the
system-info volume refresher (#1231), which tracks each running actor's
volume specs so ClusterTrustBundle rotations rewrite projected files live.
Sandboxes outlive an atelet restart, so actors keep running afterwards, but
live refresh breaks until the next Run/Restore re-registers them. An actor
that never suspends keeps a stale egress MITM trust bundle and loses TLS
through the egress gateway on the next CA rotation (#1226).
More data sources will need the same treatment (JWTs, certificates), so this
should be solved once rather than per subsystem.
For comparison, kubelet persists nothing here: on restart it re-fetches its
pods' specs from kube-apiserver and reconstructs the rest from disk, so
recovery and enrollment are the same operation.
## Proposal
Move functionality that requires tracking per actor state (e.g. volume injection) into individual ateoms rather than atelet.
Ownership is split like so:
- **ateom** owns per-actor state and initiative: which volumes each of its
actors declares, when to refresh, and how to place files into the sandbox.
- **atelet** stays the node's only control-plane principal: one
ClusterTrustBundle informer and one ateapi client per node, serving
requests from ateoms over the node-local credential broker socket.
This is a similar shape to atunnel's relationship to atelet. ateom generates an actor key, sends a CSR to atelet over UDS, and atelet authenticates the worker and asserts identity to ateapi. This proposal generalizes this to every SystemInfo volume data source.
### Interface
Extend the credential broker service on atelet:
- `WatchTrustBundle(name)`: server stream. atelet's informer fans each event
out to open streams. When initializing the stream, each ateom provides what trust bundles
it needs, and atelet only provides those specific trust bundles.
- `MintJWT`, `SignCertificate`: called on ateom's schedule to generate/renew. Follows similar auth flow to
how certs are generated for atunnel. Keys are generated in ateom and never cross the socket.
ateom's RunWorkload/RestoreWorkload request carries the volume's data sources
(today only the mounts cross the boundary) and ateom does the initial
population before starting containers.
### State and recovery
- ateom holds actor state for exactly the sandbox's lifetime, in memory.
- atelet restart: streams drop, ateoms reconnect with backoff and
resubscribe. Recovery falls out of client retry with no persisted state
anywhere. While disconnected, files keep their last-good contents.
- ateom restart: not survivable today regardless of this design, since the
sandbox plumbing dies with it.
### Alternative Options
**A. ateom talks to the control plane directly.** Worker pods get their own
informer and ateapi client.
- Pro: ateom owns state; no intermediary
- Con: puts RBAC and ateapi credentials in the pod that shares a kernel
with the actor and one watch per pod instead of per node.
- Rejected: a structural security regression.
**B. atelet keeps everything and pushes to ateom.**
- Pro: boundary unchanged. Smallest change from #1231.
- Con: atelet becomes a timer service holding per-actor state, so the restart
gap stays and grows with each data source. For certificates it must ship keys
over the socket or schedule renewals for keys it cannot see.
- Rejected: entrenches the
state-ownership problem this issue exists to remove.
**C. atelet rebuilds its registry from ateapi on restart** (kubelet model).
- Pro: no persisted state.
- Con: a new ateapi RPC and a per-node actor listing on every restart
while ateapi is being scaled out. Keeps B's timer and key problems.
- Rejected: pays ateapi load to reconstruct state ateom already has.
**D. atelet persists a spec slice on the node.**
- Pro: cheapest.
- Con: Have to maintain derived local state and keep consistent
across crashes and churn.
- Rejected: trades a design problem for an operational one.
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.