agent-substrate / agent-substrate/substrate

DeleteActor never cleans the actor's node directory

Aberta
#641 3 comentários 0 reações 1 responsável Reivindicada por @igooch Ver no GitHub
area/node kind/bug prio/P1
Linguagem predominante
Go
Estrelas
1.8k
Forks
316
Merge médio
2d 43min
PRs com merge (30d)
287

Descrição

## Expected Behavior

Deleting an actor releases everything the actor held, including its node-local state: after
`DeleteActor`, `/var/lib/ateom-gvisor/actors//` should be removed (or at minimum contain
nothing that marks the actor as still placed on the node).

This matters doubly once #463 Phase 2 lands: the image-cache GC's root set is the per-bundle
`rootfs-overlay.json` specs — "spec on disk" means "actor placed here; do not evict its image."
A deleted actor should not root an image.

## Actual Behavior

`DeleteActor` is a pure control-plane operation: it deletes the store record and cleans up volumes
(`cmd/ateapi/internal/controlapi/delete_actor.go`), but nothing ever tells the node. atelet's RPC
surface is only `Run`/`Checkpoint`/`Restore`, and `resetActorDirs` runs lazily — at the start of the
next `Run`/`Restore` or the end of a `Checkpoint` for the **same actorUID**. A deleted actor's UID
never comes back, so `/var/lib/ateom-gvisor/actors//` persists until the node is replaced.

What leaks depends on the state the actor was deleted from (the store allows SUSPENDED or CRASHED,
`cmd/ateapi/internal/store/ateredis/ateredis.go`):

- **Deleted from CRASHED**: a crash means no `Checkpoint` ran after the last `Run`/`Restore`, so the
**entire bundle dir survives** — the actor's writable `upper/` layer and the
`bundles//rootfs-overlay.json` spec. Post-#463-Phase-2, that leaked spec is
indistinguishable from a live one, so the dead actor's image is **rooted forever**: GC logs it as
in-use on every pass and it can never be evicted, regardless of disk pressure. Hit twice during
Phase 2 GC validation. The GC is behaving as designed (ambiguity must resolve toward retention —
evicting a mounted lowerdir is catastrophic); the defect is that deletion leaves the "I am placed
here" marker behind. Filed separately from #463 so this over-retention isn't mistaken for a GC bug.
- **Deleted after a clean suspend** (the common path): the bundle was wiped at `Checkpoint`, but the
`ActorPath` skeleton, `sandbox-assets.json`, and any `local-checkpoint/` contents are left
behind.

## Steps to Reproduce the Problem

1. On a Kind cluster with substrate installed, run an actor, then crash it (e.g. kill its
workload) so it lands in CRASHED.
1. `kubectl ate delete actor ` — succeeds (CRASHED satisfies the delete precondition).
1. On the node: `docker exec ls /var/lib/ateom-gvisor/actors//bundles//`
→ `rootfs-overlay.json`, `upper/`, `work/`, `rootfs/` still present, forever.
1. (Variant for the suspended path) Suspend an actor, delete it, and observe `local-checkpoint/`
and `sandbox-assets.json` survive under the actor's dir.
1. (Post-#463-Phase-2) Put the node under disk pressure: GC logs the deleted actor's image as
rooted on every pass; it is never evicted.

## Anything else we need to know?:

The fix belongs in the deletion path, not in GC: something should remove `ActorPath(uid)` when the
actor is deleted. Cleanup should key on **deletion** (the actor record no longer exists), not on
the CRASHED transition — whether a crashed actor should be recoverable from its node-local state
is an open design question on #292, and deletion-keyed cleanup is safe under either answer. Two
candidate approaches:

- **Control plane calls atelet at delete time** (a new teardown RPC). Direct, but the control plane
must still know — and be able to reach — the actor's last node when the delete happens.
- **atelet reconciles `ActorsDir` against the control plane** (periodic sweep removing dirs for
actors that no longer exist). Self-healing — it also catches missed deletes and actors crashed on
dead workers (#618) — but it gives atelet a control-plane client it deliberately doesn't have
today.

A full "delete all" of the directory is safe: nothing can reference a deleted actor's node-local
state, because the actor record that would point to it is gone.

Related: #615 (node-side debris on an abnormal path), #172 (local snapshot storage management
umbrella).

## Specifications

- Version: main @ e1fa0040 (2026-07-28); GC interaction observed with the #463 Phase 2 POC branch
(`phase2-gc-record-first` @ a06ee866)
- Platform: Kind (v1.36.1, Debian GNU/Linux 13 (trixie), 6.8.0-117-generic (arm64),
containerd://2.3.1) and GKE (v1.35.5-gke.1163012, c3-standard-4, 6.12.85+, containerd://2.1.7) —
behavior is identical on both

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.