agent-substrate / agent-substrate/substrate

DeleteActor never cleans the actor's node directory

Đang mở
#641 3 bình luận 0 reaction 1 người được giao Được @igooch nhận Xem trên GitHub
area/node kind/bug prio/P1
Ngôn ngữ chính
Go
Star
1.8k
Fork
316
Merge trung bình
2 ngày 43 phút
Pull request đã merge (30 ngày)
287

Mô tả

## Expected Behavior

Deleting an actor releases everything the actor held, including its node-local state: after
`DeleteActor`, `/var/lib/ateom-gvisor/actors//` should be removed (or at minimum contain
nothing that marks the actor as still placed on the node).

This matters doubly once #463 Phase 2 lands: the image-cache GC's root set is the per-bundle
`rootfs-overlay.json` specs — "spec on disk" means "actor placed here; do not evict its image."
A deleted actor should not root an image.

## Actual Behavior

`DeleteActor` is a pure control-plane operation: it deletes the store record and cleans up volumes
(`cmd/ateapi/internal/controlapi/delete_actor.go`), but nothing ever tells the node. atelet's RPC
surface is only `Run`/`Checkpoint`/`Restore`, and `resetActorDirs` runs lazily — at the start of the
next `Run`/`Restore` or the end of a `Checkpoint` for the **same actorUID**. A deleted actor's UID
never comes back, so `/var/lib/ateom-gvisor/actors//` persists until the node is replaced.

What leaks depends on the state the actor was deleted from (the store allows SUSPENDED or CRASHED,
`cmd/ateapi/internal/store/ateredis/ateredis.go`):

- **Deleted from CRASHED**: a crash means no `Checkpoint` ran after the last `Run`/`Restore`, so the
**entire bundle dir survives** — the actor's writable `upper/` layer and the
`bundles//rootfs-overlay.json` spec. Post-#463-Phase-2, that leaked spec is
indistinguishable from a live one, so the dead actor's image is **rooted forever**: GC logs it as
in-use on every pass and it can never be evicted, regardless of disk pressure. Hit twice during
Phase 2 GC validation. The GC is behaving as designed (ambiguity must resolve toward retention —
evicting a mounted lowerdir is catastrophic); the defect is that deletion leaves the "I am placed
here" marker behind. Filed separately from #463 so this over-retention isn't mistaken for a GC bug.
- **Deleted after a clean suspend** (the common path): the bundle was wiped at `Checkpoint`, but the
`ActorPath` skeleton, `sandbox-assets.json`, and any `local-checkpoint/` contents are left
behind.

## Steps to Reproduce the Problem

1. On a Kind cluster with substrate installed, run an actor, then crash it (e.g. kill its
workload) so it lands in CRASHED.
1. `kubectl ate delete actor ` — succeeds (CRASHED satisfies the delete precondition).
1. On the node: `docker exec ls /var/lib/ateom-gvisor/actors//bundles//`
→ `rootfs-overlay.json`, `upper/`, `work/`, `rootfs/` still present, forever.
1. (Variant for the suspended path) Suspend an actor, delete it, and observe `local-checkpoint/`
and `sandbox-assets.json` survive under the actor's dir.
1. (Post-#463-Phase-2) Put the node under disk pressure: GC logs the deleted actor's image as
rooted on every pass; it is never evicted.

## Anything else we need to know?:

The fix belongs in the deletion path, not in GC: something should remove `ActorPath(uid)` when the
actor is deleted. Cleanup should key on **deletion** (the actor record no longer exists), not on
the CRASHED transition — whether a crashed actor should be recoverable from its node-local state
is an open design question on #292, and deletion-keyed cleanup is safe under either answer. Two
candidate approaches:

- **Control plane calls atelet at delete time** (a new teardown RPC). Direct, but the control plane
must still know — and be able to reach — the actor's last node when the delete happens.
- **atelet reconciles `ActorsDir` against the control plane** (periodic sweep removing dirs for
actors that no longer exist). Self-healing — it also catches missed deletes and actors crashed on
dead workers (#618) — but it gives atelet a control-plane client it deliberately doesn't have
today.

A full "delete all" of the directory is safe: nothing can reference a deleted actor's node-local
state, because the actor record that would point to it is gone.

Related: #615 (node-side debris on an abnormal path), #172 (local snapshot storage management
umbrella).

## Specifications

- Version: main @ e1fa0040 (2026-07-28); GC interaction observed with the #463 Phase 2 POC branch
(`phase2-gc-record-first` @ a06ee866)
- Platform: Kind (v1.36.1, Debian GNU/Linux 13 (trixie), 6.8.0-117-generic (arm64),
containerd://2.3.1) and GKE (v1.35.5-gke.1163012, c3-standard-4, 6.12.85+, containerd://2.1.7) —
behavior is identical on both

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.