agent-substrate / agent-substrate/substrate

Local snapshot node pinning breaks worker fungibility and causes unrecoverable resume errors if the free workers are on a different node

Offen
#589 1 Kommentar 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
area/scheduling kind/bug
Vorherrschende Sprache
Go
Sterne
1.8k
Forks
316
Ø Merge
2 T. 43 Min.
Gemergte PRs (30 T.)
287

Beschreibung

## Summary

Local snapshots are tied to the specific nodes where the actor was running when paused. However, workers that belong to the same worker pool can be spread across different nodes in the cluster. As a result, the hard dependency between paused actors (local snapshots) and specific nodes cannot always be satisfied when no workers are available on that node.

This issue is broader than autoscaling, but it completely breaks autoscaling operations. We need to remove or mitigate this hard node dependency when there are no workers available for that cluster/node.

## More Details

1. **Worker Pool Spread vs. Pinned Local Snapshots**

- In a multi-node cluster, pods in a `WorkerPool` are distributed across various nodes.
- When an actor is paused with a local snapshot, its metadata records the specific node VM where the local snapshot resides
- Because any subsequent resume requires a worker pod on that exact node, workers across the pool are no longer fungible for that actor. If all worker pods on that node are busy or none exist, the actor cannot resume—even if there are plenty of free workers on other nodes in the same worker pool.

2. **Problems for Autoscaling**

(see https://github.com/agent-substrate/substrate/issues/198#issuecomment-5109957709)
- **Scale-Down Strands Paused Actors**: When a worker pod or node is removed during scale-down (HPA, `kubectl scale, etc`), a paused actor's local snapshot remains pinned to a node that may no longer have any free worker pods (or any worker pods at all).
- **Scale-Up Gives No Locality Guarantee**: If paused actors with local snapshots cannot resume because no worker pods are available on their pinned node, scaling up the worker pool may not unblock them (if the new workers land on a different node)

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.