agent-substrate / agent-substrate/substrate

feat: WorkerPool autoscaling — demand-reactive capacity for warm worker pools

Open
#198 11 comments 0 reactions 1 assignee Claimed by @laoj2 View on GitHub
area/api area/scheduling kind/feature
Dominant language
Go
Stars
1.8k
Forks
316
Avg merge
2d 43m
Merged PRs (30d)
287

Description

## Summary

`WorkerPool` replicas are **static** today: a human sets `spec.replicas` and the controller applies it verbatim. There is no mechanism that grows or shrinks a pool in response to actor demand. This issue opens a discussion on an autoscaling story for WorkerPools — what it should optimize for, why "just attach an HPA" is not sufficient for substrate's workload shape, and a concrete approach we've been exploring (a **warm-buffer** model with a **fast scale-up trigger** and a **slow scale-down**). The goal here is to align on direction and on the **in-tree vs. out-of-tree boundary**.

Autoscaling is already named as future work in `docs/roadmap.md` (*"Worker horizontal autoscaling: Ability to rapidly scale up nodes and warm Pods to meet actor demand"*) and `docs/architecture.md` (*"Autoscaling: We will need to be able to automatically scale the number of workers up and down based on demand…"*), but there's no design or tracking issue for it yet.

## Current state

- **`WorkerPoolSpec` carries only `replicas` + `ateomImage`**. No min/max bounds, no target utilization, no metric, no demand input.
- **The `scale` subresource is exposed**: the CRD declares
`+kubebuilder:subresource:scale:specpath=.spec.replicas,statuspath=.status.replicas`,
but HPA integration is incomplete; no pool-occupancy/capacity metrics exist
- **A full pool fails fast**: when no idle worker exists, `AssignWorkerStep` returns `FailedPrecondition "no free workers available"`, which the router maps to **HTTP 503**. There is no queue/wait (tracked separately in #27) and no capacity creation in response.

Net: the substrate partially provides the **lever** (the scale subresource) but no **control loop** and partial **signals** to drive one.

## Why autoscaling matters

1. **The density premise requires elasticity.** Substrate's whole value is multiplexing many actors onto few warm workers. Without autoscaling, operators must statically over-provision for peak (wasting money on idle workers) or under-provision (and serve 504 timeouts under load). Elastic pools are what let density translate into actual cost savings.
2. **Cost — scale-to-zero / scale-to-floor for idle pools.** Idle actors are just snapshots in object storage; a pool serving no active actors is pure waste. Autoscaling (down to a small floor, or to zero for cold-start-tolerant pools) directly reduces spend.
3. **Activation-latency SLO under bursts.** Agent workloads tend to wake in correlated storms (sub-agent tasks/CI fan-out). Maintaining warm headroom and refilling it reactively is what helps the storm pass.
4. **Eliminating avoidable 503s.** Today an empty pool returns 503 with no recourse. Pairing capacity creation with the request-parking work in #27 turns "no capacity → error" into "no capacity → brief wait → served."
5. **Operability.** Operators shouldn't hand-tune `replicas` per pool per time-of-day. A control loop (with sane bounds) mandatory for running this in production.

## The core constraint - why a naive metric→HPA isn't enough

The binding latency is **pod start** (schedule + image pull + `ateom` boot + readiness — seconds to minutes; plus node provisioning if the cluster is full).

**Pure metric-loop reactivity is bounded by detection lag.** Prometheus scrape (~15–30s) + KEDA/HPA poll & stabilization (tens of seconds) is perfectly fine for **smooth/diurnal** demand, but too slow as the *sole* up-path for **spiky wake-storms** with a lean buffer — the buffer empties and stays empty through the lag window.

So "reactive enough" should be read as *"fast enough to refill the buffer before the next comparable burst,"* not *"fast enough to serve this burst."*

## Proposed approach (for discussion)

### A warm-buffer invariant, replenished on consumption

Each pool targets a small number of **idle (warm) workers**. When a resume consumes a slot and the buffer dips below target, emit a refill — sized **net of in-flight provisioning** (anti-windup), so the loop doesn't pile on scale-ups while pods are still booting:

```
deficit = max(0, target - (free + in_flight)) // buffer refill
scale_to = clamp(current + deficit, reservation_floor, max_replicas). // rate-limited per step
```

When preemption becomes available, preemptible actors will be deducted from the deficit.

Image

### Where should this live?

Substrate is intentionally low-opinion, so the policy could reasonably live **outside** the core. Much of the above is buildable today as an external controller — it can drive the existing `scale` subresource and read pool occupancy via a new `WatchWorkers` API.

### Scale-to-zero and node headroom

- **Scale-to-zero** is attractive for idle/cold-start-tolerant pools, but the first post-zero request pays full cold start; keep a per-pool floor for latency-sensitive pools - Will handle in a follow-up issue
- **Pod scale-up only helps if a node has room.** A pod request that lands `Pending` behind cluster-autoscaler/Karpenter is not fast. Warm **node** headroom is a complementary lever worth calling out.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.