Enable auto-scaling rules to remain active during rolling update deployment
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Background
Currently, auto-scaling (CheckReplicaHandler) only runs in READY state and rolling update (DeployingProvisioningHandler) only runs in DEPLOYING state. These two are mutually exclusive by lifecycle. The goal is to allow auto-scaling rules to continue evaluating and updating `desired_replica_count` while a rolling update is in progress, so that the rolling update FSM naturally adapts to the changing target.
## Current Architecture
- `CheckReplicaHandler` targets READY only → calculates `desired_replica_count` → transitions to SCALING
- `ScalingHandler` targets SCALING only → creates/terminates routes → transitions to READY
- `DeployingProvisioningHandler` targets DEPLOYING → runs rolling update FSM each cycle
- Rolling update FSM already reads `target_replica_count` fresh each cycle (rolling_update.py:82), so it can naturally adapt to changed values
## Why It Already Almost Works
The rolling update FSM in `RollingUpdateStrategy.evaluate_cycle()` reads `deployment.replica_spec.target_replica_count` every cycle. If auto-scaling updates `desired_replica_count` in DB during DEPLOYING, the next FSM cycle would pick up the new value and adjust surge/unavailability budgets automatically:
- **Scale up (4→6):** still_needed increases, FSM creates more new-revision routes
- **Scale down (6→4):** still_needed decreases, FSM drains more old-revision routes, fewer new-revision routes needed
## Required Work
### 1. Allow auto-scaling rule evaluation during DEPLOYING
CheckReplicaHandler currently targets only READY state. It needs to also evaluate deployments in DEPLOYING state, but only for the purpose of updating `desired_replica_count` — NOT triggering a SCALING transition. The scaling action itself should be left to the rolling update FSM.
- Option A: Extend CheckReplicaHandler to target both READY and DEPLOYING, with different transition logic (READY → SCALING for READY state, no lifecycle change for DEPLOYING state — just update desired_replica_count in DB)
- Option B: Extract the desired_replica_count calculation into a separate handler that runs regardless of lifecycle state, while keeping the SCALING transition only for READY state
### 2. Guard route eviction for deploying-revision routes
RouteEvictionHandler operates independently of deployment lifecycle. During DEPLOYING, new-revision routes that are temporarily UNHEALTHY (warming up) could be evicted. The eviction handler should be aware of the deploying revision and avoid evicting new-revision routes that are still in the warm-up window.
### 3. Handle scale-down below current new-revision route count
If auto-scaling scales down from 6 to 2 during a rolling update that already has 4 healthy new-revision routes, the FSM currently has no mechanism to terminate excess new-revision routes — it only drains old-revision routes. The FSM needs logic to handle `new_healthy > desired` by draining excess new-revision routes as well.
### 4. Completion condition with dynamic desired count
The current completion check is: `old_active == 0 AND new_healthy >= desired`. With a changing desired count, verify this still works correctly. For example, if desired drops from 6 to 4 and we have 4 new_healthy + 2 old_active, the FSM should drain the old routes and complete — not get stuck.
### 5. Observability
Log when the FSM detects a desired count change mid-deployment. This helps operators understand why the rolling update changed pace.
## Key Files
- `sokovan/deployment/strategy/rolling_update.py` — FSM: evaluate_cycle, _compute_routes_to_create/terminate
- `sokovan/deployment/handlers/replica.py` — CheckReplicaHandler (needs DEPLOYING support)
- `sokovan/deployment/executor.py` — calculate_desired_replicas, scale_deployment
- `sokovan/deployment/route/handlers/route_eviction.py` — RouteEvictionHandler (needs deploying-revision awareness)
- `data/deployment/types.py` — ReplicaSpec.target_replica_count
## Acceptance Criteria
- Auto-scaling rules continue to evaluate and update desired_replica_count during DEPLOYING state
- Rolling update FSM naturally adapts to changed desired count each cycle (already works by design)
- Scale-down during deployment correctly drains excess new-revision routes when new_healthy > desired
- Route eviction does not interfere with new-revision routes during warm-up
- Deployment completes correctly regardless of desired count changes mid-deployment
JIRA Issue: BA-5290
Contributor guide
Assessment
This issue has not been assessed yet.