Introduce ReplicaGroup and resolve scaling/deploy race
- Dominant language
- Python
- Stars
- 670
- Forks
- 183
- Avg merge
- 17h 7m
- Merged PRs (30d)
- 358
Description
## Background
The current deployment model has a race condition between scaling and deploy handlers because they operate on the same `desired_replica_count` field through separate locks (`LOCKID_DEPLOYMENT_AUTO_SCALER`, `LOCKID_DEPLOYMENT_CHECK_REPLICA`, `LOCKID_DEPLOYMENT_DEPLOYING`). Specifically:
- `_evaluate_deployment_scaling` selects scale-in candidates revision-blind, which can terminate routes belonging to a new revision being rolled out.
- `target_replica_count` carries two meanings (rollout final goal vs current revision's current goal) which the scaling and rolling handlers interpret differently.
- `Route.traffic_ratio` is stored but never applied to routing logic; it is fundamentally placed at the wrong level (per-route instead of per-group).
- Blue-Green deployment strategy raises `NotImplementedError`. Canary is not supported.
- The existing 2-pointer model (`current_revision_id`, `deploying_revision_id`) on the deployment cannot represent multiple parallel groups needed for canary or blue-green.
## Goals
1. Eliminate the race between scaling decisions and deploy progression by unifying them into a single mutator path.
2. Introduce `ReplicaGroup` as a first-class entity that owns the per-group `desired_replica_count`, `traffic_weight`, `current_revision_id`, `target_revision_id`, and lifecycle phase.
3. Limit each deployment to at most 2 groups, enforced structurally by reconciler step ordering (cleanup before creation), not by DB constraint or lock.
4. Add 5 sub_steps to the deploy axis: `INITIALIZING`, `PROVISIONING`, `PROMOTING`, `DRAINING`, `COMPLETED`. Each strategy uses these sub_steps differently (rolling skips DRAINING; canary/BG go through all 5).
5. Remove `Route.traffic_ratio` in favor of `ReplicaGroup.traffic_weight`.
6. Implement Blue-Green and Canary strategies on the new infrastructure.
## Non-goals
- New user-facing API design (no `/rollout` endpoint, etc.)
- UX changes
- Per-group coordinator (group state is handled within deployment-level reconciler)
## Approach
The work is split into 6 stories under this Epic, executed in dependency order:
1. Model setup (ReplicaGroup, Route.group_id, Deployment pointers, traffic_ratio removal, migration)
2. Deploy handler rewrite (5-stage sub_step, rolling on ReplicaGroup)
3. Scaling handler rewrite (absorb CheckReplica, revision-specific desired counts)
4. Reconcile handler refinement (invariant recovery)
5. Blue-Green strategy implementation
6. Canary strategy implementation
The core invariant — that each deployment has at most 2 ReplicaGroups, with `target_group_id` set only during rollouts — is maintained by the reconciler's fixed step order (cleanup → transition → initialize → strategy step → drift correction) within a deployment-level dispatch lock.
JIRA Issue: BA-6232
Contributor guide
Assessment
This issue has not been assessed yet.