Remove the transition_role=queue_storage_target requirement: re-resolve auto runtimes at the flip, or allow a quiesced-cluster flip
- Dominant language
- Rust
- Stars
- 32
- Forks
- 5
- Avg merge
- 15h 21m
- Merged PRs (30d)
- 21
Description
## Problem
`enter-mixed-transition` requires a live runtime registered with `transition_role=queue_storage_target`. The role exists because a 0.6 runtime binds its execution engine once at startup: an auto runtime started pre-flip resolves canonical and can only *downgrade* to drain-only when the flip lands — it never starts executing queue-storage work. So the gate demands an explicit witness that someone will execute the new plane; `storage_capability=queue_storage` alone can't prove that.
In practice this forces every embedding application to plumb `TransitionWorkerRole::QueueStorageTarget` through its own config/deploy machinery for a one-time operation, then remove it (or leave a permanently misleading builder call). Verified on 0.6.5 that a live queue-storage-*capable* auto runtime is refused, and that a fully quiesced cluster (zero fresh heartbeats) is also refused.
## Proposal 1 — auto runtimes re-resolve at the flip (removes the role)
Runtimes already observe transition state changes — that's how auto runtimes downgrade to `canonical_drain_only` after the flip. Instead of only downgrading, an auto runtime that observes `mixed_transition` could re-resolve and bring up its queue-storage runtime in place (internally close to "restart the client": a runtime started fresh during `mixed_transition` already resolves to queue storage, so the resolution logic exists; this is supervisor-level re-init, not new engine code).
The gate then relaxes to "≥1 live runtime with `storage_capability=queue_storage`", and `TransitionWorkerRole` can be deprecated for operators. This also fixes two adjacent sharp edges observed in a staged-transition rehearsal:
- a pre-flip maintenance leader never promotes queue-storage `deferred_jobs` (its promotion path branches on its startup-resolved engine), so due queue-storage jobs sit unpromoted until leadership moves to a post-flip runtime;
- the "roll all workers after finalize" step stops being load-bearing.
## Proposal 2 — allow the flip on a quiesced cluster (much smaller)
If **no** runtime has a fresh heartbeat, allow `enter-mixed-transition`. With nothing live there is no fleet to strand: nothing can execute canonical work either, and any 0.6 runtime started after the flip resolves to queue storage at startup. This gives operators a role-free path: scale the fleet to zero, flip, (optionally migrate the deferred backlog, see #456), finalize, scale back up. For small/bursty deployments a brief quiesce window is often easier than coordinating a witness runtime.
Implementation wrinkle: the gate lives in the database (`awa.storage_enter_mixed_transition`, installed by v014), so relaxing it in SQL means a new migration on the 0.6 line — awkward with 0.7 owning v041–v043. Alternative: a CLI/`awa-model`-side path (e.g. `enter-mixed-transition --quiesced`) that performs its own liveness check and the state transition directly, refusing if any fresh heartbeat exists.
The two compose: quiesced clusters flip with no role today; live clusters keep the witness until proposal 1 lands, after which the role disappears entirely.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the enter-mixed-transition entry point, the awa.storage_enter_mixed_transition function installed by v014, and the TransitionWorkerRole::QueueStorageTarget handling. Compare the two proposals and inspect the mentioned awa-model --quiesced path. Done means the chosen transition path safely handles live or quiesced runtimes without stranding canonical work, with its liveness and state-transition behavior covered by the relevant tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, rust
- Domain
- backend, cli, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100