hardbyte / hardbyte/awa

Remove the transition_role=queue_storage_target requirement: re-resolve auto runtimes at the flip, or allow a quiesced-cluster flip

Open
#457 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
32
Forks
5
Avg merge
15h 21m
Merged PRs (30d)
21

Description

## Problem

`enter-mixed-transition` requires a live runtime registered with `transition_role=queue_storage_target`. The role exists because a 0.6 runtime binds its execution engine once at startup: an auto runtime started pre-flip resolves canonical and can only *downgrade* to drain-only when the flip lands — it never starts executing queue-storage work. So the gate demands an explicit witness that someone will execute the new plane; `storage_capability=queue_storage` alone can't prove that.

In practice this forces every embedding application to plumb `TransitionWorkerRole::QueueStorageTarget` through its own config/deploy machinery for a one-time operation, then remove it (or leave a permanently misleading builder call). Verified on 0.6.5 that a live queue-storage-*capable* auto runtime is refused, and that a fully quiesced cluster (zero fresh heartbeats) is also refused.

## Proposal 1 — auto runtimes re-resolve at the flip (removes the role)

Runtimes already observe transition state changes — that's how auto runtimes downgrade to `canonical_drain_only` after the flip. Instead of only downgrading, an auto runtime that observes `mixed_transition` could re-resolve and bring up its queue-storage runtime in place (internally close to "restart the client": a runtime started fresh during `mixed_transition` already resolves to queue storage, so the resolution logic exists; this is supervisor-level re-init, not new engine code).

The gate then relaxes to "≥1 live runtime with `storage_capability=queue_storage`", and `TransitionWorkerRole` can be deprecated for operators. This also fixes two adjacent sharp edges observed in a staged-transition rehearsal:

- a pre-flip maintenance leader never promotes queue-storage `deferred_jobs` (its promotion path branches on its startup-resolved engine), so due queue-storage jobs sit unpromoted until leadership moves to a post-flip runtime;
- the "roll all workers after finalize" step stops being load-bearing.

## Proposal 2 — allow the flip on a quiesced cluster (much smaller)

If **no** runtime has a fresh heartbeat, allow `enter-mixed-transition`. With nothing live there is no fleet to strand: nothing can execute canonical work either, and any 0.6 runtime started after the flip resolves to queue storage at startup. This gives operators a role-free path: scale the fleet to zero, flip, (optionally migrate the deferred backlog, see #456), finalize, scale back up. For small/bursty deployments a brief quiesce window is often easier than coordinating a witness runtime.

Implementation wrinkle: the gate lives in the database (`awa.storage_enter_mixed_transition`, installed by v014), so relaxing it in SQL means a new migration on the 0.6 line — awkward with 0.7 owning v041–v043. Alternative: a CLI/`awa-model`-side path (e.g. `enter-mixed-transition --quiesced`) that performs its own liveness check and the state transition directly, refusing if any fresh heartbeat exists.

The two compose: quiesced clusters flip with no role today; live clusters keep the witness until proposal 1 lands, after which the role disappears entirely.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the enter-mixed-transition entry point, the awa.storage_enter_mixed_transition function installed by v014, and the TransitionWorkerRole::QueueStorageTarget handling. Compare the two proposals and inspect the mentioned awa-model --quiesced path. Done means the chosen transition path safely handles live or quiesced runtimes without stranding canonical work, with its liveness and state-transition behavior covered by the relevant tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, rust
Domain
backend, cli, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.