P0: Restore the exact service pair within bounded compensation budgets

Problem

Automatic compensation for failed exact-SHA release run 31289994036 did not produce terminal recovery evidence even though ECS later converged to the captured prior pair.

The direct #97 release failure was the stale /courses/ canonical smoke assertion, not an ECS failure. Compensation began afterward. The controller spent about 160 seconds restoring and waiting for web before it issued the worker restore. It then gave the singleton worker the unrelated 180-second general-stage budget. That wait expired at about 183 seconds, while ECS completed the exact worker restore about 280 seconds after the worker restore began and within the accepted 420-second singleton-worker behavior.

Current _compensate() is therefore serial in the harmful dimension: bind web receipt -> wait web terminal -> bind worker receipt -> wait worker terminal. This issue replaces that recovery sequence with receipt-bound, back-to-back restore initiation and independently bounded cooperative observation. It does not reinterpret the failed run as successful and does not solve the defect by increasing the generic timeout.

Normative references

• AWS development deployment and Terraform: immutable web/worker deployment, exact SHA, automatic failed-readiness recovery, and preserved rollback target.
• Migration and rollout: immutable rollback, worker reconciliation, and restore rehearsal.
• Verification strategy: deployment fault injection, exact deployed-SHA checks, rollback/restore evidence, and redaction.
• Development release runbook: receipt identity, zero-work predecessor retirement, terminal pair proof, worker singleton, fixed 3600-second session, and recovery reserve.
• #96 accepted baseline: bounded monotonic receipt/adoption behavior and exact ACTIVE/DRAINING predecessor rules; this issue builds on it rather than weakening it.
• #97 runtime evidence: exact timings and attribution for the compensation false terminal failure.
• #101: separate forward-rollout web task/ALB coherence hardening, explicitly outside this issue.

Scope

Recovery budgets

Introduce three distinct code-owned recovery bounds:

• web recovery budget: exactly 240 seconds, with a hard maximum of 240;
• singleton-worker recovery budget: exactly 420 seconds, with a hard maximum of 420;
• whole recovery-phase cap: exactly 720 seconds, with a hard maximum of 720.

The existing 180-second general-stage/public-health budget, 240-second forward-web budget, 420-second forward-worker budget, 180-second browser-smoke budget, and 3600-second deployer session remain separate and unchanged. Recovery values may be represented by explicit typed configuration/CLI boundaries, but the development workflow supplies only the reviewed fixed values, exposes no workflow-dispatch/operator input for them, and rejects malformed or over-maximum values.

A single recovery-phase monotonic deadline starts before the first attempted-predecessor observation or recovery mutation. Each workload receives its own absolute deadline when recovery work for that workload begins: min(workload start + workload recovery budget, phase deadline). Capture of an ambiguous/lost attempted update, restorative receipt acquisition, and stabilization for that workload share that one deadline; it is never reset. Terminal pair proof and prior-SHA public health also have to finish inside the 720-second phase cap.

Deterministic initiation and cooperative observation

Use this deterministic controller sequence; do not use threads, processes, asyncio, or concurrent SDK calls:

1. Validate the captured exact prior pair, restore workload allowlist, and all recovery budgets before mutation.
2. In fixed web -> worker order, for every workload that was actually mutated:
   • establish its independent absolute deadline;
   • reconcile the actually attempted candidate when needed;
   • issue the exact prior UpdateService;
   • bind and retain the new restorative receipt.
3. Do not perform a receipt-bound stabilization wait, public-health check, terminal proof, evidence write/upload, or deliberate sleep between successfully binding the web receipt and issuing the worker restore. Receipt acknowledgement and its immediate exact reconciliation are part of binding and remain allowed.
4. After every eligible restore has been attempted, cooperatively observe all bound receipts in fair web -> worker rounds. Each pending workload is evaluated against only its own absolute deadline and receipt/predecessor set. Sleep at most once per round, bounded by the poll interval, the earliest pending workload deadline, and the phase deadline.
5. A completed or failed workload cannot consume, reset, extend, or suppress the other workload's final observation. Each pending workload gets exactly one final service observation at its own deadline with no later sleep/read. A provider response returned after its workload or phase deadline is rejected.
6. After receipt waits, perform one exact pair terminal proof and, for a non-bootstrap prior release, exact-SHA public health. Only that complete proof can classify compensation as passed.

If one workload's restore binding or observation fails, the controller retains that bounded error and continues the other already-authorized exact restore when its captured phase remains attributable and the phase deadline permits it. It never retries the failed mutation or substitutes an identity. Invalid recovery context, invalid allowlist/budget, expired phase cap, or a global attribution contradiction stops further mutation. Any retained workload error makes recovery fail even if a later read happens to show convergence.

A web-only forward failure continues to leave the captured worker untouched: no forced worker deployment is allowed. The final pair verifier must prove its captured PRIMARY identity and singleton state with an empty recovery-predecessor allowance. Artifact-finalization recovery continues to restore both workloads because both belong to the failed terminal release.

Exact success and failure contract

A recovery succeeds only when:

• every mutated workload has one newly bound restorative receipt targeting the exact captured prior task-definition ARN and desired count;
• a force-new A -> A restore still binds a new deployment ID; the captured terminal and attempted deployments remain predecessors, never proof of restorative success;
• each restorative receipt becomes the unique exact PRIMARY, reports COMPLETED, exact desired/running/pending counts, zero failed tasks, and only its receipt-carried recognized zero-work predecessors remain or disappear;
• the exact prior web/worker task definitions, desired counts, primary receipt IDs, active task definitions/image digest/source identity, and singleton worker pass terminal verification;
• the prior public endpoint reports the exact prior SHA and ready state, except for the accepted 0/0 bootstrap case where public health is not applicable;
• the phase completes no later than its absolute 720-second cap.

Missing/duplicate receipts, an unknown/third/cross-paired deployment, wrong task definition or count, failed restorative receipt, duplicate/missing PRIMARY, unsafe predecessor, response after deadline, public-health mismatch, or any unclassified error fails closed. Ultimate ECS convergence without every receipt and terminal proof is not recovery success.

For worker capture, acknowledgement, cooperative observation, and terminal proof, both the service and the sum across all recognized deployments must keep running + pending <= 1. Final active-task proof must find exactly the captured worker task when desired count is one, or none for bootstrap. A transient or final worker overlap fails immediately and cannot be masked by later convergence.

Compensation never creates or preserves a rollback-eligible success record. The original promotion/rollback failure remains the release result even when compensation passes. Artifact-finalization recovery removes the local failed-release record only after exact recovery passes and still ends the workflow red.

Redacted evidence and runbook

Record bounded evidence sufficient to prove:

• recovery mode and fixed 240/420/720 budgets;
• fixed restore-initiation order and the fact that all eligible restore receipts were attempted before cooperative stabilization;
• one safe receipt summary per bound workload: workload, new deployment ID, allowlisted binding reason, and carried-terminal boolean;
• per-workload passed, receipt_deadline_expired, or contract_contradiction outcome;
• exact terminal-pair, public-health, and worker-singleton result;
• total recovery result and whether a workload was intentionally untouched.

Evidence and CLI failures must not include raw AWS responses, provider exception text, request/response bodies, URLs with query strings, headers, cookies, credentials, tokens, environment values, task logs, or recovery-context contents. Keep the existing reason precedence: any contradiction/unknown error collapses to contract_contradiction; receipt_deadline_expired is retained only when every retained recovery error is that allowlisted deadline reason. Safe exact task-definition/deployment identifiers and counts remain permitted for operator recovery.

Update the development release runbook with the incident attribution, fixed sequence, deadline formulas, equality behavior, exact success/failure rules, worker invariant, triage evidence, and conservative session calculation.

Acceptance criteria

Configuration and reserve

• Distinct recovery constants/defaults and hard maxima are code-owned at web 240, worker 420, and whole phase 720 seconds; the general 180, forward web 240, forward worker 420, browser 180, and role session 3600 values are unchanged.
• Typed configuration/CLI validation rejects boolean, malformed, zero, negative, poll-incompatible, or over-maximum recovery values and impossible phase/workload combinations before any mutation.
• The workflow supplies the reviewed fixed recovery values and offers no dispatch input, repository variable, environment override, or arbitrary larger operator value.
• The existing conservative envelope remains at most 180 + 120 + 240 + 180 + 420 + 180 + 360 + 720 = 2400 seconds, leaving at least 1200 seconds of the fixed credential session; compensation and finalization recovery share the same mutually exclusive 720-second cap.
• The phase deadline starts once before recovery work; per-workload absolute deadlines use the stated formula and no acknowledgement, retry, poll, terminal check, health check, or evidence operation can reset or exceed them.

Receipt-bound pair restoration

• When both services were mutated, the operation trace is exact restore/bind web, exact restore/bind worker, then cooperative service observation; no stabilization wait, public health, terminal proof, evidence write/upload, or deliberate sleep appears between the two restorative updates.
• Recovery uses deterministic single-threaded cooperative observation, not threads/async/processes, and checks pending workloads fairly against independent deadlines.
• Every restorative update binds one new exact receipt, including A -> A; attempted and captured terminal deployments remain only allowlisted predecessors.
• A failure isolated to one workload does not steal the other's budget or prevent its already-authorized exact restore, but it does prevent compensation success and is never retried or inferred away.
• A web-only failure performs no worker mutation and terminally proves the untouched captured worker PRIMARY and singleton with no predecessor allowance.
• Artifact-finalization recovery runs through the same bounded coordinator and intentionally restores both exact workloads.

Terminal proof and safe failure

• Receipt success requires the exact unique restorative PRIMARY, target/counts, COMPLETED, zero failed tasks, and only absent or exact allowlisted zero-work predecessor remnants under the accepted #96 rules.
• Terminal success proves the exact prior task definitions/counts/receipt IDs, active task image digest and source identity, public prior SHA/readiness when enabled, and the worker singleton.
• Worker service counts and all recognized deployment counts maintain aggregate running + pending <= 1 at capture, acknowledgement, every cooperative observation, and terminal proof; any overlap is an immediate contradiction.
• Every pending workload gets one inclusive final observation at its own absolute deadline and no later sleep/read; a response after either the workload deadline or phase cap fails.
• Missing/duplicate/wrong receipts or PRIMARYs, third/cross-paired identities, unsafe predecessor states, failed tasks/rollouts, inexact counts, health/SHA mismatch, phase expiry, and unknown errors fail closed even if ECS later converges.
• Failed compensation writes no rollback-eligible record. Passed compensation restores the exact pair but the original release still ends red; finalization recovery removes its local record only after exact recovery and also ends red.

Evidence, regression safety, and documentation

• Evidence records the allowlisted plan/budgets/order, receipt summaries, per-workload outcomes, untouched-workload state, terminal/public/singleton proof, and total result.
• Redaction tests prove sentinel secrets and raw provider messages cannot appear in evidence, CLI output, exception text, or artifacts; only the documented safe identifiers, counts, booleans, and reason codes remain.
• Promotion compensation, rollback compensation, and artifact-finalization recovery all use the same coordinator; forward promotion/rollback still wait for exact web ECS/public health before worker mutation and retain their existing budgets.
• The runbook documents run 31289994036 as a smoke-caused failed release whose recovery controller timed out before later exact convergence; it does not call the release or compensation successful.
• No template, HTML/Jinja, public page, application model/data, migration, Terraform, IAM/OIDC, DNS, ECS deployment configuration, or AWS resource changes are included; no HTML is minified.
• Focused deployment/controller/workflow tests, normal quality/Django gates, workflow/YAML checks, git diff --check, and make test-playwright-core pass through uv.

Operational / HUMAN verification

• [HUMAN] An accepted push to main automatically deploys its exact full SHA with all required jobs green, fixed recovery arguments visible in the reviewed workflow/controller evidence, exact terminal web/worker state, exact-SHA health, and inspected desktop/mobile smoke screenshots.
• [HUMAN] In one separately authorized development-only post-worker failure drill, both restorative receipts are bound before either stabilization wait, the worker restore begins without waiting for web terminal completion, and the prior pair is restored inside 720 seconds with no worker overlap.
• [HUMAN] The failure drill ends red with redacted evidence and no rollback-eligible candidate record; exact prior SHA/readiness and terminal pair are proven, then a clean automatic push/approved promotion restores the intended exact current release.
• [HUMAN] If the bounded recovery drill cannot prove every receipt, singleton observation, terminal fact, or artifact, automatic deployment is disabled and the issue remains open for loud recovery; later ECS convergence is not accepted as a pass.

Deterministic test scenarios

1. Recorded incident timing: fake clock/ECS keeps web restorative work terminal at 160 seconds and worker terminal at 280 seconds. Both restore receipts are bound before waiting; the exact prior pair passes under independent 240/420 budgets, whereas operation ordering proves worker did not wait behind web.
2. Independent equality deadlines: web completes on its exact 240-second final observation and worker on its exact 420-second final observation. Both pass with no extra sleep/read and within the 720-second phase cap.
3. Worker deadline: worker remains in progress at its final 420-second observation. Recovery fails with receipt_deadline_expired, performs no later poll, emits no success record, and cannot be rescued by a later terminal fixture.
4. One workload fails: web binding contradicts or times out while worker remains attributable. The worker exact restore still receives its own budget and is observed, but overall recovery fails and never retries/invents web identity.
5. Receipt and predecessor matrix: new exact receipt, zero-count initialization, partial acknowledgement reconciliation, safe ACTIVE/DRAINING zero-work remnants, force-new A -> A, lost/invalid acknowledgement capture, missing/duplicate receipt, failed candidate, wrong/third/cross-paired identity, and unsafe predecessor cases produce the specified result.
6. Singleton worker: service aggregate, individual deployment, cross-deployment, stale read, acknowledgement, final-deadline, and active-task fixtures prove at most one running-plus-pending worker; every overlap fails on its first observed contradiction.
7. Untouched worker: failure before worker mutation restores only web, performs no worker update, and exact terminal proof requires the unchanged worker PRIMARY with an empty predecessor allowlist.
8. Phase cap and health: slow but valid receipts cannot make terminal/public health extend past 720 seconds; bootstrap 0/0 skips public health but still proves exact zero terminal state.
9. Entry-point parity: promotion failure, rollback failure, and finalization failure share the coordinator and evidence schema; forward success ordering remains unchanged.
10. Configuration/workflow/redaction: malformed and excessive values fail before mutation; fixed 240/420/720 and 3600-second session are asserted; sentinel credentials/provider payloads are absent from all surfaced errors and evidence.

Non-goals

• No generic increase to the 180-second stage/public-health budget and no reuse of a single global service timeout.
• No change to forward 240/420 stabilization behavior, migration, smoke, artifact-upload, or credential-session budgets.
• No parallel threads, async SDK calls, concurrent mutations, blind retry, extra UpdateService, pre-sleep, unbounded wait, or manual success override.
• No weakening of exact receipt, PRIMARY, task-definition/count, digest/SHA, zero-work predecessor, terminal pair, public-health, or worker-singleton checks.
• No acceptance of eventual ECS convergence after a receipt/deadline/terminal failure.
• No reverse database migration and no rollback of registrations, enrollments, content, or other database writes.
• No #101 task/ENI/ALB-target coherence work; #101 governs the forward pre-worker gate and long-worker revalidation.
• No redesign of abrupt runner/control-plane-loss break-glass recovery.
• No ECS capacity, deployment percentage, health-check, task sizing, application-startup, database, Terraform, IAM/OIDC, DNS, or AWS resource change.
• No template, HTML/Jinja, CSS, page-design, or HTML-minification change.
• No production-account or production-host work.

Dependencies and sequencing

• Satisfied prerequisite: closed #96 supplies the authoritative receipt binding, bounded predecessor retirement, ACTIVE/DRAINING zero-work, terminal verification, and reason-redaction baseline.
• Incident evidence only: closed #97 supplies the observed recovery timing and cause attribution; #102 must not reopen or reinterpret #97.
• No dependency on #101: #101 changes forward web task/ALB coherence before worker mutation. #102 changes recovery initiation/polling after a release failure. Neither interface is required for the other's product behavior.
• #101 and #102 are likely to touch deploy/release.py, deploy/aws_gateway.py, deployment tests, and the release runbook. That is merge-sequencing risk, not a product dependency: implement each from current main, keep commits isolated, and rerun independent QA after any conflict resolution.
• No other open implementation dependency blocks engineering.

Start at the current _compensate() controller and read the linked AWS/Terraform, migration, verification, and development release runbook specifications. Trace existing recovery configuration, receipt handling, observation, terminal proof, and evidence paths before checking how the acceptance criteria map to tests or workflow steps. Done means bounded web/worker restoration, exact terminal proof, redacted evidence, and an explicitly failing original release result.