Gate live restore and startup on reconciled recovery receipts

Parent epic: #66
Backup proof prerequisite: #264
Privacy replay/fence prerequisite: #258
Delivery/outbox prerequisite: #49
Immutable application-pair boundary: #102
Active-content prerequisite: not yet filed/accepted by the owning content boundary

Product outcome

Gate one live restore/startup generation so no restored database state becomes usable until the exact current backup proof, privacy replay, delivery/outbox reconciliation, active-content pointers, runtime/image identity, database-schema identity, and consumer-release policy all agree.

The controller is orchestration only. It owns a generation-bound activation decision and consumer hold/release coordination; it does not create or verify backups, replay tombstones, mutate delivery state, resolve Relay ambiguity, edit content releases/pointers, rebuild projections, replace durable-job leases, operate AWS, or infer live authority.

PM status — BLOCKED / retain needs grooming

This issue is not implementation-ready and must retain needs grooming.

The exact accepted producer interfaces do not yet exist:

• #264 specifies but has not delivered/accepted BackupVerificationReceipt, requirements, runtime/schema identities, and its pure evaluator.
• #258 is open and its specified foundation is deliberately production-disabled; no accepted live RestoreFenceReceipt, authoritative tombstone snapshot/high-watermark, production replay adapters, or startup consumer exists.
• #49 is open and defines no accepted generation-bound delivery/outbox restore-reconciliation receipt or safe-state arithmetic for this controller to consume.
• no accepted content-owned issue defines a side-effect-free active-release/pointer validation receipt for restore.
• #66 retains HUMAN decisions for the exact activation consumers and degraded behavior, live checkpoint durability, operator/release authority, and evidence custody.

Engineering must not create provisional producer DTOs inside this issue, duplicate another domain's state machine, or interpret today’s models as equivalent evidence. Re-groom only after the producer owners accept the interfaces below, or after a separately filed issue explicitly narrows this to an execution-disabled synthetic foundation.

Normative authority

• _docs/PROCESS.md: role-separated lifecycle, dependency ordering, versioned verification, and sensitive-artifact rules.
• _docs/specs/01-platform-architecture.md and _docs/architecture/app-boundaries.md: one Django deployment, orchestration-to-domain dependency direction, application services, durable after-commit work, and no circular domain imports.
• _docs/specs/03-github-content-and-people.md: validated candidate releases, atomic active-release pointers, route manifests, last-known-good retention, and no partial activation.
• _docs/specs/05-events-registration-email.md: website logical-delivery ownership, Relay authority, guarded projections, idempotency, reconciliation, and ambiguity without automatic resend.
• _docs/specs/07-security-privacy-operations.md: encrypted backup/restore proof, tombstone replay, historical-delivery holds, active-content validation, redacted evidence, and approved RPO/RTO.
• _docs/specs/08-aws-development-terraform.md: AWS/Terraform ownership and exact sealed runtime/image/deployment identity.
• _docs/specs/09-migration-rollout-roadmap.md: migration compatibility, worker/send holds, immutable rollback, dynamic-write preservation, reconciliation, and cutover sequencing.
• _docs/specs/10-verification-strategy.md: restore/fault/concurrency/crash tests, no resurrection/resend, exact-image deployment proof, and HUMAN live evidence.
• #66: parent ownership map, cycle-breaking order, approved target baseline, and remaining HUMAN authority.
• #264: provider-neutral backup requirements/result/receipt/evaluator contract.
• #258: privacy-owned tombstone ledger, replay/reconciliation checkpoints, restore-fence receipt, and activation evaluator.
• #49: delivery-owned logical intents, Relay projections, ambiguity resolution, and reconciliation.
• #102: exact immutable web/worker release pair and receipt-bound deployment recovery. This issue must compose with it, never weaken or replace it.

Required one-way ownership and import direction

The eventual controller belongs in a dependency-light orchestration boundary. It consumes safe immutable DTOs/evaluators from owners and calls only accepted owner services:

#264 backup receipt/evaluator ---------\
#258 privacy fence receipt/evaluator ---\
#49 delivery reconciliation receipt ----> recovery activation controller -> consumer gate seams
content pointer validation receipt -----/
sealed runtime/schema/release facts ----/

consumer gate seams -> web/auth/jobs/scheduler/delivery/export/projection/cache/search/deploy finalization

privacy, email_app, content, content_sync, jobs, and other domain apps must not import the controller to make their business state valid. Their orchestration-facing gate adapters may consume only a scalar generation/decision identity through an accepted common seam. The controller must not import provider SDKs, Terraform, Studio/API presentation, or domain models for direct mutation.

Exact prerequisite receipt inputs

Re-grooming requires each producer to publish a frozen, canonical, versioned, side-effect-free evaluation interface. The controller accepts no mapping, model instance, raw provider payload, exception, log scrape, mutable “current” row, or free-form metadata in place of these inputs.

1. Backup proof — owned by #264

• exact schema-1 BackupVerificationReceipt and exact BackupVerificationRequirements digest;
• exact environment, provider key/contract, verifier contract, snapshot identity/manifest/result digests, recovery/provider/verification times, encrypted complete resource arithmetic, and receipt digest;
• exact sealed runtime identity (VERSION, full source SHA, image digest) and database-schema identity (migration graph/applied digests, counts, pending zero);
• evaluate_backup_receipt(...) == verified against the current requirements, runtime, schema, and explicit evaluation instant, including exact recovery-point and receipt-age bounds.

The controller neither calls a cloud provider nor interprets provider state. Missing, stale, partial, unavailable, ambiguous, unencrypted, wrong-environment, wrong-runtime/schema, count-mismatched, noncanonical, unsupported, or digest-drifted backup evidence blocks.

2. Privacy replay/fence proof — owned by #258/privacy

• one immutable RestoreFenceReceipt for the exact authoritative tombstone-ledger snapshot/high-watermark and the exact restore fence generation;
• exact fence/plan/snapshot/release/schema/register/manifest/replay-definition/outbox-definition identities, aggregate selected/applied/already-safe/reconciled counts, completion time, and receipt digest;
• evaluate_activation(...) == ready against the current authoritative ledger state and current restore/runtime/schema identity;
• proof that no newly applicable or unsealed tombstone, unresolved hold/exception, missing domain/outbox definition, checkpoint drift, or unresolved replay row exists.

The controller never reads tombstone subjects, plans replay batches, mutates privacy/domain rows, expires a tombstone, releases a hold, or treats a receipt as live authority.

3. Delivery/outbox reconciliation proof — owned by #49/email_app

#49 or a separately groomed owner issue must first accept an immutable, canonical, versioned, generation-bound safe DTO/evaluator. At minimum it must bind:

• exact restore event/generation, selected logical-intent/job/projection manifest digest, delivery contract and Relay projection/reconciliation versions, evaluation checkpoint, aggregate counts, completion time, and receipt digest;
• exact arithmetic proving every selected historical item is held or terminal/reconciled safe under the accepted owner policy;
• a state of safe only when no item is pending, leased/in-flight, retryable, ambiguous, callback-lagged, missing, changed-request, unknown, or otherwise sendable-but-unreconciled;
• current revalidation that can detect newly applicable or changed delivery work after receipt creation.

The final owner contract must say which Relay reads/reconciliation operations are permitted during the recovery-only lane. This controller has no send, retry, resend, manual-resolution, suppression, callback-transition, or provider-fallback method. Provider acceptance is never delivery, ambiguity is never inferred safe, and Datamailer/SES can never become a sender.

4. Active-content proof — owned by content/content_sync

A separately filed and accepted owner issue must define an immutable, canonical, versioned, generation-bound validation receipt/evaluator. At minimum it must bind:

• the exact active content release/source commit, pointer set, route-manifest digest, required asset/search/graph projection identities, validation policy/version, evaluation checkpoint, completion time, and receipt digest;
• one coherent current pointer set with no missing/duplicate/conflicting/superseded release, partial candidate, stale source, path-claim conflict, or manifest/projection drift;
• current side-effect-free revalidation against the restored database and retained content artifacts.

Validation is not activation or repair. The controller never chooses a release, edits pointers/path claims, runs a sync, activates a candidate, rebuilds search/graph/cache, or replaces the last known good release.

5. Current release, schema, restore, and policy facts — orchestration/deploy boundary

The controller must bind the same exact sealed schema-2 runtime identity and database-schema identity accepted by #264, plus #102's exact current immutable web/worker release-pair identity where deployment finalization is in scope. Legacy/local/partial identity, APP_VERSION, mutable tags, reconstructed timestamps, Git/clock inference, pending migrations, mixed web/worker pairs, or merely eventual convergence blocks.

An accepted code-owned activation policy must enumerate every consumer key exactly once, its contract version, required hold/release checkpoint, dependency receipts, and safe blocked behavior. Unknown, duplicate, missing, disabled, or runtime-registered consumers block the generation. Settings, environment variables, database rows, plugins, command flags, or operator input cannot weaken this policy.

Recovery activation generation and checkpoint binding

The future controller owns one immutable generation envelope containing only:

• controller schema/contract version;
• opaque restore-event UUID and opaque activation-generation UUID;
• environment class;
• exact backup requirements/receipt/snapshot digests;
• exact privacy fence/plan/snapshot/receipt digests;
• exact delivery and content receipt/checkpoint digests;
• exact runtime, database-schema, application-pair, and activation-policy digests;
• creation/evaluation UTC seconds, state, allowlisted reason code, and canonical generation digest.

The authoritative restore-event/generation and live checkpoint must come from the separately accepted restore boundary and must survive or sit outside the database being restored. They may not be inferred from a restored auto-increment row, mutable “latest” pointer, wall clock, deployment tag, or whichever receipt appears newest. The precise durable store, monotonicity rule, restore-source binding, supersession rule, and disaster behavior remain a #66 HUMAN/live decision.

Every producer receipt, recovery job, controller decision, consumer release intent, and consumer acknowledgement must match this exact generation and digest set. Cross-generation receipt mixing, older-success reuse, receipt replacement in place, changed evidence under one idempotency key, or a checkpoint from another restore/environment/runtime blocks.

Hold surfaces and recovery-only execution lane

The code-owned policy must enumerate, and integration tests must prove, holds for all of these surfaces:

1. public web reads that could expose restored database state;
2. authentication, session establishment/refresh, account use, and private learner/staff reads;
3. general durable-job leasing/execution and worker startup;
4. ordinary scheduler dispatch;
5. delivery submission, retry/resend, callback-driven release, and delivery-purpose work;
6. exports and staff/admin/API mutations;
7. domain projections, derived-state repair, and public/private materialization;
8. cache, search, graph, asset, and route-manifest rebuild/swap/invalidation;
9. deployment success/finalization and rollback-eligible release recording.

Liveness and the minimum approved recovery/readiness diagnostic may remain available without declaring application readiness. The exact public maintenance response, route list, status code/copy, authentication behavior, callback ingestion behavior, and operator diagnostic surface are unresolved HUMAN/product inputs; no engineer may invent them under this issue.

Privacy replay, delivery reconciliation, content validation, and controller evaluation require a recovery-only lane while ordinary work remains held. That lane must be a closed code-owned allowlist of exact generation-bound handlers. It cannot run arbitrary jobs, schedules, commands, callbacks, sends, exports, projection rebuilds, or domain mutations beyond the owning service operation explicitly accepted for recovery. Every recovery handler revalidates generation, lease/fence, receipt/plan/checkpoint, runtime/schema, and policy before work and before success.

Required controller state machine

The accepted implementation must use one guarded state machine with states exactly equivalent to:

• blocked: generation exists but cannot begin evaluation;
• collecting: exact producer evidence/recovery checkpoints are incomplete;
• evaluating: all claimed inputs are being side-effect-free revalidated;
• ready: one immutable controller decision receipt was atomically created for the exact generation;
• releasing: generation-bound consumer release intents exist and acknowledgements are incomplete;
• released: every required consumer acknowledged the exact generation and deployment finalization proof passed;
• failed: a contradiction, unsafe failure, or exhausted bounded attempt permanently failed this generation;
• superseded: a new authoritative restore generation replaced this one before release.

Only guarded forward transitions are allowed. failed and superseded are terminal. Drift while collecting/evaluating/ready/releasing fails or supersedes the generation; it never edits an old receipt/checkpoint into success. Resume creates or revalidates a new fenced attempt/generation according to the accepted live checkpoint contract. Exact idempotent replay returns the original safe outcome; a changed action or input under the same key conflicts.

There is no --force, skip-consumer, ignore-receipt, accept-ambiguous, use-latest, stale-evidence grace, partial-ready, development bypass, emergency override, direct SQL, environment toggle, or eventual-convergence success path.

Transaction, durable work, release, and re-hold semantics

The eventual service must:

1. lock the exact generation/revision and revalidate every current identity, receipt evaluator, policy entry, hold, and checkpoint;
2. atomically create the immutable controller decision receipt, audit/idempotency result, and one bounded durable release intent per accepted consumer (or create none on failure);
3. wake release work only after commit; never hold a database transaction across domain work, provider/network work, process startup, or consumer acknowledgement;
4. have each release handler reload and revalidate generation, decision digest, runtime/schema, policy, lease/fence/revision, and its own consumer checkpoint before and after calling the accepted gate seam;
5. record immutable generation-bound consumer acknowledgements with exact arithmetic; duplicate execution is a no-op and changed evidence conflicts;
6. hold deployment finalization until every required acknowledgement and #102 exact-pair/runtime/readiness proof succeeds.

Partial release is never overall success. The accepted live design must define how an already-awakened consumer is synchronously prevented from further protected work if another consumer fails or evidence drifts during releasing; silence, a later restart, or eventual shutdown is not fail-closed re-holding. This handshake is a mandatory unresolved producer/consumer interface, not permission for the controller to mutate domain state.

Security and evidence contract

• Live start/release requires the separately approved environment, maintenance window, operator/deployer role, restore source, evidence destination, go/no-go/abort authority, and maximum scope. A valid receipt alone grants no authorization.
• All operations use capability checks, expected revision, exact idempotency, bounded attempts, fenced leases, transactions, after-commit wakeups, and immutable audits/checkpoints.
• Safe DTOs, logs, audits, jobs, exceptions, receipts, screenshots, and artifacts may contain only contract versions/digests, full source SHA/image/schema digests, opaque UUIDs, code-owned consumer/definition keys, UTC status times, aggregate bounded counts, and allowlisted states/reasons.
• Exclude raw/normalized email, names/profile/submission/content/message values, account/domain/database/resource identifiers, IP/header/query/body, URLs/ARNs, SQL, provider/AWS payloads, tombstone subject keys or reversible/low-entropy hashes, tokens, cookies, credentials/secrets, exception text, environment values, and free-form diagnostics.
• Unknown code/state/version/consumer/input, noncanonical evidence, excessive structures/counts, clock regression, sensitive canary, or emission/storage failure blocks without exposing the unsafe value.
• There is no AWS SDK/import/call, Terraform/IAM/KMS/RDS/S3 mutation/readback, credential lookup, backup/restore execution, provider payload, or production data in this website issue.

Required verification after re-grooming

The final acceptance matrix must include at least:

1. every producer receipt absent, malformed, corrupt, stale, future, unsupported, partial, ambiguous, superseded, wrong-environment/generation/runtime/schema/policy, and one-second boundary cases;
2. exact valid set, duplicate receipt, cross-generation mixing, changed receipt under one idempotency key, new tombstone/delivery/content drift before evaluation, after ready, and during release;
3. every controller transition, invalid/backward transition, exact replay, concurrent evaluators, stale revision/lease/fence, crash before/after controller commit, crash before/after consumer mutation/acknowledgement, retry/exhaustion, and new generation supersession;
4. recovery-only allowlisted job versus every ordinary job/schedule/send/export/rebuild path; worker contention; transaction rollback; after-commit wakeup; no network inside a transaction;
5. each named consumer attempting startup/use before ready, during releasing, after its own acknowledgement but before all peers, after release, and after drift/failure; deployment finalization stays blocked until the complete exact set passes;
6. delivery states including pending, leased/in-flight, retryable, provider-accepted, delivered, ambiguous, suppressed, dead, hard-bounced, complained, missing callback, and changed request; no path sends or automatically resends;
7. content pointer missing/duplicate/conflicting/stale/partial/superseded and asset/search/graph/manifest drift; no path repairs, activates, or rebuilds;
8. SQLite transaction/idempotency/concurrency/migration checks, exact current migration identity, full Django safety coverage, import/dependency/system checks, container/startup behavior, and the versioned graph-selected verification plan;
9. sensitive-value, Unicode/control, oversized JSON/count, unknown-key/state/code, exception/provider failure, log/audit/job/receipt/artifact redaction, and unavailable evidence sink;
10. composition with #102 proving an exact web/worker pair, singleton, exact SHA/image/schema/readiness, no rollback-eligible record before complete release, and no acceptance of later convergence.

Browser/screenshots are not_applicable only if the final implementation adds no route, template, response, navigation, health/readiness payload, operator surface, or rendered maintenance/denial state. Any HTTP/render change requires re-grooming with exact URLs, status/cache/robots/auth behavior, accessible copy/focus semantics, desktop 1440×900 and mobile 390×844 Playwright scenarios, and independent inspected screenshots under .tmp/screenshots/issue-267/.

HUMAN and live dependencies before engineering

• #264 is implemented, independently tested, PM-accepted, merged, and its exact receipt/evaluator interface is frozen.
• #258's production key/ledger/snapshot/replay adapters and exact live restore-fence receipt/evaluator are separately accepted; privacy owner approves retention-versus-backup horizon, hold/exception behavior, retained proof, and release authority.
• #49 and its Relay prerequisites are accepted, including the exact generation-bound delivery/outbox reconciliation receipt, safe-state arithmetic, ambiguity handling, and no-resend rule.
• A content-owned issue is filed and accepted for the exact side-effect-free active-release/pointer validation receipt/evaluator and retained artifact/projection rules.
• The service owner approves every activation consumer, recovery-only handler, blocked/degraded behavior, re-hold handshake, owner/escalation, and review cadence.
• The infrastructure/release owner approves authoritative live generation/checkpoint durability, environment, restore source, exact runtime/application pair, maintenance window, go/no-go/abort authority, RPO/RTO clock boundaries, evidence destination/retention, and maximum scope/cost.
• The exact public maintenance/readiness/auth/callback/operator behavior is accepted if any route/process remains reachable while held.
• #102's current exact-pair interface remains intact; live #267 acceptance composes with its current exact-SHA deployment/recovery evidence rather than substituting for #102's remaining HUMAN drill.

No live AWS/provider/restore/replay/startup/release action is authorized by this issue or these checkboxes. Missing HUMAN authority means blocked, not a default or override.

Explicit non-goals

• No production/protected-data backup, restore, rollback, replica promotion, tombstone replay, delivery reconciliation, content repair/activation, projection/cache/search rebuild, workload startup, deploy, AWS/provider call, or drill.
• No backup/provider adapter, tombstone/retention/key policy, Relay projection or ambiguity resolution, content pointer business rule, application-pair recovery, domain mutation, job lease/state-machine replacement, or infrastructure resource ownership.
• No website sender, SES/Datamailer fallback, dual sender, automatic resend, or interpretation of provider acceptance as delivery.
• No raw provider/AWS evidence, credentials, secrets, production identifiers/data, PII, tombstone subject material, database rows/exports, or unsafe free-form diagnostics.
• No relaxed readiness, partial activation, eventual convergence, latest-receipt selection, stale receipt reuse, runtime/settings registration, mutable operator bypass, direct SQL, destructive reverse migration, or whole-site rollback that loses dynamic writes.
• No Studio/admin API/public UX, dashboard, alert, runbook execution, or browser redesign in the blocked intake. Any such surface requires its own exact permissions, denial, privacy/cache/robots, API parity, and browser contract.

Re-grooming exit

Once every producer interface and HUMAN decision above is accepted, a PM must reread the actual versions and rewrite this issue against those exact names, fields, states, error codes, dependency directions, consumer seams, and live checkpoint behavior. Only then may needs grooming be removed and engineering begin. Until then, this issue is a precise dependency/decision record, not an implementation specification.

Start with _docs/PROCESS.md, _docs/specs/01-platform-architecture.md, and parent issue #66, then review the prerequisite contracts in #264, #258, and #49. There is no implementation-ready completion target yet: re-grooming is the next step only after the producer receipts, active-content owner issue, activation policy, and remaining HUMAN decisions are accepted.