Run the full web.dtcdev.click migration, fault, restore, and rollback rehearsal

Parent epic: #10

PM disposition

GROOMED / P0 / HUMAN / DEPENDENCY-BLOCKED / MILESTONE-7 AGGREGATE. Do not dispatch #73.

This issue owns one final, production-shaped rehearsal on web.dtcdev.click after the accepted Milestone 1–6 application, migration, delivery, security/privacy/accessibility, infrastructure, and verification contracts are integrated on one immutable application release candidate. It aggregates fresh and provenance-valid producer evidence into the frozen #77 release report and records a development-rehearsal go/no-go. It does not implement missing domain behavior or authorize a production cutover.

Normative authority

• _docs/PROCESS.md
• _docs/specs/open-decisions.md, decision 1, and closed #226 for the direct-sync authority
• _docs/specs/07-security-privacy-operations.md
• _docs/specs/08-aws-development-terraform.md
• _docs/specs/09-migration-rollout-roadmap.md, especially Milestones 6–8, rollback, and data-migration controls
• _docs/specs/10-verification-strategy.md, especially release-critical scenarios, production safety, and the acceptance report
• _docs/runbooks/development-release.md
• Accepted issue-specific contracts referenced by the frozen #72/#77 traceability matrix

Canary and sender boundary

#73 is not a prerequisite for the controlled development courses canary.

The authoritative sequence is:

accepted #48/#49 ordinary delivery and exact deployed Relay/OpenAPI contract
+ accepted automated/source portion of #50
+ accepted #60 side-effect-disabled course migration/freeze/outbox rehearsal
+ credential/callback/reconciliation/alarm/allowlist/simulation safeguards
  -> separately authorized one-recipient development `courses` canary
  -> #50 HUMAN evidence and acceptance/closure
  -> #73 Milestone-7 full aggregate rehearsal
  -> #74 separately authorized production cutover

Milestone 6 places the canary before Milestone 7. Every #73 migration/import, restore, application-image rollback, fault, and browser run keeps website, Relay, Datamailer, webhook, worker, and provider outbound paths disabled or uses contract-faithful fakes/zero-write dry-run. Consequently, #73 cannot prove a live canary and grants no sender, provider, credential, recipient, DNS, AWS, or production authority. Any dependency text that uses successful #73 as an input to authorizing the canary is circular and must not be followed.

Exact entry gates

An engineer/operator lane may start only after all of the following are accepted, integrated, and pinned by exact merge/source/schema/deployment identity:

1. Traceability/report contract: #72 is accepted; #76's release-critical producer graph is complete; and #77's deterministic schema, checksum, freshness, redaction, manual-evidence, exception, and go/no-go consumer contract is accepted. #73 produces a report instance; it does not invent or hand-edit the schema.
2. Application/domain release candidate and direct-sync authority: every required content, Course/Cohort, Event, account/Slack, Studio/admin API, privacy, security, accessibility, search/link/SEO, and compatibility producer named by the frozen #72/#77 matrix is accepted. The direct-sync chain #219/#253/#272/#273–#278 is accepted and deployed: every #276 source/family reader unit has completed parity, cutover, observation, and its approved application-reader rollback window; #278 has closed that compatibility window, preserved successor provenance, removed staged authority, and recorded the exact last-compatible reader/image/schema/artifact identities. Open coordination epics do not substitute for their required accepted children.
3. Course migration rehearsal: #60 is accepted for the same compatible schema/application envelope, including pinned production-like source authorization, import/delta reconciliation, write-freeze/outbox classification, backup restore, compatible image rollback, and one-sender/no-duplicate proof with all real sending disabled.
4. Delivery milestone: #48/#49 and Relay #1/#2/#3 are accepted and deployed at exact contract identities; #50's automated source/runtime slice is accepted; the separately authorized controlled development courses canary has passed all safeguards; and #50's HUMAN gate is accepted. #73 neither runs nor substitutes for that canary.
5. Operations/recovery: #66's required children and owning-domain inputs are accepted, including the #269 authorized synthetic restore/rollback/fault/expiry drill and clean return; current application release identity, readiness, alert, backup, restore, privacy-tombstone, delivery/outbox, #272 direct-sync validation, #276/#278 application-reader compatibility, and immutable image-recovery receipts all agree.
6. Infrastructure and compatibility: the non-activating #71 consumer/path-map contract and the required #78/#94 development-control/Terraform evidence are accepted. The production-shaped plan proposes no unreviewed resource, sender, DNS, credential, or provider mutation.
7. Human rehearsal authorization: named rehearsal, service, infrastructure, data-migration, privacy, delivery, security/accessibility, release/on-call, evidence-custody, abort, and exception owners approve the exact development environment, immutable release candidate, authorized anonymized production-like snapshots, maintenance window, evidence destinations, cost/scope cap, and go/no-go/abort rules.

A local candidate, open issue, green narrow test, scheduled regression, published image, stale artifact, or verbal approval is not an accepted dependency.

Scope

• Freeze one immutable rehearsal identity: website source SHA, image/config digests, migration leaves, direct-sync source commit/tree plus SyncLog/source-status/#272 receipt identities, #276 authority-manifest and #278 successor-provenance/contract identities, authorized course snapshot/schema/checksum, Relay commit/OpenAPI identity, Terraform source/plan identity, and all report/producer schema versions.
• Revalidate the accepted #273–#276 direct-sync and public-reader evidence on the frozen application image: exact source commits/trees, immutable SyncLog and source status, source-owned current/draft/soft-delete state, partial-recovery status, route/path ownership, counts, assets, routes, links, search, graph, canonicals, #272 receipt, and #276 authority-manifest units. Do not create or activate a site-wide content candidate, swap a content pointer, invoke content rollback, or synchronize an arbitrary older source revision.
• Consume and revalidate #60's production-like course import/final-delta evidence against the frozen envelope; rerun any component whose source, schema, application, policy, or freshness envelope changed.
• Execute full URL/link/SEO, security/privacy, accessibility, Studio/admin API parity, load, failure-mode, backup/restore, immutable-image rollback, worker/outbox reconciliation, cache/private-bypass/invalidation/WAF/cost/allowance, and readiness checks.
• Exercise write-freeze/delta, DNS/edge/TTL/sitemap/monitoring/cutover/rollback runbooks without activating production or submitting development URLs.
• Generate the deterministic #77 report instance, checksum, structured exception register, inspected synthetic screenshots, and named HUMAN go/no-go record. Retain the exact #276/#278 evidence and then-compatible application images for audit and generic application rollback only; historical checked/staged artifacts are not a live selector, request-time fallback, pointer target, or source-rewind mechanism.

Acceptance criteria

• All exact entry gates above are accepted and identity-pinned; the dependency graph contains no canary/#50/#60/#73 cycle.
• Direct-sync source/log/status/receipt identities and every #276 authority-manifest unit reconcile across current/draft/soft-delete state, routes, assets, links, search, graph, canonicals, and public-reader outputs. Record-atomic upserts, conditional source-scoped stale sweeps, and observable partial recovery preserve accepted outputs without a site-wide candidate, activation pointer, content rollback, dual reader, or request-time fallback.
• Course tables/IDs/mappings/accounts/enrollments/curriculum/submissions/reviews/scores/certificates/calendars/email-history/outbox reconcile with no unexplained difference under the exact accepted #60 envelope.
• Full URL/SEO, security/privacy, accessibility, Studio/admin API parity, browser, load, cache/edge, fault, readiness, and operational suites pass, or each allowed exception has owner, rationale, risk, mitigation, expiry/review date, and explicit approval.
• Backup restore meets the approved RPO/RTO, reapplies accepted tombstones, preserves accepted direct-sync current state, immutable SyncLog/source status, successor provenance, #272 validation, and #276 public-reader authority, validates every startup hold, and releases no historical or ambiguous delivery work without accepted reconciliation.
• Immutable application-image rollback uses only the exact then-compatible application/reader pair allowed by the current post-#278 schema, preserves direct rows plus synthetic post-cutover-like registrations/enrollments and logical-delivery idempotency, performs no staged write or direct-row rewind, never restores Datamailer/direct SES, never dual-sends, and returns to one exact healthy application release.
• Production-shaped Terraform and exact freeze/delta/DNS/edge/TTL/email/sitemap/monitoring/cutover/rollback runbooks have named owners and quantitative go/no-go/abort triggers, with no apply or production mutation.
• web.dtcdev.click remains noindex, emits production canonicals only, uses synthetic/redacted evidence, and performs no live delivery or broad-recipient action throughout.
• The frozen #77 report instance is deterministic, checksum-bound, complete, fresh, redaction-clean, and records the separate HUMAN #73 go/no-go. Missing, red, stale, mismatched, unowned, or expired evidence yields NO-GO.

Required scenarios

1. Revalidate frozen direct-sync source/log/status/#272 receipt identities and every #276 reader-parity unit, together with the accepted course dry-run/apply/final-delta envelope; compare exact safe counts/checksums and reject drift, ambiguity, missing mappings, unobserved partial recovery, or reader/provenance mismatch. Do not request an arbitrary older-SHA sync or staged content rollback.
2. Fault immutable checkout, parse/direct-upsert, source-scoped stale sweep, SyncLog/source-status/reconciliation, derived search/graph, workers, Relay fake/zero-write submission/callback/reconciliation, OIDC, database, edge/cache/invalidation/WAF, backup/restore, and credential-expiry paths; prove observable partial state, documented degradation, alerts, holds, abort, and forward recovery without changing public authority or resurrecting staged actions.
3. Restore and roll back the exact then-compatible application/reader pair after synthetic post-cutover-like registrations/enrollments and logical intents; reconcile workers/outbox before resume and prove no loss, staged resurrection, direct-row rewind, duplicate delivery, or changed-request replay. Consume #276's already accepted old-reader rollback evidence and #278's compatibility-window closure; do not reopen that retired reader.
4. Load representative public reads, registration/enrollment, Studio/admin API, and job paths to the approved #26 thresholds and exact #266 queries/windows/exclusions.
5. Remove, stale, mismatch, duplicate, or contaminate each report artifact class and prove the #77 consumer yields NO-GO without exposing protected values.

Browser and screenshots

Run the graph-selected full Playwright suite at desktop/mobile over the frozen development rehearsal candidate. Cover representative homepage/content/docs/FAQ/Wiki/podcast/people/course/Event/account/registration/enrollment/learner/Studio paths, compatibility aliases, safe denial/error/degraded states, accessibility, noindex/canonical behavior, and network egress denial. The independent tester stores and inspects required screenshots under .tmp/screenshots/; all data is synthetic and no email, profile text, token, secret, provider identifier/payload, source locator, or protected value appears.

Explicit non-goals

• No implementation of missing product/domain, report-generator, migration, privacy, delivery, observability, infrastructure, or compatibility behavior under #73.
• No site-wide ContentRelease candidate/prepare/ready/activate/rollback graph, active pointer, automatic content rollback, arbitrary older-SHA sync, direct-row rewind, dual reader/writer, or request-time fallback.
• No controlled canary, sender enablement, real recipient, provider/Relay write, credential readback, Datamailer queue access, direct SES call, dual sending, or fallback sender.
• No production/AWS/DNS/edge/indexing/deployment mutation, production data access, write freeze, legacy retirement, or destructive schema contraction.
• No waiver based on a normal CI run, scheduled regression, source-only artifact, stale prior rehearsal, eventual convergence, or verbal approval.

Lifecycle

After every entry gate passes, the PM freezes the exact identities and evidence contract. An explicitly authorized engineer/operator prepares one isolated uncommitted #73 rehearsal candidate and evidence pack. A separate tester independently validates the frozen plan, runs every selected check, inspects screenshots and redacted artifacts, and posts a terminal tester report. PM acceptance follows only on complete green/approved evidence. A focused commit may then use Closes #73, be locally merged/pushed, and be observed by on-call. #74 remains a separate HUMAN production authority and stays open until its own observation/retention obligations pass.

Start with _docs/PROCESS.md and the referenced specs, runbooks, and frozen #72/#77 traceability contracts; verify that the exact entry gates, including #60, #50, #66, and #71, are accepted before any rehearsal work. Done means producing the deterministic #77 report, evidence and HUMAN go/no-go for web.dtcdev.click without production mutation, live delivery, or unauthorized cutover.