Execute production cutover, monitor parity, and retire legacy writes safely

Parent epic: #10

Normative authority:

• _docs/specs/open-decisions.md §18, the owner-approved article/non-article cutover boundary;
• _docs/specs/04-courses-and-cohorts.md, especially compatibility and high-risk migration checks;
• _docs/specs/05-events-registration-email.md, especially durable Relay delivery, sender/provider boundaries, and the prohibition on Datamailer fallback;
• _docs/specs/07-security-privacy-operations.md, for privacy, targets, alarms, rollback, retention, and redacted evidence;
• _docs/specs/09-migration-rollout-roadmap.md, Milestone 8, rollback, and data-migration controls;
• _docs/specs/10-verification-strategy.md, including deployed HUMAN gates and the release acceptance report;
• _docs/PROCESS.md; and
• the production operations boundary handed off by #50.

PM disposition

GROOMED / P0 / HUMAN / PRODUCTION NO-GO / DEPENDENCY-BLOCKED. Do not dispatch a repository-only #74 engineer lane.

#74 coordinates the separately authorized production cutover, production Relay purpose/sender activation, Datamailer operational freeze/drain/retirement, observation, and rollback. It is not itself owner authorization. An issue body, closed dependency, green local test, development canary, rehearsal, elapsed observation time, or PM report cannot grant production/provider/credential/DNS/protected-data authority.

No current criterion is proved. Keep this issue open through the complete production observation and read-only retention obligations.

Outcome

After every accepted source/runtime/rehearsal/infra input and explicit owner go decision, execute one source- and image-pinned production cutover while the old static sites and course platform continue serving and all website, Relay, and Datamailer outbound paths remain disabled. Reconcile the final content and course delta, smoke the new stack internally, switch canonical DNS/edge routing, activate only approved compatibility redirects, reconcile email state, enable each approved Relay purpose exactly once with one active sender, monitor quantitative product/SEO/data/email/security signals, roll back safely when a trigger fires, and retain recoverable legacy artifacts/databases through the approved window before staged operational retirement.

The static article preservation boundary from #29 remains strict. Any non-article URL/SEO change allowed by that decision must still be independently classified, accepted, monitored, and reversible; it is not implicit #74 scope.

Protected authority boundary

Only named, explicitly authorized operators may:

• approve the exact production window, immutable release identities, go/no-go/abort roles, exceptions, and quantitative rollback thresholds;
• access protected production snapshots, rows, queues, Relay/provider state, credentials, alarms, DNS/edge, Search Console, or infrastructure state;
• freeze old writes or Datamailer intake, drain/classify live work, activate production Relay purposes/senders, execute a real canary, switch DNS/redirects, submit the sitemap, roll back, or retire a production component.

All evidence posted to this issue is redacted and contains no credential, secret, recipient, registration/profile value, provider payload, raw protected-data locator, reversible identity digest, or production record.

#50 produces accepted source/runtime inventory, routing, send-disabled history/reconciliation, and one-active-sender runbook evidence. Per #50, production Relay/provider/sender activation and production Datamailer freeze/drain/retirement happen only here under separate explicit operator authority. #74 does not reopen a Datamailer or direct-SES sending path, including during rollback.

Exact dependency and execution DAG

“Accepted” means independently tested, PM-accepted, committed, merged, pushed, green, and identified by immutable source/API/deployment evidence where applicable. Open issue prose, a local candidate, a synthetic test, or a development rehearsal is not an accepted production input.

Accepted policy baselines
  #23 privacy/retention decision (closed)
  #26 SLO/RPO/RTO targets (closed; #66 supplies measurement/alarms/runbook owners)
  #29 article/non-article scope decision (resolved; operational thresholds remain open)

Course/data/compatibility lane
  accepted current Course/Cohort/account/management inputs -> #60 side-effect-disabled rehearsal
  accepted #60 map/reconciliation -> #71 inactive course-host redirect implementation/rehearsal

Email lane
  accepted Relay contracts + #48 -> #49 ordinary/recovery delivery
  accepted owning domain triggers/policies + #49 -> #50 source/runtime integration evidence

Release lane
  accepted release-critical verification/inventories + #60 + #50
  + exact production-shaped infra/Relay/OpenAPI/credential/alarm evidence
    -> #73 full development migration/fault/restore/rollback rehearsal

Authorized production lane
  accepted #29 operational thresholds + #60 + #50 + #71 + #73
  + one immutable green release candidate/current-main release record
  + explicit owner window/runbook/roles/exceptions authorization
    -> #74 internal smoke -> DNS/edge/eligible redirect switch
    -> Datamailer freeze/classification/reconciliation
    -> separately approved production Relay canary and per-purpose activation
    -> quantitative observation/rollback/retention

Later destructive contraction
  accepted #74 observation evidence + separately resolved/groomed #287
    -> any destructive legacy Course schema contraction

#23 and #26 are accepted policy inputs, not open blockers and not operational evidence. #29 remains open for its quantitative operational threshold criterion and does not authorize cutover. #49 is transitive through #50 but its exact accepted Relay contract/deployment identity must be frozen. #60 is rehearsal-only and cannot activate a sender, redirect, or production system. #73 is a full development rehearsal, not production proof. #71 supplies the inactive/rehearsed course-host redirect and is activated only under #74’s explicit DNS gate. #287 is downstream: #74 may disable legacy writes and retire live service use after the approved window, but it cannot perform destructive schema contraction or delete recovery evidence under #287’s unresolved decision.

Execution scope after every gate passes

1. Freeze the exact website source SHA, image/config digests, active content commits, migration leaves, Relay commit/OpenAPI/deployment identity, route manifests, and rollback identities.
2. Record named commander, application, data-migration, DNS/edge, email/Relay, security, SEO/Search Console, and on-call owners; exact window; go/no-go checks; exception owners/expiry; and abort authority.
3. Keep old serving systems available while all outbound mechanisms are disabled; take the approved backups/snapshots and run final content sync and course delta.
4. Reconcile counts, checksums, stable mappings, transformations/rejections, scores/statistics, certificates, links/routes/canonicals, active commits, Datamailer history/outstanding classifications, website delivery intents/jobs, and Relay state. Any unknown, mismatch, or ambiguous item is a no-go.
5. Enable the exact new production revision behind its edge endpoint and perform internal TLS, health/readiness, auth, dynamic-write, noindex-removal, canonical, sitemap, robots, privacy/cache, Studio/API, and representative desktop/mobile smoke before public routing changes.
6. Switch canonical DNS/edge using the reviewed reversible change. Activate only #71 paths whose destination and method/auth behavior passed; preserve legacy direct compatibility where required and return true 404 for unknown paths. Permanent redirects start only after destination smoke.
7. Freeze new Datamailer intake, classify/drain or explicitly hold every outstanding logical item, import only send-disabled read-only history, reconcile website intents and Relay, and prove no new/requeue/fallback Datamailer or direct-SES path exists.
8. After separate production sender/purpose authorization and exact credential/callback/reconciliation/alarm gates, execute the named redacted production canary. Enable each approved Relay-backed purpose exactly once only after its canary gate; prove one active sender per purpose and no duplicate or missed logical delivery.
9. Submit only final canonical production sitemap URLs. Observe the complete quantitative window. Roll back on a trigger without losing post-cutover registrations/enrollments or duplicating delivery.
10. Disable legacy writes and retire old serving/sender runtime only after the approved continuous observation/redirect/data/email gates pass. Retain legacy artifacts, databases, mappings, evidence, and compatible rollback capability read-only through the approved retention window. Destructive contraction/deletion is not #74 scope.

Acceptance criteria

• [HUMAN] The owner explicitly authorizes the exact immutable release, cutover window, runbook revision, named roles, exceptions/expiry, go/no-go/abort authority, production sender/purpose set, DNS/edge change, and quantitative rollback/observation thresholds.
• Every dependency in the frozen DAG is accepted at exact merge/source/API/deployment identities; the selected current-main release CI, publish, deploy, database readiness, and deployed smoke are terminal green with no missing/cancelled gate.
• Final snapshots/sync/delta run with every website, Relay, Datamailer, worker, webhook, and provider outbound path disabled; counts/checksums/mappings/rejections/scores/certificates/routes/links/canonicals/active commits and email/outbox state reconcile with no unexplained difference.
• Internal production smoke proves TLS, health/readiness, auth, representative dynamic writes, Studio/admin API, private/cache boundaries, production canonicals, noindex removal, /robots.txt GET/HEAD contract, /sitemap.xml GET/HEAD, and desktop/mobile critical routes before the public switch.
• The reversible DNS/edge switch preserves every classified path; approved redirects are one hop and activate only after destination smoke; unsafe compatibility APIs remain direct; unknown paths remain 404; rollback identities are recorded.
• Datamailer intake is frozen and all outstanding items are safely drained, held, cancelled, or classified without silent send/drop/promotion; imported history is read-only/send-disabled; no website path can submit/requeue through Datamailer or SES.
• [HUMAN] Exact production Relay/OpenAPI/credential/callback/reconciliation/alarm evidence and the separately authorized redacted canary pass before purpose activation; each approved purpose starts exactly once with one active sender and no duplicate/missed logical message.
• Monitoring covers 404/5xx, redirects, robots/canonicals/sitemap/crawl/Search Console/organic entrances, registrations/enrollments, auth, dynamic writes, score/certificate integrity, content freshness, queue/Relay state, accepted-versus-delivered, ambiguity/backlog/duplication, cache/invalidation/WAF/allowance, and security/operator failures against approved quantitative triggers.
• Application rollback selects exact immutable identities, keeps compatible Django dynamic endpoints and expanded data, preserves post-cutover writes, pauses/reconciles workers and Relay, invalidates under the rollback identity, and never reverses migrations destructively, restores Datamailer, calls SES, mutates a Relay request, or dual-sends.
• Legacy course writes and old serving/sender runtimes are retired only after the full continuous observation gates pass; artifacts/databases/mappings remain read-only and recoverable through the approved window; any destructive schema contraction is deferred to separately resolved/groomed #287.
• The final redacted acceptance report contains every identity, reconciliation, route/SEO, migration, privacy/security, Studio/API, test/browser, Terraform/deployment, backup/restore/rollback, canary, monitoring, exception, observation, and retention artifact required by spec 10, with independent role provenance.

Required validation and evidence

1. Execute the timed runbook against the exact frozen candidate with signed-off go/no-go/abort checkpoints; do not substitute an older scheduled or development result.
2. Before and after routing changes, probe representative homepage, docs, FAQ, Wiki, Blog/Podcast/Books/People, course compatibility/canonical/API/calendar/certificate, Event/registration, learner/account/Slack, Studio/admin API, /robots.txt, and /sitemap.xml states at desktop/mobile and inspect redacted screenshots/headers/canonicals.
3. Exercise the approved rollback procedure after representative synthetic post-cutover-like dynamic writes in the accepted rehearsal; in production, execute rollback only when the authorized trigger/commander requires it. Reconcile all writes and delivery state before resume.
4. Verify Datamailer freeze/late-item/interrupted-drain/restart/hold/classification and website-job/Relay reconciliation evidence; prove the development canary is not reused as production proof and the production canary cannot reach a broad recipient.
5. Maintain the approved continuous monitoring and retention ledger. Missing telemetry, an incomplete interval, rollback, mismatch, unknown consumer, legacy-authoritative write, dual sender, or unexplained exception resets or fails the applicable gate rather than being treated as elapsed success.

Lifecycle

This broad production operations issue is not a normal repository implementation lane. Any residual source-only change discovered during readiness review must be filed and PM-groomed as a separate child and complete the normal engineer → independent tester/screenshots → PM → focused commit → local no-ff merge/push → on-call lifecycle before becoming a #74 input.

A separately authorized cutover commander and named operators execute #74 only after the PM dependency freeze and explicit owner authorization. On-call alone observes post-push/release CI. #74 remains open throughout live observation and read-only retention; criteria are checked only by their owning role against redacted exact evidence.

Explicit non-goals

• No repository feature redesign, inferred business trigger/purpose/template, article URL redesign/SEO experiment, speculative redirect, or unrelated enhancement.
• No production/protected-data/provider/credential/AWS/DNS/Search Console access or mutation without exact named authority.
• No cutover from a red/cancelled/missing gate; no stale release, mutable tag, synthetic-only proof, verbal approval, or development canary as production evidence.
• No duplicate worker/scheduler/sender, blind ambiguity retry, writable Datamailer compatibility, Datamailer/SES rollback sender, import-triggered send, or broad-recipient canary.
• No destructive reverse migration, legacy database/artifact deletion, #287 contraction, or loss of recoverability under #74.

Start with _docs/specs/09-migration-rollout-roadmap.md, Milestone 8, the frozen dependency DAG, and the evidence from #50, #60, #71, and #73; this issue explicitly says not to dispatch a repository-only engineer lane. Done means named owners and explicit authorization exist, every dependency and production gate passes, the cutover is observed with rollback readiness, and the redacted acceptance report and retention obligations are complete.