Epic: Coordinate observability, targets, backup/restore, and failure readiness

Parent epic: #8

Product outcome

Make release-critical behavior measurable and recoverable without turning one issue into a cross-repository implementation lane. The completed children must provide a redacted correlation contract, an executable target/query catalog, a safe backup-verification receipt and snapshot-provider protocol, fail-closed live restore/startup activation, Terraform-owned backup/alarms/dashboard resources, and rehearsed restore/rollback/fault/expiry procedures against the approved service and recovery targets.

#66 is the operations coordination epic and final HUMAN readiness record. It is not engineer-sized. Its six delivery children are #264 through #269. Additional owning-domain prerequisite contracts, including #284, remain prerequisites rather than new #66 delivery children. Every delivery passes the full role-separated lifecycle in _docs/PROCESS.md.

Normative authority

• _docs/PROCESS.md: issue lifecycle, separate engineering/tester/PM/on-call roles, versioned verification, and sensitive-artifact rules.
• _docs/specs/01-platform-architecture.md and _docs/architecture/app-boundaries.md: one Django deployment, owning-domain services, durable after-commit work, safe degradation, and immutable release identity.
• _docs/specs/07-security-privacy-operations.md: redacted observability, approved service targets, backup/recovery, failure behavior, and the no-resurrection/no-historical-resend gates.
• _docs/specs/08-aws-development-terraform.md: infrastructure-repository ownership, encrypted RDS/S3/logging, metrics/alarms/dashboard, immutable deployment, and OIDC boundaries.
• _docs/specs/09-migration-rollout-roadmap.md: rehearsal, worker holds, exact image rollback, active-content reconciliation, and production cutover gates.
• _docs/specs/10-verification-strategy.md: fault injection, backup verification, restore, rollback, reconciliation, redaction, and release evidence.
• _docs/specs/open-decisions.md, resolved decision 15, and closed #26: the initial service and recovery targets are approved.
• #64/#258 own privacy authority, tombstones, replay, and the privacy activation receipt; #284 defines the non-activating email recovery input that recovery-specific #49 work must implement; #49 owns logical delivery and Relay reconciliation; #102 owns immutable application-pair recovery.

Approved target baseline

Closed #26 supersedes the stale decision dependency. These targets are approved:

• 99.9% monthly availability for public reads and registration/enrollment submission;
• p95 cached public response below 500 ms and uncached HTML below 1 second at the edge region under normal load;
• 99% of approved transactional intents accepted by Relay within 5 minutes, excluding Relay outage or suppression; provider acceptance and delivery remain distinct;
• GitHub content freshness below 15 minutes after an accepted main-branch commit;
• production database RPO at most 15 minutes and service RTO at most 4 hours;
• development RPO 24 hours and RTO one business day.

The numbers do not become operational commitments until their versioned queries, windows/exclusions, evidence, alarms, runbooks, named owners, escalation paths, and exercises pass. #66 owns that implementation and acceptance work; it does not reopen #26.

Current baseline and gap

The repository already has bounded request/correlation/job context, redacted AppEvent output, one generic EMF counter, a small staff chart, liveness/readiness, durable jobs, sealed release identity, deployed smoke, and #102's exact immutable web/worker recovery controller. These are substrates, not #66 acceptance.

#264 and #265 have frozen uncommitted engineer candidates, but neither has passed independent testing, PM acceptance, merge/push, or on-call; they are not accepted interfaces. There is no accepted site-wide event/correlation schema, complete target/query/owner catalog, authoritative backup-verification receipt, production snapshot provider, global restore fence, privacy-tombstone replay integration, delivery/outbox release decision, active-content restore validator, comprehensive alarm/dashboard matrix, credential-expiry inventory, or successful restore/fault drill.

Remaining HUMAN decisions and authority

The parent retains decision and human. Engineering may define safe schemas, protocols, tests, and deny-by-default seams, but must not infer:

1. the named accountable and escalation owner for every target, plus exact measurement window, exclusions, review cadence, alert destination, and rollback/review trigger;
2. the AWS account, region, exact resources, operator/deployer roles, evidence destinations, maintenance authority, and maximum permitted cost/scope for each live operation;
3. backup retention, PITR window, automated verification cadence, encrypted verification-artifact retention, authoritative restore source, isolated target, go/no-go/abort rules, and the exact RPO/RTO clock boundaries;
4. #64/#258's production subject-key and authoritative tombstone-ledger snapshot/high-watermark, retention-versus-backup horizon, legal-hold/exception handling, retained proof, and release authority;
5. the exact activation consumers and safe degraded state for web, authentication, workers, scheduler, delivery work, exports, projections, cache/search rebuilds, and deployment finalization;
6. #49's delivery/outbox safe states, ambiguity resolution and release authority, plus the content owner's active-release/pointer validation contract;
7. the credential/certificate/provider inventory, warning lead time, rotation/reconciliation owner, and fail-closed expiry response;
8. the fault/restore environment, authorized operator and maintenance window, synthetic data boundary, evidence audience, abort authority, and approval to perform any live AWS or provider-affecting drill.

No issue label, implementation history, repository access, or prior incident silently supplies these decisions.

Child delivery map

1. #265 — observability and correlation contract — version the redacted request/job/delivery/content/course/event event schema, bounded labels, failure-safe emission, and domain integration seam. It owns no target values, AWS resources, backup/restore, or runbook execution.
2. #266 — service-target and query catalog — map every approved #26 target and release-critical failure to an exact versioned query, source, window/exclusions, threshold, evidence cadence, owner/escalation slot, alert/runbook key, and dashboard panel key. Domain apps own their measurements; the catalog does not mutate domain state.
3. #264 — backup-verification receipt and snapshot-provider protocol — highest-priority cycle breaker for #258. Define a repository-only, versioned, provider-neutral BackupVerificationReceipt and snapshot-provider result/protocol with canonical safe fields, digests, runtime/schema identities, timestamps, counts, allowlisted error codes, synthetic verifier/provider, and fail-closed compatibility tests. It performs no AWS read/write, backup, restore, privacy replay, or activation.
4. #267 — live restore/startup activation controller — consume the accepted backup receipt/provider result, #258 privacy receipt, recovery-specific #49 delivery/outbox reconciliation result, active-content validation, and current runtime/schema identity. Hold every named activation consumer until all exact inputs revalidate, and fail closed on absence, drift, ambiguity, or partial coverage. It does not own tombstones, delivery state, content mutations, or infrastructure resources.
5. #268 — AWS backups, alarms, and dashboard — in DataTalksClub/aws-infra, implement the approved encrypted backup/PITR/final-snapshot settings, verification execution and artifact destination, query-backed alarms, dashboard, notification routing, expiry monitors, IAM, retention, and cost controls from the final #266 catalog and #264 contract. Source/policy tests do not count as apply, restore, or alarm-firing evidence.
6. #269 — restore, rollback, fault, and expiry runbook plus drill — compose #267 with #102 recovery and #268 infrastructure; define exact restore/RPO/RTO clocks, workload holds, rollback/fault/expiry scenarios, escalation/abort rules, redacted evidence, and clean return-to-service. Execute only after explicit HUMAN authorization and all dependencies are accepted.

The parent closes only after all children and required owning-domain integrations are accepted. Child issues close only their own bounded contracts and reference #66.

Cycle-breaking order and parallel lanes

1. Repair the shared website base, then rebase and complete separate tester/PM/merge/on-call lifecycles for frozen #264 and #265. They are independent contracts; neither staged candidate is an accepted dependency.
2. After #264 is accepted and green, #284 may deliver its source-only, non-activating email recovery input. PM then re-grooms the recovery-specific portion of #49 against #284's exact merged identity. The privacy path proceeds independently through #254 -> #281 -> #255/#283 -> #257. #258 is re-groomed only after accepted #264, #284, recovery-specific #49, #255, and #257 interfaces. #258 must not depend on #267.
3. After #265 is accepted and green, #266 may deliver its bounded repository-only catalog slice with unresolved rows explicitly blocked and Refs #266. Domain owners and HUMAN owners can complete their source/query/owner inputs in parallel.
4. #267 is re-groomed only after accepted #264, #258, recovery-specific #49, the separately accepted content-owned active-pointer validator, #102 composition, and its exact HUMAN consumer/hold/authority packet. It consumes owner receipts one way; it does not define them.
5. To avoid a cycle, final #266 closure binds RPO/RTO query semantics to accepted source contracts—#264 for recovery-point evidence and #267 for the generation/controller completion clocks—not to #268 live resources or #269 drill results. #268 and #269 later prove live execution of those catalogued semantics; they are downstream evidence, never prerequisites for #266's source contract.
6. #268 is re-groomed only after final zero-gap closure_ready #266, accepted #264/#265 source identities, #78/#94, and the complete HUMAN packet. External DataTalksClub/aws-infra source authority and protected plan/apply/readback authority are separate grants. Website repository access grants neither.
7. #269 is last. Re-groom it only after accepted #264/#266/#267 contracts, exact applied/read-back #268 state, accepted #258/#49/content/#102 evidence, and the complete protected-operation packet. Its successful authorized rehearsal and clean return are required for parent acceptance.

No child may create a reverse import or dependency from privacy, email_app, content, or a domain app into deployment/infrastructure code merely to make orchestration convenient.

Ownership boundaries

• core/the observability substrate owns common context, schema, redaction, bounded labels, and safe emission; each domain owns the meaning and correctness of its events and aggregates.
• The target/query catalog owns declarative measurement and evidence mapping, not domain business logic or alert resources.
• #64/#258 privacy owns tombstones, replay, privacy checkpoints, and its exact activation evaluation/receipt. It does not verify backups or activate workloads.
• #284 freezes only a non-activating email recovery input vocabulary. Recovery-specific #49/email_app owns its runtime implementation, logical-delivery and Relay-projection reconciliation. #258 and #66 may consume an accepted safe receipt/evaluator; they may not define email-state meaning, send, retry, resolve ambiguity, or create a direct provider fallback.
• content owns active-release and pointer validation through its service. Restore orchestration consumes a safe result and never edits content rows directly.
• deploy/#102 owns immutable image identity, exact web/worker compensation, and release records. Child 4 composes with that controller and does not weaken or replace its receipt/terminal checks.
• jobs owns generic durable jobs, leases, scheduler, and heartbeats. The activation controller gates them through an accepted seam rather than duplicating their state machine.
• DataTalksClub/aws-infra owns Terraform, AWS backup resources, IAM, alarms, dashboard, notification destinations, and encrypted operational storage. The website repository owns provider-neutral contracts and application behavior only.
• A named HUMAN operator owns live authorization, maintenance window, go/no-go/abort, evidence custody, and final drill acceptance.

Parent acceptance criteria

• All six children are independently groomed when their exact prerequisites exist, delivered, tested, PM-accepted, merged, and observed according to the dependency order above; bounded Refs source phases do not count as child closure.
• Every approved #26 target and release-critical failure has one versioned query/source, threshold/window/exclusion policy, review cadence, named owner/escalation, alert, runbook, dashboard panel, and tested evidence path.
• Request/job/delivery/content/course/event correlation is end-to-end, deterministically redacted, cardinality-bounded, failure-safe, and excludes sensitive values from logs, metrics, traces, audits, screenshots, and artifacts.
• The provider-neutral backup receipt and snapshot-provider protocol rejects missing, stale, partial, mismatched, unencrypted, wrong-runtime/schema, or noncanonical evidence and interoperates with #258 without a dependency cycle.
• Live restore/startup holds every accepted activation consumer until backup, tombstone replay, delivery/outbox reconciliation, active-content validation, runtime/schema identity, and checkpoint arithmetic all revalidate; no bypass or eventual-convergence inference can mark it ready.
• AWS source, policy, apply/readback, alarm firing, dashboard, retention, encryption, IAM, cost, backup-verification, isolated-restore, and evidence-destination gates are each classified and independently evidenced; source validation alone is not live proof.
• Restore meets the approved environment-specific RPO/RTO, retains the correct active content, reapplies erasure tombstones, prevents erased-data use/projection/export, and releases historical delivery work only after accepted reconciliation with no resend.
• Immutable image rollback preserves post-cutover registrations/enrollments and idempotency, proves the exact VERSION/source SHA/image digest and terminal pair, and does not weaken #102.
• GitHub, search/graph, worker, Relay submission/callback/reconciliation, OIDC, database, edge, backup, restore, and credential-expiry faults produce the documented safe degradation, alert, escalation, and recovery behavior.
• Credential, certificate, callback-secret, and provider-client expiry monitors fire at the accepted lead times and the tested rotation/reconciliation procedure fails closed without exposing secret material.
• [HUMAN] Named service, infrastructure, content, privacy, delivery, release/on-call, and escalation owners approve their exact rows, destinations, authorities, review cadence, and exception handling.
• [HUMAN] An authorized synthetic production-like backup/restore/rollback/fault/expiry rehearsal uses the approved environment and maintenance window, meets RPO/RTO, exercises abort and rollback, returns to a known-good exact release, and retains redacted evidence with no production data, credential, provider payload, resend, or resurrection.

Required scenario families

1. Complete/missing/malformed/duplicate correlation; nested request-to-job and callback/reconciliation chains; emission backend unavailable; sensitive/cardinality canaries.
2. Every target at healthy, exact boundary, breach, excluded interval, missing data, stale query/catalog version, absent owner/destination, and dashboard/query disagreement.
3. Backup receipt before/exactly-at/after freshness and RPO bounds; wrong environment/runtime/schema/source; partial count; checksum/digest drift; provider ambiguity; encrypted artifact missing/expired; exact replay.
4. Restore before/during/after erasure and delivery work; missing/new tombstone; ambiguous outbox; stale content pointer; checkpoint crash/retry; one consumer attempts early startup; every consumer remains held until the exact current receipt set passes.
5. AWS plan/apply/readback drift, denied IAM, unavailable provider, failed verification, backup/PITR/final-snapshot retention boundary, alarm no-data/breach/recovery, notification failure, and cost/scope cap.
6. GitHub/search/worker/Relay/OIDC/database/edge failure; immutable rollback with new registrations/enrollments; credential-expiry warning/breach; drill abort; failed recovery remains loud; clean return to the intended exact release.

Explicit non-goals

• No monolithic implementation under #66 and no collapsing of independent website, privacy, delivery, content, deployment, or infrastructure ownership.
• No reopening approved #26 target values and no claim that a target is operational before its measurement, ownership, alert, runbook, dashboard, and exercise exist.
• No production/protected-data backup, restore, replay, drill, AWS mutation, provider call, credential readback, or workload activation without the named HUMAN authority and normal lifecycle.
• No raw email, profile/submission content, account/domain identifier, IP/header, token, credential, secret, provider payload, database row/export, query string, or response body in operational evidence.
• No direct transport provider or legacy-sender path, automatic resend of ambiguity, dual sender, or acceptance of provider-accepted as delivered.
• No database-only tombstone assumption, destructive reverse migration, whole-site rollback that loses dynamic writes, arbitrary --force/skip/ignore override, silent partial readiness, or eventual convergence accepted as proof.
• No browser/product redesign. If a child introduces an operator or maintenance surface, that child owns the graph-selected browser tier, accessible states, private/no-store/noindex behavior, and inspected redacted desktop/mobile screenshots.

Delivery convention

Each accepted commit references #66 and follows its owning issue's explicit closure rule. #266's first source slice uses Refs #266; #268 source delivery uses Refs #268 until protected apply/readback evidence passes; #269 source delivery uses Refs #269 until the authorized rehearsal and every HUMAN gate pass. No source commit, green source CI run, Terraform validation, or synthetic fixture closes a live-evidence issue. Because owner/authority, live AWS, backup/restore, and drill evidence remain HUMAN, #66 keeps human/decision and stays open until every automated and manual criterion above has accepted evidence.

Start with _docs/PROCESS.md and the listed architecture, security, Terraform, rollout, and verification specifications, then review delivery children #264 through #269. This parent is coordination-only and explicitly not engineer-sized; work is complete only after the bounded children, required owning-domain integrations, human decisions, and authorized readiness evidence are accepted.