Fence restores with erasure tombstone replay and outbox reconciliation

Authoritative lifecycle hold — dependency reconciliation 2026-08-30

needs grooming; engineering is not authorized. This block supersedes the provisional implementation-ready and locally defined backup/outbox-adapter prose below. Privacy path: accepted #255 + #257. Recovery path: accepted/merged/green #264 → #284 → recovery-specific #49 implementation. PM may re-groom #258 only after those exact merged interfaces exist; #258 must consume them and must not define reverse backup ownership or email state-to-safe semantics. #66 alone owns live restore/startup activation and the combined rehearsal. Production subject keys, protected data, provider work, restore execution, and HUMAN decisions remain disabled.

Parent epic: #64
Restore/backup owner: #66
Backup receipt prerequisite: #264
Privacy authority/kernel/retention prerequisites: #254, #255, #257
Delivery/outbox prerequisites: #284, then #49

Product outcome

Provide a durable, non-PII erasure-tombstone ledger contract and a fail-closed restore/recovery fence so a restored or rolled-back system cannot authenticate or expose an erased subject, rebuild an erased projection, export erased data, or dispatch historical delivery work before deterministic tombstone replay and outbox reconciliation succeed.

This issue has two gates. The bounded backend foundation below is implementation-ready only after accepted #254/#255/#257 and is deliberately execution-disabled for every production domain, restore path, subject-key scheme, and delivery adapter. It may be delivered with Refs #258. Live restore/startup integration and final closure remain blocked on #66, #284, #49, accepted production erasure/domain adapters, and the HUMAN decisions listed below.

Normative authority

• _docs/PROCESS.md: role separation, dependency ordering, versioned verification, sensitive-artifact rules, and Refs delivery for remaining HUMAN work.
• _docs/specs/01-platform-architecture.md, _docs/architecture/app-boundaries.md, and _docs/architecture/shared-primitives.md: one Django deployment, owning-domain services, atomic durable intents, after-commit work, at-least-once execution, and fenced leases.
• _docs/specs/07-security-privacy-operations.md: full-erasure direction, deletion propagation, non-PII reconciliation evidence, tombstone replay, restore hold, no historical resend, redaction, and approved RPO/RTO.
• _docs/specs/09-migration-rollout-roadmap.md: immutable release identity, worker/send holds, rollback reconciliation, and no destructive reverse migration or Datamailer/SES fallback.
• _docs/specs/10-verification-strategy.md: restored-backup replay, no resurrection/resend, fault injection, migration/concurrency, and redacted HUMAN evidence.
• _docs/specs/05-events-registration-email.md, #284, and #49: #284 owns the non-activating generation/receipt interface; #49 owns its future runtime implementation; Relay remains authoritative for transport, ambiguity never permits automatic resend, and Datamailer/direct SES are never fallback senders.
• _docs/specs/open-decisions.md, closed #23, #254, and parent #64: full erasure is approved, but tombstone expiry, legal-hold interaction, processor receipts, retained proof, and override/escalation remain HUMAN decisions.
• #255 owns the domain-neutral privacy aggregate and opaque subject binding; #257 owns the retention/hold registry and runner. This issue consumes those contracts and does not replace them.

Decision and dependency boundary

The following are not derivable and must not be invented:

1. the production tombstone subject-key/pseudonymization scheme, its resolver, rotation/migration rules, collision behavior, and whether/how it may relate to #255's opaque subject binding;
2. exactly which tombstone/request/linkage fields constitute approved non-PII retained proof;
3. tombstone retention or expiry relative to every backup/PITR/replica/export/object-store horizon, legal/security holds, processor obligations, and decommissioning;
4. legal-hold/exception behavior, release authority, escalation deadlines, and whether any override is ever permitted;
5. processor receipt requirements and requester-safe evidence;
6. #66's exact authoritative ledger storage/snapshot transport, restore-event/backup identity, startup/deployment activation interface, checkpoint durability, and approved operator runbook.

The foundation therefore registers no production subject-key definition, domain replay adapter, outbox adapter, ledger snapshot provider, startup hook, command, scheduler, restore controller, or override. Test definitions use random opaque UUIDs only. No email, normalized identifier, account/provider/domain primary key, raw or reversible hash, keyed digest of a subject, lookup table back to PII, or free-form value is accepted as a tombstone key or evidence field.

A database table inside the restored database is not by itself proof that tombstones survived the restore. The production authoritative ledger/snapshot must be supplied through #66's separately accepted restore boundary and must cover a horizon at least as strong as the restored source; until that contract exists, every production fence remains blocked.

Ownership boundaries

• privacy owns tombstone/replay/fence models, immutable definition registries, services, safe DTOs, audit/redaction, and test-only handlers.
• Each production domain owns subject resolution and re-erasure of its source records, projections, caches, search documents, exports, and queued domain state through its accepted service/adapter. privacy never imports or mutates domain models directly.
• email_app/#49 owns logical-delivery and Relay-projection reconciliation. Relay remains authoritative; this issue never submits, retries, resends, suppresses, or calls a provider.
• jobs owns generic durable jobs and leases. core owns generic revision, idempotency, operation, audit, and redaction primitives. Do not duplicate them.
• #66 owns backup verification, restore/rollback/replica-promotion orchestration, authoritative pre-activation input, workload holds, deployment/startup integration, RPO/RTO, and the final synthetic rehearsal.

Execution-disabled foundation contract

Immutable code registries

Definitions are immutable, deterministically enumerated dataclasses. Duplicate keys with different definitions, malformed keys, unknown versions, unsafe bounds, or import-order disagreement fail system checks.

• TombstoneKeyDefinition: bounded scheme_key, positive contract version, and declared opaque-key type. #258 registers only a test scheme whose value is a random UUID. There is no hash/email/provider/domain-key mode and no production definition.
• RestoreReplayAdapterDefinition: bounded domain_key, positive contract version, accepted #254 manifest-entry digest, supported tombstone-key scheme/version, maximum batch size and attempts, and plan_batch, apply_batch, and reconcile_batch methods. The adapter returns only counts, allowlisted codes, UUIDs, and SHA-256 digests. Only synthetic test adapters are registered.
• OutboxReconciliationAdapterDefinition: bounded outbox_key, positive contract version, maximum batch size/attempts, and side-effect-free plan plus reconcile methods. It may classify/reconcile historical work but has no send method. Only a synthetic test adapter is registered; the accepted #49 implementation of #284 supplies the future production definition.
• RestoreSnapshotProviderDefinition: validates a canonical snapshot manifest supplied by the future #66 boundary. Only a synthetic test provider exists. It cannot read production backup/provider state.

No runtime setting, database row, environment variable, command option, request parameter, or plugin may add/replace a definition or weaken a bound.

Tombstone ledger

Add an append-only privacy.ErasureTombstone foundation model with:

• random UUID primary key and random opaque tombstone_ref UUID;
• test-only subject_key_ref UUID plus pinned key-scheme key/version; this field is structurally present for protocol testing but cannot be populated by a production path until the HUMAN key decision is accepted;
• opaque privacy-request/erasure-operation UUID references only, with no cascade back to erased subject data;
• pinned #254 authority/register and #255 manifest versions/digests, erasure generation, and required ordered domain-manifest digest;
• state exactly pending, sealed, or blocked; bounded allowlisted code; created/sealed timestamps; and an immutable evidence digest over safe canonical fields.

A logical tombstone is unique for the accepted (key scheme/version, subject key ref, erasure generation) identity. Exact duplicate/concurrent creation converges through core.IdempotencyRecord; changed policy/manifest/domain set/key/generation conflicts. sealed is immutable. A later production erasure service must create/seal the tombstone and complete the erasure request atomically, or leave both explicitly nonterminal/blocked; this foundation has no production caller and cannot mark a real erasure complete.

There is no tombstone delete, expiry, compaction, subject lookup, re-identification, merge, edit, force-seal, force-skip, or override service. The foundation rejects non-test creation and any key that is not the exact registered opaque test UUID type.

Authoritative snapshot and restore fence

Add immutable safe records for:

• TombstoneLedgerSnapshot: opaque snapshot UUID, schema/registry versions and digests, monotonic safe high-watermark, ordered active-tombstone manifest digest/count, key-definition digest, source identity digest, creation time, and validation state/code. It stores no tombstone subject list, cursor, raw backup identifier, provider payload, or subject-key values in safe evidence.
• RestoreFence: opaque restore-event UUID; environment class restricted to synthetic test in this slice; exact source/deployed full SHA, image digest, database schema/migration digest, backup-identity digest, #254/#255/#257 identities, ledger snapshot/high-watermark/digest, replay-definition digest, and outbox-definition digest; revision/generation; state exactly blocked, planned, replaying, reconciling, ready, or failed; allowlisted safe code and timestamps.
• RestoreReplayPlan and one immutable plan row per required replay domain/outbox definition: exact ordered tombstone/input manifest digest, selected count, adapter/control versions, bounds, and plan digest.
• RestoreReplayCheckpoint: fence/generation/plan/domain-or-outbox key, sequential batch number, opaque batch/job/correlation UUIDs, input/result/evidence digests, selected/applied/already-safe counts, attempt, state, and allowlisted code. A succeeded checkpoint is immutable and unique for its batch identity.
• RestoreFenceReceipt: one immutable receipt for a ready fence containing only the exact fence/plan/snapshot/release/schema/definition identities, aggregate counts, completion time, and canonical receipt digest. It grants no authority by itself; #66 must consume and revalidate it through its future exact activation interface.

The foundation stores only canonical bounded values. Snapshot and backup identities are digests of future #66 safe manifests, never provider responses, object URLs, credentials, database contents, record IDs, or production exports.

Planning, replay, reconciliation, and activation semantics

Shared services require an injected deny-by-default authorization policy, expected revision, and core.IdempotencyRecord key. There is no HTTP, Studio, admin API, or management-command adapter.

1. create_restore_fence(...) accepts only a validated snapshot-provider result and exact sealed runtime/source/schema/policy identities. It creates a blocked fence, operation, audit, and idempotency result atomically. Missing, stale, unreadable, malformed, regressed-high-watermark, unknown-key-scheme, or digest-mismatched input remains blocked.
2. create_replay_plan(...) deterministically enumerates every required #255/#254 manifest domain and every required outbox definition. Each adapter plans against the same immutable tombstone snapshot and restore identity. Missing/extra/unknown/disabled definitions, an unsealed/pending tombstone, an expired/held/unknown tombstone disposition, count disagreement, or any unsafe output blocks publication. No ready plan exists on partial planning.
3. start_replay(...) revalidates every identity and plan digest, claims one cross-restore fence, creates per-row execution state and one bounded durable intent per first batch, and wakes only after commit. A restore event/fence generation cannot overlap another replay of the same backup/snapshot/environment identity.
4. The versioned durable handler processes at most one configured batch per invocation. It reloads fence/plan/snapshot/definition/lease/revision state before adapter invocation and again before completion. Long work renews its fenced lease. It never holds a database transaction during adapter planning or external work.
5. apply_batch may call only the owning domain's accepted service. Mutation identity is (fence UUID, generation, plan digest, adapter key/version, batch UUID, input digest). Crash before domain commit safely retries; crash after domain commit but before checkpoint commit calls reconcile_batch with that exact identity and never blindly repeats mutation.
6. A successful replay batch must reconcile exactly selected = reapplied + already_absent. A successful outbox batch must reconcile exactly selected = held_or_terminal_safe + reconciled_safe. Zero is valid only for an explicitly selected adapter. Unknown, skipped, expired, excluded, ambiguous, sendable-but-unreconciled, or count/digest mismatch is blocked, never added to success.
7. Tombstone replay completes before outbox reconciliation begins. Outbox reconciliation must prove every selected historical logical intent/job/projection is held or in an accepted safe reconciled state. pending, leased, retryable, ambiguous, callback-lagged, missing, changed-request, or unknown state blocks; it never calls Relay or turns uncertainty into safe. Future #49 reconciliation may resolve state through its own accepted service without automatic resend.
8. mark_ready(...) revalidates the authoritative snapshot high-watermark, all identities, every immutable checkpoint, exact aggregate arithmetic, no active/unknown hold or exception, no unsealed tombstone, no new applicable tombstone outside the plan, no unresolved domain/outbox row, and no lost lease/fence. It then creates one receipt atomically. Any drift requires a new plan/generation; old plans and succeeded checkpoints are never edited.
9. The foundation exposes a pure evaluate_activation(receipt, current_identity) -> blocked|ready seam for tests. Absence, corruption, mismatch, staleness, superseded snapshot, unknown current tombstone, or failed revalidation returns blocked. It has no production startup/deployment hook. #66 must later prove that web, auth, workers, scheduler, sends, exports, projection/cache/search rebuilds, and deployment activation cannot bypass that result.

Exact idempotent replay returns the original safe DTO. Same key with changed action, restore identity, snapshot, plan, generation, revision, adapter version, batch input, or bounds conflicts. A stale/lost fence, lease, generation, revision, checkpoint owner, or receipt cannot invoke an adapter or commit success. There is no --force, ignore-hold, ignore-tombstone, skip-domain, skip-outbox, accept-ambiguous, arbitrary cutoff, subject selector, duration override, direct SQL, production schedule, or emergency bypass.

Failure and evidence contract

Explicit transient test codes may retry within the pinned maximum. Permanent, unknown, unsafe, exhausted, ambiguous-after-reconcile, zero-progress, snapshot/control unavailable, corrupted receipt, plan/count/digest drift, or replay/outbox definition loss blocks the fence. A failed fence never becomes ready by changing its old rows; resume requires full invariant recheck and a new fenced generation.

Safe DTOs, audits, logs, jobs, exceptions, checkpoints, receipts, and verification artifacts contain only schema/register/manifest versions and digests; full source SHA/image/schema digests; opaque restore/fence/plan/tombstone/operation/job/batch/correlation UUIDs; definition keys/versions; UTC status timestamps; aggregate integer counts; allowlisted states/codes; and pass/fail.

Never persist, log, audit, return, screenshot, or attach names, emails, usernames, IPs, raw account/provider/domain IDs, raw or reversible subject hashes, subject-key lists, database cursors, candidate rows, field values/content, URLs, credentials/tokens/cookies, SQL, stack traces, provider payloads, message bodies, production backup manifests, or production exports. Protocol serializers reject unknown keys/types, sensitive-looking names, email/URL/credential/provider canaries, free-form strings, noncanonical Unicode/control characters, floats/NaN, cycles, or excessive depth/items/bytes; redaction alone is not acceptance.

Scope now — backend foundation only

• Extend the accepted #254 register schema only as needed to represent this issue's production-disabled definition and HUMAN gates; do not duplicate policy authority.
• Add the privacy models, constraints, migrations, registries, DTOs, services, test-only handler, audit/redaction integration, system checks, and internal architecture documentation above.
• Add only synthetic non-PII tombstones/snapshots/restores and explicitly test-registered adapters/providers.
• Prove deterministic planning, bounded replay/reconcile mechanics, crash recovery, immutable checkpoints, idempotency, fencing, safe evidence, and fail-closed activation evaluation without any production integration.

Explicit non-goals

• No production subject-key scheme/resolver, tombstone creation, ledger persistence/snapshot transport, domain replay adapter, outbox adapter, restore/startup/deploy hook, scheduler, command, route, API, OpenAPI, template, navigation, dashboard, alert, or runbook execution.
• No production/protected-data backup, restore, rollback, replica promotion, projection rebuild, dry run, replay, erasure, export, subject lookup, login change, cache/search mutation, provider call, message submission, suppression change, or processor receipt.
• No direct Relay/Datamailer/SES/provider fallback; no email send/retry/resend and no claim that provider state was reconciled.
• No tombstone expiry/retention, legal-hold release, exception disposition, override/bypass/escalation policy, RPO/RTO interpretation, processor policy, requester evidence, or legal invention.
• No domain behavior owned by accounts/#251, courses, events, content, email/#49, search/cache, exports, audit/log storage, #66, or future erasure-adapter issues.
• No claim of live restore safety, commit, push, merge, deployment, or external side effect outside the normal lifecycle.

Acceptance criteria — execution-disabled foundation

• #254 remains the sole hand-edited authority. Every tombstone key/retention/expiry/hold/override/processor/#66 interface gap is represented as human_required or missing dependency, never as an accepted default.
• The exact ledger/snapshot/fence/plan/checkpoint/receipt models and portable constraints exist; sealed tombstones and succeeded checkpoints/receipts are immutable through supported services.
• Only the opaque random-UUID test key scheme, synthetic snapshot provider, synthetic replay adapters, and synthetic outbox adapter can register; startup/system checks reject production/runtime registration and conflicting definitions.
• Production tombstone creation, restore planning/replay, outbox reconciliation, activation evaluation consumption, schedules, commands, and adapters are absent/disabled. Any production environment/input fails before mutation.
• Tombstone creation/sealing, fence creation, planning, replay start/resume, batch completion, outbox transition, and receipt creation enforce authorization, expected revision, exact idempotency, pinned identities, atomic durable intents, and after-commit wakeups.
• Plans are deterministic and all-or-blocked. Missing/extra/unknown domains, keys, adapters, tombstones, controls, pending erasure, stale policy/schema/runtime identity, snapshot regression, or count/digest drift cannot publish a runnable plan or ready receipt.
• Work is bounded to one batch per job, lease/fence/revision/generation protected, immutable after success, crash/restart resumable, and reconciled after an ambiguous commit without duplicate domain mutation.
• Replay precedes outbox reconciliation; unknown/pending/in-flight/retryable/ambiguous/changed delivery state blocks and no adapter interface can send or automatically resend.
• Exact selected/applied/already-safe arithmetic and aggregate checkpoint reconciliation are mandatory; unknown skip categories, zero progress, partial coverage, newly applicable tombstones, or unexplained differences block readiness.
• Missing/corrupt/stale/mismatched/superseded receipt or current identity evaluates blocked; no safe DTO or receipt grants authorization or bypasses future #66 revalidation.
• Audit, log, exception, job, snapshot, checkpoint, receipt, idempotency, fixture, migration, and verification output passes PII/credential/URL/provider/domain-ID/reversible-hash/free-text/Unicode/control/excessive-JSON/NaN canaries.
• Migrations pass forward, reverse, fresh install, migrate-back/forward, constraint introspection, rollback injection, and real concurrent transaction tests on SQLite and production-like PostgreSQL; migration drift is clean.
• Focused privacy/core/jobs/register tests, import/system checks, formatting/lint/type checks, container tests, and the graph-selected versioned verification plan pass with uv/Make.
• Render contract is exactly no URL, response, template, context processor, static asset, navigation, OpenAPI, email, or browser-visible change. The independent tester records Playwright/screenshots not_applicable only if the frozen change graph proves it; any render impact restores the graph-selected browser tier and inspected desktop/mobile screenshots.

Required verification scenarios

1. Malformed/duplicate/conflicting registries; runtime override; unknown key/provider/adapter version; source/schema/policy/manifest drift; unsafe bounds; import-order disagreement; production registration/input rejection.
2. Exact duplicate versus changed/concurrent tombstone creation; pending/blocked/sealed tombstone; sealing transaction rollback; changed erasure generation/domain manifest; mutation attempt after seal.
3. Missing/corrupt/unreadable/stale/regressed snapshot; before/exactly-at/after high-watermark; tombstone added during planning/replay/readiness; unknown/expired/held/exception tombstone; raw subject/hash/email/provider-key rejection.
4. Snapshot before, during, and after synthetic erasure; restored source before and after tombstone; empty/one/many tombstones; empty/one/many domains; missing/extra domain; duplicate/concurrent restore event; rollback and replica-promotion identity classes.
5. Same plan inputs twice versus changed order, count, input digest, key/adapter/control version, source SHA/image/schema, registry/policy/manifest, bounds, or plan expiry; exact plan digest comparison.
6. Batch size 1 and maximum; final partial batch; zero progress; apply twice; two workers/fences; stale lease/fence/generation/revision; renewal loss; crash before domain commit; crash after domain commit/before checkpoint; reconcile success/unknown; retry/exhaustion/permanent/unsafe failure.
7. Historical outbox states pending, leased/in-flight, retryable, provider-accepted, delivered, ambiguous, suppressed, dead, hard-bounced, complained, missing callback, and changed request hash; only accepted reconciled safe outcomes progress and no path sends.
8. Account/login use, public/private projection, cache/search rebuild, export, Slack-state projection, worker/scheduler/send activation test seams all remain blocked until the exact receipt matches; missing/corrupt/stale receipt and a newly applicable tombstone re-block.
9. Selected/checkpoint/aggregate arithmetic, unknown code/count key, excessive counts/JSON, and every sensitive-output canary across protocol, persistence, audit, job, exception, log, and artifacts.
10. Migration/constraint/fresh/apply-twice and true concurrent transaction tests on SQLite/PostgreSQL; injected rollback proves fence/plan/execution/checkpoint/job/idempotency/audit atomicity.

Browser and screenshot disposition

Expected for the foundation: not_applicable. It changes no rendered route or API response. The engineer records pending_independent_tester until the independent tester recomputes the frozen graph; only that tester may mark Playwright/screenshots not_applicable.

If later #66 integration adds a public/learner/operator maintenance or denial state, that separately groomed render slice must test the exact affected routes at desktop 1440×900 and mobile 390×844, inspect screenshots under .tmp/screenshots/issue-258/, prove private/no-store/noindex where applicable, accessible status/focus/recovery copy, no identity/value leakage, and no error/debug page.

Later live activation and HUMAN closure gate

Live work requires separate accepted issues/contracts for the production key scheme and retained proof, each domain erasure/replay adapter, ledger durability/snapshot transport, #284 contract plus #49 runtime outbox reconciliation, and #66 restore/startup/deployment sequencing. Each must pin exact versions, bounds, hold/exception authority, safe failure codes, evidence, and rollback, and pass its own engineer/tester/PM lifecycle.

Before any live or production-like restore, an authorized HUMAN operator/privacy owner must approve the exact environment, #254 register/policy version, subject-key scheme, tombstone retention versus backup horizon, hold/exception snapshot, authoritative ledger snapshot/high-watermark, source/deployed/schema identities, replay/outbox plan digest, maintenance and workload-hold window, maximum scope, RPO/RTO, evidence destination, escalation, and whether an override exists. Silence never authorizes an override; absent authority means no override and a blocked fence.

[HUMAN] Independently rehearse a synthetic restore through #66 using a pre-erasure snapshot, erasure during the snapshot horizon, every replay/outbox checkpoint crash boundary, and exact redacted evidence. Prove before activation that no account use, public/private output, export, projection/cache/search rebuild, Slack state, worker, or historical send can resurrect. The rehearsal must meet the approved #26 RPO/RTO and contain no subject value, reversible digest, credential, provider payload, message body, or production data.

Delivery convention

After accepted #254/#255/#257, engineer implementation of only this backend foundation, independent tester PASS, and PM acceptance, commit with Refs #258; retain decision and human and leave #258 open. Final Closes #258 requires accepted #284/#49/#66 and production adapter/key/retention/override decisions plus the redacted HUMAN rehearsal above. No push/merge/deploy occurs outside the orchestrator lifecycle.

Start with _docs/PROCESS.md and the cited architecture, security, migration, and verification specifications, then review the accepted #254, #255, #257, #264, #284, and #49 boundaries. The intended result is an execution-disabled Django privacy foundation with immutable tombstone, snapshot, fence, plan, checkpoint, and receipt records, using only synthetic test definitions; production restore and activation remain blocked.