Create private privacy-export artifacts and correction orchestration

Authoritative lifecycle hold — dependency reconciliation 2026-08-30

needs grooming; engineering is not authorized. This block supersedes any statement below that #254/#255 alone make the backend slice implementation-ready. Exact order: accepted/merged/green #254 → #281; then #255 and #285 may be delivered independently; only after both #255 and #285 are accepted/merged/green may PM rewrite #256 against their exact contracts and remove needs grooming. Source contract #285 is non-activating. Production adapters, private artifact infrastructure, requester/staff surfaces, protected data, and HUMAN policy remain later gates.

Parent epic: #64
Authority/register prerequisite: #254
Kernel prerequisite: #255
Export/correction input prerequisite: #285
Production adapter/infrastructure prerequisites: separately accepted domain adapters and private artifact-store infrastructure

Product outcome

Produce one private, expiring, machine-readable portable-export bundle for one verified privacy subject and orchestrate corrections through the exact owning domain services. Export and correction work is manifest-pinned, deterministic, bounded, resumable, idempotent, revision-safe, and fail-closed on missing policy, adapters, shared-record rules, storage, authorization, or proof. The privacy app stores only opaque identities, state, counts, digests, and safe codes in the database; export bytes and sealed correction commands never enter logs, audits, jobs, operations, screenshots, issue evidence, or ordinary backups.

This issue has two delivery gates. The backend contract/orchestration slice is implementation-ready after accepted #254 and #255 and uses only test adapters plus a test artifact store. It introduces no production domain adapter, production storage backend, requester/staff surface, or real export/correction. That slice is committed with Refs #256; #256 stays open. Production closure requires accepted domain adapters, accepted private-store infrastructure, the applicable HUMAN policy rows, and the later requester/Studio surfaces.

Normative authority

• _docs/PROCESS.md: role separation, versioned verification, sensitive-artifact rules, and Refs delivery for remaining gates.
• _docs/specs/01-platform-architecture.md, _docs/architecture/app-boundaries.md, and _docs/architecture/shared-primitives.md: one deployment, owning-domain services, durable after-commit work, revision/idempotency/audit primitives, and one-way imports.
• _docs/specs/04-courses-and-cohorts.md and 05-events-registration-email.md: minimized course-registration snapshots, historical immutability, learner/shared relations, event registration/export ownership, and formula-safe tabular exports.
• _docs/specs/06-studio-and-admin-api.md: capability/object/field authorization, high-risk export/correction, expected revision, idempotency, audit, and future Studio/API parity.
• _docs/specs/07-security-privacy-operations.md: private/no-store exports, portable export/correction rights, exact account-export inclusions and Slack-secret exclusion, propagation/cleanup, redaction, encryption, and bounded retention.
• _docs/specs/08-aws-development-terraform.md: workload-owned encrypted S3 exports/artifacts with public-access block, lifecycle, and least privilege; this issue does not provision it.
• _docs/specs/10-verification-strategy.md: privacy, security, storage-failure, concurrency, migration, and evidence requirements.
• Closed #23, parent #64, and #254: full-erasure direction, approved facts, accepted policy/manifest identities, and explicit HUMAN gaps.
• #255: request/right/domain-work state, proof, manifest, retry/reconcile, safe evidence, and adapter kernel consumed here.

Dependency and activation disposition

Engineering must not start until #254, #281, #255, and #285 are independently accepted, merged, and green. The contract slice registers no production export/correction adapter and no production artifact store; startup/system checks must prove production execution is disabled. Test-only adapters/stores may exercise the complete protocol with synthetic canaries.

Production execution additionally requires all of the following for every manifest-required domain:

1. an accepted #254 row for access/portable_export and/or correction, including exact field projection, shared-record disposition, exception/hold status, and adapter contract/version;
2. an independently delivered domain adapter that calls only that domain's query/application service;
3. accepted account/accountless proof plus requester-safe result/download policy; staff initiation also requires accepted #32/#33 high-risk controls;
4. an accepted workload-owned private artifact-store backend and key/lifecycle infrastructure; and
5. no unresolved policy, legal hold, exception, proof, deadline, manifest, adapter, storage, or revision block.

An unknown or missing item blocks before reading domain PII or creating an object. No adapter is inferred from installed apps or database models.

Ownership and model additions

Extend the privacy app from #255. It may depend on core, jobs, and a provider-neutral artifact-store protocol; it must not import an owning domain, Studio/API/view, AWS SDK adapter, or provider module. Domain apps register adapters toward privacy and call only their own services.

Add revisioned, UUID-keyed metadata models (exact naming may follow existing #255 conventions, but fields/states are normative):

• PrivacyExport: one per portable-export generation; request/work/operation references, public opaque export_id, pinned policy/manifest/schema versions and digests, state pending|assembling|available|expired|deleting|deleted|blocked, generation, artifact-store backend/version, opaque object-key digest (never the key in audit/result DTOs), content digest, byte/record/domain counts, created/available/expires/deleted timestamps, download-attempt count, and safe result/error code.
• PrivacyExportPart: one per required domain, unique (export, domain_key); ordinal, adapter/schema versions, manifest-entry/input/payload digests, record and uncompressed-byte counts, state pending|prepared|written|blocked, and safe result code. It stores no payload or record locator.
• PrivacyCorrectionBatch: request/work/operation references, public opaque batch ID, pinned policy/manifest versions/digests, state pending|validated|processing|partial|completed|blocked|cancelled, generation, sealed-command object digest, expected revision-set digest, domain counts, timestamps, and safe result/error code. partial is nonterminal and resumable, never reported as completed.
• PrivacyCorrectionPart: one per domain, unique (batch, domain_key); ordinal, adapter/schema version, immutable command/input digest, expected domain revision token digest, state pending|validated|running|retry_wait|succeeded|conflict|denied|blocked, attempt/generation, result/evidence digest, numeric safe counts, next-attempt timestamp, and allowlisted result/error code. It stores no field name, old/new value, free-text reason, domain record ID, or provider payload.

Database/application constraints enforce coherent timestamps, positive generations, exact terminal requirements, one active generation per request/right, immutable pinned identities, and no available export without every required export part written and a verified final digest. Reuse #255 Operation, IdempotencyRecord, AuditEvent, PrivacyDomainWork, durable jobs, revisions, and blocks; do not create parallel generic primitives.

Portable bundle format — version 1

The canonical artifact is one non-ZIP64 ZIP file named privacy-export-<opaque-export-id>.zip. The filename contains no user/domain identifier. It uses UTF-8 entry names, deflate compression, fixed permissions, fixed timestamps, no comments, no extra fields, no symlinks, no absolute/parent paths, and deterministic member order. Duplicate members, encrypted members, nested archives, macros, executables, media, HTML, CSV, spreadsheets, and arbitrary uploaded files are forbidden.

Members are exactly:

1. manifest.json; then
2. domains/000-<domain_key>.json through domains/NNN-<domain_key>.json, ordered by normalized ASCII domain_key ascending.

All JSON is UTF-8 without BOM, LF-terminated, object keys lexicographically sorted, compact separators, JSON primitives only, finite integers only (no float/NaN), and preserves the adapter-returned Unicode scalar values without lossy transliteration. Strings containing invalid Unicode/control characters are rejected. Arrays retain the owning adapter's declared semantic order; record arrays must be sorted by a stable export-local key, never a database primary key.

manifest.json requires schema_version: 1, opaque export/request references, generated/expires timestamps in UTC, policy/notice/manifest versions and digests, subject class (not subject value), rights, limits profile privacy-export-v1, and parts[]. Each part records ordinal, domain key, adapter/schema versions, member name, record count, uncompressed bytes, and lowercase SHA-256 payload digest. It contains no names, emails, domain IDs, URLs, correction values, internal object key, or error prose.

Each domain member requires schema_version: 1, domain_key, adapter_contract_version, payload_schema_version, and records[]. A record contains an export-local opaque record_ref, a bounded allowlisted record_type, and fields defined by the accepted domain schema. Relationships use export-local references and allowlisted roles only. Raw model names/PKs, internal URLs, secret fields, credentials, tokens, Slack invite/access URLs, provider payloads/identifiers, IPs, staff-only notes, abuse/security internals, and data about another subject are forbidden unless an exact accepted #254 field projection explicitly authorizes a minimized value.

The assembler validates every part against its immutable registered JSON-shape validator, scans sensitive-name/value canaries, recomputes counts/digests, writes to a bounded stream, closes the object, then re-reads/verifies metadata and the complete artifact digest before available. It never assembles the full archive in a database row, job payload, log, or system temporary directory. Any invalid part deletes/quarantines the incomplete object and blocks; no partial bundle becomes downloadable.

Shared-record and third-party minimization

The default disposition is exclusion. Every domain payload schema must classify each field/relationship as subject_owned, subject_authored_shared, third_party, public_source, or operational. Only subject_owned is exportable by default. The other classes require an exact accepted #254 projection/version; unknown classification or mixed ownership blocks that domain before payload creation.

• Peer-review/complaint/vote relations do not expose another learner's identity, contact, submission, private feedback, assignment locator, or stable cross-export pseudonym. Requester-authored material is included only when the accepted domain policy provides a minimized projection that cannot reveal the other subject; otherwise it is excluded and the manifest records only a safe omission code/count.
• A requester’s own cohort-scoped certificate state/name is domain-owned export data when the accepted adapter allows it. Another person's certificate, signer/staff identity, private verification locator, and bulk-operation metadata are excluded. The already-issued real-name exception affects retention/publication, not permission to export another person.
• Minimized course-registration shared-profile snapshots and separately registration-owned email/target/comment/notice/consent evidence follow spec 07. Later profile correction never rewrites the immutable historical snapshot; only an accepted legal privacy workflow may do so.
• Public GitHub-authored material is referenced/minimized only under its accepted public-source policy; this issue does not edit public Git history or decide publication removal.

No code may silently replace an excluded value with a hash or stable pseudonym: reversible/linkable digests remain personal data. Safe omission codes and numeric counts are the only orchestration evidence.

Limits profile privacy-export-v1

Generation fails closed with export_limit_exceeded before availability when any bound is crossed:

• at most 32 domain parts, 10,000 records per part, and 50,000 records total;
• at most 8 MiB uncompressed per part, 64 MiB uncompressed total, and 32 MiB final stored object;
• JSON nesting depth 16, 128 fields per record, 1 MiB per string, and 4 MiB aggregate strings per record;
• at most 60 seconds of adapter execution per part and 15 minutes wall-clock per generation, enforced by durable worker leases/timeouts rather than request threads;
• compression ratio at most 20:1 when verifying the final archive.

Adapters must paginate deterministically within these limits and return exact counts. There is no truncation, sampling, split multi-bundle result, or “successful with omitted tail.” A later larger-export product requires a separately groomed delivery mechanism.

Private storage, encryption, expiry, and download contract

The provider-neutral PrivacyArtifactStore protocol supports bounded streaming put, immutable finalize, verified head/read, and idempotent delete. Production registration requires a dedicated workload-owned S3 prefix/bucket with TLS, public-access block, bucket-owner enforcement, no ACLs, no public/CloudFront origin, no replication, no event payload logging, least-privilege web-read/worker-write-delete roles, and SSE-KMS using a workload-owned customer-managed key. Bucket keys may be enabled; key policy grants only the exact runtime roles and approved operators. No application-managed archive password, shared secret, public URL, email attachment, or direct provider return is allowed. Versioning is disabled for this short-lived prefix; lifecycle is a defense-in-depth abort/delete rule, not the primary expiry mechanism.

The object key is a random opaque value under a fixed private prefix and never contains user/request/domain data. The database stores the protected locator needed by the store plus its digest; logs/audits/results expose only the digest. Local/unit tests use an in-memory or project-local .tmp/ fake; no production backend or credential is added by this slice. Terraform/provider work is a separate prerequisite and must prove encryption/public-denial/lifecycle policies without creating a real export.

Version-1 availability is 24 hours from verified finalize and permits at most three authorized stream starts. Expiry is authoritative in the application even if cleanup is late. The later download adapter must revalidate current subject proof, request ownership, request/export state and revision, policy/manifest identity, and active blocks on every attempt; it streams through Django over an authenticated private/no-store/noindex/referrer-safe response, never redirects to a presigned URL, never accepts a caller-supplied object key, and emits application/zip plus RFC-safe attachment disposition and X-Content-Type-Options: nosniff. Range requests are rejected. Cross-object, expired, replay-exhausted, missing-object, proof, and authorization failures are non-enumerating.

This backend slice implements/test-drives the download authorization/service DTO but creates no URL/view/template/API. Download count is consumed atomically at stream authorization, so an interrupted attempt counts; exact idempotent replay of the same start command returns the same bounded result without consuming twice. Regeneration is a new explicitly authorized generation, never resurrection of an expired object.

At expiry or earlier request invalidation, a durable cleanup moves available|expired|blocked to deleting, performs idempotent delete, verifies absence, then records deleted; downloads deny from expires_at regardless of object state. Cleanup begins at expiry, retries with fenced leases, and must delete bytes within 24 hours of expiry under the version-1 operational target. Persistent/unknown delete or head failure remains blocked and alerts with opaque IDs/codes only; it is never recorded as deleted. Incomplete uploads are aborted/deleted. After verified deletion, only bounded counts/digests/state/audit evidence may remain under the accepted #254 audit policy. If that retention policy is unresolved, production activation remains blocked. Export bytes and sealed correction objects are excluded from database/RDS backups and ordinary verification artifacts.

Correction command and orchestration contract

A correction batch is a manifest-pinned set of domain commands ordered by domain_key, then adapter-defined stable export-local record/field key. Each domain adapter declares immutable CorrectionFieldDefinitions: public field key, value schema and size, mutability (mutable|historical_immutable|shared|derived|public_source), required capability, owning service command/version, revision contract, and accepted #254 policy/projection. Duplicate or conflicting definitions fail startup. Unknown, derived, historical, shared, public-source, cross-object, cross-domain, or exception/hold fields fail closed unless an exact accepted policy and owning command authorize that correction.

The input uses export-local record references resolved only inside the owning adapter; callers can never supply model labels, database IDs, object-store keys, or service names. Each command requires expected revision/If-Match, canonical old-value digest where supported, new typed value, bounded allowlisted reason code (no free text), and an idempotency key. Before any durable work is created, orchestration validates proof/authorization, exact request/right/manifest/adapter versions, every field definition, object ownership, expected revision snapshot, holds/exceptions, and all limits. Validation returns only non-enumerating safe codes.

Because correction values may be PII and resume must survive a crash, the canonical per-domain command set is stored only as a short-lived sealed object through the same private artifact store, encrypted under the export KMS boundary. Database/job/audit/operation rows retain only locator/content/revision-set digests and counts. The sealed command object expires at the earlier of batch completion plus one hour or 24 hours after creation; correction blocks if safe cleanup cannot be guaranteed. No command value appears in an idempotency result.

Each domain part calls only its owning application service. That service rechecks subject/object/field authorization and expected revision in its own transaction and uses (correction_part UUID, generation, command digest) as the stable mutation identity. Success atomically commits the domain mutation and domain audit. After an ambiguous crash, reconcile must establish the exact committed result before any retry; blind repeat is forbidden. Permanent denial/conflict is not retried. Explicit transient safe codes use #255 durable retry bounds.

There is no impossible cross-domain transaction or compensating rewrite. Preflight prevents known invalid batches before mutation; a concurrent change or runtime failure may still leave some domain parts succeeded. Overall state becomes nonterminal partial/blocked, exposes safe per-domain codes only, and resumes only failed/unexecuted parts after proof, policy, manifest, authorization, revisions, and sealed-command digest are revalidated. Succeeded parts are immutable and reconciled, never rerun or rolled back. Completion requires every required part succeeded and verified command-object cleanup scheduled. Exact replay returns the prior bounded DTO; changed fields/value/revision/reason/action with the same key conflicts.

Audit, observability, and failure behavior

Every create/preflight/start/part prepare/write/finalize/authorize-download/expire/delete, correction validate/execute/reconcile/retry/conflict/denial/resume, stale revision, idempotency conflict, and terminal outcome uses the shared audit writer. Audits/logs/metrics/jobs/operations contain only opaque request/export/batch/work IDs, safe actor class, domain key, policy/manifest/adapter/schema versions, generations/revisions, numeric counts, digests, and allowlisted result/reason codes. Metrics are aggregate counts/bytes/age/status only.

Reject rather than redact-and-accept unknown JSON keys/types, free-form strings in evidence, sensitive field names, email/URL/token/credential/provider shapes in orchestration metadata, noncanonical data, excessive depth/size, digest mismatch, storage/backend drift, KMS/access denial, stale lease/revision, or unavailable proof. Safe storage unavailable/denied/throttled/timeout/missing/ambiguous responses never produce available or deleted. No exception includes an object key, content fragment, correction value, storage URL, provider response, or another subject's existence.

Scope — backend contract slice

• Add the metadata models/constraints/migrations, DTOs, versioned export/correction adapter extensions, schema registries/system checks, deterministic bounded archive assembler/validator, artifact-store protocol, services, durable handlers, cleanup, and authorization seam described above.
• Reuse #255 request/domain work, operation, idempotency, audit, proof, block, retry/reconcile, and revision primitives.
• Register only explicit test adapters and test store; add internal architecture/security documentation for this contract.
• Add synthetic model/service/protocol/archive/storage/concurrency/migration/security tests.

Explicit non-goals

• No production account, course, event, email/Relay, content/GitHub, audit/log, search/analytics, processor, or provider adapter; no #251 absorption and no cross-domain model import.
• No real protected data, requester action, export, correction, object, bucket, KMS key, credential, provider/AWS mutation, or production execution.
• No Studio/admin API/self/accountless/public API/management command, URL, view, template, navigation, email, attachment, notification, OpenAPI, or browser behavior. Child 7 owns requester/staff presentation and #32/#33 gates.
• No retention runner, erasure/anonymization, tombstone/restore fence, projection/cache/search invalidation, processor deletion/receipt, publication removal, or public Git-history edit.
• No legal rule, lawful basis, shared-record permission, exception/hold release, appeal/deadline outcome, requester-evidence wording, or post-erasure proof retention invention. Unknowns block.
• No CSV/spreadsheet/PDF/HTML/media/native file export, multi-bundle pagination, public/presigned URL, bucket provisioning, application-managed encryption key, or backup inclusion.

Acceptance criteria — backend contract slice

• Exact export/part and correction-batch/part metadata, constraints, revisions, states, timestamps, digests, counts, and terminal invariants exist and pass SQLite plus production-like PostgreSQL checks.
• privacy-export-v1 bundles are byte-deterministic for identical canonical input, have exactly the allowed members/order/JSON contract, validate all digests/counts/schemas, preserve valid Unicode, and reject traversal, duplicate/nested/ZIP64/executable/invalid-Unicode/float/control/oversize/compression-bomb content.
• Empty, one, and many test domains produce the exact manifest order; a missing/extra/unknown/version-drifted adapter or field policy, shared/third-party/unsafe value, truncation, or any limit breach blocks and leaves no available partial object.
• Artifact-store tests prove bounded streaming, immutable finalize, full digest verification, private locator handling, KMS/access/missing/throttled/timeout/ambiguous failure behavior, incomplete-upload cleanup, expiry, three-attempt fencing, regeneration isolation, idempotent deletion, absence verification, and persistent-delete blocking.
• Production system checks fail closed when the private store/key/lifecycle proof or any production adapter is absent/mismatched; the slice registers only test implementations and performs no network/provider call.
• Correction definition/command validation proves allowlisted typed fields, ownership, policy, reason, expected revision, old-value digest, and service command; immutable/derived/shared/public-source/cross-object/unknown fields deny with no mutation or enumeration.
• Exact replay and concurrent duplicate export/correction commands create one generation/object/work set; changed payload/revision/action conflicts; stale lease/revision cannot finalize/download/delete or commit a correction.
• Correction tests cover zero/one/many domains, atomic preflight, one-domain transactional success, concurrent stale change, partial cross-domain success, permanent denial/conflict, transient retry, exhaustion, crash before/after domain commit, reconcile-before-retry, resume of only pending/failed parts, and no compensation/rerun of success.
• Sealed correction commands are encrypted/private, digest-bound, short-lived, deleted on completion/expiry, absent from database/job/audit/idempotency/log/exception/verification outputs, and unavailable after cleanup.
• Audit/log/metric/operation/job/artifact metadata canaries prove no PII, another subject's data, raw domain IDs, object keys/URLs, Slack secret/URL, credential/token, provider payload, archive content, correction field/value, or free-text error leaks.
• Migrations pass forward/reverse/fresh/apply-twice on SQLite and PostgreSQL; focused privacy/core/jobs/security tests, format/lint/type/container checks, and graph-selected versioned verification pass using uv/Make.
• Render contract is exactly no URL/template/static/navigation/OpenAPI/response change. Independent tester recomputes the graph and may mark Playwright/screenshots not_applicable only if that remains true.

Acceptance criteria — production closure

• Every accepted #254 production domain has an independently delivered adapter with exact export schema/field ownership/shared-record/correction definitions, owning-service tests, and no missing required domain.
• Private S3/KMS/lifecycle/role infrastructure is independently accepted and proves public/CloudFront/presigned denial, encryption, least privilege, no replication/version residue, cleanup alarms, and backup/evidence exclusion without real export content.
• Accepted account and accountless proof/requester-result rules, applicable #32/#33 high-risk staff controls, and child-7 download/status surfaces are delivered; download remains private/no-store/noindex/non-enumerating and authorization is rechecked.
• [HUMAN] Authorized privacy/legal owner accepts every shared/third-party/public-source projection, exception/hold interaction, requester-safe omission/result wording, artifact/metadata retention, download/appeal/escalation rule, and the exact register/manifest version. Silence or a source observation is not approval.
• A fully synthetic production-like rehearsal covers every enabled domain plus expiry/deletion/storage denial/correction conflict/retry/restore boundary and proves no third-party/secret/PII leakage in operational evidence.
• If a later requester or staff surface changes rendering, independent desktop/mobile Playwright and inspected screenshots cover available/partial/blocked/expired/replay-exhausted/download/correction-conflict states, keyboard/focus/status/error behavior, 200% reflow, headers, no overflow, and no error/debug page.

Required verification scenarios

1. Canonical empty/one/many-domain bundles; order/digest reproducibility; Unicode and boundary sizes; every forbidden archive/JSON shape; formula-like strings remain inert JSON strings and no tabular member exists.
2. Subject-owned versus subject-authored-shared/third-party/public/operational fields; peer review, vote, complaint, certificate, minimized registration snapshot, consent evidence, public-source, secret/provider and stable-pseudonym canaries.
3. Store success, denial, missing, throttle, timeout, ambiguous finalize/delete/head, corrupted readback, key/backend drift, expiry race, three simultaneous download starts, proof revocation, cross-object request, interrupted stream, cleanup retry, and regeneration.
4. Correction mutable/immutable/shared/derived/unknown fields, wrong subject/object, wrong expected revision/old digest, concurrent writer, duplicate changed/same idempotency key, partial domain failure, ambiguous commit/reconcile, retry exhaustion, resume, sealed-command expiry/deletion, and audit canaries.
5. Migration/constraint introspection and forward/reverse/fresh/apply-twice on both databases; crash/rollback injection proves metadata/job/audit/idempotency/object state cannot claim unavailable work as complete.

Browser and screenshot disposition

Expected for the backend slice: not_applicable. The engineer records pending_independent_tester until the independent tester recomputes the final graph. Any route, response, OpenAPI, template, or static impact restores the graph-selected browser tier and independent desktop/mobile screenshots; UI work is otherwise deferred to child 7.

Delivery convention

After independent tester pass and PM acceptance of the backend contract slice, commit with Refs #256 and leave #256 open. Do not register production adapters/store or enable real execution. Only after every production-closure criterion and normal lifecycle gate passes may the final focused commit use Closes #256.

Start with _docs/PROCESS.md, the listed privacy architecture and verification documents, and prerequisites #254, #281, #255, and #285; engineering is explicitly blocked until those are accepted, merged, and green. Done first means grooming and rewriting this issue against the accepted contracts, with production adapters, artifact storage, policy, and requester or Studio surfaces remaining later gates.