DataTalksClub / DataTalksClub/website

Define the non-activating privacy export and correction input contract

Open
#285 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation enhancement foundation integration operations P0 security testing
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Parent epic: #64
Authority-register prerequisite: #254
Kernel-input prerequisite: #281
Blocked consumer: #256

Product outcome

Publish one immutable, machine-validated, non-activating export/correction input contract between the accepted privacy authority/kernel contracts and #256. It defines complete per-domain source-field coverage, privacy ownership/classification and minimization dispositions, deterministic portable-payload schemas and identities, bounded omission/result/reason vocabularies, and correction mutability/owning-service/revision definition slots.

The artifact never grants a production export or correction. Every production projection, payload, correction, service, and runtime-registration allowance is false; unresolved policy and unimplemented domain-service contracts stay explicit blockers. A sealed in-memory synthetic test seam supplies later #256 tests with one valid enabled shape without creating a production activation or discovery path.

Why this is a distinct prerequisite

#254 owns legal/source authority and source-flow inventory. #281 owns policy/domain identity, the six rights, the six candidate adapter domains, their exhaustive right/domain matrix, and shared block/exception codes. Neither artifact defines which source fields may enter a portable payload, how shared/third-party/public/operational fields are minimized or omitted, a payload schema identity, or which field may be corrected through which owning service and revision contract.

#255 owns the runtime request aggregate and adapter protocol; it must not invent those domain definitions. #256 currently assumes all of them and was correctly returned to grooming in its PM dependency audit. Adding them to #254 or #281 would broaden accepted ownership and evidence envelopes. This issue supplies the missing source-only interface and does not modify those artifacts.

Normative authority

  • _docs/PROCESS.md: raw intake, PM grooming, role separation, immutable verification envelopes, source-only screenshot disposition, and sensitive-artifact rules.
  • _docs/specs/01-platform-architecture.md, _docs/architecture/app-boundaries.md, and _docs/architecture/shared-primitives.md: domain ownership, application-service mutation boundaries, immutable identities, revision/idempotency conventions, and one-way imports.
  • _docs/specs/04-courses-and-cohorts.md: deliberately minimized immutable registration snapshots, separately owned registration evidence, peer/shared records, certificates, and no silent historical rewrite.
  • _docs/specs/05-events-registration-email.md: event/accountless data ownership, consent/evidence, export safety, and Relay/legacy-email boundaries.
  • _docs/specs/06-studio-and-admin-api.md: owner/capability/field authorization, expected revision, idempotency, export/correction audit, and no bulk member-profile export.
  • _docs/specs/07-security-privacy-operations.md: exact account-export inclusions, Slack-secret exclusion, correction through the accounts service, third-party minimization, private/no-store artifacts, redaction, and fail-closed privacy behavior.
  • _docs/specs/10-verification-strategy.md: privacy contract, schema, authorization, correction/export, drift, redaction, and source-only evidence requirements.
  • Closed #23 and parent #64: accepted facts, full-erasure direction, narrow issued-certificate exception, adapter separation, and unresolved HUMAN/legal gates.
  • Accepted source-only #254: sole authority/source-flow identity. Accepted #281: sole kernel policy/domain/right/matrix and shared block/exception identity.
  • #255 and #256 are downstream consumers. Their prose and frozen candidates are not authority for this contract.

Dependency gate

Depends on independently tested, PM-accepted, merged, and green-on-call source-only deliveries of #254 and #281. Full HUMAN/legal closure of #254 is not required for a disabled structural artifact, but the exact accepted register/schema/version/digest/inventory commit/flow/gate identities and exact #281 contract/policy/domain-manifest/matrix identities are required.

Engineering must not copy a provisional candidate digest, infer from issue prose, or use a mutable branch/tag. Any accepted #254/#281 identity, source-field reference, flow classification, right disposition, domain mapping, HUMAN gate, or shared-code change invalidates this artifact and requires a new contract version and fresh verification.

#255 and this issue may proceed independently after #281 is green: neither supplies an interface required by the other. #256 depends on both accepted #255 runtime primitives and this accepted source contract. Domain adapters and HUMAN/legal activation remain later prerequisites.

Exact owned files

This issue owns exactly six new source files:

  • _docs/legal/privacy-export-correction-input-contract.json — sole hand-edited technical contract;
  • _docs/legal/privacy-export-correction-input-contract.schema.json — JSON Schema Draft 2020-12 structure;
  • scripts/privacy_export_correction_input_contract.py — canonicalization, frozen DTOs, and the sole non-activating source loader;
  • scripts/validate_privacy_export_correction_input_contract.py — read-only semantic/cross-contract validator;
  • scripts/tests/privacy_export_correction_input_contract_test_support.py — sealed in-memory synthetic enablement helper;
  • scripts/tests/test_privacy_export_correction_input_contract.py — identity, coverage, payload, correction, seal, drift, reachability, and redaction tests.

It modifies or shadows no #254/#281 artifact. There is no generated Markdown/CSV/YAML copy, Django setting, application constant, database seed/migration/fixture, runtime registry/plugin, environment/path override, mutable latest alias, production adapter, or duplicate legal/policy authority.

Canonical identity

The canonical JSON has only these top-level sections:

  • schema_version: integer 1;
  • contract_version and contract_digest;
  • authority_register;
  • kernel_input_contract;
  • activation;
  • right_domain_coverage;
  • field_projection_catalog;
  • payload_schema_catalog;
  • correction_definition_catalog;
  • omission_catalog;
  • result_catalog;
  • reason_catalog;
  • synthetic_fixture_envelope.

Versions are bounded immutable ISO-date-plus-positive-revision identifiers. Digests are lowercase SHA-256 over UTF-8 NFC canonical JSON with sorted object keys, compact separators, preserved Unicode, integers only, no BOM, and the accepted #254 self-digest omission rule. The contract digest excludes only contract_digest; each projection, payload-schema, correction-definition, and synthetic-envelope digest excludes only its own digest, while the parent includes every child digest. Ordered identifier arrays use one validated canonical order.

Duplicate JSON keys or semantic identities, unknown fields, noncanonical bytes/Unicode/order, floats/NaN, mutable aliases, a reused version with changed bytes, or any digest disagreement fails before a DTO is returned.

authority_register pins exact accepted #254 path/schema/register version/digest/inventory commit, used flow IDs, source-reference freshness identity, and applicable HUMAN gates. kernel_input_contract pins exact accepted #281 path/schema/contract/policy/domain-manifest versions/digests, the portable_export and correction right identities, six candidate domains, applicable 12 matrix rows, and referenced shared block/exception codes. The loader reads both local canonical artifacts and rejects drift.

Exact non-activation contract

activation is fixed for schema version 1:

  • state: non_activating;
  • production_export_allowed: false;
  • production_correction_allowed: false;
  • production_payload_registration_allowed: false;
  • production_service_registration_allowed: false;
  • runtime_loading_allowed: false;
  • synthetic_test_enablement_allowed: true only through the explicit test-support helper and its marker type;
  • successor_contract_required_for_activation: true;
  • blocking_human_gate_ids: the complete applicable #254/#281 blocker union;
  • superseded_by: null.

Every canonical projection/schema/correction row sets its production allowance false and state to disabled_non_activating, disabled_human_required, disabled_missing_domain_contract, or an exact negative/no-work state. No production row may contain a callable/import path, non-null adapter/command/revision contract version, object/store identity, subject value, or executable default. Settings, environment, registry discovery, installed apps, database state, fixture discovery, or fallback cannot alter this state.

Complete right/domain and source-field coverage

right_domain_coverage has exactly 12 rows: the Cartesian product of portable_export|correction and the six #281 candidate domains accounts|courses|events|legacy_datamailer|content_publication|operations. Each row pins the exact #281 matrix row, contributing #254 flows, source inventory states, HUMAN/future-issue blockers, and one state:

  • disabled_definition_present;
  • disabled_human_required;
  • disabled_missing_domain_contract;
  • not_applicable_authority_backed.

All set production_request_allowed: false. A not-applicable row requires an exact #254 negative assertion and accountable owner; it is not a runtime skip or successful empty result. Missing, duplicate, extra, regrouped, or more-permissive coverage fails. Absent Relay, inactive analytics, transient country suggestion, aggregate-only records, public-source workflows, and legacy Datamailer remain distinguishable and cannot be inferred into or out of a domain.

field_projection_catalog covers every source field/data member referenced by the contributing accepted #254 flows exactly once for its owning domain, including explicit negative/absent fields. Each stable row contains:

  • bounded public projection_key, positive row version/digest, domain and exact #254 source references;
  • source state and one privacy classification: subject_owned, subject_authored_shared, third_party, public_source, operational, or secret_credential;
  • one ownership state: verified_subject, mixed_or_shared, not_subject, not_persistent, or unresolved;
  • one minimization disposition: candidate_direct, candidate_minimized, omit, human_required, or not_applicable;
  • exact applicable authority/gate IDs, omission code, future domain-adapter issue, and payload/correction definition references;
  • production_projection_allowed: false.

Classification may record source-observable ownership structure; it may not turn that observation into legal permission. candidate_direct and candidate_minimized describe a schema shape only and remain disabled. Mixed/unknown ownership blocks unless an accepted authority supplies a specific minimized projection. A hash or stable pseudonym is never substituted for omitted personal data.

The validator scans current source metadata only—never rows or production data—and rejects a stale/missing source field, unclassified new source field, duplicate ownership, classification elevation, hidden grouping, missing negative row, or cross-domain ownership. Exact schema field names/source symbols are allowed only in the bounded source-reference and definition-key positions; examples, defaults, production values, and record identifiers are forbidden.

Portable payload schema identities

payload_schema_catalog contains exactly one immutable schema definition for each of the six domains, even when its state is disabled or authority-backed no-work. Each definition has a stable schema ID, positive schema version, schema digest, exact projection-row set, canonical member ordering, structural JSON shape, record/reference rules, limits inherited by identity from the contract envelope, and blocker set.

The portable domain shape is declarative only: schema_version, domain_key, adapter_contract_version, and ordered records. Each record uses an export-local opaque record_ref, bounded allowlisted record_type, and fields named only by catalogued projection keys. Relationships use export-local references plus allowlisted roles. Raw model labels/PKs, stable cross-export pseudonyms, internal/provider identifiers, URLs, credentials/tokens/secrets, staff/security internals, another subject's data, free-form schema extensions, arbitrary binaries/files, executable/tabular content, and undeclared fields are unrepresentable.

Each field shape pins JSON type, nullability, scalar/string/array/object bounds, Unicode/control handling, semantic ordering, classification, minimization disposition, and omission behavior. It contains no example/default/production value. Schema identity remains non-null and digest-bound even when disabled; adapter contract version remains null. A payload schema is not permission to read a source or produce bytes.

Canonicalization is deterministic: UTF-8 NFC, sorted object keys, integers only, finite bounded depth/items/bytes, no unknown members, and exact declared array ordering. The validator rejects schema/digest/version drift, projection mismatch, unsafe type/shape, implicit additional properties, payload examples, unbounded values, or a schema capable of representing forbidden metadata.

Omission contract

The omission catalog is exact and non-dispositive:

  1. not_subject_owned
  2. third_party_excluded
  3. shared_policy_unresolved
  4. public_source_separate_workflow
  5. operational_metadata_excluded
  6. secret_or_credential_excluded
  7. inactive_or_absent_source
  8. field_policy_unresolved
  9. unsupported_payload_kind

Each code declares permitted classifications, whether a numeric count may be emitted, authority references, and production_completion_allowed: false. Omission evidence is only code plus nonnegative count. It never contains a field value, subject/domain record ID, stable hash, free-form explanation, or permission to treat a required domain as complete. Unknown/custom/free-form codes fail.

Correction definitions

correction_definition_catalog covers every projection row exactly once. Each definition pins domain, projection key/version/digest, classification/minimization identity, and one mutability state:

  • mutable_candidate;
  • historical_immutable;
  • shared;
  • derived;
  • public_source;
  • operational;
  • secret_credential;
  • human_required;
  • not_applicable.

Every row also contains nullable slots for owning application-service key, public command key, command contract version/digest, typed input-schema identity, required capability, expected-revision mode/version, old-value-digest mode, result-evidence schema identity, and future owning issue. In version 1, all executable service/command/revision versions and digests are null, registration state is blocked_unimplemented, and production_correction_allowed is false. Stable routing keys and accountable app names are metadata, never callable/import paths.

A mutable_candidate records only source-observed editability; it is not correction authority. Historical immutable, shared, derived, public-source, operational, secret/credential, unknown, cross-object, or cross-domain rows cannot be elevated. Immutable course-registration snapshots stay immutable; later profile correction cannot silently rewrite them. Public-source correction/removal remains its separately accepted source workflow. Every correction requires a future owning-service contract plus exact expected-revision semantics; no direct model write or accounts-to-domain import can be represented.

Bounded result and reason catalogs

The result catalog contains exactly:

  • definition_loaded;
  • definition_blocked;
  • synthetic_projection_valid;
  • synthetic_projection_rejected;
  • synthetic_correction_valid;
  • synthetic_correction_rejected.

Only definition_loaded|definition_blocked can be returned by the canonical loader. Every synthetic_* code is test-only. All codes set production_terminal_success_allowed: false; none means a requester export/correction succeeded.

The contract references applicable #281 shared block codes by exact identity rather than redefining them. Its export/correction-specific reason catalog is exactly:

  1. coverage_mismatch
  2. projection_definition_missing
  3. projection_classification_unknown
  4. minimization_disposition_missing
  5. payload_schema_missing
  6. payload_schema_mismatch
  7. payload_shape_invalid
  8. omission_contract_mismatch
  9. correction_field_unknown
  10. correction_mutability_denied
  11. correction_service_missing
  12. correction_service_mismatch
  13. correction_revision_missing
  14. correction_revision_mismatch
  15. synthetic_fixture_invalid
  16. unsafe_metadata
  17. definition_drift

Reasons are bounded diagnostic categories only. They do not decide a right, approve minimization, release a hold/exception, authorize a service, resolve an ambiguous mutation, or activate production. Unknown/custom/free-form result or reason codes fail.

Fail-closed source loader

scripts/privacy_export_correction_input_contract.py exposes load_non_activating_export_correction_contract() with no path, settings, environment, profile, allow_enabled, fixture, registry, or override argument. It resolves only the canonical repository path; validates byte/size/depth/duplicate-key/Unicode/schema/digests/order first; then validates exact #254/#281 identities, coverage, source references, cross-links, catalogs, non-activation, and redaction; and returns frozen non-runnable DTOs only after every check passes.

The loader imports no Django/application/provider module and performs no settings setup, database/network/provider access, discovery, cache across source changes, write, or mutation. Failures expose only bounded path/identity/reason/count/digest metadata and never echo rejected input. There is no warning-only mode, partial DTO, unknown-field preservation, last-known-good fallback, mutable cache, or skip_validation.

The validation module may accept an in-memory document for mutation tests, but it cannot return a runnable/active definition. Production/application modules may not import either script or load either canonical/test artifact; static import/dependency/system-check tests prove zero startup, settings, URL, command, job, registry, model, migration, provider, database, or render reachability.

Sealed synthetic test-only seam

scripts/tests/privacy_export_correction_input_contract_test_support.py owns one helper, build_synthetic_enabled_export_correction_contract(). Nothing outside scripts/tests/ may import or discover it.

The helper first requires a fully validated canonical non-activating DTO, then creates one in-memory synthetic_test_only domain with reserved projection keys covering subject-owned direct/minimized, shared/third-party omission, immutable, derived, and mutable correction cases. It supplies bounded non-production payload/command/revision schema identities and values, exact test-only versions/digests, and an unforgeable-in-production marker type owned by that test module.

It rejects canonical domain/flow/field/authority IDs, source references, production-like identifiers or values, environment/settings input, filesystem paths, serialization as the canonical schema, registry handoff, or a caller-selected production allowance. It performs no file write, Django setup, database/network/provider work, source mutation, or real correction/export. The canonical loader rejects the enabled object and every serialized enabled mutation. #256 must define and test its own runtime injection seam later; this fixture proves the source shapes only.

Redaction and drift boundaries

The schema/validator/test helper accept only schema metadata and synthetic canaries. No real/reversible subject value, email/name/username, IP, provider/domain record ID, cursor/object key, URL/endpoint, credential/token/cookie/session, Slack secret, message/content body, raw SQL, production count, free-form reason/error, or protected export/correction value may appear in the canonical artifact, fixture object, exceptions, output, logs, reports, or evidence.

Exact source field/symbol names and approved privacy-contact authority references are allowed only in schema-designated metadata positions and can never carry a value. Sensitive-looking runtime values at every other string/object boundary fail before DTO construction and are not echoed. Cross-contract validators reject #254/#281 version/digest/source/flow/gate/right/domain/matrix/code drift, source-reference freshness drift, and newly uncovered fields.

Scope

  • Add only the exact six source-only files.
  • Bind exact immutable contract/child identities to accepted #254/#281 artifacts.
  • Cover all 12 portable-export/correction domain pairs and every contributing source field/member exactly once.
  • Publish disabled projection/classification/minimization, deterministic payload schema, omission, correction-definition, and bounded result/reason contracts.
  • Provide one fail-closed non-runnable loader and one sealed in-memory synthetic test seam.
  • Prove canonical identity, cross-contract/source drift, redaction, and zero runtime/render reachability.
  • Supply the stable source artifact required to re-groom #256; do not implement #255 or #256.

Explicit non-goals

  • No change to #254/#281 artifacts, candidates, reports, schemas, generated inventory, HUMAN gates, legal pages, or accepted digests.
  • No legal/privacy decision, notice/lawful-basis wording, shared/third-party permission, correction entitlement, hold/exception release, deadline/appeal/escalation rule, requester proof, artifact retention, or production activation.
  • No Django app/model/migration/service/registry/setting/URL/view/template/static/OpenAPI/Studio/admin/command/job/scheduler/domain adapter/artifact store/provider/runtime code.
  • No production/test database, protected data, requester, AWS/S3/KMS, GitHub mutation, Google/Slack/Datamailer/Relay/provider call, export bytes, correction command/value, object, receipt, or domain mutation.
  • No fake production adapter/service/revision contract, callable path, production-like fixture, guessed schema permission, wildcard/default mapping, or workaround around #255, domain-adapter issues, #32/#33, #49, #66, or HUMAN gates.
  • No retention runner, erasure/tombstone/restore, propagation/processor receipt, cache/search invalidation, public-source edit, browser/render change, commit, push, merge, deployment, or external operational action outside the normal lifecycle.

Acceptance criteria

  • The exact six files exist with one hand-edited contract and no shadow authority, manifest, schema copy, runtime registry, alternate loader, or discoverable fixture.
  • Contract and every child version/digest are canonical, deterministic, immutable, and pinned to exact accepted/merged/green #254/#281 identities and current source-reference freshness.
  • Activation is exactly non-activating: every production export/correction/payload/service/runtime allowance is false, executable contract slots are null, and no settings/environment/path/database/registry/default can change that.
  • The exact 12 right/domain rows preserve #281 dispositions and #254 flow/negative coverage without elevation, regrouping, wildcard, inference, or runtime skip.
  • Every contributing source field/member is classified once with exact ownership/minimization/authority/omission/payload/correction references; missing/new/duplicate/cross-domain/elevated rows fail.
  • Each of six domains has one deterministic payload schema identity/digest whose shape can represent only catalogued bounded export-local data, never raw IDs, secrets, third-party data, unsafe files, examples/defaults, or extensions.
  • The exact nine omission codes carry only safe counts, match allowed classifications, and cannot complete a required domain or conceal missing coverage.
  • Every projection has one correction definition with exact mutability and null/blocked service/command/revision slots; no direct model write, callable path, historical/shared/derived/public/secret elevation, or production correction can be represented.
  • The exact six results, 17 specific reasons, and referenced #281 shared blocks are complete, bounded, non-dispositive, and reject unknown/free-form/universal-success semantics.
  • The canonical loader has no override/activation seam, returns frozen non-runnable DTOs only, fails before partial load on malformed/tampered/stale/unsafe input, and never echoes a sensitive canary.
  • The sealed helper creates only its reserved in-memory synthetic domain and marker; canonical IDs, source refs, production-like data, serialization, non-test import/discovery, and canonical-loader acceptance all fail.
  • Static dependency/import/system-check tests prove zero application/runtime/render reachability and no model, migration, URL, command, job, registry, settings, provider, database, or page change.
  • Focused validators/tests, formatting/lint/type checks, migration drift, repository quality, and graph-selected versioned verification pass using uv/Make.
  • Independent tester proves the frozen graph is source-only and records Playwright/screenshots not_applicable; any runtime/render path is a scope failure and returns the candidate to engineering.
  • Engineer/tester/PM reports record exact base/head, upstream/contract/child/fixture digests, 12-row and field-coverage counts, changed paths, graph/plan/artifact digests, commands/counts, and no pending/skipped required component.

Required verification scenarios

  1. Each #254/#281 path/version/digest/source commit/flow/gate/right/domain/matrix/code changed, absent, provisional, reordered, or stale; load fails before DTO construction.
  2. Contract/child self-digest tamper, duplicate/unknown key, noncanonical order/Unicode/bytes, malformed version/digest, float/control/depth/item/byte boundary, or version reused with changed semantics.
  3. Every missing/duplicate/extra/reclassified right/domain/flow/source-field/projection/payload/correction/omission/result/reason/cross-reference; no wildcard, grouping, inference, or fallback succeeds.
  4. Active/enabled/production-allowed/runtime-load flag, non-null executable adapter/service/revision slot, removed HUMAN blocker, elevated shared/third-party/HUMAN row, hidden direct-write/callable path, or schema accepting undeclared/raw/secret/other-subject data.
  5. Payload empty/one/many records and declared types/order/Unicode/bounds versus raw PK/model/provider IDs, stable hash, unsafe nested object/file/archive/tabular/executable content, default/example/additional property, and count/digest mismatch.
  6. Correction mutable/immutable/shared/derived/public/operational/secret/unknown fields; missing/wrong owner/service/command/revision/old-value definition; cross-domain/object reference; no canonical case becomes executable.
  7. Synthetic helper with canonical IDs/source refs, production-like values, invalid bounds/digests, environment/path input, serialization, non-test import/discovery, or canonical-loader handoff; every boundary fails without echo.
  8. Email/name/IP/URL/token/cookie/session/Slack/provider/domain-ID/object-key/SQL/free-text/production-value canaries at every document, schema, loader, helper, exception, output, log, report, and artifact boundary.
  9. Source scan/import graph/system checks prove application code cannot discover either loader/artifact/helper and that no model, migration, URL, command, job, settings, provider, or rendered route changed.

Required verification commands

At minimum, plus the generated change-selective plan:

uv run --frozen python scripts/validate_privacy_authority_register.py
uv run --frozen python scripts/validate_privacy_kernel_input_contract.py
uv run --frozen python scripts/validate_privacy_export_correction_input_contract.py
uv run --frozen python -m unittest scripts.tests.test_privacy_authority_register scripts.tests.test_privacy_kernel_input_contract scripts.tests.test_privacy_export_correction_input_contract
make migrations-check
make verification-quality

The independent tester recomputes the plan and reruns focused validation/tests. Playwright/screenshots are not_applicable only after the tester independently proves zero runtime/render impact.

Delivery and downstream order

After #254 → #281 are merged and green, this issue follows engineer → independent tester → PM acceptance → focused commit with Closes #285 → local no-ff merge/push → on-call green. #255 may be re-groomed/delivered in parallel after #281. #256 remains needs grooming and may be rewritten only after both #255 and #285 are accepted, merged, and green, against their exact paths/APIs/versions/digests. Production domain adapters, private artifact storage, #32/#33 surfaces, and HUMAN/legal decisions remain later closure gates; this issue activates none of them.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with _docs/PROCESS.md and the accepted #254 and #281 artifacts, then review the six owned files listed in the issue. Run scripts/tests/test_privacy_export_correction_input_contract.py and the read-only validator to understand the required identity, coverage, non-activation, drift, and redaction checks. Done means the canonical contract and schema validate against both prerequisites without enabling production export, correction, registration, or runtime loading.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.