DataTalksClub / DataTalksClub/website

Define the non-activating privacy propagation input contract

Open
#282 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation email enhancement foundation integration operations P0 security testing
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Parent epic: #64
Upstream contract prerequisites: #254, #281, #255, #256, #257, #258
Blocked runtime consumer: #259

Product outcome

Publish one immutable, machine-validated non-activating privacy-propagation input contract between the accepted privacy authority/kernel/export/retention/restore contracts and #259. It gives #259 one canonical identity for complete destination coverage, exact source-action mappings, bounded projection inputs, empty production adapter/verifier slots, safe outcome/reason vocabularies, bounds/review state, and synthetic-only test fixtures.

Version 1 is deliberately incapable of production execution. It does not decide whether a privacy action is legally required, declare any processor confirmation sufficient, enable a destination, register an adapter/verifier, load from runtime settings, or process a real subject. It exists so later #259 engineering can consume an accepted interface rather than invent policy and authority while building models and jobs.

Why this is a distinct prerequisite

#254 owns the sole privacy authority register and observed/planned source flows. #281 owns the non-activating kernel policy/domain manifest, right/domain matrix, and block/exception catalogs. Neither owns propagation destination definitions, source-action projections, adapter/verifier slots, retry/reconciliation bounds, or receipt/result vocabularies. #255–#258 will own the runtime source aggregates and action identities that propagation consumes.

Putting this contract into #259 would let runtime engineering create its own authority and would require changing already accepted upstream evidence envelopes. Putting it into #281 would expand that issue's exact four-file boundary and confuse kernel routing with destination propagation. This issue is therefore source-only and separately delivered before #259 is re-groomed.

Normative authority

  • _docs/PROCESS.md: raw intake, PM grooming, immutable evidence, role separation, and source-only screenshot disposition.
  • _docs/specs/01-platform-architecture.md, _docs/architecture/app-boundaries.md, and _docs/architecture/shared-primitives.md: one deployment, owning-domain services, durable after-commit side effects, idempotency, audit, and redaction.
  • _docs/specs/05-events-registration-email.md and #49: Relay owns delivery/provider events and suppression; ambiguity never permits automatic resend; no direct SES/Datamailer path.
  • _docs/specs/07-security-privacy-operations.md: correction/withdrawal/restriction/objection/erasure, internal-copy invalidation, processor propagation, non-PII evidence, suppression safety, and restored-backup fencing.
  • _docs/specs/09-migration-rollout-roadmap.md and _docs/specs/10-verification-strategy.md: immutable release identity, protected-data gates, fault injection, processor ambiguity, redaction, and restore evidence.
  • Closed #23 and parent #64: only the accepted facts recorded there; no additional legal/processor semantics may be inferred.
  • Accepted, merged, green artifacts from #254, #281, #255, #256, #257, and #258: sole machine identities consumed by this contract.
  • #259 audit comment: https://github.com/DataTalksClub/website/issues/259#issuecomment-5468272618

Dependency gate

Engineering is blocked until the source-only/runtime-interface deliveries of #254, #281, #255, #256, #257, and #258 have each passed independent testing, PM acceptance, merge/push, and green on-call. This issue uses their exact canonical paths, schema/contract versions, digests, source commit, and exported action identities. Issue prose, mutable branches/tags, worktree paths, frozen candidates, or provisional digests are not consumable authority.

If an upstream contract does not publish a required stable action or projection identity, that owning issue must be re-groomed and redelivered; this issue may not manufacture it. Any upstream version/digest/path/action drift invalidates this contract and requires a new contract version and full verification.

#49 is not required for this non-activating source artifact, but remains mandatory before any later email/Relay destination can be activated. HUMAN/legal closure remains mandatory before any production destination has enabled policy, terminal-confirmation semantics, receipt retention, retry/escalation deadlines, or protected-data execution.

Canonical files and ownership

This issue owns exactly four new source files:

  • _docs/legal/privacy-propagation-input-contract.json — sole hand-edited canonical propagation input;
  • _docs/legal/privacy-propagation-input-contract.schema.json — JSON Schema Draft 2020-12 structure;
  • scripts/validate_privacy_propagation_input_contract.py — fail-closed canonical loader and semantic/cross-contract validator;
  • scripts/tests/test_privacy_propagation_input_contract.py — identity, drift, coverage, non-activation, and redaction tests.

It does not modify or shadow any #254/#281/#255–#258 artifact. There is no generated Markdown copy, Python application constant, Django setting, environment override, database seed, migration, fixture discovered by production, registry plugin, mutable latest alias, or duplicate destination policy.

Canonical identity

The JSON has only these required top-level sections:

  • schema_version — integer 1;
  • contract_version and contract_digest;
  • upstream_contracts;
  • activation;
  • source_action_catalog;
  • destination_manifest;
  • projection_input_catalog;
  • adapter_contract_slots;
  • receipt_verifier_slots;
  • outcome_catalog;
  • reason_catalog;
  • bounds_catalog;
  • synthetic_fixture_contract.

contract_version is a bounded immutable ISO-date-plus-positive-revision identifier. contract_digest is lowercase SHA-256 over UTF-8 canonical JSON using the accepted #254 canonical ordering, separators, Unicode, and self-digest omission rule; only contract_digest is omitted. Object keys and every ordered catalog/manifest array have one validated canonical order. Duplicate JSON keys, duplicate semantic identities, reordered rows, unknown fields, noncanonical Unicode, mutable aliases, or a version reused with changed bytes fail.

upstream_contracts has exactly one row for each of #254, #281, #255, #256, #257, and #258. Each row pins issue number, canonical repository path, schema version, artifact/API version, lowercase SHA-256 digest, accepted full source commit, and the exact exported identity names used here. The validator loads every repository artifact locally and rejects missing files, unmerged/provisional identities, digest/version/path mismatch, missing export, or semantic drift before returning a contract DTO.

Exact non-activation contract

activation is fixed in version 1:

  • state: non_activating;
  • production_processing_allowed: false;
  • production_destination_registration_allowed: false;
  • production_adapter_registration_allowed: false;
  • production_receipt_verifier_registration_allowed: false;
  • runtime_loading_allowed: false;
  • synthetic_test_loading_allowed: true only through the validator's explicit test-only loader and exact synthetic fixture identity;
  • blocking_human_gate_ids: complete union of applicable upstream authority/processor/retention/receipt/hold/deadline gates;
  • superseded_by: null.

The schema and validator reject active/enabled/accepted states, a production allowance, an empty blocker union, a missing applicable gate, environment/settings/runtime overrides, or wording/fields that imply approval. Production application modules may not import this validator or artifact; repository tests prove no startup, URL, command, job, registry, provider, database, or application-service discovery path reaches it.

Source-action catalog

The catalog contains exactly these #259 action keys, each pinned to the accepted owning upstream contract's exported action name/version and evidence/input digest rules:

  1. withdraw_optional_processing
  2. correct
  3. restrict
  4. object
  5. retention_dispose
  6. erase
  7. export_expire

No action aliases, wildcard, fallback, free-form action, or inferred mapping is allowed. withdraw_optional_processing, restrict, and object remain HUMAN-blocked wherever upstream policy is unresolved. correct and export_expire must resolve only to #256 identities; retention_dispose only to #257; erase only to #258; request/kernel identities resolve only through #255/#281. A catalog row records identity and compatibility only and always sets production_request_allowed: false.

Complete destination manifest

Each destination row has a stable bounded destination_key, positive destination_version, class exactly one of:

  • internal_projection
  • internal_cache
  • internal_search
  • internal_export
  • internal_queue
  • external_processor

It pins accountable upstream domain/service, exact #254 flow IDs, #281 domain/right rows, supported source-action keys, source projection identities, authority/HUMAN gate IDs, dependency issue IDs, review state, and evidence type. State is exactly disabled_human_required, disabled_missing_contract, disabled_missing_adapter, or no_propagation_work; version 1 has no enabled state.

Every accepted #254 flow and every #281 domain/action possibility is covered exactly once either by one destination or an explicit no_propagation_work row backed by an upstream negative assertion and owner. Absent Relay, inactive analytics, transient non-stored country suggestion, aggregate/public content, legacy Datamailer, and any future upstream flow cannot disappear through grouping or runtime inference. Unknown, duplicate, extra, newly applicable, or unclassified coverage fails validation. A no_propagation_work row is not a runtime skip and cannot be selected as success.

Projection-input catalog

For every supported (source_action, destination_key) pair, one projection-input row pins:

  • upstream artifact/action/schema/version/digest and exact source generation/revision/evidence-digest fields;
  • propagation operation/work opaque UUID fields and immutable execution generation;
  • destination/version, adapter-slot identity, verifier-slot identity where external, operation-map version, and canonical plan/input/request/idempotency digest recipes;
  • only allowlisted opaque UUIDs, bounded codes, UTC timestamps/deadlines, nonnegative integer counts, booleans, versions, and lowercase SHA-256 digests;
  • maximum serialized depth/items/bytes and deterministic ordering;
  • state production_disabled and production_projection_allowed: false.

Rows never contain or permit a subject value, reversible subject hash, email/name/username, IP, domain record key/ID, cursor/object key, endpoint/URL, credential/token/cookie/session, provider identifier/payload, message/content/body, SQL, free-form reason/error, or production value. They are schemas for future safe inputs, not concrete production records. Unknown fields/types and sensitive-looking keys/values fail before a DTO is returned and are not echoed.

Empty adapter and verifier slots

There is exactly one adapter slot for every destination row. It records expected destination/action compatibility and future owner issue, but in version 1 has adapter_contract_version: null, adapter_definition_digest: null, registration_state: blocked_unimplemented, and production_registration_allowed: false.

Every external-processor destination additionally has exactly one receipt-verifier slot with verifier/schema/key/transition versions and digest all null, state blocked_unimplemented, and production registration false. Internal destinations have an explicit not_applicable verifier slot and cannot later become external without a new contract version. A non-null slot, fake production adapter/verifier, callable/import path, provider endpoint, key material, callback route, or runtime discovery hook fails.

Bounded outcome and reason catalogs

The outcome catalog is structural and non-dispositive. It contains exact namespaces for adapter attempt and receipt observation, matching the future #259 state-machine vocabulary but granting no universal success:

  • adapter attempt: blocked, transient, permanent, ambiguous, confirmed;
  • receipt observation: accepted, confirmed_applied, confirmed_absent, confirmed_suppressed, rejected, unsupported, ambiguous, expired, superseded.

Every code declares its layer, whether mutation may have begun, whether reconciliation is mandatory, and terminal_success_allowed: false in version 1. In particular accepted is never completion, confirmed is not self-verifying, timeout/connection loss after possible invocation is ambiguous, and expiry/supersession never turns pending work into success. A future activating contract must bind destination-specific safe terminal receipt and postcondition semantics through new owner-approved versions.

The reason catalog contains exactly these bounded safe structural codes:

  • authority_human_required
  • upstream_contract_missing
  • upstream_contract_mismatch
  • source_action_unsupported
  • source_generation_mismatch
  • destination_unclassified
  • destination_disabled
  • adapter_unavailable
  • adapter_contract_mismatch
  • receipt_verifier_unavailable
  • receipt_contract_mismatch
  • policy_or_manifest_drift
  • projection_input_invalid
  • pre_mutation_transient
  • result_ambiguous
  • receipt_missing
  • receipt_invalid
  • receipt_expired
  • postcondition_unverified
  • retry_exhausted

Codes are redacted identifiers only. None decides a right, releases a hold/exception, authorizes retention/anonymization, classifies a provider response, approves retry after possible mutation, or activates processing. Unknown/custom/free-form reason or result codes fail.

Bounds and review state

Every production destination bound that depends on processor/legal/operational authority—batch size, timeout, attempts, lease, ordered retry delays, reconciliation/escalation deadlines, receipt retention/expiry, callback replay/sequence rules, and review/effective dates—is present but null with its exact blocking gate in version 1. Null is permitted only in these schema-designated disabled fields. No default, environment fallback, inherited global value, or synthetic value can fill a production row.

The synthetic fixture contract has a separate digest and exact test_ key namespace, opaque UUID/digest-only inputs, bounded nonzero test values, and an unmistakable synthetic_test_only marker. The validator's explicit test loader accepts it only when passed directly by tests; it is not in the production manifest and cannot be discovered through settings, imports, startup checks, registries, commands, jobs, or database state. Synthetic outcomes demonstrate validation/state-machine inputs only and are never presented as provider evidence or production authority.

Fail-closed loaders and validator

The validator is read-only and has no network, database, provider, or production access. Its canonical loader validates size/depth/duplicate keys/Unicode/schema/digests/order first, then exact upstream identities and exports, then coverage/cross-references/non-activation/redaction. It returns one immutable safe DTO only after all checks pass; otherwise it returns a bounded safe code/path without echoing rejected content.

There is no permissive mode, warning-only drift, unknown-field preservation, partial load, last-known-good fallback, mutable cache, runtime override, or skip_validation. The explicit synthetic loader requires the accepted base contract plus exact test fixture digest/marker and rejects production-like keys or values. Loader APIs live in the validation script only and are not application interfaces.

Scope

  • Add only the exact four canonical source files.
  • Pin the six accepted upstream contract identities and seven source actions.
  • Provide complete, deterministic destination and projection-input coverage with all production rows disabled.
  • Provide null production adapter/verifier slots, bounded non-dispositive outcome/reason catalogs, unresolved bounds, and a separately digested synthetic-only fixture contract.
  • Prove canonical identity, fail-closed cross-contract drift, redaction, and zero runtime/render reachability.
  • Supply the stable source artifact required to re-groom #259; do not implement #259.

Explicit non-goals

  • No change to #254/#281/#255–#258 files, schemas, candidates, reports, HUMAN checkboxes, runtime APIs, or accepted digests.
  • No legal/privacy/processor decision, notice/lawful-basis wording, destination approval, confirmation meaning, deadline, retry/escalation policy, receipt retention, hold/exception release, suppression disposition, or production activation.
  • No Django app/model/migration/service/registry/setting/URL/view/template/static/OpenAPI/Studio/admin/command/job/scheduler/callback/adapter/verifier/provider/runtime code.
  • No production/test database, protected data, requester, AWS, Relay, Datamailer, SES, search/cache/export/queue invalidation, processor call/receipt, tombstone replay, or domain mutation.
  • No fake production manifest/adapter/verifier, production-like fixture, guessed upstream identity, wildcard/default mapping, or bypass around #49, #66, domain adapters, or HUMAN gates.
  • No browser/render change, commit, push, merge, deployment, or external operational action outside the normal issue lifecycle.

Acceptance criteria

  • The exact four files exist with one hand-edited authority and no shadow manifests, policy constants, generated copy, or runtime integration.
  • Contract and synthetic-fixture versions/digests are canonical, deterministic, immutable, and bind exact accepted/merged/green #254/#281/#255–#258 paths, versions, digests, source commit, and exported identities.
  • Activation is exactly non-activating; every production processing/destination/adapter/verifier/runtime-loading allowance is false and cannot be changed by settings, environment, registry, database, fixture, import, or default.
  • The seven source actions map exactly to owning accepted upstream exports with no alias/fallback/inference and production_request_allowed: false.
  • Every upstream flow and domain/action possibility is covered exactly once by a disabled destination or authority-backed no_propagation_work; unknown/new/duplicate/missing coverage fails.
  • Every supported action/destination pair has one bounded projection-input row containing only safe opaque identities, versions, counts, timestamps, and digests; sensitive/domain/provider/free-form fields fail without echo.
  • Adapter slots are null/blocked for every destination; external verifier slots are null/blocked and internal slots explicit not-applicable; no callable/runtime registration can be represented.
  • The exact adapter/receipt outcome and 20-reason catalogs are complete, non-dispositive, and set terminal production success false; unknown/free-form codes and universal success mappings fail.
  • All decision-bearing production bounds/review values remain schema-designated null with blockers; synthetic bounded values are separately identified/digested and cannot populate a production row.
  • Canonical and synthetic loaders fail closed on malformed/oversized/duplicate/unknown/reordered input, digest/version/path/export drift, incomplete coverage, permissive flags, non-null production slots, and every sensitive canary.
  • Repository dependency/import/system-check tests prove no runtime, setting, startup, URL, command, job, registry, database, provider, or render path loads the contract or fixture.
  • Focused tests, formatting/lint/type checks, migration-drift check, repository quality checks, and graph-selected versioned verification pass using uv/Make.
  • Independent tester confirms the frozen change graph is source-only and records Playwright/screenshots not_applicable; any runtime/render path is a scope failure and returns the candidate to engineering.
  • Engineer/tester/PM reports record exact base/head, six upstream identities, contract/fixture digests, changed-path set, plan/graph/artifact digests, commands/counts, and no pending/skipped required component.

Required verification scenarios

  1. Each upstream path/version/digest/source commit/export altered, absent, provisional, reordered, or stale; loading fails before DTO construction.
  2. Contract/fixture self-digest tamper, duplicate JSON key, unknown field, array/key reorder, Unicode/control/floating-point/cycle/depth/item/byte boundary, reused version with changed semantics.
  3. Missing/duplicate/extra flow, domain/action pair, destination, projection pair, adapter/verifier slot, outcome, reason, blocker, or negative-coverage assertion; no fallback succeeds.
  4. Active/enabled/production-allowed/runtime-load flag, removed HUMAN blocker, non-null production adapter/verifier/bound, callable path, environment/default override, or synthetic row inserted into production coverage.
  5. Each source action mapped to a wrong owner/export/version; withdrawal/restriction/objection elevation; receipt accepted or unverified confirmed treated as success; timeout treated as safe retry; expiry treated as completion.
  6. Names/emails/usernames/IPs/subject hashes/domain IDs/cursors/object keys/URLs/endpoints/tokens/cookies/sessions/AWS/Relay/provider payloads/message content/SQL/free text and sensitive-looking keys at every object/string boundary; bounded failure output never echoes the canary.
  7. Source scan/import graph/system checks prove the application cannot discover either loader/artifact and that no model, migration, URL, command, job, settings, provider, or rendered route changed.

Required verification commands

At minimum, plus the generated change-selective plan:

uv run --frozen python scripts/validate_privacy_authority_register.py
uv run --frozen python scripts/validate_privacy_kernel_input_contract.py
uv run --frozen python scripts/validate_privacy_propagation_input_contract.py
uv run --frozen python -m unittest scripts.tests.test_privacy_authority_register scripts.tests.test_privacy_kernel_input_contract scripts.tests.test_privacy_propagation_input_contract
make migrations-check
make verification-quality

The independent tester reruns focused validators/tests and every graph-selected component. Screenshots and Playwright are not_applicable only after independent proof of zero runtime/render impact.

Delivery and downstream order

After #254 → #281 → #255 and the accepted #256/#257 lanes → audited/accepted #258 are merged and green, this issue proceeds engineer → independent tester → PM acceptance → focused commit with Closes #282 → local no-ff merge/push → on-call green. Then PM rewrites #259 against the exact file/schema/contract/fixture identities and removes needs grooming only if every runtime dependency is accepted. #49 and applicable HUMAN/legal decisions remain required before any production email/processor destination or protected-data execution is enabled.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read the accepted artifacts from #254, #281, and #255–#258 first, then inspect the four owned files: _docs/legal/privacy-propagation-input-contract.json, its schema, scripts/validate_privacy_propagation_input_contract.py, and scripts/tests/test_privacy_propagation_input_contract.py. Run the validator and tests after all dependency gates are satisfied. Done means canonical identity, coverage, drift, non-activation, and redaction checks pass without adding runtime discovery or production execution paths.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend-api-design, security, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.