DataTalksClub / DataTalksClub/website

Contract legacy course-registration schema after the rollback window

Open
#287 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

courses data-migration decision needs grooming operations P0 testing
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Parent epic: #54. Split from #242 during PM dependency reconciliation.

PM disposition

DECISION REQUIRED / FAIL-CLOSED / NOT ENGINEER-READY.

This issue owns only the future destructive contract phase for legacy course-registration schema and compatibility behavior. It does not choose an exit policy, execute a migration, or treat elapsed time as evidence. Until the authorized owner selects E1, E2, E3, or an equally exact replacement, retain both decision and needs grooming and make no destructive change.

Decision requested

The owner should reply with one code:

  • E1 — two releases plus 30 clean days (PM recommendation). After the target writer/read cutover is deployed, require two distinct immutable production releases to complete green on-call observation. The second release starts a 30-consecutive-day observation window. Contract only if every required measurement below is continuously complete and clean throughout that window.
  • E2 — one release plus 14 clean days. One immutable target-native production release completes green on-call observation and starts a 14-consecutive-day window with the same evidence. This removes redundant compatibility storage sooner, but provides neither a second-release compatibility proof nor E1's longer chance to observe low-frequency readers, jobs, exports, and rollback faults.
  • E3 — retain compatibility storage indefinitely. Disable every legacy-authoritative writer after target cutover, but retain the classified legacy columns, constraints needed by the projection, and read projection. No destructive migration is authorized; PM closes #287 as intentionally retained once writer cutover and compatibility tests are accepted. Duplicate PII remains inside export, correction, erasure/anonymization, restore-tombstone, access-control, and retention coverage for as long as it exists.

An alternative is valid only if it states the number of distinct deployed releases, the consecutive observation duration and start event, every required zero/error threshold, the restore/rollback proof, and whether compatibility storage is removed or retained.

Normative authority

  • _docs/PROCESS.md
  • _docs/specs/04-courses-and-cohorts.md, Registration and Expand-and-contract model migration
  • _docs/specs/07-security-privacy-operations.md, Privacy baseline
  • _docs/specs/09-migration-rollout-roadmap.md, Rollback, Data migration controls, and Member-profile expand and contract
  • _docs/specs/10-verification-strategy.md, Course/Cohort, onboarding, privacy, migration, and rollback verification
  • _docs/architecture/app-boundaries.md
  • Parent #54; additive source #242; target flows #243–#246; profile foundations #247/#248; production-like rehearsal #60; aggregate registration provenance #133

Current main still carries the legacy CourseRegistration columns and campaign/email uniqueness contract. That is characterization evidence only, not permission to freeze a future drop list before #242 and the target flows establish their accepted schema fingerprint.

Hard prerequisites under E1 or E2

PM may replace this decision packet with an engineer-ready contract issue only when all of the following are true:

  1. The owner has selected E1/E2 or an equally exact destructive policy.
  2. #242 and #243–#246 are accepted, integrated, deployed, and green; their transitive Course/Cohort, account/profile, authorization, delivery, and management foundations are present. #60 has accepted the production-like forward/backward migration and compatibility rehearsal.
  3. Every public, learner, Studio, admin API, export, privacy, job, and management reader/writer in the accepted graph uses the target-native service/schema. The legacy-authoritative writer is disabled. Service-maintained rollback projections, if any, are separately identified and must compare exactly with target state.
  4. The observation window begins only at the option's defined release event. Missing telemetry, an incomplete day, a release rollback, an unexplained mismatch, a legacy-authoritative write, a quarantine row, or #133 drift resets the consecutive window to zero after correction and a new green deployment.
  5. The exact contract inventory is frozen from the deployed schema: every legacy column, constraint, index, projection, model/form/serializer/admin field, reader, writer, export/privacy consumer, fixture, migration-history dependency, and compatibility test is classified remove, retain, or replace. Nothing unclassified may be dropped.

E3 authorizes no contract implementation. It retains the compatibility inventory and makes target-only writer enforcement plus continuing projection/privacy verification the terminal disposition.

Required operational evidence

Evidence is aggregate, redacted, immutable-release-bound, and free of protected row values:

  • exact source pin, website SHA, VERSION/image digest, task definitions, database schema/migration leaves, target-schema fingerprint, #133 source/schema/policy/native-boundary fingerprint, and start/end timestamps for each observed release and day;
  • green deployment/readiness/smoke and required CI verdict for each counted release, with no cancellation or unresolved job treated as green;
  • continuous zero legacy-authoritative writes from every HTML/API/Studio/admin/job/import/export path; service-owned rollback-projection writes are not counted as legacy authority but must be explicitly enumerated and compare exactly;
  • continuous zero target-versus-legacy read mismatches, zero unresolved registration quarantine rows, zero unknown/unclassified consumers, and zero unexplained #133 total/checksum/pointer/native-boundary drift;
  • deterministic reconciliation with source/target counts, stable-ID mapping cardinality, duplicates/missing relations, reason counts, and safe checksums; dry-run/apply-twice/restart/concurrency/source-drift evidence without raw identity or registration values;
  • a verified backup restore into an isolated production-like environment, followed by the recorded pre-contract application-image rollback against the expanded schema, including registrations created after cutover; registration, enrollment, privacy/export, delivery-intent idempotency, and #133 totals remain correct and no legacy sender is restored;
  • graph-selected Django, migration, security/privacy, compatibility, API/OpenAPI, container, and browser evidence for the final classified removal. Screenshots are not_applicable only when the final graph proves no render impact.

Exact contraction boundary after E1/E2 passes

The later engineer-ready issue must use a separately reviewed staged contract:

  1. freeze the accepted inventory and evidence digests;
  2. deploy code that has no legacy-authoritative reader/writer while the legacy storage still exists;
  3. re-run final reconciliation under an explicit write freeze and verify a fresh restorable backup;
  4. remove only inventory rows classified remove, activate only already-proved target constraints, and preserve migration-history compatibility required for maintained installs;
  5. deploy/readiness/smoke the exact contracted image and resume writes only after success.

Before destructive DDL commits, normal immutable-image rollback remains available. After a committed drop, an application revision that requires the removed schema is not a valid rollback target. Recovery is forward-fix, or a whole verified backup restore while writes remain frozen; do not reverse a migration by fabricating deleted values, merge a stale restored database with newer writes, or silently re-enable legacy/Datamailer writers. A contract failure is a failed release, never a reason to waive reconciliation.

Retention and migration consequences

Every option retains historical primary keys and timestamps, stable legacy-ID mappings, immutable Cohort/Course/campaign and normalized-email snapshots, minimized shared-profile snapshots, comment and privacy/marketing evidence, historical accepted_newsletter, #133 revisions/displacement/provenance, and bounded non-PII reconciliation evidence. Contract cleanup is not legal deletion/anonymization and must not rewrite those records.

Under E1/E2, a duplicate legacy PII column may be dropped only when the inventory proves its accepted target-owned value/evidence is complete and its removal is covered by export, correction, deletion/anonymization, restore-tombstone, and audit tests. Ambiguous, quarantined, legally retained, or unmatched data is retained and blocks contraction; it is never guessed or discarded.

Under E3, retained duplicate data remains a live privacy and security obligation indefinitely. It cannot become a second authority, be exposed through a compatibility mutation, or fall outside rights/retention/backup controls.

Acceptance criteria for this decision issue

  • [HUMAN] The authorized owner records E1, E2, E3, or an equally exact replacement.
  • PM records the choice and either re-grooms an exact destructive implementation after all gates/evidence pass (E1/E2), or records the retain-indefinitely disposition with no destructive work (E3).
  • No field, mapping, evidence, constraint, reader, writer, projection, or rollback support is removed before those steps.

Explicit non-goals

  • No implementation, migration, production/protected-data inspection, deployment, backup operation, provider call, workflow dispatch, commit, push, merge, or timer-only approval in this decision issue.
  • No change to public registration, profile, learner, Studio/admin API, export, email/Relay/Datamailer, CourseInterest (#286), Course/Cohort, URL/host, or #133 product behavior.
  • No deletion of educational history, consent/privacy evidence, mappings, provenance, immutable snapshots, or reconciliation records; no legal erasure/retention redesign owned by #64.
  • No exact drop list inferred from today's legacy schema; the accepted deployed target schema and consumer inventory define that later list.

Lifecycle

Keep decision and needs grooming. The next actor is the authorized owner, not engineering. After an owner choice, PM records the disposition but still leaves E1/E2 implementation blocked until the selected deployed evidence exists; only then may PM produce the separately reviewable engineer-ready contract. Every later implementation follows engineer → independent tester → PM acceptance → focused commit/local no-ff merge/push → on-call.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with _docs/PROCESS.md and the referenced migration, privacy, architecture, and verification specifications, then review parent #54 and split issues #242–#246. This issue is not implementation-ready: done means the authorized owner records E1, E2, E3, or an equally exact policy and PM records the disposition; no schema change is authorized yet.

Written by the indexing model from the issue text.

Assessment

Tech stack
django, python
Domain
api, backend, databases, security
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.