DataTalksClub / DataTalksClub/website

Implement reviewed pre-2024 cohort mapping and historical-import support

Open
#224 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug courses data-migration P0
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Parent decision: #15

PM disposition

GROOMED / P0 / REPOSITORY SLICE / OPERATIONAL HOLD.

This issue closes the remaining repository implementation gap for resolved decision #15. It owns code, migrations, deterministic synthetic import behavior, and characterization updates only. It does not run a production or protected-data import. After this repository slice is independently verified, accepted, committed with Closes #224, merged, pushed, and green, #51 may consume its exact merge SHA and mapping digest.

The previous 135-path candidate is stale evidence: it is frozen on 9a491cd9baeb1e723bd05e8dacb20978fbaf9d33, while the audited remote base is face8e4808d65afbf0374d1ced7a88079950d663. Its candidate-specific acceptance does not survive that base change. Do not replay or rebase the patch wholesale. Dispatch remains on HOLD until the current release-recovery lane establishes one clean green origin/main; then reconstruct the smallest complete slice from that exact SHA and generate a new versioned verification plan.

Normative authority

Exact repository scope

Reviewed mapping and fail-closed creation
  • Add exactly these seven confirmed pre-2024 edition identities to the explicit course-family catalog:
    • de-zoomcamp-2022 -> de-zoomcamp / 2022
    • de-zoomcamp-2023 -> de-zoomcamp / 2023
    • mlops-zoomcamp-2022 -> mlops-zoomcamp / 2022
    • mlops-zoomcamp-2023 -> mlops-zoomcamp / 2023
    • ml-zoomcamp-2021 -> ml-zoomcamp / 2021
    • ml-zoomcamp-2022 -> ml-zoomcamp / 2022
    • ml-zoomcamp-2023 -> ml-zoomcamp / 2023
  • Remove live regex/year-stripping family inference from Cohort.save(). Saving a bare Cohort without an explicit Course parent fails loudly and atomically.
  • Historical import and production/local seed/import adapters must resolve the reviewed catalog and pass course= and catalog year explicitly. An unknown, duplicate, conflicting, or year-mismatched legacy edition fails before partial mutation.
  • Preserve the accepted compatibility API behavior without moving #51/#52/#59 into this issue: a reviewed legacy slug resolves through the catalog; a genuinely new, unmapped create request may receive an explicit same-slug Course parent through the existing adapter, but no runtime regex/year stripping or fuzzy family inference is allowed.
ML Zoomcamp 2021 certificate source support
  • Add repository-side discovery and parsing for the confirmed courses/mlzoomcamp-2021 certificate CSV/JSON source shape.
  • Accept the historical certificate identifier from the existing link field or the confirmed 2021 text-field shape, with strict type checks and deterministic behavior.
  • Synthetic fixtures prove the 2021 edition imports certificate URLs like the other supported editions. This issue does not locate, download, inspect, or execute against protected production data.
Durable legacy learner identifier
  • Add an accounts-owned AccountLegacyIdentifier schema for source zoomcamp-scoring and the normalized lowercase 40-hex sha1(email) identifier.
  • Enforce one owner per (source, identifier), a protected account relation, queryability by source/identifier and account/source, and database/application validation portable across supported backends.
  • Persist the identifier in the same atomic historical-learner resolution transaction. Exact replay is idempotent; an identifier that resolves to a different account fails closed with no partial learner/enrollment/import mutation.
  • Treat the digest as an opaque legacy join key. Do not emit source email, digest values, account candidates, or learner content in logs, reports, screenshots, or issue comments.
Necessary compatibility adaptations
  • Update only the fresh-base runtime authoring paths, shared synthetic factories, and copied characterization fixtures that directly relied on Cohort.save() inference so they construct an explicit Course/Cohort relationship.
  • Regex parsing is permitted only in a clearly test/import-authoring-only helper, never in a model, view, API service, or other live family-resolution path.
  • Regenerate only adoption inventory/patch evidence made stale by this exact slice. Preserve current source pin and unrelated current-main behavior.
  • The engineer handoff must classify every changed path as product implementation, migration, direct authoring adapter, mechanical characterization adaptation, focused test, or generated adoption evidence. Any other path is out of scope.

Dependencies and delivery order

  • #15 is resolved and supplies the product decision. There is no open product dependency.
  • The current HOLD is operational, not a new issue dependency: reconstruct only after one clean green current-main recovery SHA exists.
  • #51 is a hard downstream consumer. It starts only after this repository slice closes on an exact green merge SHA; #51 owns the remaining Course -> Cohort structural/preflight/rollback migration work.
  • #230, #231, and #234 do not supply an interface required by #224 or #51. They may be reconstructed as independent lanes from the same recovered base, but each must refresh after any preceding merge that overlaps its candidate. #230 later feeds #244/#60. #231 and #234 join accepted #51 at #247.
  • #247 remains blocked until accepted, integrated #224, #231, #234, and #51 are all recorded by exact merge SHA.
  • #60 owns the separately authorized production-like snapshot rehearsal, actual full historical/course import and reconciliation, backup/restore, freeze/delta, compatible-image rollback, and send-disabled provider boundary. No production-execution remainder stays in #224.

Shortest gating path: #224 -> #51; in parallel capacity, reconstruct and integrate #230, #231, and #234 from current green bases; then dispatch #247 only after #51 + #231 + #234 are accepted and integrated. #230 is not a #247 dependency.

Acceptance criteria

  • The engineer records the exact clean green start SHA, current source pin, final accounts/courses migration leaves, seven-row catalog serialization/digest, and complete changed-path classification.
  • All seven confirmed pre-2024 slugs resolve exactly once to the reviewed family/year; unknown, duplicate, conflicting, stale, or year-mismatched legacy input fails before mutation.
  • Cohort.save() contains no runtime slug/year inference and a bare Cohort without course fails loudly; every in-scope live authoring adapter supplies an explicit parent without regex/fuzzy inference.
  • The existing compatibility course-create request remains supported through its bounded explicit-parent adapter, with reviewed legacy slugs using the catalog and genuinely new slugs never being mistaken for a historical mapping.
  • ML Zoomcamp 2021 certificate CSV/JSON discovery and both confirmed certificate-field shapes are covered by deterministic synthetic import tests; malformed/non-dictionary/non-string input is ignored or rejected safely as specified, never executed.
  • Each successfully resolved historical learner has one queryable zoomcamp-scoring SHA-1 legacy identifier; exact replay is idempotent and an account/identifier conflict rolls back atomically.
  • Migration forward/reverse/apply-twice, constraint, backend-portability, PII/log canary, historical-import, mapping, certificate, identity-conflict, runtime-authoring, factory, and copied-characterization tests pass with no migration drift.
  • Adoption verification remains green at the unchanged CMP source pin and documents only this target-owned mapping/import/factory overlay.
  • No protected/production data or provider is accessed; no production import/reconciliation, Course -> Cohort structural redesign, route/capability/profile/certificate-API redesign, source-pin advance, deployment, or destructive contraction is present.
  • A separate tester validates the frozen candidate and the complete graph-selected verification plan. Because the intended product render is unchanged, screenshots are not_applicable only if the independently recomputed graph proves no render impact; otherwise the tester captures and inspects every selected desktop/mobile state.
  • PM accepts the fresh tester-final report before a focused Closes #224 commit is locally merged, pushed, and observed green by on-call.

Required scenarios

  1. Each of the seven pre-2024 editions, multiple editions converging on one family, an unknown slug, duplicate/conflicting catalog data, and catalog/source year mismatch.
  2. Direct bare Cohort persistence, reviewed-legacy API/import creation, genuinely new compatibility API creation, local/development seed/import paths, and repeated invocation.
  3. ML Zoomcamp 2021 CSV plus JSON link-field and text-field certificate shapes; missing, malformed, wrong-container, wrong-scalar, unmatched, and repeated sources.
  4. Historical learner with recovered real email, synthetic learner without it, exact replay, invalid SHA-1, same identifier/same account, same identifier/different account, and injected failure proving transaction rollback.
  5. Fresh migration, forward/reverse/apply-twice, constraint violations, current factory matrix, copied course characterization, adoption manifest/inventory verification, and PII canaries.

Explicit non-goals

No reimplementation of #51; no #230 peer-review fix; no #231 country/email change; no #234 certificate identity resolver/API behavior; no #247 MemberProfile; no #60 protected snapshot or production-like execution; no live migration/import; no learner-row inspection; no public page, heading, navigation, CSS, or design change; no CMP source-pin advance; no provider/network side effect; no commit, merge, push, or deployment during engineering/testing/PM review.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with _docs/PROCESS.md and Specifications 04, 07, 09, and 10, then trace Cohort.save(), the historical-import and seed adapters, and the accounts/courses migration leaves named in the issue. Run the migration, mapping, certificate, identity-conflict, runtime-authoring, factory, characterization, and adoption checks using synthetic fixtures only. Done means all listed scenarios pass atomically without protected-data or production-provider access.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, database, security, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.