DataTalksClub / DataTalksClub/website
Implement reviewed pre-2024 cohort mapping and historical-import support
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Parent decision: #15
PM disposition
GROOMED / P0 / REPOSITORY SLICE / OPERATIONAL HOLD.
This issue closes the remaining repository implementation gap for resolved decision #15. It owns code, migrations, deterministic synthetic import behavior, and characterization updates only. It does not run a production or protected-data import. After this repository slice is independently verified, accepted, committed with Closes #224, merged, pushed, and green, #51 may consume its exact merge SHA and mapping digest.
The previous 135-path candidate is stale evidence: it is frozen on 9a491cd9baeb1e723bd05e8dacb20978fbaf9d33, while the audited remote base is face8e4808d65afbf0374d1ced7a88079950d663. Its candidate-specific acceptance does not survive that base change. Do not replay or rebase the patch wholesale. Dispatch remains on HOLD until the current release-recovery lane establishes one clean green origin/main; then reconstruct the smallest complete slice from that exact SHA and generate a new versioned verification plan.
Normative authority
_docs/PROCESS.md- Open decision 4, resolved by #15
- Specification 04 — Course/cohort migration and high-risk checks
- Specification 07 — identity, privacy, and safe evidence
- Specification 09 — data-migration controls
- Specification 10 — verification strategy
Exact repository scope
Reviewed mapping and fail-closed creation
- Add exactly these seven confirmed pre-2024 edition identities to the explicit course-family catalog:
de-zoomcamp-2022 -> de-zoomcamp / 2022de-zoomcamp-2023 -> de-zoomcamp / 2023mlops-zoomcamp-2022 -> mlops-zoomcamp / 2022mlops-zoomcamp-2023 -> mlops-zoomcamp / 2023ml-zoomcamp-2021 -> ml-zoomcamp / 2021ml-zoomcamp-2022 -> ml-zoomcamp / 2022ml-zoomcamp-2023 -> ml-zoomcamp / 2023
- Remove live regex/year-stripping family inference from
Cohort.save(). Saving a bare Cohort without an explicit Course parent fails loudly and atomically. - Historical import and production/local seed/import adapters must resolve the reviewed catalog and pass
course=and catalog year explicitly. An unknown, duplicate, conflicting, or year-mismatched legacy edition fails before partial mutation. - Preserve the accepted compatibility API behavior without moving #51/#52/#59 into this issue: a reviewed legacy slug resolves through the catalog; a genuinely new, unmapped create request may receive an explicit same-slug Course parent through the existing adapter, but no runtime regex/year stripping or fuzzy family inference is allowed.
ML Zoomcamp 2021 certificate source support
- Add repository-side discovery and parsing for the confirmed
courses/mlzoomcamp-2021certificate CSV/JSON source shape. - Accept the historical certificate identifier from the existing link field or the confirmed 2021 text-field shape, with strict type checks and deterministic behavior.
- Synthetic fixtures prove the 2021 edition imports certificate URLs like the other supported editions. This issue does not locate, download, inspect, or execute against protected production data.
Durable legacy learner identifier
- Add an accounts-owned
AccountLegacyIdentifierschema for sourcezoomcamp-scoringand the normalized lowercase 40-hexsha1(email)identifier. - Enforce one owner per
(source, identifier), a protected account relation, queryability by source/identifier and account/source, and database/application validation portable across supported backends. - Persist the identifier in the same atomic historical-learner resolution transaction. Exact replay is idempotent; an identifier that resolves to a different account fails closed with no partial learner/enrollment/import mutation.
- Treat the digest as an opaque legacy join key. Do not emit source email, digest values, account candidates, or learner content in logs, reports, screenshots, or issue comments.
Necessary compatibility adaptations
- Update only the fresh-base runtime authoring paths, shared synthetic factories, and copied characterization fixtures that directly relied on
Cohort.save()inference so they construct an explicit Course/Cohort relationship. - Regex parsing is permitted only in a clearly test/import-authoring-only helper, never in a model, view, API service, or other live family-resolution path.
- Regenerate only adoption inventory/patch evidence made stale by this exact slice. Preserve current source pin and unrelated current-main behavior.
- The engineer handoff must classify every changed path as product implementation, migration, direct authoring adapter, mechanical characterization adaptation, focused test, or generated adoption evidence. Any other path is out of scope.
Dependencies and delivery order
- #15 is resolved and supplies the product decision. There is no open product dependency.
- The current HOLD is operational, not a new issue dependency: reconstruct only after one clean green current-main recovery SHA exists.
- #51 is a hard downstream consumer. It starts only after this repository slice closes on an exact green merge SHA; #51 owns the remaining Course -> Cohort structural/preflight/rollback migration work.
- #230, #231, and #234 do not supply an interface required by #224 or #51. They may be reconstructed as independent lanes from the same recovered base, but each must refresh after any preceding merge that overlaps its candidate. #230 later feeds #244/#60. #231 and #234 join accepted #51 at #247.
- #247 remains blocked until accepted, integrated #224, #231, #234, and #51 are all recorded by exact merge SHA.
- #60 owns the separately authorized production-like snapshot rehearsal, actual full historical/course import and reconciliation, backup/restore, freeze/delta, compatible-image rollback, and send-disabled provider boundary. No production-execution remainder stays in #224.
Shortest gating path: #224 -> #51; in parallel capacity, reconstruct and integrate #230, #231, and #234 from current green bases; then dispatch #247 only after #51 + #231 + #234 are accepted and integrated. #230 is not a #247 dependency.
Acceptance criteria
- The engineer records the exact clean green start SHA, current source pin, final
accounts/coursesmigration leaves, seven-row catalog serialization/digest, and complete changed-path classification. - All seven confirmed pre-2024 slugs resolve exactly once to the reviewed family/year; unknown, duplicate, conflicting, stale, or year-mismatched legacy input fails before mutation.
-
Cohort.save()contains no runtime slug/year inference and a bare Cohort withoutcoursefails loudly; every in-scope live authoring adapter supplies an explicit parent without regex/fuzzy inference. - The existing compatibility course-create request remains supported through its bounded explicit-parent adapter, with reviewed legacy slugs using the catalog and genuinely new slugs never being mistaken for a historical mapping.
- ML Zoomcamp 2021 certificate CSV/JSON discovery and both confirmed certificate-field shapes are covered by deterministic synthetic import tests; malformed/non-dictionary/non-string input is ignored or rejected safely as specified, never executed.
- Each successfully resolved historical learner has one queryable
zoomcamp-scoringSHA-1 legacy identifier; exact replay is idempotent and an account/identifier conflict rolls back atomically. - Migration forward/reverse/apply-twice, constraint, backend-portability, PII/log canary, historical-import, mapping, certificate, identity-conflict, runtime-authoring, factory, and copied-characterization tests pass with no migration drift.
- Adoption verification remains green at the unchanged CMP source pin and documents only this target-owned mapping/import/factory overlay.
- No protected/production data or provider is accessed; no production import/reconciliation, Course -> Cohort structural redesign, route/capability/profile/certificate-API redesign, source-pin advance, deployment, or destructive contraction is present.
- A separate tester validates the frozen candidate and the complete graph-selected verification plan. Because the intended product render is unchanged, screenshots are
not_applicableonly if the independently recomputed graph proves no render impact; otherwise the tester captures and inspects every selected desktop/mobile state. - PM accepts the fresh tester-final report before a focused
Closes #224commit is locally merged, pushed, and observed green by on-call.
Required scenarios
- Each of the seven pre-2024 editions, multiple editions converging on one family, an unknown slug, duplicate/conflicting catalog data, and catalog/source year mismatch.
- Direct bare Cohort persistence, reviewed-legacy API/import creation, genuinely new compatibility API creation, local/development seed/import paths, and repeated invocation.
- ML Zoomcamp 2021 CSV plus JSON link-field and text-field certificate shapes; missing, malformed, wrong-container, wrong-scalar, unmatched, and repeated sources.
- Historical learner with recovered real email, synthetic learner without it, exact replay, invalid SHA-1, same identifier/same account, same identifier/different account, and injected failure proving transaction rollback.
- Fresh migration, forward/reverse/apply-twice, constraint violations, current factory matrix, copied course characterization, adoption manifest/inventory verification, and PII canaries.
Explicit non-goals
No reimplementation of #51; no #230 peer-review fix; no #231 country/email change; no #234 certificate identity resolver/API behavior; no #247 MemberProfile; no #60 protected snapshot or production-like execution; no live migration/import; no learner-row inspection; no public page, heading, navigation, CSS, or design change; no CMP source-pin advance; no provider/network side effect; no commit, merge, push, or deployment during engineering/testing/PM review.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with _docs/PROCESS.md and Specifications 04, 07, 09, and 10, then trace Cohort.save(), the historical-import and seed adapters, and the accounts/courses migration leaves named in the issue. Run the migration, mapping, certificate, identity-conflict, runtime-authoring, factory, characterization, and adoption checks using synthetic fixtures only. Done means all listed scenarios pass atomically without protected-data or production-provider access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, database, security, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100