Consolidate every data ingest into one clear set of prod scripts

Every legacy and upstream data source should reach the database through one clear set of scripts in scripts/prod/. Most of the code already exists and works — it is scattered across four different places, so nobody can see what is covered and what is not.

Update 2026-09-03, end of day: scripts/prod/ now holds six scripts (import_cmp_content.py, import_cmp_learners.py, import_events.py, import_legacy_zoomcamp.py, import_testimonials.py, sync_content.py), each with its own tests and a Make target, following one shared convention (SYNC_MODEL, BOOTSTRAPPS_EMPTY_DATABASE, checked by scripts/tests/test_prod_conventions.py). A full local dry run of the migration plan ran the whole sequence end to end against the real production export today (_docs/runbooks/production-data-migration.md) — results below are from that run, not estimates.

The two sync models

A reader opening scripts/prod/ should be able to tell at a glance which scripts they would ever run twice.

Git-synchronized — upstream keeps changing, we re-sync:

• Course repositories (ai-dev-tools-zoomcamp, llm-zoomcamp, ml-zoomcamp) → modules, units, homework, projects.
  One shared path for both transports: content_sync/course_repository_ingest.py, driven by the signed push webhook and content_sync/management/commands/pull_course_repositories.py. Repository list comes from registered ContentSource rows, not a hardcoded list. Its snapshot-transport and draft-detection halves are now extracted into content_sync/snapshot.py/content_sync/drafts.py so other sources (faq, docs) can reuse them instead of duplicating.
• DataTalksClub/content → wiki, podcast, articles, people, books.
  Still file-backed: served from content/public_projection/*.json, built by scripts/build_public_projection.py. The database pipeline (content/models.py, content/services.py, content_sync/dtc_content/) is real and tested but has no serving caller. Confirmed today this is fine to defer past launch — nothing else shipped today depends on it, and the project's own roadmap already treats the cutover as separately groomed work. Publishing already works without it: bump the pin in build_public_projection.py, rebuild, commit, review.

One-time import — frozen history, imported once at migration:

• Pre-2023 Zoomcamp history → 7 editions, submissions, certificates, users, enrollments.
  Moved into scripts/prod/import_legacy_zoomcamp.py, with tests and a Make target. Verified today against all 7 real editions (not just one): 1,207s total, e.g. de-zoomcamp-2023 alone landed 3,132 homework submissions, 298 project submissions, 256 graduates. Idempotent on replay.
• CMP export — course content → courses, cohorts, homework, questions, projects, criteria, campaigns.
  scripts/prod/import_cmp_content.py + courses/services/cmp_content_import.py. The empty-database defect below is fixed: it now bootstraps its own reviewed families and cohorts rather than depending on seed_local_courses running first. Verified today: 16 of 21 cohorts imported (5 correctly owner-deferred, see below), 9 repository modules rebound to CMP's homework slugs.
• CMP export — learner data → split by what's actually built.
  - [x] Accounts and emails — scripts/prod/import_cmp_learners.py. Resumable (proven with a real kill -9 mid-run and clean resume against the actual export) and deduplicated against the legacy importer's rows (a real dry run found 879 people who would have landed as two separate accounts and been locked out at sign-in as verified_owner_ambiguous; fixed and reverified against the real export — collision count dropped from 880 to the one genuinely ambiguous pair). Verified: 20,009 accounts, 0 rows in any of the six forbidden tables.
  - [ ] Enrollments, submissions, answers, registrations, criteria responses, peer reviews, project evaluation scores, wrapped statistics — no importer exists yet. Confirmed by today's dry run: 9 of the 11 tables this line originally named have zero CMP-sourced rows. That's roughly 470,000 of the ~510,000 learner rows in the export. This is the largest remaining piece of work in this issue.
• Luma + Eventbrite → historical registration aggregates.
  Adapters in events/importers.py, driven from scripts/prod/import_events.py (now its own script, not just orchestrator-embedded logic). Counts only; no attendee row is ever written. Unchanged: only 3 of 421 provider mappings are activated, the other 380 sit at mapping_review_required. The "surface incomplete coverage" defect below is fixed — this is now a visible, reported gap rather than a silent one.
• Event identity manifest → 421 events, 1,684 aliases.
  Now has its own script, scripts/prod/import_events.py, alongside the existing manage.py import_event_identities management command. Works, idempotent, verified today. A real allocator bug was found and fixed in the same lane: the seeding migration that used to write these rows never advanced EventPublicIdSequence, which would have raised event_public_id_allocator_invalid on the very first from-zero production import.

Found today, not in the original scope

• A2 — llm-zoomcamp-2026 has 2 unreconciled repository homework rows (homework-06, homework-07) that don't pair against any CMP slug. Confirmed still present in today's dry run; must resolve before the course-content import can be called fully correct for that cohort.
• Migration history collapsed from 93 files to 10, schema equivalence proven by structural diff across all 100 tables, not asserted.
• The projection's hardcoded record counts are gone. Publishing a new article/podcast/person used to require a Python edit; it's now checked by referential integrity and manifest self-consistency instead.

Defects to fix alongside

• courses/services/local_cmp_content_import.py:200 — now stages under the project-local .tmp/.
• courses/services/development_content_transport.py:19 — same fix.
• scripts/prepare_local_data.py CMP-before-repository-pull ordering — resolved by switching to the reconciling importer, verified against the real export in production order.
• scripts/load_rds_export.py deleted outright — confirmed removed, including its test coverage retired with it.
• Empty-database bootstrap ordering made explicit and checked — BOOTSTRAPPING_ENTRY_POINTS/SYNC_MODEL/COURSE_CATALOGUE_ORDER in scripts/prod/__init__.py, enforced by scripts/tests/test_prod_conventions.py.
• "3 of 421 activated" now surfaces — import_events.py reports activation_coverage on every run.

Constraints

• Production exports at /data/tmp/rds-export/ are read in place. Never copied into the worktree, never committed.
• Never import django_session, socialaccount_socialaccount, socialaccount_socialapp, socialaccount_socialapp_sites, socialaccount_socialtoken, accounts_token. import_cmp_learners.py now writes this forbidden set explicitly for its own purpose rather than reusing review_import's SENSITIVE_TABLES, which is scoped to a different (sanitized-review) allowlist and excludes the entire learner payload if reused here.
• A member's email address is visible to admins in Studio and nowhere else.
• Plain scripts, direct ORM, no framework. Call existing services rather than reimplementing them — in particular the CMP reconciliation logic.
• Slug authority: CMP's slug wins for modules-format cohorts.

Decisions already taken

• Content moves into the database — "all the content".
• One course path only, handling legacy and new formats (landed in 793ed1f).
• Course inventory: no MLOps 2026 edition; de-zoomcamp-2026 is correctly finished.
• The five owner-deferred cohorts (ai-bootcamp-2025, ai-buildcamp-2, ai-buildcamp-3, ai-hero-2025, ai-hero-2026) are correctly skipped, not a defect: all five are visible: false in CMP itself, with no dates set — CMP's own owners haven't published them either. Awaiting an explicit publication decision per cohort before any reviewed family/title mapping is written for them.

Start with scripts/prod/ and _docs/runbooks/production-data-migration.md, then inspect import_cmp_learners.py and the unchecked CMP learner tables. Review the existing tests and dry-run conventions before determining the scope; this issue is done only when the remaining learner data and the two unreconciled repository rows have an agreed, tested migration path.