Project gallery enrichment: structured per-course content, gated website integration, production importer

Summary

Owner request (2026-09-19): "I want to run a batch enrichment job to add this. we can do it once after each cohort finishes." — about the /courses/projects learner-project gallery; the public Streamlit gallery is product inspiration only, not a data source.

During the 2026-09-20 working session the owner directed the scope past the original Phase-1 intake: define structured content per course, extract it for the submitted repositories, and add it to the website. This body reflects that direction — the original "no public page changes" non-goal predates it and no longer applies.

Shipped (branch issue-416-gallery-enrichment, commit 42eb9224)

• Additive migrations 0020/0021 and the ProjectRepoEnrichment model (liveness, effective URL, structured course-structured-v1 payload).
• Prod-parallel importer scripts/prod/import_project_repo_enrichment.py — dry-run → apply → replay honesty verified on a throwaway DB (5,224 probed submissions, 1,202 structured records) — plus runbook §22.
• Gallery integration on /courses/projects: annotated cards (summary, topic/pipeline chips, gone state), safe denials (low/baseline confidence suppressed, staff-only reviewer note), mobile-390 layout after the PM-gate repair.
• Data ownership follows the database-only rule: records live in the database and ~/prod/dtc-data/content-staging/ (canonical merge md5 87e7a625…); nothing is checked into the repo.

Gates (2026-09-20)

• Tester PASS — six checks: suites, real-data render screenshots, importer honesty (dry-run/apply/replay), behavioral contract, PII sweep, migration drift.
• PM PASS — after one repair loop (mobile-390 topic-pill ovals), re-verified at 390px and desktop.
• Publication REVIEW_QUEUE pass — 48 repos with forbidden-term hits all dispositioned as benign subject matter (zero learner-assessment-outcome leaks); the earlier 23 schema deviations were resolved by the evidence-hygiene pass; the svizor42/de_zoomcamp star-attribution question resolved (no star counts ship anywhere).

Remaining

1. Merge sequencing: this branch's migrations 0020/0021 stack on the tours/testimonials 0019_add_tour_testimonial_placement (currently untracked in the shared tree). Land that on main first, then git merge --no-ff issue-416-gallery-enrichment and push. Never push the branch alone — its 0020/0021 reference their 0019.
2. Production ingestion via the runbook §22 importer after merge.
3. Cohort cadence decision ("once after each cohort finishes") — Phase 2 / PM grooming.

Acceptance criteria (status)

• Migrations are additive only, reversible, contain no data.
• Fetch/validation fixtures (SSRF, unsafe redirect, path traversal, oversized response, secret/PII-shaped content, prompt-injection-shaped content) fail closed.
• Fetch budgets enforced and tested (≤20 files, ≤100KiB/file, ≤500KiB/submission).
• Fingerprinting is deterministic; re-running an identical selection is a safe no-op.
• Dry-run selection touches only eligible submissions; no model-provider call, no public write.
• Job/run manifest provenance without emails, credentials, full query strings, or raw repository content.
• Per-course dimension research grounded in fetched evidence, with an "other" bucket per dimension kind.
• No public page, template, or view changes — superseded by the owner's explicit add-to-website direction (see Summary).

References

• _docs/design/specs/project-gallery-enrichment.md (binding spec, full acceptance-test list)
• _docs/architecture/database-only-content.md
• _docs/architecture/app-boundaries.md
• https://datatalksclub-projects.streamlit.app/ (product-idea reference only)

Read _docs/design/specs/project-gallery-enrichment.md and the runbook §22, then inspect migration 0019 and the issue-416-gallery-enrichment branch. Confirm migration ordering before merging, run the production importer as documented, and verify the gallery integration and importer workflow remain intact; the cohort cadence is a separate PM decision.