DataTalksClub / DataTalksClub/website
Cut public content reads and projections over to direct-sync authority
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Parent epic: #38.
Source/projection baseline: #253.
Direct-sync schema/runner/ingress predecessors: #273, #274, #275.
Active-content validation consumer: #272.
Product outcome
Move each owner-approved public content family from its current checked/baked, database, or historical-reader authority to accepted direct-sync current rows only after exact source-bound parity is proved. Preserve every established route, redirect, query, rendered field, asset byte, search/graph contract, cache policy, canonical, structured-data field, feed, sitemap, and safe failure behavior.
The cutover is source/family-scoped. It is not a site-wide content activation, a mutable ContentRelease pointer, a dual writer, or a request-time fallback. Direct-sync draft/soft-deleted/disabled/unsafe rows remain non-public by construction.
PM status — BLOCKED / retain needs grooming and HUMAN gate
This issue is a bounded blocked contract, not an engineer handoff. Keep needs grooming, human, and decision until the product owner approves one exhaustive versioned public-authority/cutover manifest and every named predecessor is accepted.
The raw intake's premise needs correction: current public authority is heterogeneous and is not generally the staged ContentRelease selector.
content.public_data.public_projection()loads the checkedcontent/public_projection/manifest/tree and supplies homepage content, Blog, Podcast/transcripts, Books, People, Events content, Wiki pages/search/graph, media, route aliases, feeds, and section-sitemap inputs. It is process-cached and source-pin/count/digest validated.- public Event pages are hybrid: checked event content is combined with database-owned canonical identity, Q&A, registration totals, and other event-domain behavior. This issue must never replace those event-domain owners with editorial direct rows.
content.docs_projectionand checked docs assets independently own/docs/pages/navigation/assets.content.faq_dataand the checked FAQ projection/assets independently own/faq/.- public course/cohort pages are database-owned by
courses; the checked 12-course projection also feeds bounded homepage/sitemap/review behavior. The manifest must state which course-owned surfaces, if any, are direct-sync content rather than course-domain reads. content.queries.resolve_public_document/assetselects activeContentReleaserows, but current main uses it only in fixture/test surfaces; it is not the reader behind the public route families above.
Engineering must not replace all of these with one guessed selector or describe a passing fixture query as a public cutover.
Normative authority and vocabulary correction
_docs/PROCESS.md: issue lifecycle, dependency ordering, versioned verification, independent tester screenshots, PM acceptance, and HUMAN handling._docs/specs/open-decisions.mddecision 1 and #226: direct sync is source lock → immutable checkout → bounded parse/direct upsert → source-scoped draft/soft-delete → immutable log/status. It explicitly rejects ordinary site-wide candidate/activate/rollback._docs/specs/01-platform-architecture.mdand_docs/architecture/app-boundaries.md: public requests perform no source/network work; public mutations stay in owning services; after-commit side effects are durable._docs/specs/02-url-link-seo-compatibility.md: exact routes, query behavior, redirects, canonical/SEO metadata, sitemaps, assets, cache isolation, parity, and rollback-window requirements._docs/specs/03-github-content-and-people.md: repository ownership, parsing/sanitization/provenance, relations, public route/link/search/graph/assets, freshness, and safe diagnostics remain authoritative. Its staged content candidate/pointer/rollback wording is stale where it conflicts with #226._docs/specs/04-courses-and-cohorts.md: Course/Cohort and learner/registration ownership cannot be absorbed into content._docs/specs/07-security-privacy-operations.md: last-known-safe behavior, bounded freshness, monitoring, redaction, and no provider work on public reads._docs/specs/09-migration-rollout-roadmap.mdand_docs/specs/10-verification-strategy.md: expand/reconcile/cutover/contract, portability, exact-image evidence, URL/SEO/search/graph/cache/browser/rollback gates.- #219 owns preservation/repair of historical
ContentReleasemigration state. Historical rows are compatibility provenance, not future direct authority. - #253 owns the final reproducible source-to-checked-projection bytes, pins, counts, transforms, manifest/tree, and semantic census. No
face8e4or currently known stale-source digest may be blessed here. - #273 owns the additive current-row/path-claim/source-status schema and public-safe visibility invariant.
- #274 owns deterministic record-atomic direct upsert, conditional stale sweeps, partial recovery, and final state/evidence digests.
- #275 owns authenticated source ingress, immutable checkout, leases/locks, reconciliation, and freshness.
- #272 owns the immutable direct-sync active-content validation receipt. Its accepted receipt is required before a family may be declared live authority.
- #44 owns unified search and graph behavior. Its staged vocabulary must be reconciled with #226; the search/graph authority unit cannot cut over before #44 is re-groomed and accepted.
- #109 owns positive edge caching and durable invalidation. If it remains unaccepted, this issue may preserve existing zero-TTL/no-store behavior but must not introduce or claim positive-cache operation.
Implementation must amend stale staged-release passages in specifications 01, 03, 06, 09, and 10 only where needed to describe the accepted direct-read cutover. It must not erase historical migration facts.
Required HUMAN public-authority/cutover manifest
The owner must approve one canonical, code-reviewed manifest with schema version, stable path, canonical JSON bytes, SHA-256, approver, and approval timestamp. It is build/test authority, not a mutable database setting or operator feature flag.
For every possible source and public family—preferred content, legacy main, Docs, FAQ, Podwiki, course-owned sources, and any explicit retained checked/baked source—the manifest must state:
- stable source UUID/key, exact repository, branch, adapter/contract versions, ownership namespaces, and permitted record kinds;
- disposition:
direct_sync,retained_checked,course_domain,event_domain,retired, or another explicitly approved closed enum; - current reader and artifacts/tables; future reader and exact direct source/path/visibility query invariant;
- exhaustive owned canonical paths, aliases, hubs, details, assets, feeds, sitemaps, search documents, graph nodes/edges, homepage/relationship dependents, and cross-source joins;
- the exact accepted #253 source commit/tree, artifact/tree/count/provenance digests and any reviewed parity transform;
- the direct-sync successful commit/tree/log/evidence/visible/draft/ownership digests and accepted #272 receipt requirements;
- cutover unit, dependency edges, total order, hold point, responsible owner, observation window, and quantitative go/no-go/rollback thresholds;
- exact retained rollback reader/artifact/application image, schema-compatibility window, maximum rollback age, post-cutover-write treatment, and final retirement condition;
- cache class, TTL/stale policy, query-key policy, ETag/content-type/length behavior, invalidation/change-set ownership, and safe failure policy;
- search/graph build identity and coupling, asset custody/storage/byte identity, route/alias ownership, canonical/SEO/robots/sitemap/feed expectations, and representative browser/screenshot routes.
No runtime discovery, moving branch, “all content,” guessed route prefix, or unbounded wildcard ownership is allowed. Missing, duplicate, overlapping, contradictory, or unclassified path/source/family ownership blocks all affected cutover. The manifest must explicitly classify sources/families that remain checked or domain-owned; silence is not approval to migrate them.
Dual-read parity phase
“Dual read” means a side-effect-free comparison harness or shadow observation while the current reader remains the sole response authority. It never means dual writing, mixing fields from two readers, fallback on a miss/error, or serving one viewer from a different authority.
For one frozen manifest unit, the parity run must bind:
- exact application source SHA, runtime/schema identities, database snapshot/checkpoint, manifest digest, #253 projection identities, direct source/log/evidence identities, and evaluation time/locale;
- only a terminal accepted direct-sync state with no current partial/failed/running/conflicting evidence and an accepted #272 receipt within its policy;
- two clean independent evaluations producing byte-identical canonical evidence.
The comparison inventory is exhaustive, not representative-only:
- all canonical paths, aliases, unknown/malformed paths, GET/HEAD/unsafe methods, exact accepted/unknown/duplicate/malformed queries, pagination/filter/season behavior, redirects and raw-query preservation;
- hub/detail/homepage membership, ordering, dates/timezones, pagination, empty/missing/error/fallback states, relation targets, fragments, and no-JavaScript reachability;
- every public DTO/render input: stable key, content kind, source provenance, title/summary/body/sanitized HTML, timestamps, images, alt text, canonical/SEO fields, edit/source links, and adapter-owned metadata;
- exact asset inventory and bytes, SHA-256, path, storage identity, content type, size, GET/HEAD/range/error behavior, and absence of remote request work;
- search corpus/query grammar, filters, ranking/tie order, fragments, JSON schema, empty/malicious/Unicode/overlong results, privacy and no-store behavior;
- graph nodes/edges/weights/order, deep links, deterministic JSON bytes, graph HTML, and unavailable/degraded behavior;
- route claims, one-hop aliases, canonical/alternate/OG/Twitter/JSON-LD/breadcrumb values, robots, root/section sitemaps, feeds, lastmod, internal/external links, and development noindex policy;
- response status, content type, cache directives,
Vary, ETag/conditional behavior, credential/private bypass, 404/405/5xx behavior, and query cache isolation.
Canonical parity evidence stores bounded counts and digests plus safe mismatch codes/path digests, never source bodies, rendered private data, credentials, provider payloads, exception text, or production records. A mismatch is not waived because the page “looks right.” Every difference must either be eliminated or be an explicit owner-approved transform already bound by #253 and the authority manifest.
Cutover ordering and authority switch
- Accept #219, #253, #273, #274, #275, #272, the authority manifest, and any family-specific predecessor such as #44 before that unit starts.
- Recompute complete dual-read parity on the exact candidate base/head and freeze the report.
- Cut only one manifest unit or one explicitly coupled dependency group. A code-owned reader selector changes from its named current authority to its named direct authority. It is reviewed and deployed with the application; there is no runtime operator pointer, hidden activation endpoint, or request-controlled source choice.
- Direct selectors require enabled source plus accepted published visibility/path ownership and must exclude draft, soft-deleted, disabled, cross-source, unresolved, quarantined, historical, partial-diagnostic, and raw sync rows by default.
- A unit becomes live only after exact-image local verification, independent tester/PM acceptance, deployment, healthy/ready identity, public smoke, parity re-observation, cache/search/graph/asset/SEO checks, and its HUMAN observation window pass.
- The next dependent unit cannot start its authority switch until the prior unit reaches its recorded hold point. Independent non-overlapping units may be prepared in isolated lanes but may not bypass their manifest order.
- Historical/baked readers remain read-only and reachable only by the exact preapproved rollback image during the compatibility window. They are never request-time fallback.
- Contract/removal of checked projections, historical selectors, staged services, or tables is a later separately groomed child after every consumer and rollback window is closed.
A route must never combine a direct detail with stale checked relationships, search/graph identity, media, alias, sitemap, or homepage entry unless the manifest explicitly models that bounded temporary state and parity proves it. Coupled outputs switch together.
Cache and change propagation
Cutover preserves the accepted route-cache registry and request classifier. It does not broaden a public cache class, add query variants, or treat cache/robots as authorization.
- Current zero-TTL/no-store/private behavior remains exact unless #109 is accepted and the manifest names the new class.
- A query-bearing, search, authenticated, preview, management, error, malformed, unsafe, or credential-shaped response remains private/no-store according to specification 02.
- For any positively cached route, each committed direct mutation exposes an immutable bounded changed-path/dependent-family receipt. A durable after-commit service invalidates/coalesces every affected detail, hub, alias destination, feed, sitemap, search/graph surface, and stable asset before the source's accepted stale bound expires. Partial runs invalidate successful direct mutations truthfully; rejected records retain prior bytes.
- Provider/invalidation failure cannot alter direct rows or cause fallback. It retries durably, reports safe status, and serves only within the approved stale policy; missing evidence fails closed to zero TTL.
- Stable asset paths may change bytes only under the manifest's checksum/custody and invalidation rule. Otherwise use an immutable versioned storage identity. A database asset record with missing/mismatched retained bytes is non-public and blocks cutover.
Search and graph
Search and graph are derived projections, not public content activation pointers. The accepted #44 design must build from one manifest-bound snapshot of public-safe direct rows, validate routes/fragments/relations and deterministic ordering, then atomically replace only the derived search/graph index after success.
A search/graph build failure keeps the prior derived projection available only under its documented degraded/freshness policy and never rolls direct content rows back. It records the source-state and build digests and alerts. It cannot silently mix an unbounded set of source revisions, expose draft rows, or claim complete status. Search queries remain private/no-store and retain current grammar, ranking, JSON, filter, zero-result, Unicode, malicious-input, and accessibility behavior. Wiki graph JSON/HTML/deep links and episode graph behavior remain exact.
If the HUMAN manifest retains checked search or graph temporarily, that disposition and maximum mismatch/freshness window must be explicit. #276 remains open until every accepted search/graph unit reaches its final manifest disposition.
Routes, assets, SEO, and rendered behavior
- Preserve the exact explicit Django route families; do not replace them with a database catch-all or global slash/extension normalizer.
- Keep canonical hub/detail paths, one-hop aliases, event numeric identity, podcast ID/slug behavior, People-catalogue 404 policy, Docs trailing slashes, FAQ anchors/JSON, Wiki special/search/graph/feed/assets, course routes, and unsafe-method behavior exactly as specification 02 records.
- Global path claims and the authority manifest must agree before a row is public. Cross-source document/asset/alias collision fails closed.
- Templates receive the same code-owned view DTOs; raw ORM models, source Markdown/YAML/HTML, diagnostics, sync logs, or ownership metadata are never passed through.
- Preserve sanitized HTML and link policy, production-origin canonicals, metadata, structured data, images, alt/fallback, breadcrumbs, internal links, feeds, sitemaps, robots, lastmod, and development noindex. This issue ships no SEO experiment or content/design rewrite.
- Public requests perform no GitHub/provider/network checkout, Markdown/source parsing, projection rebuild, content mutation, sync/status mutation, cache invalidation, or job enqueue.
Rollback compatibility
Rollback is an exact application/reader-manifest rollback within the approved schema/artifact compatibility window. It is not ContentRelease.activate, an active pointer swap, arbitrary older-SHA sync, database rewind, hard delete, or automatic reversal of direct-sync mutations.
The HUMAN manifest must prove the retained reader can still run against the expanded schema, bind its exact checked/historical artifacts, and state how direct-sync changes made after cutover are treated. If rollback would omit newer public-safe direct content beyond the approved threshold, it is unsafe and must halt/escalate rather than silently serve an unbounded stale snapshot. Rolling forward a corrected source commit remains the normal recovery.
Rollback verification must show no dual writer, no direct-row mutation, no staged service resurrection, no route/SEO/cache-policy drift, no search/graph/assets mismatch, and no loss of event/course domain data. Final removal is forbidden until the observation and rollback windows pass.
Acceptance criteria after blockers are resolved
- [HUMAN] The owner-approved authority/cutover manifest is checked, canonical, digest-bound, exhaustive across all possible sources/families, and has no missing/overlapping ownership or unresolved cutover/rollback value.
- [AUTO] #219, #253, #273, #274, #275, #272 and every manifest-named family dependency are accepted; the issue body replaces their placeholders with exact model/service/receipt/pin/digest interfaces before
needs groomingis removed. - [AUTO] A current-authority inventory test proves every route/view/helper/search/graph/asset/sitemap/homepage consumer is classified and fails on new unclassified readers or direct ORM bypass.
- [AUTO] For each unit, two clean exhaustive dual-read evaluations produce the same evidence digest and zero unexplained mismatch across records, ordering, routes, queries, redirects, assets, search, graph, cache, SEO, feeds, sitemaps, error states, and response DTOs.
- [AUTO] Direct public selectors enforce enabled-source, exact ownership/path claim, published visibility, accepted source/evidence state, and exclusion of draft/soft-deleted/disabled/quarantined/cross-source/historical/raw rows in portable bounded-query tests.
- [AUTO] Unit switching is deterministic and code-owned; missing/unknown/conflicting manifest, invalid #272 receipt, stale/partial/failed current evidence, selector mismatch, or post-plan drift fails closed before release. There is no request-time fallback, dual writer, runtime activation pointer, or provider work.
- [AUTO] Search/graph rebuild and swap meet accepted #44 parity, deterministic/failure/freshness contracts and never expose drafts or roll back direct rows. If retained checked, the exact approved disposition is tested instead.
- [AUTO] Assets preserve complete inventory, bytes/digests/content type/size/path/GET/HEAD/range/error behavior and fail closed on missing/mismatched storage; routes/aliases/path claims have no collision, chain, loop, soft 404, or catch-all fallback.
- [AUTO] Cache/header tests preserve the accepted registry for anonymous/credential/query/error/unsafe cases. Any positive cache adds complete after-commit changed-path/dependent invalidation, partial-run, retry/coalescing, stale-bound, poisoning, and missing-evidence zero-TTL tests.
- [AUTO] Canonical/OG/Twitter/JSON-LD/breadcrumb/internal-link/fragment/robots/root-and-section-sitemap/feed/lastmod parity passes for every final and alias; development stays noindex/nofollow and no source family introduces an SEO experiment.
- [AUTO] Event and course boundary tests prove direct editorial reads cannot create, replace, infer, or leak Event identity/Q&A/registration data, Course/Cohort/learner state, accounts, email, Studio data, or private joins.
- [AUTO] Fault/concurrency coverage includes current-reader drift, direct-row tamper, source disable, status/receipt transition, partial run, missing asset, search/graph failure, cache invalidation failure, selector/manifest conflict, concurrent public read, and rollback-image compatibility with no mixed or fallback response.
- [AUTO] Focused content/docs/FAQ/Wiki/event/course/query/source-adapter/search/graph/cache/link/SEO tests, migration drift, portability, quality/type, complete Django, full Playwright, container, manifest/tree reproducibility, and versioned verification-plan checks pass. This shared cross-surface cutover cannot use a narrow backend-only browser disposition.
- [AUTO] Independent tester captures and inspects desktop (~1440×900) and mobile (~390×844) screenshots under
.tmp/screenshots/issue-276/for homepage plus each cut unit's hub/detail and applicable Docs, FAQ, Event, Course, Wiki search/graph, asset/fallback, empty/no-result/error, and no-JavaScript states. Images must show the expected responsive page, correct content/media/canonical behavior, no overflow, no debug/error page, and no sensitive data. - [HUMAN] In the authorized environment, one manifest unit at a time passes exact-image migration/readiness/identity, public GET/HEAD/redirect/query/search/graph/asset/SEO/cache smoke, parity re-observation, monitoring, and its full observation window before the next unit advances.
- [HUMAN] A controlled rollback rehearsal restores the exact approved reader/image without direct-row mutation, staged activation, provider fallback, route/SEO/cache drift, or event/course data loss; forward recovery then returns to the same accepted direct state.
- Engineer/tester/PM/on-call reports record exact base/head, graph/plan/report/artifact/manifest/parity/receipt digests, commands/counts, screenshot paths, unit/hold-point outcome, and every rerun/reuse/skip/not-applicable disposition. HUMAN evidence remains redacted and issue-scoped.
Explicit non-goals
- No engineering while
needs groomingremains and no owner decision inferred from current files, stale issue text, or source repository layout. - No source-repository repair or pin approval (#253), direct-sync schema/runner/ingress implementation (#273–#275), active-content receipt design (#272), or historical migration repair (#219).
- No broad public-query rewrite, database catch-all router, generic serializer, dual writer, request-time fallback, mixed checked/direct DTO, runtime activation pointer, hidden preview/activate/rollback endpoint, or staged
ContentReleaseresurrection. - No arbitrary older-SHA sync, automatic database rollback, hard delete, cross-source adoption, fuzzy identity join, or use of email/display name/title as identity.
- No route, search UX/ranking, graph schema, content, design, canonical, SEO, cache-class, sitemap, feed, or asset-path redesign.
- No replacement of Event/Course/account/learner/registration/Q&A/email ownership by editorial content rows.
- No public request source/network access, parsing, sync, mutation, rebuild, invalidation, or job creation.
- No early deletion of checked projection files, historical releases/children/path claims, compatibility selectors/services, tests, or rollback artifacts.
- No production data, AWS/provider mutation, commit, push, merge, deployment, or Actions run during PM grooming.
Grooming exit
PM may remove needs grooming only after:
- the HUMAN manifest is approved with exact canonical bytes/digest and every source/family/cutover/rollback value;
- #219, #253, #273, #274, #275, and #272 are accepted with exact interfaces and identities;
- #44 is re-groomed/accepted for any search/graph unit and #109 is accepted for any positive-cache unit;
- PM rebases the current-reader audit against then-current main, replaces every placeholder with exact selectors, paths, counts, commits, receipts, reports, thresholds, and screenshots, and confirms the issue is one safely deliverable lane or splits it into ordered children.
Until then, no engineer may dispatch #276.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with _docs/PROCESS.md, _docs/specs/open-decisions.md, and the authority rules in #253 and #273-#275. First verify that #219, #253, #273, #274, #275, #272, and any family-specific predecessor are accepted, then identify the code-owned readers and manifest units; done requires an approved versioned authority manifest, exact parity evidence, and the stated HUMAN cutover gates.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend, backend-api-design
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 20/100