Adopt secure direct-sync content ingestion and management

Parent epic: #4
Owner architecture decision: #226 (closed)
Historical migration gate: #219
Projection/source baseline: #253
Restore evidence consumer: #272

Product outcome

Replace the repository’s staged ContentRelease prepare/ready/activate/rollback implementation with the owner-approved direct-sync architecture:

allowlisted source lock
  → authenticated immutable checkout
  → bounded parse/classify/validate
  → source-owned direct upsert
  → source-scoped draft/soft-delete transition
  → immutable SyncLog + source status

GitHub remains read-only from the website. Draft/unpublished state is the public visibility boundary. There is no site-wide content candidate, activation pointer, or rollback graph, and direct sync does not pretend to provide one atomic snapshot across every source or content type.

This issue is now the parent epic for the bounded delivery slices below. Child implementation commits use Refs #38; #38 closes only after the accepted source inventory has migrated, public authority has cut over, management parity is delivered, and the obsolete staged path is safely retired.

PM status — returned to grooming / do not dispatch

needs grooming is required. The old issue body and title prescribed a rejected architecture, while current code and several specifications still implement it. No engineer may start a broad #38 lane or reinterpret current ContentRelease behavior as direct sync.

The execution architecture and safe partial-recovery rules are derivable from #226 and accepted repository security contracts. Two owner/HUMAN decisions still block the complete epic:

1. the exhaustive source rollout manifest—among DataTalksClub/content, legacy main, docs, FAQ, Podwiki, and course-owned sources—which repositories become live direct-sync sources in which order, which remain checked/baked, and the exact accepted branch/adapter/ownership partition for each; and
2. the public-authority cutover—when database direct-sync rows replace checked projection reads for each route/search/graph/asset family, and how their source commits must match the final reproducible #253 projection during dual-read/reconciliation.

Those decisions do not justify inventing a staged release graph. They must be recorded before the source-specific runner/cutover children are groomed.

Normative authority and superseded vocabulary

• _docs/PROCESS.md: parent/child lifecycle, dependency ordering, migrations, independent testing, screenshots, and release evidence.
• _docs/specs/open-decisions.md, decision 1 as resolved by #12/#226: the direct-sync flow above is authoritative and explicitly rejects ordinary ContentRelease candidate/activation/rollback.
• _docs/specs/01-platform-architecture.md and _docs/architecture/app-boundaries.md: one Django deployment, business mutations in owner services, scalar durable jobs, after-commit side effects, and one-way dependencies.
• _docs/specs/03-github-content-and-people.md: source ownership, immutable checkout, parser/sanitizer/provenance, route/link/search/graph safety, no database override, and freshness remain authoritative. Its ContentRelease, preview, activation, pointer, and rollback passages are stale where they conflict with #226.
• _docs/specs/06-studio-and-admin-api.md: capability, UUID, optimistic revision, idempotency, masking, audit, pagination, and Studio/admin parity remain authoritative. Candidate preview/activate/rollback actions are removed from this product surface.
• _docs/specs/07-security-privacy-operations.md: GitHub content freshness below 15 minutes, safe diagnostics, last-known-safe behavior, monitoring, and no secret/PII leakage.
• _docs/specs/09-migration-rollout-roadmap.md and _docs/specs/10-verification-strategy.md: expand/reconcile/contract, cross-database migration tests, fault/concurrency/crash cases, exact-image release evidence, and fail-closed operations.
• #156 supplies the accepted provider-neutral raw-body HMAC/delivery fence. It does not supply a GitHub endpoint or sync.
• #103 supplies the current network-free dtc-content adapter against an already verified checkout. Its staged preparation wrapper is not retained.
• #219 preserves and repairs historical staged-release rows. Those rows are migration provenance only under the new architecture.
• #253 must provide a reproducible checked projection/source baseline before any public-authority cutover.
• #272 is downstream: it consumes accepted direct-sync status/evidence. It does not define #38’s models or authorize live sync.

Canonical direct-sync state model

ContentSource

One immutable UUID/stable-key row owns one exact allowlisted repository/branch/adapter/mount/path/resource policy. Secrets remain references, never values. Direct-sync bookkeeping must include:

• revision and enabled state;
• last attempted commit and UTC time;
• last successful commit and UTC time, changed only by terminal success;
• last remote-head observation commit/time;
• current last_sync_status and immutable last-log reference;
• source lock owner/fence/time and one pending-follow-up bit;
• exact freshness target (14 minutes for the existing dtc-content contract, satisfying the below-15-minute specification bound);
• direct-sync contract, adapter, parser/rendering, selection, and source-status evidence digests required by #272.

Repository, branch, adapter, mount, and allowlists are changed only through the future high-risk management child with revision/idempotency/reauthentication. A sync request cannot supply or widen them.

SyncLog

One UUID log row represents one source/run/input identity. Statuses are exactly the #226 reference workflow states:

• queued → accepted durable work not yet leased;
• running → exact source lock/fence held and processing begun;
• success → every selected ownership partition completed with zero rejected records and eligible stale transitions completed;
• partial → one or more records/partitions failed while at least one bounded direct mutation committed; recovery is required;
• failed → the run could not produce an accepted completion, including checkout/global validation/transaction/system failure;
• skipped → exact unchanged remote head after a prior complete success, or a second request coalesced behind the held source lock. The reason code distinguishes these cases; a coalesced request also sets the one pending-follow-up bit.

success, partial, failed, and skipped are terminal and immutable. Unknown/backward transitions fail closed. A queued log becomes the running log rather than creating a duplicate. A stale queued/running lease becomes terminal failed through a fenced watchdog/reconciler; it is never silently overwritten.

Each log binds schema/contract versions; source; trigger (webhook, reconcile, manual, or follow_up); safe delivery/request/idempotency digests; requested/started/finished times; immutable commit and tree identity; parser/render/selection policy digests; bounded created/updated/unchanged/drafted/rejected counts; partition-completion and visible/draft/ownership state digests; redacted allowlisted diagnostics; and one canonical log/evidence digest. It stores no raw body, signature, delivery ID, repository payload, source content, secret, token, exception text, or unbounded per-record values.

Source status and #272 mapping

ContentSource.last_sync_status mirrors only the newest accepted run transition. Freshness is an orthogonal derived state, not another mutable success value.

• success and a verified unchanged-head skipped map to #272 complete while freshness remains within policy.
• queued/running map to in_progress.
• partial maps to partial.
• failed maps to failed.
• missing evidence maps to unavailable; duplicate/contradictory current evidence maps to conflicting.

A coalesced lock-skip is not complete evidence. A later partial/failed/in-progress run invalidates a previously complete #272 observation until recovery succeeds. The last successful commit remains available for diagnostics but never masks current unsafe status.

Direct application, atomicity, and partial recovery

The owner accepted direct upsert and its mixed-version tradeoff. The safe target boundary is:

1. Authenticate/fence the request, fetch one immutable allowlisted commit, and perform repository-wide path/resource/classification checks before database content mutation.
2. Parse in deterministic code-owned ownership partitions. A partition is one adapter-defined content kind/namespace with an exhaustive inventory and collision policy.
3. Apply one record plus its owned dependent relations/assets atomically. No network, checkout, provider, parser process, or external asset write occurs inside that transaction.
4. A valid record may commit even if a later record fails. That accepted partial visibility is reported as partial; it is never relabelled success or hidden behind an old status.
5. An invalid/rejected existing record retains its prior public-safe row unchanged. A new invalid record creates no public row. Error input is not persisted in diagnostics.
6. Run a partition’s draft/soft-delete sweep only after its complete inventory parsed and every record in that partition reached an accepted result. Any classification, collision, parse, or apply error suppresses that partition’s stale sweep, so omission under partial evidence cannot unpublish unrelated prior content.
7. Records absent from a complete source-owned inventory transition through the adapter’s existing non-public state (is_published=false, status=draft, or an explicit source-owned soft-delete flag); they are never hard-deleted by sync. Cross-source rows and manually owned operational fields are never changed.
8. Mark success and advance last_successful_commit only after all eligible sweeps and final source/evidence digests commit. A crash before this marker leaves running/failed evidence and is recoverable by replay.

Recovery is a full deterministic replay of the same immutable commit. Already applied records compare unchanged, failed records retry, suppressed sweeps run only when their partition becomes complete, and exact replay converges without duplicate rows, relations, assets, logs, jobs, or audits. No separate mutable per-record resume cursor is authoritative; record-level idempotency makes restart-from-beginning safe.

There is no automatic rollback and no sync of an arbitrary older SHA. Correction is a new accepted branch-head commit (including a Git revert committed upstream) or exact replay of the current failed/partial commit. An out-of-order webhook, older observed head, operator parameter, restored ContentRelease, or “last known good” pointer can never roll database state backward implicitly.

Ownership, uniqueness, and collision rules

• Every direct row carries immutable source UUID/stable key, source record key, adapter/content kind, normalized source-path digest, last applied commit, source checksum, and published/draft ownership state.
• (source, content_kind, stable_record_key) is unique. Exact replay updates that row; it never creates a second identity.
• Exact public paths, stable asset paths, and promoted cross-source identities remain globally unique. Duplicate identities within one checkout, two records claiming one route, or two sources claiming one owner/path fail before the conflicting write and make the affected run/partition unsafe.
• One source cannot update, draft, delete, adopt, or resolve a row owned by another source. Source-transfer requires a separately reviewed manifest/migration, not sync-time guesswork.
• Person/account/member/course identities never join by display name or email. Existing exact source keys and adapter-owned relation policy remain the only content relationship authority.
• Public readers always filter the adapter’s accepted published/non-draft contract; a raw direct-sync row, partial diagnostic, historical release child, or disabled source is never public by default.

GitHub ingress, idempotency, locking, and reconciliation

• Reuse #156: verify sha256= HMAC over exact bounded raw bytes in constant time before parsing; resolve the configured secret reference without storing/logging it; then require an allowlisted GitHub push, exact repository, branch ref, and full commit identity.
• The delivery fence key is the configured namespace plus safe delivery UUID/digest, bound to the raw-body digest. Exact replay returns the prior safe result; the same key with changed bytes conflicts; concurrent duplicates converge. Provider IDs/payloads are not copied into SyncLog.
• Delivery fencing, source+commit/run idempotency, durable-job deduplication, and manual Idempotency-Key are distinct layers. Every durable job receives scalar UUID/SHA/digest values, uses a fenced lease, and wakes only after commit.
• Exactly one source lock exists. A concurrent request sets one pending-follow-up bit and produces/coalesces safe skipped evidence. Lock release atomically consumes that bit and emits at most one follow-up job after commit.
• Scheduled reconciliation reads only configured source identities, observes immutable remote head, skips exact unchanged complete state, and queues newer head. Older/out-of-order delivery never applies. A missing webhook is detected soon enough to meet the source’s 14-minute target.
• Checkout is isolated and bounded: exact commit/tree, ancestry/ref validation, no repository code execution, traversal/symlink/submodule escape, unsafe file/protocol/HTML/script, alias/entity bomb, file/byte/depth/node/time overrun, and missing required reference all fail closed.
• GitHub/network outage, missing credential reference, checkout failure, invalid content, worker crash, or reconciliation failure changes status/evidence and alerts; it does not invent content, enable another source, or fall back to a direct provider path.

Historical ContentRelease migration and rollback compatibility

Current main still stores documents/relations/assets beneath ContentRelease, selects active_release, swaps ActiveContentPath, and exposes prepare/activate/rollback services. Direct sync must use expand → reconcile → cut over → contract:

1. #219’s 0004/0005/0006 repair and actual applied-state classification must be accepted first.
2. Add new direct-sync source/log/status/ownership storage without dropping or rewriting any historical source, release, child, path-claim, digest, status, primary key, or pointer.
3. Deterministically copy/reconcile each enabled source’s one historically active public state into new source-owned current rows, preserving route/content/asset/relation bytes and provenance. Missing, duplicate, conflicting, unsupported-digest, or non-active ambiguity quarantines the source and blocks cutover.
4. Run direct sync and historical reads in a migration-only comparison phase. No dual writer and no request-time fallback between mismatched stores.
5. Cut each accepted public family only after exact counts/digests/routes/search/graph/assets and query behavior match its approved source/#253 authority.
6. Retain historical releases/children read-only for the approved compatibility/audit window. The old activate/rollback entry points become unavailable and cannot mutate direct rows.
7. Drop staged tables/fields/services only in a later contract migration after every consumer, query, test, Studio/API action, job, receipt, and rollback reference is removed and independent migration evidence passes.

Historical rollback means only that pre-cutover data remains readable/auditable while the migration is reversible before contract. It does not preserve an operator “activate old release” product feature. Post-cutover correction follows forward source history as defined above.

Security, diagnostics, and operations

• Source configuration and manual sync require accepted content-management capabilities, optimistic revision, exact idempotency, reauthentication/confirmation where #32/#33 require it, and immutable redacted audit.
• Webhook ingress is anonymous only through the exact signature/source/event contract. It has no session, CSRF bypass beyond that bounded endpoint, arbitrary repository/branch/path/SHA, preview, activation, rollback, force-old-commit, or source-configuration capability.
• Diagnostics use allowlisted codes, path/field pointers bounded and sanitized to source-relative safe grammar, fixed counts/digests, and safe timestamps. They exclude raw body/signature/payload, secret refs/values, tokens, checkout paths, source content, HTML/Markdown/YAML, email/member/profile/course activity, IP/header/query, SQL, provider response, exception/stack text, and production data.
• Status/freshness/failure/partial/lock age/pending follow-up/last successful and attempted commit digests are observable. High-cardinality source paths, commits, delivery IDs, and error text are not metric labels.
• No public request performs GitHub/network work. No database transaction spans network/provider/subprocess work. Draft/unpublished filtering and safe cache behavior do not depend on an external provider being available.

Bounded child delivery plan

Slice A — direct-sync schema and historical expand/reconcile

Execution-disabled models/migrations for SyncLog, source status/evidence, and source-owned current rows; deterministic preservation/copy of historical active state; no network, sync runner, public cutover, or UI. Depends on final #219 acceptance and the owner source rollout manifest.

Slice B — deterministic direct-upsert runner and partial recovery

Network-free service consuming an already verified checkout; exact states/transitions, ownership partitions, per-record atomicity, safe partial behavior, conditional draft sweeps, replay/idempotency, evidence digests, and adapter contract. First source-specific adoption requires #253’s repaired immutable pin/baseline. No webhook, scheduler, public cutover, or UI.

Slice C — secure GitHub ingress, checkout, jobs, and reconciliation

Reuse #156, add exact GitHub push parsing/source authorization, immutable checkout, delivery/run/job fences, one source lock/follow-up, scheduled head reconciliation, freshness/alerts, and safe outage behavior. Depends on accepted Slices A/B and source-specific credential/branch policy. No public-reader or management surface.

Slice D — projection/search/graph/assets and public-reader cutover

Source-by-source dual-read comparison followed by explicit query ownership cutover; deterministic projection/search/graph/asset rebuild/invalidation only after commit; no staged pointer. Depends on #253, accepted source runner, and the HUMAN public-authority manifest. Render/browser/screenshots are required.

Slice E — Studio and /api/v1/admin/ content management parity

Status/history/freshness/diagnostics, source configuration, manual sync, safe retry, and edit-on-GitHub using shared services and #32/#33 capability/audit/idempotency conventions. Remove candidate preview/activate/rollback actions. Depends on #32/#33 and Slices A–C; public cutover is not required for read-only status but is required before claiming end-to-end source authority. Render/browser/screenshots are required.

Slice F — staged-path contract removal and spec/runbook reconciliation

Remove obsolete activation/rollback services, queries, tests, and finally staged schema only after all consumers migrate; reconcile specs 01/03/06/08/09 and runbooks to #226 and accepted source inventory. Depends on accepted Slices A–E and #272 interface adoption. No product behavior may be deleted ahead of its accepted replacement.

These must be separate raw child issues filed by the orchestrator and independently PM-groomed. The parent itself is not an engineering lane.

Parent acceptance criteria

• Owner records the exhaustive source rollout and public-authority/cutover manifests; no source, branch, adapter, ownership partition, or baked/direct boundary is inferred.
• #219 is fully classified and accepted before the first direct-sync schema migration; every historical row/digest/pointer remains preserved through expand/reconcile.
• ContentSource/SyncLog implement the exact status, provenance, count, digest, lock/follow-up, freshness, terminal immutability, and #272 mapping above.
• Network-free runner proves per-record atomic direct upsert, source ownership, safe conditional draft/soft-delete, partial visibility, deterministic replay/recovery, collision denial, no older-SHA rollback, and no public draft leak.
• GitHub webhook/reconciliation proves authenticate-before-parse, exact allowlists, immutable checkout, four idempotency layers, one source lock/follow-up, out-of-order denial, below-15-minute recovery, outage/crash behavior, and complete redaction.
• Each accepted source has exact source/adapter/branch/path/resource/provenance evidence and transitions from baked/historical state without a route/content/search/graph/asset regression.
• Public query/projection/search/graph/assets use the accepted direct-sync authority only after exact #253/source parity; no request-time provider dependency, mixed fallback, stale pointer, or competing writer remains.
• Studio and admin API expose identical capability-controlled source/status/history/diagnostic/manual-sync behavior; no candidate preview/activate/rollback action remains.
• #272 consumes canonical complete direct-sync evidence one way; partial/failed/in-progress/stale/conflicting state blocks restore/startup validation.
• Expand/reconcile/cutover/contract migrations pass forward/reverse/apply-twice, SQLite and PostgreSQL constraints/concurrency, populated historical fixtures, migration drift, and exact count/digest preservation.
• Every child supplies a versioned verification plan and focused/full evidence. Backend-only schema/runner/ingress slices mark screenshots not_applicable; public cutover and Studio/API slices capture and inspect desktop/mobile, authorized/denied, stale/partial/failed/empty states, JavaScript/no-JavaScript where applicable, accessibility, noindex/no-store, and no overflow/debug/error page.
• Specs/runbooks no longer describe ContentRelease activation/rollback as product behavior, and staged code/schema are removed only after all accepted replacements and consumers pass independently.

Explicit non-goals

• No GitHub commit, branch, pull request, arbitrary repository, arbitrary SHA, source execution, request-time network fetch, or provider fallback.
• No resurrection, emulation, or renaming of candidate/ready/activate/rollback as “direct sync”; no cross-source atomic snapshot claim.
• No automatic rollback, old-SHA force sync, direct SQL, hard deletion, cross-source adoption, display-name/email identity join, partial-as-success, stale-success masking, or fallback to historical rows after cutover.
• No public cutover before #253/source parity; no blessing or hand editing of non-reproducible projection bytes.
• No combined parent implementation, speculative source enablement, production credential/provider mutation, workflow run, deploy, or protected data operation while this parent retains needs grooming.
• No Studio/admin implementation before #32/#33, no #272 restore controller behavior, and no #267/#269 activation/rehearsal behavior.

Dependencies and exact next actions

1. #219 disposition: its code is merged, but the issue remains open because the actual applied-0004 state and exact-image deployment failure are unclassified. On-call must obtain the authorized redacted state and determine data/contract versus runtime/infrastructure. Fix and accept #219 if data/contract; resolve the owning runtime recovery if infrastructure. Do not supersede or close #219 from #38, and do not begin a direct-sync schema migration first.
2. Product owner records the two source/public-authority manifests above.
3. Orchestrator files the six raw child intakes from this parent. PM grooms them in order; needs grooming remains on every raw child.
4. Complete #253 before source-specific runner parity/public cutover; complete #32/#33 before management mutation surfaces.
5. #272 is re-groomed after accepted Slices A/B and the owner source/public/freshness decisions. #267/#269 remain downstream.

Start with _docs/PROCESS.md and the direct-sync decision in #226, then compare the current implementation with #253 and downstream evidence expectations in #272. Do not begin implementation while the source rollout manifest and public-authority cutover remain undecided; this epic is done only after those decisions are recorded, child slices are delivered, migration and reconciliation evidence exists, and the staged path is retired.