Add secure GitHub content ingress, immutable checkout, and reconciliation

Parent epic: #38
Prerequisite foundation: #156 (closed)
Schema/state slice: #273
Network-free runner slice: #274

Product outcome

Add Slice C of the owner-approved direct-sync architecture: one bounded anonymous GitHub push endpoint authenticates an exact raw delivery, authorizes it against one configured content source, durably requests sync, and returns without doing provider or content work. A leased worker then resolves a configured read-only credential, proves and materializes one immutable repository/branch/commit/tree checkout, and calls the accepted #274 direct-upsert runner. A code-owned reconciler detects missed webhooks, newer remote heads, stale work, and freshness failures within the source's 14-minute target.

The website remains read-only toward GitHub. Existing public content remains last-known-safe during GitHub, credential, checkout, queue, worker, or parser failure. This slice introduces no public-read cutover, source-management UI/API, staged release activation, or provider fallback.

PM disposition — precise blocked contract / do not dispatch

Keep needs grooming. #156 is implemented and supplies the raw-body HMAC/delivery fence, but this issue cannot yet name an engineer-safe implementation against guessed downstream state or production trust policy:

1. #273 must be accepted and this issue must be updated with its exact ContentSource, ContentSyncLog, source-lock/fence, pending-follow-up, remote-head/freshness, transition, recovery-link, and evidence interfaces.
2. #274 must be accepted and this issue must be updated with its exact verified-checkout value, runner entry point, fence/lease contract, terminal outcomes, and retry/error taxonomy.
3. The product/security owner must approve the first live source policy: exact repository immutable identity, repository owner identity, branch/ref, accepted event, sender/actor identities, secret-reference resolver, GitHub read-credential type/reference/scope, checkout transport, credential rotation/failure behavior, and whether non-fast-forward branch movement is ever accepted. No value may be inferred from the stale #103 pin, a payload, a mutable login alone, or an environment variable name already used by the course-repository compatibility path.
4. The jobs/operations owner must freeze the exact code-owned handler names, scalar payloads, source-lock lease/renewal periods, retry/backoff/attempt bounds, reconciliation cadence/batch bound, stale-log watchdog threshold, alert reason codes/sink, and transport response contract/route. The invariants and 14-minute deadline below are settled; these interface values are not.

No engineer may add a generic GitHub endpoint, reuse the course-repository codeload worker as this source's authority, mint or inspect a real credential, enable a source, or implement against placeholder model/job names. After all four items are resolved, PM must insert the exact accepted identities and interfaces, remove needs grooming, and post ## Grooming Complete.

Normative authority

• _docs/PROCESS.md: issue lifecycle, dependency ordering, isolated implementation, migrations, versioned verification, independent tester/PM gates, redaction, and no commit before acceptance.
• _docs/specs/open-decisions.md, decision 1 as resolved by #12/#226, and #38: direct sync, one source lock, truthful partial recovery, no ordinary candidate/activation/rollback graph.
• _docs/specs/01-platform-architecture.md and _docs/architecture/app-boundaries.md: business mutations in shared services, scalar durable jobs, provider work only after commit, no public-request GitHub work, and one-way content_sync/jobs ownership.
• _docs/specs/03-github-content-and-people.md: explicit source ownership, HMAC push safety, immutable bounded checkout, source allowlists, coalescing, reconciliation, parser/sanitizer limits, and no database editorial override. Its staged ContentRelease wording is superseded where it conflicts with #226/#38.
• _docs/specs/07-security-privacy-operations.md: below-15-minute GitHub freshness, last-known-safe behavior, safe structured diagnostics, alerting, runtime secret ownership, and no secrets/PII/provider payloads in evidence.
• _docs/specs/09-migration-rollout-roadmap.md and _docs/specs/10-verification-strategy.md: portable constraints, idempotency/concurrency/crash testing, external-failure drills, and exact release evidence.
• _docs/architecture/shared-primitives.md and current jobs contracts: durable database intent, after-commit best-effort wakeup, code-owned handlers/schedules, scalar payload validation, fenced leases, renewal, bounded retry, and scheduler ownership.
• #156: reuse authenticate_and_fence_webhook_delivery; do not fork HMAC or raw-delivery persistence.
• #273 owns direct-sync storage and source/log transitions. #274 owns verified-input parsing/planning/direct mutation. This slice owns only ingress, provider checkout, orchestration, locks, follow-ups, reconciliation, watchdogs, and freshness alerts around those accepted services.
• #272 is a downstream evidence consumer. This issue emits the accepted source/log evidence but does not implement restore/startup validation.

Trust configuration required before implementation

One code/configuration-owned source policy binds all of the following as a single revisioned contract:

• ContentSource UUID/stable key and expected revision;
• immutable GitHub repository numeric/node identity plus exact owner/name, owner identity, exact branch and full refs/heads/... ref;
• closed adapter ID, mount/path/file/resource limits, accepted commit hash algorithm/grammar, and runner contract/version digests;
• exact allowed GitHub event (push) and exact allowed sender/actor immutable identities plus canonical logins used only as consistency checks;
• endpoint webhook-secret reference and resolver, selected without reading unauthenticated payload bytes;
• read-only checkout credential reference, credential kind, installation/repository scope, allowed GitHub API/archive hosts and redirect policy; and
• enabled state, 14-minute freshness target, lock/watchdog/job policy identifiers, and alert policy.

A request may select none of these values and may not widen them. Unknown, missing, duplicate, disabled, stale-revision, contradictory, or partially configured policy fails closed. Secret and credential references are opaque configuration identifiers; their resolved values never enter a model, job payload, log, metric, audit, exception, issue, fixture, or artifact.

GitHub webhook transport and authenticate-before-parse boundary

The final grooming pass must insert the one exact POST route and bounded response table. Irrespective of route spelling, the adapter contract is:

1. Accept only POST, the frozen content type/body limit, one exact X-Hub-Signature-256, one bounded X-GitHub-Delivery, and exact X-GitHub-Event: push. The route is operational, anonymous only through this contract, CSRF-exempt only at this path, private/no-store, non-indexable, and never cached.
2. Select the webhook namespace and secret reference from trusted endpoint configuration only. Verify HMAC-SHA256 over the untouched raw bytes with #156 before JSON decoding, source lookup, actor lookup, job creation, or provider access. Missing/invalid/ambiguous input returns only the frozen safe reason/status; authentication failures do not reveal whether a source, repository, actor, or credential exists.
3. Use #156's delivery fence keyed by the configured namespace and delivery identity, bound to the raw-body SHA-256. Exact replay returns the original bounded outcome; the same delivery identity with changed bytes conflicts; concurrent duplicates converge. The raw delivery ID itself does not enter ContentSyncLog.
4. Only inside the authenticated fenced command, decode exactly one JSON object with bounded depth/items/string sizes and reject duplicate keys, invalid Unicode/non-finite values, trailing bytes, or any alternate serialization. Retain no decoded payload after the command.
5. Require mutually consistent repository immutable ID, full_name, owner identity, repository name; exact configured ref; deleted=false; one lowercase full non-zero after commit; and, when present/required by the frozen payload contract, matching head_commit.id. Require the configured sender/actor immutable ID and matching canonical login. Missing, renamed, transferred, forked, deleted, tag, pull-request, non-push, unallowlisted actor, or contradictory identity fails closed.
6. Re-read the source enabled state and revision while atomically creating/reusing the source/run request and durable intent. Return a bounded accepted/replayed/coalesced/rejected outcome without cloning, fetching, parsing, checking remote head, or waiting for a worker.

No session/API credential grants authority to this endpoint. GitHub signature validity alone does not authorize a repository, branch, actor, commit, adapter, or source.

Four distinct idempotency layers

These identities must remain separate and each exact-key/different-intent reuse must conflict:

1. Delivery fence: #156 namespace plus delivery identity, bound only to the exact raw-body digest. It authenticates/deduplicates transport and persists no provider ID/payload.
2. Source/run fence: the accepted #273 service creates or returns one immutable ContentSyncLog for the exact source revision, trigger, commit/input identity and contract digests. Webhook replay and reconciliation convergence cannot create duplicate runs; a newer head is a different run. A terminal partial/failed recovery is a new linked run, never a reopened log.
3. Durable-job fence: current jobs.dispatch_after_commit owns one handler plus deduplication-key hash bound to one immutable scalar payload. The final key must bind the accepted sync-log/source/fence identity; changed payload bytes conflict. Disposable Django-Q wakeups are not authority and may duplicate safely.
4. Manual/reconciliation request fence: a provider-neutral service boundary accepts a bounded explicit idempotency identity for future trusted manual callers and a deterministic code-owned identity for each scheduled observation window. It is bound to exact source/revision/trigger/observed-head intent and converges into layer 2. This issue adds no manual HTTP, Studio, admin API, or command surface; Slice E will call the accepted seam under its own capabilities/reauthentication/audit rules.

A layer never substitutes another: a new delivery may reference an existing run; multiple requests may reuse one durable job; and a retrying job retains its original run identity.

Immutable checkout and provider boundary

Provider/network work occurs only in a leased durable worker after database commit and outside every content/database transaction:

1. Reload the exact source, log, expected revision, input identity, job lease, and source fence. Resolve the credential reference at execution time. Missing/revoked/wrong-scope credentials are safe retryable or terminal outcomes according to the frozen policy; no anonymous/public archive fallback is allowed.
2. Query the allowlisted provider boundary for the configured repository/ref. Bind provider repository immutable identity, exact ref, current remote-head commit, requested commit object, tree identity, ancestry/reachability, and repository visibility to the configured source. Never trust payload names/URLs as fetch targets.
3. An exact webhook commit is eligible only if it is the current authorized branch head under the frozen fast-forward/non-fast-forward policy. An older/out-of-order commit never applies. If a newer head already exists, record safe coalesced/out-of-order evidence, set the pending follow-up, and let the follow-up re-observe current head rather than carrying an attacker-selected SHA.
4. Materialize only the exact verified commit/tree through the frozen read-only transport into a fresh isolated bounded directory/value. Enforce allowlisted HTTPS hosts, TLS, redirect limits, connect/read/total time, archive/transfer/file/byte/depth/node limits, normalized paths, no absolute/traversal/case-collision path, no duplicate entry, no symlink/hardlink/submodule/device/FIFO/socket, and no repository hooks, filters, LFS commands, submodule commands, package scripts, or source executable invocation.
5. Verify the materialized tree/commit/repository/ref identity again and pass only #274's immutable VerifiedCheckout plus scalar database/service identities. The checkout path, response, token, headers, provider exception, and source bytes never cross into a durable payload or ContentSyncLog.
6. Clean the isolated materialization on success, failure, cancellation, or timeout. Cleanup failure is a bounded safe diagnostic and alert; it never changes a failed run into success.

Checkout/global validation failure produces failed with zero content mutation. A transient provider failure follows the frozen bounded retry policy while truthful queued/running evidence remains observable; exhaustion terminalizes safely. No database transaction spans DNS, TLS, HTTP, git/archive processing, parsing, or runner execution.

One source lock, coalescing, and follow-up

• There is exactly one database-portable compare-and-set source lock for one source, with opaque owner UUID, random fence token, acquired/renewed/expiry times, and the accepted #273 consistency constraints. Do not depend on PostgreSQL advisory locks, select_for_update, or process-local mutexes.
• A worker may transition the same queued log to running only after atomically acquiring both its durable-job lease and the source fence against the expected source revision/input. Every renewal and mutation validates both current fences; a stale owner cannot run, terminalize, release, or emit a follow-up.
• Exact same-input concurrency returns the existing run/job. A different request while the source lock is held produces the accepted coalesced skipped evidence, atomically sets one pending-follow-up bit, and does not apply content or claim complete evidence.
• Lock release terminalizes the owned log first, then atomically clears only the matching fence. In that transaction it consumes the one pending bit at most once and creates at most one follow-up durable intent after commit. The follow-up carries the source identity/revision and re-observes the configured remote head; it does not trust/store a pending payload SHA.
• A wakeup failure cannot lose work: durable intent remains sweepable. Duplicate wakeups and worker retry cannot duplicate a log, checkout application, follow-up, or audit/evidence row.
• A lost/expired job lease or source fence makes the worker stop before another mutation. The watchdog uses compare-and-set on the exact stale log/job/source fences, terminalizes stale queued/running evidence as failed, releases only the matching lock, then schedules one new linked recovery/follow-up run when policy permits. It never overwrites the old log or fabricates success from current rows.

Reconciliation, 14-minute freshness, and alerts

A single code-owned fenced scheduler registers the final named reconciliation task. Its bounded batches select only enabled configured sources in stable order; each source operation is idempotent and independently failure-isolated.

For each source, reconciliation resolves the configured credential and observes the exact allowlisted repository/ref head:

• exact head equal to the newest complete successful commit creates/reuses truthful unchanged-head skipped evidence and does not run #274;
• a newer authorized head creates/reuses one queued reconciliation run and durable job;
• a head already queued/running/coalesced converges without duplicates;
• an older, missing, contradictory, renamed/transferred, unauthorized, deleted, or non-fast-forward head fails closed under the frozen source policy, preserves last-known-safe content, and alerts; and
• partial, failed, stale queued/running, missing terminal evidence, or pending-follow-up state remains non-complete and is recovered through a new linked run, never by masking it with an older success.

The configured service-level objective is exact: an accepted branch-head change must be observed and either reach terminal complete evidence or raise a freshness alert no later than 14 minutes after the earliest trustworthy local event (authenticated delivery receipt or successful remote-head observation). The code-owned schedule cadence, queue-age budget, lease/retry budget, and alert evaluation must also prove by fake-clock worst-case tests that a webhook lost immediately after one scheduler tick is observed/alerted within 14 minutes of the remote change. Provider commit timestamps never reset, excuse, or fabricate this local clock. last_remote_head_observed_at advances only after an authenticated provider observation and binds its commit.

Alert conditions include: age at/over 14 minutes without matching complete evidence; source lock/log/job lease past the frozen threshold; partial/failed run; retry exhaustion; missing/revoked credential; repeated checkout/provider/reconciler failure; pending follow-up not consumed; non-fast-forward/older/contradictory head; and cleanup failure. Emit bounded reason, source opaque ID, status, safe age/count, and correlation/job/log opaque IDs only. Commit, delivery, repository/path, actor, credential, provider response, error/exception text, and source content are not metric labels or alert payloads.

Diagnostics, evidence, and failure behavior

• Store only the #273/#274 bounded fields and allowlisted reason codes. Webhook receipt may advance the accepted attempted/receipt time only through the exact service transition; last successful commit/time advances only on terminal success.
• Never persist or log raw body/signature/delivery ID/header/payload, sender fields, secret/credential reference or value, token, repository response, checkout/absolute/temp path, GitHub URL/query, source content/HTML/Markdown/YAML, email/profile/member/course activity, IP, SQL, exception, traceback, or unbounded record value.
• Structured logs/metrics use bounded operation/status/reason, opaque source/log/job/correlation IDs where permitted, duration/age/count buckets, and no high-cardinality commit/path/delivery/error labels.
• Authentication/authorization failures reveal no policy membership. Provider/config/outage failure never enables another source, selects an alternate branch/repository/credential, falls back anonymously, runs repository code, rolls content backward, or mutates current public reads.
• Existing last-known-safe public content remains served through the current authority. This slice does not claim freshness complete for coalesced, queued, running, partial, failed, unavailable, conflicting, or stale evidence.

Acceptance criteria after blockers are resolved

• The final body embeds the approved first-source policy revision/digest and exact repository/owner/branch/ref/event/actor identities, secret and credential reference/resolver/rotation contracts, checkout transport/hosts/limits, non-fast-forward rule, and enabled-state rollout; fixtures use synthetic values only.
• The final body names accepted #273 source/log/lock/status/evidence fields and transition/recovery services, accepted #274 verified-checkout/runner/error interfaces, and exact jobs handler/schedule/payload/dedupe/lease/retry/watchdog/alert/route/response contracts.
• Endpoint tests prove method/media/body/header bounds, raw-byte HMAC before parse, constant-time #156 reuse, duplicate-key/hostile JSON denial, exact repository/ref/commit/event/actor allowlists, disabled/stale policy denial, exact replay, changed-body conflict, concurrent convergence, bounded indistinguishable errors, private/no-store, and no provider call on any rejected request.
• The four idempotency layers are independently tested for replay, changed-intent conflict, concurrency, rollback, outer-transaction rollback, wakeup failure, duplicate wakeup, retry, and cross-layer convergence with no duplicate log/job/checkout/follow-up/evidence.
• Provider tests prove configured credential resolution/scope, immutable repository/ref/commit/tree/ancestry binding, current-head/out-of-order/non-fast-forward behavior, host/redirect/TLS/time/size limits, no anonymous fallback, no source execution, and cleanup on every terminal/interrupt path.
• Archive/checkout adversarial fixtures reject traversal, absolute/backslash/control/case-collision/duplicate paths, symlink/hardlink/submodule/special files, decompression/alias/entity bombs, over-count/byte/depth/node/time inputs, wrong tree/commit/repository/ref, and missing required content before #274 mutation.
• Source-lock tests prove one portable lock/fence, renewal, exact-owner release, stale-owner denial, different-input coalescing, one pending bit, atomic consume, at-most-one after-commit follow-up, lost wakeup recovery, crash at each lock/log/job transition, and fenced watchdog/new-log recovery.
• Fake-clock reconciliation tests cover missed webhook immediately after a tick, exact unchanged head, newer head, duplicate observation, older/out-of-order/non-fast-forward head, provider/credential outage, partial/failed/stale work, and batch boundaries; complete-or-alert is reached within 14 minutes without using provider timestamps.
• Failure/redaction canaries inspect database rows, job payloads, idempotency records, audits, logs, metrics, alerts, exceptions, and verification artifacts and find none of the forbidden fields/values above. Last-known-safe reads remain byte/query-identical throughout injected outage and crash cases.
• SQLite tests cover all portable service/constraint behavior; the supported PostgreSQL path covers concurrency and fence races. Migration drift, jobs scheduler/lease/dispatch/execution, #156, content/content-sync adapter, security/redaction, graph-selected quality/Django, and container verification pass under the versioned plan.
• Playwright and screenshots are not_applicable only if the final change graph confirms no public/management URL, template, navigation, copy, cache/public selector, or rendered-state impact. Any such impact requires PM re-grooming and the normal core/full browser plus independent desktop/mobile screenshot gate.
• Engineer evidence records exact base/head, graph/plan/report digests, synthetic source-policy digest, idempotency key matrix, provider/checkout fault matrix, fake-clock deadline proof, commands/counts, redaction artifact, and every rerun/reuse/skip/not-applicable disposition; independent tester recomputes and verifies the full contract before PM acceptance.

Explicit non-goals

• No public reader, projection/search/graph/sitemap/feed/SEO/cache/asset-delivery cutover, route invalidation, request-time provider call, dual reader, or fallback to staged/historical rows.
• No Studio, Django admin, /api/v1/admin/, CLI/manual-sync surface, source-configuration mutation, credential-management UI, preview, edit-on-GitHub, activate, rollback, or force-SHA action. The provider-neutral manual idempotency seam has no caller here.
• No ContentRelease candidate creation, ready/activate/pointer/rollback mutation, staged child write, staged-table removal, or reinterpretation of historical rows as direct current rows.
• No arbitrary repository/branch/ref/path/SHA/URL/host/adapter/actor/credential from a request; no tag/PR/deletion event; no automatic old-SHA rollback; no implicit acceptance of force-push/non-fast-forward history.
• No repository write, commit status/check, branch/PR/comment mutation, source script/hook/filter/LFS/submodule/package execution, arbitrary network fetch, anonymous archive fallback, hard delete, cross-source adoption, or fuzzy identity join.
• No invention or storage of provider/production credentials, owner/source policy, alert provider, infrastructure/IAM/Terraform/deployment secret, production data, new package without separate justification, unrelated refactor, commit, push, merge, deploy, or Actions rerun during grooming.

Required final grooming handoff

After #273 and #274 are accepted and the owner approvals/interfaces above are recorded, PM must replace every placeholder with exact accepted model/service/handler/schedule/route/response/credential/source-policy values, verify the 14-minute budget arithmetic against those values, remove needs grooming, and post ## Grooming Complete. Until then this issue is a blocked implementation contract, not an engineering handoff.

Start with _docs/PROCESS.md and the cited architecture and security specifications, then review the accepted contracts from #273 and #274. Do not implement until the source policy, handler names, scalar payloads, lease and retry values, and response contract are frozen. Done means the exact interfaces and identities are inserted, needs grooming is removed, and ## Grooming Complete is posted.