Build the offline Docs source parser and fail-closed parity census

Parent epic: #41

Normative authority:

• Spec 03 — source ownership and Docs adapter
• Spec 03 — sync safety
• Spec 02 — exact URL/link/fragment compatibility
• Development process
• #35 supplies the generic fail-closed compatibility vocabulary; #38 owns later direct sync and #44 owns later unified search.

Product outcome

Make the currently checked Docs projection auditable from one caller-supplied immutable DataTalksClub/docs checkout. This issue delivers only a deterministic, network-free parser, selection ledger, schema validation, and parity report. It must expose present source/projection drift as a typed BLOCKED result; it does not bless, repair, activate, or serve a different projection.

Public repository/edit controls are forbidden by spec 03. The existing source path, checksum, immutable revision, and moving-main edit target may be retained as management-only provenance, but no public DTO, template, page, metadata, link, or browser control may expose them.

Exact baseline to bind

Repository baseline is DataTalksClub/website@face8e4808d65afbf0374d1ced7a88079950d663.

Docs source declared by the checked projection:

• repository: DataTalksClub/docs;
• commit: 3f23e006ffdaa498bbc69697408853b6f5eb37dc;
• tree: 585e440a6c7c32e8a3936f431a8c0a1d8709b1a7;
• complete source tree: 285 blobs, including 119 Markdown files and 54 image files;
• adopted checked selection: 106 pages and 39 image assets;
• excluded Markdown census: 13 paths; unselected image census: 15 paths.

Checked target evidence at face8e4:

• content/docs_projection.json: 381,637 bytes, SHA-256 15d9491fee4683303ce3cc0ba00c702599016149c010923e200a85d37084fc8a;
• deterministic 39-file content/docs_assets/ path/checksum manifest SHA-256: 8cce1dd8b6d0339c42879817cfa9d07f45684c9084ac7445dec3c8f6056f601f;
• projection records: 106 pages, 39 assets, 106 management-only edit_url values;
• _docs/compatibility/generated-path-baseline.jsonl: 174 dtc-docs rows, all preserve: 107 HTML, 63 assets, 2 JSON, and 2 XML.

The older inventory claim that the projection digest is 1abd84ab... is stale. The checked projection also contains at least one proven source contradiction: courses/llm-zoomcamp/project.md retains the pinned raw-source checksum 6a3ae6c9... while its body contains the later AI Shipping Blog destinations absent from the declared source commit. The parser must discover and report the complete mismatch set, not special-case this example.

Moving upstream main is not an input. It is currently f6c8527d349fceec6e8e89cc255c6d484ffbf22f / tree 2e856b9a818e0c3123b9efaf3879c2b999c04899, four commits ahead of the declared pin, and changes four adopted page files plus repository-maintenance files. This issue neither adopts nor copies those bytes.

Scope and owned files

Implement the smallest self-contained offline lane in these file families:

• new content/docs_source_adapter.py: bounded source selection, frontmatter decoding, hierarchy/path derivation, asset reference selection, checksums, and canonical projection records;
• new scripts/build_docs_projection.py: a uv-run CLI that accepts an already verified checkout path plus exact commit/tree arguments and writes only to an explicit output directory under .tmp/, or compares with --check;
• new _docs/compatibility/docs-source-selection.schema.json and docs-source-selection.json: exact 285-blob classification ledger with one disposition and bounded reason for every source blob;
• new _docs/compatibility/docs-projection.schema.json: strict schema for the current version-1 projection object, page records, asset records, and management-only provenance fields;
• new focused fixtures/tests under content/tests/fixtures/docs_source_adapter/, content/tests/test_docs_source_adapter.py, and tests_ci/test_docs_projection_generator.py;
• one Makefile check target and only the minimal CI ownership/change-selection entries needed to run it.

Do not edit content/docs_projection.json, content/docs_assets/**, content/docs_projection.py, content/docs_presentation.py, views, URLs, templates, browser tests, models, migrations, source pins, or compatibility observations in this issue. An unexpected need to touch those files returns the issue to PM.

Parser and selection contract

• The adapter performs no Git, GitHub, HTTP, DNS, database, subprocess, template, or filesystem write operation. It accepts a local checkout plus scalar expected repository/commit/tree identities supplied by the CLI.
• Reject a wrong/moving SHA, wrong tree, dirty or symlinked selection, traversal, duplicate normalized path, unsupported file type, oversized file/tree, YAML alias/tag, malformed or duplicate frontmatter key, non-UTF-8 text, and unknown required field with stable content-free diagnostics.
• Classify all 285 source blobs exactly once. Adopt the exact 106 page sources rooted at index.md, activities/, courses/, and general/; every other Markdown/config/theme/tool/draft file requires an explicit ledger disposition and reason. Do not infer publication merely from .md.
• Preserve exact source bytes/checksums and decoded body. Parse only the bounded Docs frontmatter fields title, description, nav_order, parent, grand_parent, has_children, has_toc, and permalink; reject wrong scalar types and unknown publication-affecting metadata.
• Derive canonical /docs/ pretty paths, explicit permalinks, parent_path, and grand_parent_path deterministically. Missing, ambiguous duplicate-title, contradictory parent/grand-parent, cycle, orphan, duplicate public path, wrong-root permalink, query, fragment, slash, case, Unicode, or percent-escape collision fails the whole result; no title/path guessing or partial projection is allowed.
• Preserve Markdown/Liquid as source data for the existing renderer. Recognize and validate root-relative relative_url links/images without executing Liquid. Select only the exact referenced adopted JPEG/PNG/SVG files, verify content type/size/SHA-256, and reject traversal, remote/protocol-relative media, missing/duplicate asset ownership, unsafe SVG, or unreferenced asset presented as adopted.
• Emit records in deterministic source/navigation order with the exact current projection fields. edit_url is management provenance only and must be derived from the allowlisted repository plus bounded source path; public exposure is outside this adapter and forbidden.
• Canonical JSON and asset output must be byte-identical across two clean invocations from the same checkout. All diagnostics and reports are bounded, path-safe, secret-free, and contain no source bodies.

Parity and compatibility report

The check command compares a generated scratch output with the frozen checked projection/assets and separately accounts for all 174 dtc-docs compatibility rows.

• Every source blob, generated page/asset, checked page/asset, and compatibility row is classified exactly once. Missing, extra, duplicate, changed, or unexplained entries block.
• The report binds website SHA, source SHA/tree, parser/schema/selection-ledger digests, checked and generated projection/asset-manifest digests, and the checked compatibility-manifest digest.
• The present expected verdict is BLOCKED, with stable findings for every source/checksum/body/projection mismatch and every one of the 29 compatibility paths not represented by the 106-page/39-asset projection (one HTML 404, 24 other assets, two JSON, and two XML). No finding is automatically approved as redirect, retirement, or parity.
• The known courses/llm-zoomcamp/project.md declared-source/body contradiction and stale 1abd84ab... audit binding must be detected from general rules. Hard-coding those paths, URLs, or hashes as exceptions fails acceptance.
• PASS is impossible while any selected source byte cannot reproduce the checked record, any public compatibility path lacks a separately approved expectation, or any binding is stale. A blocked report never invokes a write, activation, cutover, or public-reader callback.

Acceptance criteria

• Strict schemas and loaders reject unknown versions/keys, duplicate/missing classifications, bad hashes/counts/pins, malformed paths, and hostile YAML/path/media fixtures with stable bounded codes.
• The selection ledger covers exactly 285 source blobs: 106 adopted pages, 39 adopted images, and 140 explicitly reasoned exclusions; it binds the exact source SHA/tree and regenerates canonically.
• Real-pin generation produces 106 page and 39 asset records twice with byte-identical scratch trees and no network or repository write.
• Navigation/path fixtures cover explicit permalink, nested parent/grand-parent, order, duplicate titles, ambiguity, orphan/cycle, case, Unicode, percent spelling, query/fragment, and trailing slash.
• Content fixtures cover tables, fenced code, callouts, Mermaid source, raw HTML, images, safe relative_url, malformed Liquid, unsafe HTML/URL/media, missing asset, and unsafe SVG without executing source code.
• The full real-pin report deterministically returns BLOCKED, accounts for all 285 source blobs, 145 projection records, and 174 compatibility rows, and reports the complete drift/unimplemented-path census with no auto-approval.
• Focused tests, schema checks, make docs-projection-check, formatting, lint, type checking, migration drift, Django checks, and the graph-selected verification envelope pass. For backend-only/no-render impact, screenshots are not_applicable only when the computed graph proves render.impact=false with no required captures; the graph-selected Playwright smoke tier remains required.
• The frozen checked projection/assets and all public routes/rendering remain byte-for-byte untouched.

Test scenarios

1. Parse a compact real-shaped hierarchy and assert canonical records, checksums, ordering, edit provenance, referenced assets, and exact repeat output.
2. Mutate each pin, selection, frontmatter, hierarchy, path, source-body/checksum, Liquid/media, asset, schema, and count boundary independently; assert one deterministic blocking diagnostic and no partial output.
3. Run the full supplied immutable checkout twice into separate .tmp/ outputs, compare complete trees, then compare to the checked projection/assets and the 174-row compatibility scope.
4. Tamper with generated output, checked projection, checked asset bytes, source ledger, schema, or compatibility manifest after report creation; validation must fail stale/contradictory rather than reuse evidence.

Non-goals

No source-pin advance, moving-branch adoption, upstream repository edit, public content repair, generated-artifact replacement, public edit/search control, template/view/route change, Markdown renderer redesign, database model/migration, direct upsert, webhook/reconciliation, Studio/admin API, search indexing, public authority cutover, redirect/410 approval, provider/production access, deployment, or source checkout committed to this repository. No new Python dependency.

Dependencies and handoff

Direct prerequisite #35 is closed. This offline issue is deliberately independent of open #38, #44, #253, and production/provider state because it activates nothing and accepts a caller-supplied checkout.

A later #41 phase may select a new immutable Docs revision, repair/replace the checked projection, wire direct sync, approve compatibility dispositions, switch public authority, and build unified search only after this report, #38’s source-rollout/public-authority decisions and accepted children, and #44’s applicable interface exist. Evidence from this issue cannot authorize those later actions.

Read Spec 03 and the existing projection and compatibility artifacts first. Implement and test the bounded flow across content/docs_source_adapter.py, scripts/build_docs_projection.py, the selection ledger and schemas, then run the focused tests and make docs-projection-check. Done means deterministic output from the immutable checkout, complete accounting of 285 blobs and 174 compatibility rows, and a stable BLOCKED parity report without modifying frozen projection or public rendering files.