DataTalksClub / DataTalksClub/website
Normalize the pinned Podwiki source into typed route-neutral records
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Parent: #43
Downstream projection/search owner: #44
Normative authority:
_docs/specs/03-github-content-and-people.md#source-ownershipand#podwiki-adapter_docs/specs/01-platform-architecture.md, for GitHub-owned Podwiki pages, typed links, citations, graph source, and search source_docs/specs/02-url-link-seo-compatibility.md#canonical-wiki-route, noting that route selection belongs downstream and does not rename the source identity- closed owner decision #24, which assigns ranking/indexing and compatibility projection behavior to #44
Product outcome
Add one deterministic, network-free adapter for the exact accepted DataTalksClub/podwiki checkout. It turns source Markdown/frontmatter and source-owned graph/search inputs into typed, route-neutral records that #43 can map onto the canonical public Wiki and #44 can use to build search/graph projections.
This child is the source-normalization boundary. It must preserve what the source says without assigning a website mount, resolving cross-repository public URLs, ranking search results, publishing graph JSON, or activating database/public state.
Exact accepted input and current evidence
The implementation binds:
- repository:
DataTalksClub/podwiki; - immutable commit:
988b79d0d655bf4755945c3118544cb9e0dbead6; - immutable tree:
60e8821fa16af4688e782759a4a65eb1e36f9f85; - exactly 282
_wiki/*.mdpublic source pages; - generated source inputs
graph/graph.jsonwith 1,072 nodes and 13,006 links, andsearch/search-corpus.jsonwith 2,998 documents; - the allowlisted public source asset
assets/og-default.png; and - caller-supplied immutable git-derived date evidence, rather than a network or moving-branch lookup inside the adapter.
At current website origin/main face8e4808d65afbf0374d1ced7a88079950d663, the checked artifacts are characterization evidence: wiki.json SHA-256 b2525b45a61aed28d035cd996852ffda17ae35f20325b33b66222dbcd102120b, wiki_graph.json f6ec4fb29e71dd179cdaa79526bfb4174ad25ee4991dfcba7f2fe3c5b0e99a13, and wiki_search.json 74d6373df63386aead5108e5addd5ecf2a83a885e891338ca72022e6bdcd3fef. They are not the new route-neutral output format and must not be copied or hand-edited as implementation.
The exact source contains 18,434 typed chip occurrences in source order: 9,008 wiki, 8,571 citation, 622 person, 153 podcast, and 80 book. Of the citation occurrences, 2,622 carry a timestamp. Frontmatter uses the observed allowlisted fields and includes the five accepted editorial type tags (guide, comparison, roadmap, transition, and how-to). These source-occurrence counts are the normalization census; the current public projection's 10,643 deduplicated relation rows are a downstream rendering choice, not source authority.
Scope
- Parse bounded YAML frontmatter and Markdown from the exact caller-supplied checkout without running repository code, plugins, WASM, Jekyll, JavaScript, Make targets, or network requests.
- Emit versioned typed page records with stable source key/path, title, summary, SEO fields, tags/type, ordered safe content blocks/headings, source checksum, immutable commit/tree provenance, and supplied git dates.
- Emit every typed chip occurrence in source order with an occurrence identity and bounded source pointer. Preserve target kind/key, visible alias label, citation episode key, citation timestamp/label, and relation context; do not silently discard timestamp data or collapse repeated occurrences.
- Normalize source aliases/labels,
related/related_wiki, tags, headings, fragments, citations, and mirroredperson/podcast/bookkeys as typed source values. Cross-repository keys remain opaque until their owning canonical resolvers consume them. - Validate the source graph/search input schemas and preserve their route-neutral node/document/link identities and typed fields as source seeds. #44 owns website URL localization, search-document construction, ranking/indexing, deterministic public graph projection, query/filter behavior, activation/fallback, and Lambda retirement evidence.
- Produce deterministic canonical serialization and bounded allowlisted diagnostics. Two clean supported runs over the same checkout and date evidence must be byte-identical.
Explicit non-goals
No /wiki or /podwiki route, redirect, canonical, sitemap, template, browser behavior, public asset response, checked public projection rewrite, database model/migration/write, direct-sync runner/webhook/reconciliation, candidate/activation/rollback, search ranking/query service/index, public graph JSON, cross-repository URL resolution, Person inference, Studio/admin API, GitHub write, provider/production access, Lambda change/retirement, commit, push, merge, or deployment.
The source repository name may remain Podwiki for provenance. The adapter must not encode /podwiki, /wiki, or another public mount into route-neutral identities.
Acceptance criteria
- The adapter accepts only the exact repository/commit/tree and bounded allowlisted paths above; moving branch input, changed pin/tree, symlink/submodule/traversal, unexpected executable/plugin input, unsupported file, and resource-limit excess fail before output.
- All 282 pages normalize once with unique safe source keys/paths, complete checksums/provenance, deterministic ordering, supplied git dates, allowlisted metadata, and safe typed content blocks.
- All 18,434 chip occurrences normalize in source order with the exact kind census above; all 2,622 timestamped citations retain their episode key and timestamp/label; visible aliases remain presentation labels and never become inferred identity.
-
related/related_wiki, the five accepted type tags, headings/fragments, citations, and opaque mirrored entity keys are represented explicitly. Unknown kinds/fields, malformed tokens/timestamps/frontmatter, duplicate identities, ambiguous headings, and unsafe Markdown/HTML/URLs fail with bounded deterministic diagnostics. - Graph/search source inputs validate at 1,072 nodes / 13,006 links / 2,998 documents and retain route-neutral typed seeds without website URL localization, ranking, filtering, public JSON generation, or activation.
- Exact repeated generation is byte-identical and binds schema version, adapter version, source commit/tree, input digests/counts, date-evidence digest, output counts, and one canonical output digest.
- Failure emits no partial normalized corpus or database/public mutation; diagnostics exclude source bodies, arbitrary exception text, checkout paths, credentials, tokens, private/account data, and production data.
- Tests prove there is no network/provider call, repository-code execution, GitHub write, database write, public-route change, checked public-artifact drift, or dependency on #38/#40/#44 runtime implementation.
- A current change-selective plan,
uv-backed format/lint/type/migration checks, focused parser/schema/security/determinism tests, independent tester PASS, and PM acceptance exist before commit.
Required test scenarios
- Normalize the exact pin twice with independently ordered filesystem enumeration and identical date evidence; compare canonical bytes/digest, 282 identities, metadata/type census, content blocks, 18,434 ordered occurrences, 2,622 timestamped citations, and graph/search seed counts.
- Cover every observed frontmatter field, all five type tags, wiki/person/podcast/book/citation chips, visible aliases, timestamp variants, related lists, headings/fragments, duplicate visible occurrences, and Unicode.
- Reject changed SHA/tree, missing/extra/duplicate page, unsafe key/path, symlink/submodule/traversal, YAML alias/entity/depth/node/size overrun, unknown field/kind/tag, malformed citation/timestamp/chip, ambiguous identity/heading, unsafe URL/HTML/script, invalid graph/search schema/link, and missing/inconsistent git-date evidence.
- Prove mirrored keys remain opaque and that names, labels, case folding, URLs, email/account data, or source prose never resolve a Person/Podcast/Book identity.
- Prove the adapter performs no network, provider, database, public-route/template, checked-projection, Studio/API, job, GitHub-write, or production action.
Verification and screenshots
This is a source-only parser/normalizer. Screenshots are not_applicable only when the change-selective graph and artifact comparison prove no public artifact, route, template, CSS, JavaScript, or rendered behavior changed. The independent tester still verifies the frozen diff, exact source/census, focused Django/unit gates, required repository gates, and documented smoke disposition.
Dependencies and sequencing
Satisfied authority: #24 and the immutable pinned source above. This child does not depend on #38, #40, #43, or #44 and may be implemented independently in an isolated worktree.
After acceptance, #43 consumes the typed pages/relations and owning canonical resolvers to produce the /wiki public page projection. #44 consumes the typed source/public identity contracts for downstream search, graph, cross-source link projection, query behavior, and parity. Neither downstream issue may move source parsing or token semantics back into its projection layer.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read the source-ownership and Podwiki-adapter sections in _docs/specs/03-github-content-and-people.md, then review the platform and canonical-route specs. Run the focused parser/schema/security/determinism tests against the pinned _wiki, graph/graph.json, and search/search-corpus.json inputs. Done means deterministic typed records and bounded diagnostics meet the listed counts without routes, network, database, or public-artifact changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, markdown, python, yaml
- Domain
- backend, data, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100