indexzero / indexzero/autocache
manifest: add an optional top-level `sources` map to schema v2
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Problem
A manifest records the Original Resources that a document references, under entries. It has no place for the Original Resource of the document itself.
Example: a blog post that first ran at http://blog.nodejitsu.com/a-simple-webservice-in-nodejs now lives in a new corpus. Its wayback.json maps every dead link inside the post to a Memento. The address where the post itself first ran is recorded nowhere. A consumer that wants to cite the post's own capture has no field to read.
Proposed solution
Add one optional top-level key to schema v2: sources.
sourcesis a map with the same shape asentries:urlto{ wayback, timestamp, checkedAt? }.- In the terms of RFC 7089 (Memento), each key is an Original Resource (URI-R) of the document, and each
waybackvalue is a Memento (URI-M) of that resource. entriesmaps the resources that the document references.sourcesmaps the resources that the document is. The two never overlap in purpose.- The map usually has one entry. A document merged from several original documents can list several.
applyignoressources. The key describes the document. It does not rewrite links.
Schema after the change:
{
"version": 2,
"sources": { "<url>": { "wayback": "<replay-url>", "timestamp": "<YYYYMMDDHHMMSS>" } },
"rewrites": { "<url>": "<replacement-url>" },
"entries": { "<url>": { "wayback": "<replay-url>", "timestamp": "<YYYYMMDDHHMMSS>", "checkedAt": "<ISO-8601>" } },
"exclude": [ "<url>" ]
}
Changes:
validateManifestacceptssources. It applies the same checks that it applies to eachentriesvalue. A missing key reads as an empty map.emptyManifestreturnssources: {}.canonicalizesorts the keys ofsources.writeManifestemits the key only when the map is not empty, afterversionand beforerewrites.manifest.d.tsaddssources: Record<string, ManifestEntry>toManifest.- The schema comment at the top of
manifest.js,src/waybackify/README.md,docs/CACHE.md, anddocs/REMASTER.mddescribe the key. - The schema version stays
2. The key is optional, so every existing manifest still validates.
Acceptance criteria
- A v2 manifest with
sourcesvalidates and round-trips throughreadManifestandwriteManifestbyte-for-byte. - A v2 manifest without
sourcesvalidates.writeManifestdoes not add the key. - A
sourcesvalue with a missingwaybackor a badtimestampfails with the same error text thatentriesproduces. -
applyoutput does not change whensourcesis present.
Verification
- Tests pass: new cases in
src/waybackify/test/manifest.test.js. -
pnpm -r testpasses offline.
Dependencies
None.
Files likely touched
src/waybackify/manifest.jssrc/waybackify/manifest.d.tssrc/waybackify/test/manifest.test.jssrc/waybackify/README.md,docs/CACHE.md,docs/REMASTER.md
Estimated scope
Small (3-5 files)
Alternatives considered
- A single
sourcestring. Rejected. A merged document has several Original Resources, and the map shape already exists. - A schema version 3. Rejected. The key is optional, and readers that do not know it lose nothing.
Non-goals
- Fetch, crawl, and remaster of the
sourcescaptures. That is a follow-up issue. - A CLI flag to populate
sources. That is a follow-up issue.
Open question
The schema has no way to say "this document's own URL was never archived." One option is to record that URI-R in exclude. This keeps the vocabulary at three sections. It also mixes the document's identity into a list of link policy. Decide before the first consumer needs it.
Context
- RFC 7089, Memento terminology: https://www.rfc-editor.org/rfc/rfc7089.html#section-1.1
- Schema v2 header:
src/waybackify/manifest.js:1-40 - First consumer: charlie.dev, the source-attribution footer on every historical post.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the schema and manifest operations in src/waybackify/manifest.js, then read the related typings and cases in src/waybackify/manifest.d.ts and src/waybackify/test/manifest.test.js. Add coverage for validation, round-tripping, omission of empty sources, and unchanged apply output, then update the listed README and documentation files. Run the manifest tests and pnpm -r test offline; done means all acceptance criteria pass without changing schema version 2.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- documentation, tooling
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 76/100