indexzero / indexzero/autocache

manifest: add an optional top-level `sources` map to schema v2

Open
#8 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
JavaScript
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Problem

A manifest records the Original Resources that a document references, under entries. It has no place for the Original Resource of the document itself.

Example: a blog post that first ran at http://blog.nodejitsu.com/a-simple-webservice-in-nodejs now lives in a new corpus. Its wayback.json maps every dead link inside the post to a Memento. The address where the post itself first ran is recorded nowhere. A consumer that wants to cite the post's own capture has no field to read.

Proposed solution

Add one optional top-level key to schema v2: sources.

  • sources is a map with the same shape as entries: url to { wayback, timestamp, checkedAt? }.
  • In the terms of RFC 7089 (Memento), each key is an Original Resource (URI-R) of the document, and each wayback value is a Memento (URI-M) of that resource.
  • entries maps the resources that the document references. sources maps the resources that the document is. The two never overlap in purpose.
  • The map usually has one entry. A document merged from several original documents can list several.
  • apply ignores sources. The key describes the document. It does not rewrite links.

Schema after the change:

{
  "version": 2,
  "sources":  { "<url>": { "wayback": "<replay-url>", "timestamp": "<YYYYMMDDHHMMSS>" } },
  "rewrites": { "<url>": "<replacement-url>" },
  "entries":  { "<url>": { "wayback": "<replay-url>", "timestamp": "<YYYYMMDDHHMMSS>", "checkedAt": "<ISO-8601>" } },
  "exclude":  [ "<url>" ]
}

Changes:

  • validateManifest accepts sources. It applies the same checks that it applies to each entries value. A missing key reads as an empty map.
  • emptyManifest returns sources: {}.
  • canonicalize sorts the keys of sources. writeManifest emits the key only when the map is not empty, after version and before rewrites.
  • manifest.d.ts adds sources: Record<string, ManifestEntry> to Manifest.
  • The schema comment at the top of manifest.js, src/waybackify/README.md, docs/CACHE.md, and docs/REMASTER.md describe the key.
  • The schema version stays 2. The key is optional, so every existing manifest still validates.
Acceptance criteria
  • A v2 manifest with sources validates and round-trips through readManifest and writeManifest byte-for-byte.
  • A v2 manifest without sources validates. writeManifest does not add the key.
  • A sources value with a missing wayback or a bad timestamp fails with the same error text that entries produces.
  • apply output does not change when sources is present.
Verification
  • Tests pass: new cases in src/waybackify/test/manifest.test.js.
  • pnpm -r test passes offline.
Dependencies

None.

Files likely touched
  • src/waybackify/manifest.js
  • src/waybackify/manifest.d.ts
  • src/waybackify/test/manifest.test.js
  • src/waybackify/README.md, docs/CACHE.md, docs/REMASTER.md
Estimated scope

Small (3-5 files)

Alternatives considered
  • A single source string. Rejected. A merged document has several Original Resources, and the map shape already exists.
  • A schema version 3. Rejected. The key is optional, and readers that do not know it lose nothing.
Non-goals
  • Fetch, crawl, and remaster of the sources captures. That is a follow-up issue.
  • A CLI flag to populate sources. That is a follow-up issue.
Open question

The schema has no way to say "this document's own URL was never archived." One option is to record that URI-R in exclude. This keeps the vocabulary at three sections. It also mixes the document's identity into a list of link policy. Decide before the first consumer needs it.

Context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the schema and manifest operations in src/waybackify/manifest.js, then read the related typings and cases in src/waybackify/manifest.d.ts and src/waybackify/test/manifest.test.js. Add coverage for validation, round-tripping, omission of empty sources, and unchanged apply output, then update the listed README and documentation files. Run the manifest tests and pnpm -r test offline; done means all acceptance criteria pass without changing schema version 2.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
documentation, tooling
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.