indexzero / indexzero/autocache

ledger and cache: fetch, crawl, and remaster the captures in `sources`

Open
#9 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
JavaScript
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Problem

After #8, a manifest can name the document's own Original Resource under sources. The ledger does not read that key. flatten and against walk entries only, and sourceRefs folds in entries only. As a result, cache fill never fetches the document's own capture, cache crawl never renders it, remaster build never localizes it, and neither verify command reports it missing.

Proposed solution

Treat each sources value as one more capture that the ledger owns.

  • flatten unions sources into the output manifest under sources, with the same first-seen rule that entries uses.
  • against adds each sources capture to the worklist. It dedupes by capture key, so a URL that appears in both sources of one file and entries of another counts once.
  • sourceRefs(file, { manifest: true }) includes sources values with source: 'manifest'.
  • cache fill, cache crawl, remaster build, cache verify, and remaster verify need no code change if they read the ledger through these three functions. If one of them walks entries directly, correct it to use the ledger.
Acceptance criteria
  • A content tree with one manifest whose sources names a capture that is not in any entries: cache fill fetches that capture.
  • The same tree: against lists the capture under unfetched before the fill and under cached after it.
  • remaster verify --tier static reports the capture when it is missing from the remastered root.
  • A tree with no sources anywhere produces the same worklists as before this change.
Verification
  • Tests pass: new cases in src/waybackify/test/ledger.test.js and src/waybackify-cli/test/cache-fill-command.test.js.
  • pnpm -r test passes offline.
Dependencies

Blocked by #8.

Files likely touched
  • src/waybackify/ledger.js
  • src/waybackify/ledger.d.ts
  • src/waybackify/manifest.js (sourceRefs)
  • src/waybackify/test/ledger.test.js
  • src/waybackify-cli/test/cache-fill-command.test.js
Estimated scope

Small (3-5 files)

Alternatives considered
  • A separate sources worklist. Rejected. A capture is a capture. The cache does not care why the ledger names it.
Non-goals
  • The CLI flag that writes sources. That is a separate issue.
  • Any change to apply or to link rewriting.
Context
  • flatten: src/waybackify/ledger.js:86. against: src/waybackify/ledger.js:134. sourceRefs: src/waybackify/manifest.js:291.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with flatten and against in src/waybackify/ledger.js, then inspect sourceRefs in src/waybackify/manifest.js and the related type declarations. Add ledger tests in src/waybackify/test/ledger.test.js and cache-fill coverage in src/waybackify-cli/test/cache-fill-command.test.js; run pnpm -r test offline and verify source captures appear in the stated worklists without changing trees that have no sources.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript
Domain
cli, tooling
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.