indexzero / indexzero/autocache
ledger and cache: fetch, crawl, and remaster the captures in `sources`
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 1
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Problem
After #8, a manifest can name the document's own Original Resource under sources. The ledger does not read that key. flatten and against walk entries only, and sourceRefs folds in entries only. As a result, cache fill never fetches the document's own capture, cache crawl never renders it, remaster build never localizes it, and neither verify command reports it missing.
Proposed solution
Treat each sources value as one more capture that the ledger owns.
flattenunionssourcesinto the output manifest undersources, with the same first-seen rule thatentriesuses.againstadds eachsourcescapture to the worklist. It dedupes by capture key, so a URL that appears in bothsourcesof one file andentriesof another counts once.sourceRefs(file, { manifest: true })includessourcesvalues withsource: 'manifest'.cache fill,cache crawl,remaster build,cache verify, andremaster verifyneed no code change if they read the ledger through these three functions. If one of them walksentriesdirectly, correct it to use the ledger.
Acceptance criteria
- A content tree with one manifest whose
sourcesnames a capture that is not in anyentries:cache fillfetches that capture. - The same tree:
againstlists the capture underunfetchedbefore the fill and undercachedafter it. -
remaster verify --tier staticreports the capture when it is missing from the remastered root. - A tree with no
sourcesanywhere produces the same worklists as before this change.
Verification
- Tests pass: new cases in
src/waybackify/test/ledger.test.jsandsrc/waybackify-cli/test/cache-fill-command.test.js. -
pnpm -r testpasses offline.
Dependencies
Blocked by #8.
Files likely touched
src/waybackify/ledger.jssrc/waybackify/ledger.d.tssrc/waybackify/manifest.js(sourceRefs)src/waybackify/test/ledger.test.jssrc/waybackify-cli/test/cache-fill-command.test.js
Estimated scope
Small (3-5 files)
Alternatives considered
- A separate
sourcesworklist. Rejected. A capture is a capture. The cache does not care why the ledger names it.
Non-goals
- The CLI flag that writes
sources. That is a separate issue. - Any change to
applyor to link rewriting.
Context
flatten:src/waybackify/ledger.js:86.against:src/waybackify/ledger.js:134.sourceRefs:src/waybackify/manifest.js:291.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with flatten and against in src/waybackify/ledger.js, then inspect sourceRefs in src/waybackify/manifest.js and the related type declarations. Add ledger tests in src/waybackify/test/ledger.test.js and cache-fill coverage in src/waybackify-cli/test/cache-fill-command.test.js; run pnpm -r test offline and verify source captures appear in the stated worklists without changing trees that have no sources.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript
- Domain
- cli, tooling
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100