internetarchive / internetarchive/dweb-mirror

Cache optimization improvements

Open
#159 0 comments 0 reactions 0 assignees View on GitHub
2SC/ON UpNext
Dominant language
JavaScript
Stars
330
Forks
34
PR merge metrics
No merged PRs in 30d

Description

The goal is to improve the cache performance, especially so that operations like isDownloaded on search results are quicker

- [ ] document the contract with the cache - at the top of MirrorFS.js

A member should have a quickly checkable isdownloaded
- [ ] isDownloaded is false if metadata not saved (no need to fetch it)
- [x] isDownloaded can check the downloaded flag on metadata (which should set the flag)
- depends on item check below

An item should have a cachable and quickly checkable isdownloaded
- [ ] Its true if minimumForUI files are all downloaded
- [ ] check and cache this flag as true if downloaded
- [ ] In a crawl, cant tell because files downloaded later and in parallel,
- [ ] If file not present when trying to "addCrawlInfo", check and set it and store on metadata
- [ ] maintenance should check this contract and set/clear downloaded flags
- depends on file check below

A file should never have a sha different from the sha in the metadata so:
- [ ] BLOCKED this section till #166 fixed
- [ ] when fresh (files) metadata is downloaded check sha's on any files already stored, delete if wrong,
- [x] Store downloaded flag in files metadata
- [x] files are only stored when sha matches
- [x] ArchiveFile.isDownloaded() should be true if file present, no need to check sha, but also can use downloaded flag in files metadata
- [ ] maintenance should check this contract and set/clear downloaded flags

- [ ] Quick Maintenance process should be defined (but not resetting IPFS or contenthashes)
- [ ] Quick maintenance kicked off at server start,
- [ ] Quick maintenance runnable at command line (better diagnosis possibility)

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.