internetarchive / internetarchive/dweb-mirror
Cache optimization improvements
- Dominant language
- JavaScript
- Stars
- 330
- Forks
- 34
- PR merge metrics
- No merged PRs in 30d
Description
The goal is to improve the cache performance, especially so that operations like isDownloaded on search results are quicker
- [ ] document the contract with the cache - at the top of MirrorFS.js
A member should have a quickly checkable isdownloaded
- [ ] isDownloaded is false if metadata not saved (no need to fetch it)
- [x] isDownloaded can check the downloaded flag on metadata (which should set the flag)
- depends on item check below
An item should have a cachable and quickly checkable isdownloaded
- [ ] Its true if minimumForUI files are all downloaded
- [ ] check and cache this flag as true if downloaded
- [ ] In a crawl, cant tell because files downloaded later and in parallel,
- [ ] If file not present when trying to "addCrawlInfo", check and set it and store on metadata
- [ ] maintenance should check this contract and set/clear downloaded flags
- depends on file check below
A file should never have a sha different from the sha in the metadata so:
- [ ] BLOCKED this section till #166 fixed
- [ ] when fresh (files) metadata is downloaded check sha's on any files already stored, delete if wrong,
- [x] Store downloaded flag in files metadata
- [x] files are only stored when sha matches
- [x] ArchiveFile.isDownloaded() should be true if file present, no need to check sha, but also can use downloaded flag in files metadata
- [ ] maintenance should check this contract and set/clear downloaded flags
- [ ] Quick Maintenance process should be defined (but not resetting IPFS or contenthashes)
- [ ] Quick maintenance kicked off at server start,
- [ ] Quick maintenance runnable at command line (better diagnosis possibility)
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.