filecoin-project / filecoin-project/lassie

Resumable downloads

Open
#274 3 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
118
Forks
23
PR merge metrics
No merged PRs in 30d

Description

We currently have no option to restart a download, which makes lassie pretty fussy and problematic for large downloads. If you fail, you have to start from scratch. At least with Kubo, you have the data in a blockstore so it can resume from there.

Challenges to be solved:

* If you "resume" from an existing CAR, do you have to run a traversal over it to verify that the CAR DAG it has is correct up to the point that it ends (presumably prematurely)?
* Can you "resume with bundle of blocks" where you supply a CAR (or multiple?) that have blocks that may be needed in your traversal, but the output CAR is still new?
* What do we do about HTTP retrievals in this case since we have no "I already have this" facility, do we just document this behaviour and suggest removing the HTTP retriever?

As an experiment I've been trying to download a copy of wikipedia (`bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze`) and can't get more than ~500Mb in with lassie before I get timeouts or other errors and I have no way of resuming. Kubo gets much further although it slows to a crawl for me at a certain point, but at least I know I can cancel it and start again and it'll have what it already fetched in its blockstore.

There's a general problem set of "large data" that I don't think lassie is up to the challenge of solving yet.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.