Improve data onboarding speed: `ipfs add` and `ipfs dag import|export`
- Dominant language
- Go
- Stars
- 17.1k
- Forks
- 3.2k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 11
Description
### Checklist
- [X] My issue is specific & actionable.
- [X] I am not suggesting a protocol enhancement.
- [X] I have searched on the [issue tracker](https://github.com/ipfs/kubo/issues?q=is%3Aissue) for my issue.
### Description
This is a followup on user @endomorphosis's [comment](https://github.com/filecoin-project/community/discussions/696#discussioncomment-8967972) in the Filecoin community discussions about IPFS hashing being slow.
> I noticed that when trying to index large ML models that the IPFS daemon hashing seems to be single threaded, and therefore somewhat slow when indexing large files. If this is funded, it is my hope that someone in your org can try to create a new spec, to parallelize the hashing of large files.
Per @lidel in an ipfs-steering conversation on 2 April 2024:
In my mind this is not about inventing new hashing specifications, this is about making the most popular implementation majority of ecosystem uses for data onboarding (Kubo) better. My translation:_the IPFS daemon hashing [..] slow when indexing large files_
→ Kubo's commands like `ipfs add` are not as fast as they "should be", when comparing with `sha256sum` over the number of chunks_parallelize the hashing of large files._
→ improve implementation, make core commands like `ipfs dag import|export` and `ipfs add` as fast as possible (we know they are not)Once we have reference implementation, we can add some rules of thumb how to implement UnixFs hashing and chunking to "notes for implementers" section of [wip Unix specification](https://github.com/ipfs/specs/pull/331).
Contributor guide
Research direction
The scope covers Kubo's `ipfs add` and `ipfs dag import|export` commands; begin by benchmarking their hashing and chunking against `sha256sum` over equivalent chunks. Done means large-file data onboarding is measurably faster, with implementation guidance suitable for the referenced UnixFS notes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100