ipfs / ipfs/kubo

Improve data onboarding speed: `ipfs add` and `ipfs dag import|export`

Open
#10,383 13 comments 0 reactions 0 assignees View on GitHub
effort/weeks exp/expert kind/enhancement P2
Dominant language
Go
Stars
17.1k
Forks
3.2k
Avg merge
3d 18h
Merged PRs (30d)
11

Description

### Checklist

- [X] My issue is specific & actionable.
- [X] I am not suggesting a protocol enhancement.
- [X] I have searched on the [issue tracker](https://github.com/ipfs/kubo/issues?q=is%3Aissue) for my issue.

### Description

This is a followup on user @endomorphosis's [comment](https://github.com/filecoin-project/community/discussions/696#discussioncomment-8967972) in the Filecoin community discussions about IPFS hashing being slow.

> I noticed that when trying to index large ML models that the IPFS daemon hashing seems to be single threaded, and therefore somewhat slow when indexing large files. If this is funded, it is my hope that someone in your org can try to create a new spec, to parallelize the hashing of large files.

Per @lidel in an ipfs-steering conversation on 2 April 2024:


In my mind this is not about inventing new hashing specifications, this is about making the most popular implementation majority of ecosystem uses for data onboarding (Kubo) better. My translation:

_the IPFS daemon hashing [..] slow when indexing large files_
→ Kubo's commands like `ipfs add` are not as fast as they "should be", when comparing with `sha256sum` over the number of chunks

_parallelize the hashing of large files._
→ improve implementation, make core commands like `ipfs dag import|export` and `ipfs add` as fast as possible (we know they are not)

Once we have reference implementation, we can add some rules of thumb how to implement UnixFs hashing and chunking to "notes for implementers" section of [wip Unix specification](https://github.com/ipfs/specs/pull/331).

Contributor guide

Open the contributing guide

Research direction

The scope covers Kubo's `ipfs add` and `ipfs dag import|export` commands; begin by benchmarking their hashing and chunking against `sha256sum` over equivalent chunks. Done means large-file data onboarding is measurably faster, with implementation guidance suitable for the referenced UnixFS notes.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.