ipfs / ipfs/kubo

Importer flow needs a redesign

Open
#3,638 5 comments 0 reactions 0 assignees View on GitHub
need/community-input topic/perf
Dominant language
Go
Stars
17.1k
Forks
3.2k
Avg merge
3d 18h
Merged PRs (30d)
11

Description

More and more we are getting into situations where our bottleneck for adding files is hashing speed. To get around this we need to parallelize the hashing (and also look at faster hashing functions, blake2b). This is a little bit difficult with the current way that the importer setup works now. So i'm proposing a different flow.

The current flow is that we start by building the tree, we get nodes until we fill up a lay of the tree, build the next layer, and so on. with the key here being getting data at the same time we're constructing the tree.

The flow i propose would ideally generate all the leaf blocks up front, streaming them, hashing them in parallel, and putting them to disk. Then, once that is done (or, far enough along that we can start tree construction) we take the cids of those blocks and start constructing the tree. This means that we don't have to hold blocks in memory as we're creating the tree (Which we don't necessarily have to do now, but i think it helps the memory pressure).

The downside of computing all the blocks up front is that we then have to hold all those Cids somewhere.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.