Importer flow needs a redesign
- Dominant language
- Go
- Stars
- 17.1k
- Forks
- 3.2k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 11
Description
More and more we are getting into situations where our bottleneck for adding files is hashing speed. To get around this we need to parallelize the hashing (and also look at faster hashing functions, blake2b). This is a little bit difficult with the current way that the importer setup works now. So i'm proposing a different flow.
The current flow is that we start by building the tree, we get nodes until we fill up a lay of the tree, build the next layer, and so on. with the key here being getting data at the same time we're constructing the tree.
The flow i propose would ideally generate all the leaf blocks up front, streaming them, hashing them in parallel, and putting them to disk. Then, once that is done (or, far enough along that we can start tree construction) we take the cids of those blocks and start constructing the tree. This means that we don't have to hold blocks in memory as we're creating the tree (Which we don't necessarily have to do now, but i think it helps the memory pressure).
The downside of computing all the blocks up front is that we then have to hold all those Cids somewhere.
Contributor guide
Assessment
This issue has not been assessed yet.