outputDirs in coreunix/Adder is hopelessly slow
- Dominant language
- Go
- Stars
- 17.1k
- Forks
- 3.2k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 11
Description
### Checklist
- [X] My issue is specific & actionable.
- [X] I am not suggesting a protocol enhancement.
- [X] I have searched on the [issue tracker](https://github.com/ipfs/kubo/issues?q=is%3Aissue) for my issue.
### Description
When adding large-ish directories with many files (e.g the English Wikipedia), the add command will spend >50% of the time just scanning through every file that was added (loading the root of every file) just to output events, in order to output directory cid listings - even if the --quieter flag is set.
In case of wikipedia, I see my blockstore serve 2800 gets/sec (~30MiB/s), just to output ~4-5 directory CIDs/second. The goroutine handling the add is stuck here:
```
# 0x28b6ee2 github.com/lotus-web3/ribs/integrations/blockstore.(*Blockstore).Get+0x142 /home/magik6k/github.com/lotus_web3/ribs/integrations/blockstore/ribsbs.go:208
# 0x17e600a github.com/ipfs/go-blockservice.getBlock+0xaa /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-blockservice@v0.5.0/blockservice.go:237
# 0x17e5dd8 github.com/ipfs/go-blockservice.(*blockService).GetBlock+0x318 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-blockservice@v0.5.0/blockservice.go:224
# 0x184c894 github.com/ipfs/go-merkledag.(*dagService).Get+0xb4 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-merkledag@v0.10.0/merkledag.go:81
# 0x2057b97 github.com/ipfs/go-ipld-format.(*Link).GetNode+0x97 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-ipld-format@v0.4.0/format.go:89
# 0x2057b55 github.com/ipfs/go-unixfs/io.(*HAMTDirectory).Find+0x55 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-unixfs@v0.4.4/io/directory.go:389
# 0x20637a8 github.com/ipfs/go-mfs.(*Directory).childFromDag+0x48 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-mfs@v0.2.1/dir.go:214
# 0x2063787 github.com/ipfs/go-mfs.(*Directory).childNode+0x27 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-mfs@v0.2.1/dir.go:148
# 0x206402e github.com/ipfs/go-mfs.(*Directory).childUnsync+0x6e /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-mfs@v0.2.1/dir.go:225
# 0x2063cc4 github.com/ipfs/go-mfs.(*Directory).Child+0xc4 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-mfs@v0.2.1/dir.go:202
# 0x229dd6d github.com/ipfs/kubo/core/coreunix.(*Adder).outputDirs+0x16d /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/kubo@v0.19.1-0.20230331144503-5cb8134812c3/core/coreunix/add.go:208
# 0x229ddf5 github.com/ipfs/kubo/core/coreunix.(*Adder).outputDirs+0x1f5 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/kubo@v0.19.1-0.20230331144503-5cb8134812c3/core/coreunix/add.go:214
# 0x229e40b github.com/ipfs/kubo/core/coreunix.(*Adder).AddAllAndPin+0x36b /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/kubo@v0.19.1-0.20230331144503-5cb8134812c3/core/coreunix/add.go:333
# 0x22be00b github.com/ipfs/kubo/core/coreapi.(*UnixfsAPI).Add+0x19ab /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/kubo@v0.19.1-0.20230331144503-5cb8134812c3/core/coreapi/unixfs.go:191
# 0x25620f2 github.com/ipfs/kubo/core/commands.glob..func6.1+0x172 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/kubo@v0.19.1-0.20230331144503-5cb8134812c3/core/commands/add.go:273
```
Small-file large-directory adding could be made probably ~2x faster with:
* The quieter (-Q) flag could skip outputDirs entirely
* outputDirs could load entries in parallel
* Not easy because mfs Directory.Child takes the directory lock, which in case of one giant flat directory (e.g. wikipedia) won't work well
* The adder could cache what cid is/isn't a directory
* Could make outputDirs significantly faster as now it wouldn't need to load file nodes, just directories
* En Wikipedia is ~12M files, and it would probably be sufficient to key the cache with ~8 bytes of each cid, and you'd only need to mark what is a directory, which in most datasets will be way less blocks than files.
Contributor guide
Assessment
This issue has not been assessed yet.