ipfs / ipfs/kubo

outputDirs in coreunix/Adder is hopelessly slow

Open
#9,902 0 comments 0 reactions 0 assignees View on GitHub
kind/enhancement
Dominant language
Go
Stars
17.1k
Forks
3.2k
Avg merge
3d 18h
Merged PRs (30d)
11

Description

### Checklist

- [X] My issue is specific & actionable.
- [X] I am not suggesting a protocol enhancement.
- [X] I have searched on the [issue tracker](https://github.com/ipfs/kubo/issues?q=is%3Aissue) for my issue.

### Description

When adding large-ish directories with many files (e.g the English Wikipedia), the add command will spend >50% of the time just scanning through every file that was added (loading the root of every file) just to output events, in order to output directory cid listings - even if the --quieter flag is set.

In case of wikipedia, I see my blockstore serve 2800 gets/sec (~30MiB/s), just to output ~4-5 directory CIDs/second. The goroutine handling the add is stuck here:
```
# 0x28b6ee2 github.com/lotus-web3/ribs/integrations/blockstore.(*Blockstore).Get+0x142 /home/magik6k/github.com/lotus_web3/ribs/integrations/blockstore/ribsbs.go:208
# 0x17e600a github.com/ipfs/go-blockservice.getBlock+0xaa /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-blockservice@v0.5.0/blockservice.go:237
# 0x17e5dd8 github.com/ipfs/go-blockservice.(*blockService).GetBlock+0x318 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-blockservice@v0.5.0/blockservice.go:224
# 0x184c894 github.com/ipfs/go-merkledag.(*dagService).Get+0xb4 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-merkledag@v0.10.0/merkledag.go:81
# 0x2057b97 github.com/ipfs/go-ipld-format.(*Link).GetNode+0x97 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-ipld-format@v0.4.0/format.go:89
# 0x2057b55 github.com/ipfs/go-unixfs/io.(*HAMTDirectory).Find+0x55 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-unixfs@v0.4.4/io/directory.go:389
# 0x20637a8 github.com/ipfs/go-mfs.(*Directory).childFromDag+0x48 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-mfs@v0.2.1/dir.go:214
# 0x2063787 github.com/ipfs/go-mfs.(*Directory).childNode+0x27 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-mfs@v0.2.1/dir.go:148
# 0x206402e github.com/ipfs/go-mfs.(*Directory).childUnsync+0x6e /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-mfs@v0.2.1/dir.go:225
# 0x2063cc4 github.com/ipfs/go-mfs.(*Directory).Child+0xc4 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/go-mfs@v0.2.1/dir.go:202
# 0x229dd6d github.com/ipfs/kubo/core/coreunix.(*Adder).outputDirs+0x16d /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/kubo@v0.19.1-0.20230331144503-5cb8134812c3/core/coreunix/add.go:208
# 0x229ddf5 github.com/ipfs/kubo/core/coreunix.(*Adder).outputDirs+0x1f5 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/kubo@v0.19.1-0.20230331144503-5cb8134812c3/core/coreunix/add.go:214
# 0x229e40b github.com/ipfs/kubo/core/coreunix.(*Adder).AddAllAndPin+0x36b /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/kubo@v0.19.1-0.20230331144503-5cb8134812c3/core/coreunix/add.go:333
# 0x22be00b github.com/ipfs/kubo/core/coreapi.(*UnixfsAPI).Add+0x19ab /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/kubo@v0.19.1-0.20230331144503-5cb8134812c3/core/coreapi/unixfs.go:191
# 0x25620f2 github.com/ipfs/kubo/core/commands.glob..func6.1+0x172 /home/magik6k/.opt/go/pkg/mod/github.com/ipfs/kubo@v0.19.1-0.20230331144503-5cb8134812c3/core/commands/add.go:273
```

Small-file large-directory adding could be made probably ~2x faster with:
* The quieter (-Q) flag could skip outputDirs entirely
* outputDirs could load entries in parallel
* Not easy because mfs Directory.Child takes the directory lock, which in case of one giant flat directory (e.g. wikipedia) won't work well
* The adder could cache what cid is/isn't a directory
* Could make outputDirs significantly faster as now it wouldn't need to load file nodes, just directories
* En Wikipedia is ~12M files, and it would probably be sufficient to key the cache with ~8 bytes of each cid, and you'd only need to mark what is a directory, which in most datasets will be way less blocks than files.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.