ipfs / ipfs/kubo

Consistent chunking for sequential writes to /mfs

Open
#5,129 1 comment 1 reaction 0 assignees View on GitHub
kind/enhancement topic/files topic/MFS
Dominant language
Go
Stars
17.1k
Forks
3.2k
Avg merge
3d 18h
Merged PRs (30d)
11

Description

I am attempting to improve [small write performance](https://github.com/piedar/js-ipfs-mount/issues/6) by sending optimally-sized buffers to `ipfs files write`. Ideally `--flush=false` standardizes the chunk size internally so consumers can [ignore the problem](https://github.com/piedar/js-ipfs-mount/commit/8e0cb9b3d2036d1d303a034ae2511f0548a746e9). Of course it is a difficult problem, though all the more reason to solve it correctly once.

### test case

```bash
file="/1M.random"
chunk_size=$(( 8 * 1024 ))
chunk_count=$(( 128 ))
```

For example, the pipe is fairly fast as a single write operation.

```bash
time dd if=/dev/urandom bs=$chunk_size count=$chunk_count | ipfs files write --create $file
```
```
128+0 records in
128+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731766 s, 14.3 MB/s

real 0m0.348s
user 0m0.064s
sys 0m0.036s
```

But this next way is __really__ slow because it doesn't merge sequential writes.
The loop adds pointless complexity, but it demonstrates the problem that naturally occurs when programs call write() with small buffers like 4096.

```bash
ipfs files write $file --create < /dev/null
time for chunk_num in $(seq 0 $chunk_count) ; do
dd if=/dev/urandom bs=$chunk_size count=1 | ipfs files write $file --flush=false --offset=$(( $chunk_num * $chunk_size ))
done ; ipfs files flush $file
```
```
real 0m22.955s
user 0m7.038s
sys 0m2.291s
```

The disk bottlenecks on all the small block writes, so it's at least 2x faster when `~/.ipfs/` is on tmpfs.
When `--flush=false` there is an opportunity for chunk size optimization using a write cache.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.