Consistent chunking for sequential writes to /mfs
- Dominant language
- Go
- Stars
- 17.1k
- Forks
- 3.2k
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 11
Description
I am attempting to improve [small write performance](https://github.com/piedar/js-ipfs-mount/issues/6) by sending optimally-sized buffers to `ipfs files write`. Ideally `--flush=false` standardizes the chunk size internally so consumers can [ignore the problem](https://github.com/piedar/js-ipfs-mount/commit/8e0cb9b3d2036d1d303a034ae2511f0548a746e9). Of course it is a difficult problem, though all the more reason to solve it correctly once.
### test case
```bash
file="/1M.random"
chunk_size=$(( 8 * 1024 ))
chunk_count=$(( 128 ))
```
For example, the pipe is fairly fast as a single write operation.
```bash
time dd if=/dev/urandom bs=$chunk_size count=$chunk_count | ipfs files write --create $file
```
```
128+0 records in
128+0 records out
1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0731766 s, 14.3 MB/s
real 0m0.348s
user 0m0.064s
sys 0m0.036s
```
But this next way is __really__ slow because it doesn't merge sequential writes.
The loop adds pointless complexity, but it demonstrates the problem that naturally occurs when programs call write() with small buffers like 4096.
```bash
ipfs files write $file --create < /dev/null
time for chunk_num in $(seq 0 $chunk_count) ; do
dd if=/dev/urandom bs=$chunk_size count=1 | ipfs files write $file --flush=false --offset=$(( $chunk_num * $chunk_size ))
done ; ipfs files flush $file
```
```
real 0m22.955s
user 0m7.038s
sys 0m2.291s
```
The disk bottlenecks on all the small block writes, so it's at least 2x faster when `~/.ipfs/` is on tmpfs.
When `--flush=false` there is an opportunity for chunk size optimization using a write cache.
Contributor guide
Assessment
This issue has not been assessed yet.