Benchmark compression on Raft Snapshots to reduce disk IO
Open
needs-investigation
theme/performance
type/enhancement
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
Currently raft-initiated snapshots are stored on disk as plain length-prefixed msgpack serialised records.
We hypothesise that under heavy write load with large data sizes, the disk IO of writing out large snapshots to disk regularly causes raft throughput to tank since the snapshot is written to the same device as the raft log.
Given that, it _might_ be an easy-ish performance win for IO bound write workloads if we allow optional compression. GZIP might even be OK given the async nature of the snapshot process but faster compression algorithms like Snappy or LZ4 might also reduce IO considerably with lower CPU cost.
Contributor guide
Assessment
This issue has not been assessed yet.