erigontech / erigontech/erigon
Disk preallocation of snapshot files
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
Per the networking meeting, I proposed preallocating (`fallocate`, and `fcntl` with `F_PREALLOCATE`) space for snapshot files (during initial sync). This reduces out of disk space errors that crash with SIGBUS, or unexpected behaviour later in the sync. It also potentially reduces file fragmentation.
Further to that, we could also request the system to preallocate contiguously (`F_ALLOCATECONTIG`). The startup time could be increased but there's potential performance improvements operating on snapshots that are indexes etc., especially via memory mapping.
Preallocation on most systems can be done without requiring actually zeroing or writing the data in preparation. The system marks sectors/clusters as inuse but doesn't actually touch them.
This would benefit the Downloader for predictability and some very minor improved write speed (except maybe on external USBs it could be much larger :] ). But it also has knock-on effects for later use of snapshots if the data has already been laid out in an optimized fashion.
@mh0lt
Contributor guide
Assessment
This issue has not been assessed yet.