erigontech / erigontech/erigon

Disk preallocation of snapshot files

Open
#16,743 1 comment 1 reaction 0 assignees View on GitHub
ErigonDB Networking
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

Per the networking meeting, I proposed preallocating (`fallocate`, and `fcntl` with `F_PREALLOCATE`) space for snapshot files (during initial sync). This reduces out of disk space errors that crash with SIGBUS, or unexpected behaviour later in the sync. It also potentially reduces file fragmentation.

Further to that, we could also request the system to preallocate contiguously (`F_ALLOCATECONTIG`). The startup time could be increased but there's potential performance improvements operating on snapshots that are indexes etc., especially via memory mapping.

Preallocation on most systems can be done without requiring actually zeroing or writing the data in preparation. The system marks sectors/clusters as inuse but doesn't actually touch them.

This would benefit the Downloader for predictability and some very minor improved write speed (except maybe on external USBs it could be much larger :] ). But it also has knock-on effects for later use of snapshots if the data has already been laid out in an optimized fashion.

@mh0lt

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.