erigontech / erigontech/erigon
files: format and packing changes for the next file generation
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
- [ ] `.vi` reduce size by storing offsets array as EF: https://github.com/erigontech/erigon/pull/23844
- [ ] add Compression cfg to file header: #20074. The reader takes the compression state from the config rather than the file, so every change above needs a coordinated bump. A self-describing header removes that coupling.
- [ ] Bigger bittorrent piece size to reduce RAM usage by hashes: https://github.com/erigontech/erigon/issues/21684
- [ ] Cut the file count: Merge Caplin files
- [ ] Cut the file count: Merge Blocks files
- [ ] Re-compress `.seg` with `SamplingFactor: 1`
- [ ] Commitment: one branch node per key
----
- [ ] rcache page size 16 -> 64 (to decide)
`ValuesOnCompressedPage: 16` in `db/state/statecfg/state_schema.go` is the only compression rcache has — `Compression: seg.CompressNone` on both the domain and its history. Measured on 400k real receipts pulled from `v3.1-rcache.9644-9646.v` and rewritten through `seg.PagedWriter`:
| page | size MB | B/receipt | ratio | 1-tx read, us |
|---|---:|---:|---:|---:|
| 16 | 63.1 | 157.7 | 3.36 | 6.1 |
| 32 | 59.6 | 149.1 | 3.56 | 13.6 |
| 64 | 56.8 | 141.9 | 3.74 | 23.7 |
| 128 | 54.3 | 135.8 | 3.90 | 35.5 |
| 255 | 52.1 | 130.2 | 4.07 | 86.5 |
64 is -10% (~40 GB off a mainnet archive) for 24 us on a single-receipt read. Block-scoped reads (`eth_getBlockReceipts`, `eth_getLogs`) are flat — one page decompress per ~180 receipts either way. `AccountsDomain.Hist` already uses 64.
But also need investigate zstd compression params (maybe some of them still can give good 1-page-decompression speed)
Contributor guide
Research direction
Start with db/state/statecfg/state_schema.go and the rcache measurements, then trace the .vi and .seg generation and reader paths mentioned in the checklist. Review the linked PRs and issues before deciding which format, file-count, piece-size, and compression changes can be coordinated; done requires an agreed scope and validated size, read-latency, and compatibility results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100