erigontech / erigontech/erigon

files: format and packing changes for the next file generation

Open
#23,787 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

- [ ] `.vi` reduce size by storing offsets array as EF: https://github.com/erigontech/erigon/pull/23844

- [ ] add Compression cfg to file header: #20074. The reader takes the compression state from the config rather than the file, so every change above needs a coordinated bump. A self-describing header removes that coupling.

- [ ] Bigger bittorrent piece size to reduce RAM usage by hashes: https://github.com/erigontech/erigon/issues/21684

- [ ] Cut the file count: Merge Caplin files
- [ ] Cut the file count: Merge Blocks files

- [ ] Re-compress `.seg` with `SamplingFactor: 1`
- [ ] Commitment: one branch node per key

----

- [ ] rcache page size 16 -> 64 (to decide)

`ValuesOnCompressedPage: 16` in `db/state/statecfg/state_schema.go` is the only compression rcache has — `Compression: seg.CompressNone` on both the domain and its history. Measured on 400k real receipts pulled from `v3.1-rcache.9644-9646.v` and rewritten through `seg.PagedWriter`:

| page | size MB | B/receipt | ratio | 1-tx read, us |
|---|---:|---:|---:|---:|
| 16 | 63.1 | 157.7 | 3.36 | 6.1 |
| 32 | 59.6 | 149.1 | 3.56 | 13.6 |
| 64 | 56.8 | 141.9 | 3.74 | 23.7 |
| 128 | 54.3 | 135.8 | 3.90 | 35.5 |
| 255 | 52.1 | 130.2 | 4.07 | 86.5 |

64 is -10% (~40 GB off a mainnet archive) for 24 us on a single-receipt read. Block-scoped reads (`eth_getBlockReceipts`, `eth_getLogs`) are flat — one page decompress per ~180 receipts either way. `AccountsDomain.Hist` already uses 64.

But also need investigate zstd compression params (maybe some of them still can give good 1-page-decompression speed)

Contributor guide

Open the contributing guide

Research direction

Start with db/state/statecfg/state_schema.go and the rcache measurements, then trace the .vi and .seg generation and reader paths mentioned in the checklist. Review the linked PRs and issues before deciding which format, file-count, piece-size, and compression changes can be coordinated; done requires an agreed scope and validated size, read-latency, and compatibility results.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.