erigontech / erigontech/erigon
data and indices files experiments
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 432
Description
single place to track different experiments about data/index snapshots...
- commitment: kvi vs bt -- https://github.com/erigontech/erigon/issues/21795: tldr bt for commitment looks much better
- page compression for domains:
> My benchmark is setup in such a way that it's doing query for 15k keys on the file (all page evicted); the metric I'm looking for is majorfault/op.
> so btree+interpolation works great -- for the btree mmap stuff, the cold faults amortises over the 15k queries + interpolation probes very near to actually page in kv -- so account/code/storage have like 1.1-1.2 majorf/op (M=256); for commitment it is 1.8 for M=256 and 1.3 for M=64 -- this in on big files; with small files everything warms up fast.
> page compression suffers because of page alignment - 1 compressed page might span multiple os pages (unaligned) and so has 1.5 majorf/op (although file is 33% smaller)
> The "aligned version" brings majorf/op closer to 1 but makes the file bigger (and has same cold latency has bt+interpolation)
- [lemonhash](https://github.com/erigontech/erigon/issues/21795#issuecomment-4706618123) -- 2x smaller MPHT indices (when keys are sorted)
- https://github.com/erigontech/erigon/issues/21823
- [analyse/bench: using bt instead of efi and vi](https://github.com/erigontech/erigon/issues/22472)
- faster bt by reducing M - https://github.com/erigontech/erigon/issues/21692
Contributor guide
Assessment
This issue has not been assessed yet.