erigontech / erigontech/erigon
compression: `replacement preprocessing` add support of sampling
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
Context:
`replacement preprocessing` phase used for dictionary reduction. and it's currently a bottleneck.
we already have `SamplingFactor: 4` which is used in another phase - `patterns generation from superstrings`.
Step1: Investigate how `queueLimit` affecting performance/compression_ratio.
Step2: Add one more sampling parameter to `replacement preprocessing` phase. Maybe it's not possible - then discover another optimization for this phase.
For profiling can use next command (it supports uncompressed and compressed files) or write benchmark:
```
go run ./cmd/erigon snapshots uncompress /erigon-data/snapshots/v1-001400-001500-transactions.seg | DictReducerSoftLimit=2000000 MinPatternLen=20 MaxPatternLen=32 SamplingFactor=4 OnlyKeys=true go run ./cmd/erigon snapshots compress --datadir=/erigon-data/erigon3/ --log.console.verbosity=5 --pprof /erigon-data/snapshots/v1-001400-001500-transactions.seg.compressed_s4_soft2_min20_max32 > ~/log.txt 2>&1 &
```
It's ok to make some preparation PR's: to separate refactoring from optimizations/features, etc...
Contributor guide
Assessment
This issue has not been assessed yet.