erigontech / erigontech/erigon

compression: `replacement preprocessing` add support of sampling

Open
#11,720 0 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

Context:
`replacement preprocessing` phase used for dictionary reduction. and it's currently a bottleneck.
we already have `SamplingFactor: 4` which is used in another phase - `patterns generation from superstrings`.

Step1: Investigate how `queueLimit` affecting performance/compression_ratio.

Step2: Add one more sampling parameter to `replacement preprocessing` phase. Maybe it's not possible - then discover another optimization for this phase.

For profiling can use next command (it supports uncompressed and compressed files) or write benchmark:
```
go run ./cmd/erigon snapshots uncompress /erigon-data/snapshots/v1-001400-001500-transactions.seg | DictReducerSoftLimit=2000000 MinPatternLen=20 MaxPatternLen=32 SamplingFactor=4 OnlyKeys=true go run ./cmd/erigon snapshots compress --datadir=/erigon-data/erigon3/ --log.console.verbosity=5 --pprof /erigon-data/snapshots/v1-001400-001500-transactions.seg.compressed_s4_soft2_min20_max32 > ~/log.txt 2>&1 &
```

It's ok to make some preparation PR's: to separate refactoring from optimizations/features, etc...

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.