erigontech / erigontech/erigon

db: auto-compact mdbx databases at startup

Open
#23,954 0 comments 0 reactions 1 assignee Claimed by @AskAlexSharov View on GitHub
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

At Erigon startup, add an auto-compact feature for mdbx databases.

## Plan

- [ ] **Base PR: auto-compact at startup.** If a db has `ReclaimableSpace` (`db/kv/table_sizes.go`) 3x bigger than its data, run the same compaction as `erigon db compact` (`backup.CompactDatadir`) on this db. `db.Close()` already waits for all readers to finish.
- `cmd/integration` must do it. `cmd/rpcdaemon` must not do it. `datadir.ApplyMigrations` probably already has this logic: a command which is allowed to apply migrations is also allowed to compact.
- Test it on n5. Find where it does not work well and improve it.
- Review the change, fix the findings, open the PR.
- [ ] **Draft PR (based on the base PR): compact after the initial-cycle prune.** In the initial cycle (not on chain tip), after the prune RwTx commit, compact chaindata.
- [ ] **Draft PR (based on the base PR): online compaction.** Use https://github.com/erigontech/mdbx-go/pull/223 (`Env.Copy*` with `MDBX_CP_OVERWRITE`, `Env.Defrag`) to compact the db without closing it.
- [ ] **Compare on n5** the three approaches above: stability, robustness, speed, implementation effort.
- [ ] **Draft PR (based on the base PR): retire when steps accumulate.** If `stepsInDb > 3`, run `erigon seg retire` logic (Collate + BlockingBuild + Prune, without Merge), then auto-compact after Prune.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.