erigontech / erigontech/erigon
db: auto-compact mdbx databases at startup
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
At Erigon startup, add an auto-compact feature for mdbx databases.
## Plan
- [ ] **Base PR: auto-compact at startup.** If a db has `ReclaimableSpace` (`db/kv/table_sizes.go`) 3x bigger than its data, run the same compaction as `erigon db compact` (`backup.CompactDatadir`) on this db. `db.Close()` already waits for all readers to finish.
- `cmd/integration` must do it. `cmd/rpcdaemon` must not do it. `datadir.ApplyMigrations` probably already has this logic: a command which is allowed to apply migrations is also allowed to compact.
- Test it on n5. Find where it does not work well and improve it.
- Review the change, fix the findings, open the PR.
- [ ] **Draft PR (based on the base PR): compact after the initial-cycle prune.** In the initial cycle (not on chain tip), after the prune RwTx commit, compact chaindata.
- [ ] **Draft PR (based on the base PR): online compaction.** Use https://github.com/erigontech/mdbx-go/pull/223 (`Env.Copy*` with `MDBX_CP_OVERWRITE`, `Env.Defrag`) to compact the db without closing it.
- [ ] **Compare on n5** the three approaches above: stability, robustness, speed, implementation effort.
- [ ] **Draft PR (based on the base PR): retire when steps accumulate.** If `stepsInDb > 3`, run `erigon seg retire` logic (Collate + BlockingBuild + Prune, without Merge), then auto-compact after Prune.
Contributor guide
Assessment
This issue has not been assessed yet.