erigontech / erigontech/erigon
cl/persistence/blob_storage: flat 10k-slot bucket directories reach 1.28M entries on a supernode
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
Both hot sidecar stores put every file in one flat directory per 10,000 slots: `/_` (`cl/persistence/blob_storage/blob_db.go:44-46,71-77`, `data_column_db.go:62-68`).
That directory holds up to 90,000 blob files, or 1,280,000 column files on a supernode custodying all 128 columns.
Pruning is fine — it only lists bucket names. The per-object paths are not: `GetSavedColumnIndex` and `RemoveAllColumnSidecars` each do `NumberOfColumns` `Stat`/`Remove` calls inside that directory, per availability check and per removal (`data_column_db.go:154-163,197-212`).
Fix: a per-slot leaf, `//_` — at most 9 blob or 128 column files per directory. It also lets the retention floor be exact; today the cutoff rounds down to a bucket boundary and keeps up to 9,999 extra slots.
Migration has to be an in-place rename walk, not a delete-and-refill. These files are never torrented — `db/downloader/` has no reference to either directory — and `erigon snapshots reset` is scoped to `SnapDir` (`db/datadir/reset/reset.go:43,131`), so there is no canonical copy to re-fetch; refill is p2p backfill, bounded by the retention window. Under `--caplin.blobs-archive` without `--caplin.snapgen` the hot store is the only copy of everything below that window and peers will not serve it back.
`Dirs.RenameOldVersions` (`db/datadir/dirs.go:275-347`) already walks both directories at startup and renames in place, so the hook exists; it currently matches only a `vX-` filename prefix. Note #23429 — that walk is pure waste today.
Blocked on #23413 — after the collapse this is one path function plus the pruner, not the same edit twice.
Part of #23024, item 4.
Contributor guide
Research direction
Read cl/persistence/blob_storage/blob_db.go, data_column_db.go, and Dirs.RenameOldVersions in db/datadir/dirs.go, starting with the existing path construction and startup rename walk. Coordinate with blocked issue #23413. Done means the in-place migration preserves files, the new per-slot layout limits directory size, and retention no longer rounds down to a bucket boundary.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 42/100