erigontech / erigontech/erigon

cl/persistence/blob_storage: GetSavedColumnIndex does 128 stat calls where one MDBX read would do

Open
#23,433 0 comments 0 reactions 0 assignees View on GitHub
Caplin
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

`GetSavedColumnIndex` answers "which columns do I hold for this root" by stat-ing every possible column (`cl/persistence/blob_storage/data_column_db.go:197-212`):

```go
for i := uint64(0); i < s.beaconChainConfig.NumberOfColumns; i++ {
_, filepath := dataColumnFilePath(slot, blockRoot, i)
if _, err := s.fs.Stat(filepath); os.IsNotExist(err) {
continue
} else if err != nil {
return nil, err
}
savedColumns = append(savedColumns, i)
}
```

`NumberOfColumns` is 128, and the callers are hot: `cl/das/peer_das.go:248` (`IsColumnOverHalf`), `:272`, `:360` (blob recovery), `:996` (download planning). Each call is 128 stats inside a directory holding up to 1.28M entries — #23426.

The blob store already has the right shape for this: `kv.BlockRootToKzgCommitments`, root → count, keyed by what callers hold and answering what the filesystem can only answer by enumeration.

`kv.BlockRootToDataColumnCount` is declared at `db/kv/tables.go:220` and registered in the table list at `:420`, and is never read or written anywhere in the tree, tests included. Every caplin DB creates it empty.

Wire it as root → 128-bit bitmap, 16 bytes:

- `GetSavedColumnIndex` — read the row, expand set bits
- `IsColumnOverHalf` (`:248`) and the recovery gate (`:360`) need only a count — popcount, no list built
- `:272` and `:996` diff against the custody set, which the bitmap gives directly
- `WriteColumnSidecars` / `RemoveColumnSidecars` — read-modify-write one bit, under the store's slot stripe

A count would not do: two of the four callers need the set, not its size.

The design has to make the index share the store's lifecycle — #23432 is the failure mode on the blob side, where pruning drops the files and leaves the rows.

Best after #23413, so the read-modify-write goes into the shared write path once instead of being added and then moved. It also reprices #23426 downward: the 128-stat path is the main reason the flat directory hurts.

Part of #23024.

Contributor guide

Open the contributing guide

Research direction

Start with cl/persistence/blob_storage/data_column_db.go:197-212, the callers in cl/das/peer_das.go, and the BlockRootToDataColumnCount declaration and registration in db/kv/tables.go. Trace the blob store lifecycle and shared write path, including pruning, before running the relevant persistence and DAS tests. Done means the bitmap stays consistent with sidecar files and replaces repeated stats for all listed callers.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.