erigontech / erigontech/erigon

store first entity num in snapshot file.

Open
#13,342 10 comments 0 reactions 0 assignees View on GitHub
ErigonDB
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

first entity num: each snapshot stores a particular entity. The first entity in the snapshot has a canonical identifier (calling it Num). Putting this Num of first entity in the snapshot is useful.

The API will be exposed in seg.Decompressor -- `FirstEntityNum() uint64`

why:

**index building**

Index building needs "baseDataId" which is in most cases the num of the first entity in snapshot. Currently, indexbuilder does different things to get this `baseDataId` or "firstNum":
- for milestone/checkpoint, get the first value in snapshot, unmarshal it and get id.
- headers/bodies/caplin stuff: they know easily, because blockFrom is available (=firstNum);
- borspan: a function from blockNum -> spanId is used to get first entity.
- borevents: look into first value in the snapshot, some range of bytes in it is eventId.
- transactions:
- txHashIdx -- get first body in bodies snapshot, and get baseTxId
- txnHash2BlockNumIdx -- baseId is useless (enum=false)

All these cases, can be covered by simply putting baseDataId (first entity num) into the snapshot file. Then indexbuilder reads this from snapshot file and can just use the value for baseDataId. No need for unmarshalling/calculating stuff.
This allows for a more minimal (less callback to pass) for a "SimpleIndexBuilder" impl, which creates a recsplit index providing ordinal lookups from entity num to offset (like `BeaconSimpleIdx`).
This will be useful today and for appendables too.

---

Note that this is only needed when the sharding key (root num) is not same as the entity num. e..g borcheckpoints/spans/txs. But for entities which have the sharding key and entity key the same, we can deduce first entity num from the filename itself.
So we might not need to do this for snapshots of all entities.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.