erigontech / erigontech/erigon
store first entity num in snapshot file.
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
first entity num: each snapshot stores a particular entity. The first entity in the snapshot has a canonical identifier (calling it Num). Putting this Num of first entity in the snapshot is useful.
The API will be exposed in seg.Decompressor -- `FirstEntityNum() uint64`
why:
**index building**
Index building needs "baseDataId" which is in most cases the num of the first entity in snapshot. Currently, indexbuilder does different things to get this `baseDataId` or "firstNum":
- for milestone/checkpoint, get the first value in snapshot, unmarshal it and get id.
- headers/bodies/caplin stuff: they know easily, because blockFrom is available (=firstNum);
- borspan: a function from blockNum -> spanId is used to get first entity.
- borevents: look into first value in the snapshot, some range of bytes in it is eventId.
- transactions:
- txHashIdx -- get first body in bodies snapshot, and get baseTxId
- txnHash2BlockNumIdx -- baseId is useless (enum=false)
All these cases, can be covered by simply putting baseDataId (first entity num) into the snapshot file. Then indexbuilder reads this from snapshot file and can just use the value for baseDataId. No need for unmarshalling/calculating stuff.
This allows for a more minimal (less callback to pass) for a "SimpleIndexBuilder" impl, which creates a recsplit index providing ordinal lookups from entity num to offset (like `BeaconSimpleIdx`).
This will be useful today and for appendables too.
---
Note that this is only needed when the sharding key (root num) is not same as the entity num. e..g borcheckpoints/spans/txs. But for entities which have the sharding key and entity key the same, we can deduce first entity num from the filename itself.
So we might not need to do this for snapshots of all entities.
Contributor guide
Assessment
This issue has not been assessed yet.