erigontech / erigontech/erigon
InvertedIndex v2 table format
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 465
Description
Current format: “addr -> txnums”
This format is unfriendly for: adding new events, pruning. Because it cause “random updates” (txs are updating unpredictable adds).
Requirements:
- query eth_getLogs where addr=X and txNum between A and B
- prune - less random writes
- inert batch of new events (addrs updated in given txnum) less random writes
Solution: “step+addr -> txNums”
Then:
- all prune will come to old step and inserts to new step (and both will be smaller).
- no reason to read from old step: we prune it because we have it in file already. Mean can prune sequentially (leave garbage in old step - half-pruned txnums) - it will make prune more time-determenistic
Problem: how to know current step?
—————
Gnosis around attack:
```
du -h /erigon-data/chaindata/
439G /erigon-data/chaindata/
./build/bin/mdbx_stat -efa /erigon-data/chaindata/
mdbx_stat v0.13.6-0-ga971c76a (2025-04-22T11:53:23+03:00, T-4ca2c913e8614a1ed09512353faa227f25245e9f)
Running for /erigon-data/chaindata/...
open-MADV_DONTNEED 114925568..114925568
readahead OFF 0..114925568
Environment Info
Pagesize: 4096
Dynamic datafile: 262144..1099511627776 bytes (+16777216/-33554432), 64..268435456 pages (+4096/-8192)
Current mapsize: 1099511627776 bytes, 268435456 pages
Current datafile: 470735126528 bytes, 114925568 pages
Last transaction ID: 676757
Latter reader transaction ID: 676757 (0)
Max readers: 32112
Number of reader slots uses: 22
Garbage Collection
Pagesize: 4096
Tree depth: 2
Branch pages: 1
Leaf pages: 65
Overflow pages: 11771
Entries: 11773
Page Usage
Total: 268435456 100%
Backed: 114925568 42.8%
Allocated: 114924826 42.8%
Remained: 153510630 57.2%
Used: 102941435 38.3%
GC: 11983391 4.5%
Retained: 831 0.0%
Reclaimable: 11982560 4.5%
Available: 165493190 61.7%
Status of Main DB
Pagesize: 4096
Tree depth: 2
Branch pages: 1
Leaf pages: 3
Overflow pages: 0
Entries: 121
Status of AccountChangeSet
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of AccountHistory
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of AccountHistoryKeys
Pagesize: 4096
Tree depth: 3
Branch pages: 69
Leaf pages: 15194
Overflow pages: 0
Entries: 1565335
Status of AccountHistoryVals
Pagesize: 4096
Tree depth: 3
Branch pages: 1592
Leaf pages: 23524
Overflow pages: 0
Entries: 1565335
Status of AccountIdx
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of AccountVals
Pagesize: 4096
Tree depth: 3
Branch pages: 5
Leaf pages: 356
Overflow pages: 0
Entries: 16870
Status of ActiveValidatorIndicies
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BadHeaderNumber
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BalancesDump
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BeaconBlock
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BeaconBlockHeaders
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BlockBody
Pagesize: 4096
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 1
Status of BlockBorTransactionLookup
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BlockProposers
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BlockRoot
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BlockRootToBlockHash
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BlockRootToBlockNumber
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BlockRootToKzgCommitments
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BlockRootToParentRoot
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BlockRootToSlot
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BlockRootToStateRoot
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BlockTransaction
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BlockTransactionLookup
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BorCheckpointEnds
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BorCheckpoints
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BorEventNums
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BorEventProcessedBlocks
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BorEvents
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BorFinality
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BorMilestoneEnds
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BorMilestones
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BorProducerSelections
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BorSeparate
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of BorSpans
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of CanonicalBlockRoots
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of CanonicalHeader
Pagesize: 4096
Tree depth: 4
Branch pages: 2207
Leaf pages: 493976
Overflow pages: 0
Entries: 40012000
Status of ChangeSets3
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of Code
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of CodeHistoryKeys
Pagesize: 4096
Tree depth: 2
Branch pages: 1
Leaf pages: 6
Overflow pages: 0
Entries: 619
Status of CodeHistoryVals
Pagesize: 4096
Tree depth: 2
Branch pages: 1
Leaf pages: 8
Overflow pages: 0
Entries: 619
Status of CodeIdx
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of CodeVals
Pagesize: 4096
Tree depth: 2
Branch pages: 1
Leaf pages: 44
Overflow pages: 154
Entries: 619
Status of CommitmentHistoryKeys
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of CommitmentHistoryVals
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of CommitmentIdx
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of CommitmentVals
Pagesize: 4096
Tree depth: 4
Branch pages: 1787
Leaf pages: 110653
Overflow pages: 0
Entries: 559387
Status of Config
Pagesize: 4096
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 10
Entries: 2
Status of CurrentSyncCommittee
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of DbInfo
Pagesize: 4096
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 8
Status of DevEpoch
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of DevPendingEpoch
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of EffectiveBalancesDump
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of EpochData
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of Eth1DataVotes
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of HashedAccount
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of HashedCodeHash
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of HashedStorage
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of Header
Pagesize: 4096
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 1
Status of HeaderNumber
Pagesize: 4096
Tree depth: 4
Branch pages: 6769
Leaf pages: 642960
Overflow pages: 0
Entries: 40012000
Status of HeadersTotalDifficulty
Pagesize: 4096
Tree depth: 5
Branch pages: 8805
Leaf pages: 704170
Overflow pages: 0
Entries: 40012000
Status of HighestFinalized
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of InactivityScores
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of IncarnationMap
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of IntraRandaoMixes
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of LastBeaconSnapshot
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of LastBlock
Pagesize: 4096
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 1
Status of LastForkchoice
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of LastHeader
Pagesize: 4096
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 1
Status of LogAddressIdx
Pagesize: 4096
Tree depth: 4
Branch pages: 18239
Leaf pages: 1073696
Overflow pages: 0
Entries: 232874672
Status of LogAddressKeys
Pagesize: 4096
Tree depth: 4
Branch pages: 11004
Leaf pages: 2461618
Overflow pages: 0
Entries: 232874672
Status of LogTopicsIdx
Pagesize: 4096
Tree depth: 5
Branch pages: 847479
Leaf pages: 26624088
Overflow pages: 0
Entries: 3172057989
Status of LogTopicsKeys
Pagesize: 4096
Tree depth: 4
Branch pages: 2073284
Leaf pages: 46552860
Overflow pages: 0
Entries: 3172057989
Status of MaxTxNum
Pagesize: 4096
Tree depth: 4
Branch pages: 1147
Leaf pages: 256488
Overflow pages: 0
Entries: 40012000
Status of Migration
Pagesize: 4096
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 5
Status of NextSyncCommittee
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of ParentRootToBlockRoots
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of PendingConsolidations
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of PendingConsolidationsDump
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of PendingDeposits
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of PendingDepositsDump
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of PendingPartialWithdrawals
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of PendingPartialWithdrawalsDump
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of PlainCodeHash
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of PlainState
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of PruningProgress
Pagesize: 4096
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 12
Status of RandaoMixes
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of ReceiptCache
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of ReceiptCacheHistoryKeys
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of ReceiptCacheHistoryVals
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of ReceiptCacheIdx
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of ReceiptCacheVals
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of ReceiptHistoryKeys
Pagesize: 4096
Tree depth: 3
Branch pages: 16
Leaf pages: 3204
Overflow pages: 0
Entries: 490823
Status of ReceiptHistoryVals
Pagesize: 4096
Tree depth: 1
Branch pages: 17
Leaf pages: 2580
Overflow pages: 0
Entries: 490823
Status of ReceiptIdx
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of ReceiptVals
Pagesize: 4096
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 2
Status of Sequence
Pagesize: 4096
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 2
Status of SlotData
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of StateEvents
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of StateRoot
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of StateRootToBlockRoot
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of StatesProcessingProgress
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of StaticValidators
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of StorageChangeSet
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of StorageHistory
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of StorageHistoryKeys
Pagesize: 4096
Tree depth: 3
Branch pages: 1682
Leaf pages: 34527
Overflow pages: 0
Entries: 1773508
Status of StorageHistoryVals
Pagesize: 4096
Tree depth: 4
Branch pages: 1843
Leaf pages: 26845
Overflow pages: 0
Entries: 1773508
Status of StorageIdx
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of StorageVals
Pagesize: 4096
Tree depth: 4
Branch pages: 239
Leaf pages: 9970
Overflow pages: 0
Entries: 317591
Status of SyncStage
Pagesize: 4096
Tree depth: 1
Branch pages: 0
Leaf pages: 1
Overflow pages: 0
Entries: 12
Status of TracesFromIdx
Pagesize: 4096
Tree depth: 4
Branch pages: 55971
Leaf pages: 2592031
Overflow pages: 0
Entries: 548481250
Status of TracesFromKeys
Pagesize: 4096
Tree depth: 4
Branch pages: 24319
Leaf pages: 5439597
Overflow pages: 0
Entries: 548481250
Status of TracesToIdx
Pagesize: 4096
Tree depth: 4
Branch pages: 48824
Leaf pages: 4239601
Overflow pages: 0
Entries: 926266868
Status of TracesToKeys
Pagesize: 4096
Tree depth: 4
Branch pages: 41138
Leaf pages: 8474981
Overflow pages: 0
Entries: 926266868
Status of TrieAccount
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of TrieStorage
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of TxSender
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of ValidatorBalance
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of ValidatorEffectiveBalance
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of ValidatorSlashings
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of VerkleRoots
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
Status of VerkleTrie
Pagesize: 4096
Tree depth: 0
Branch pages: 0
Leaf pages: 0
Overflow pages: 0
Entries: 0
```
Contributor guide
Research direction
Start by locating the InvertedIndex v2 implementation and tracing the current addr -> txnums format through eth_getLogs, pruning, and batched event insertion. Define how the current step is determined, then verify that address-and-range queries work while pruning and new-event batches produce fewer random writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100