erigontech / erigontech/erigon
Reduce amount of files offline-merge Caplin files on snapshotters
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
Now `mainnet.toml` and `gnosis.toml` on `release/3.6` branch of erigon-snapshot repo - have 10K-30K Caplin files
I guess we can offline-merge files on Snapshotter machines (by `seg retire` command). Just need to check that we have enough existing caplin-integrity-checks in `seg integrity` command.
Maybe do same for block files (we did attempt to merge them to 500K 1-2 years ago - but faced bug and rollback)
Example from gnosis.toml:
```
'caplin/v1.1-013500-013550-ActiveValidatorIndicies.idx' = 'c3d698fcbb8223a12e836b640630d5184412dab5'
'caplin/v1.1-013500-013550-ActiveValidatorIndicies.seg' = '687336c91378cf2afa61f219d0a1e9516b5f8b02'
'caplin/v1.1-013500-013550-BalancesDump.idx' = '4aa9aaab6ba055360a4bb6eb9a0d5dc66d016c88'
'caplin/v1.1-013500-013550-BalancesDump.seg' = '29f3e68deb96b113bd9f12a35b8ccdecb4a7c96f'
'caplin/v1.1-013500-013550-BlockRoot.idx' = 'b8afb9bf87a9fa23f267c131c1ca7b1f4f829bb9'
'caplin/v1.1-013500-013550-BlockRoot.seg' = '2ad1305a867e7873bdcb258699b7fc8052c10087'
'caplin/v1.1-013500-013550-CurrentSyncCommittee.idx' = '8b408362921a0b66266a1c127e86bbe3c18d1619'
'caplin/v1.1-013500-013550-CurrentSyncCommittee.seg' = '85494dba23185006b756772bbe61b46941f47b7d'
'caplin/v1.1-013500-013550-EffectiveBalancesDump.idx' = '14e21b716953456db3b1cb0aba2c1bb881ed8b39'
'caplin/v1.1-013500-013550-EffectiveBalancesDump.seg' = 'c03ec013f60553500bf4b8205d799b6578365582'
'caplin/v1.1-013500-013550-EpochData.idx' = 'c343858a896013c1ec8d927a25a9802543344c12'
'caplin/v1.1-013500-013550-EpochData.seg' = '52670bdfec1d9923eca73e2d8b988731d7639667'
'caplin/v1.1-013500-013550-Eth1DataVotes.idx' = 'feeaf90ca336ccc246632b7fbd725714bdc75b8f'
'caplin/v1.1-013500-013550-Eth1DataVotes.seg' = '6fb14eae1a85c9bbd66b7141c2dc974d01f3df9c'
'caplin/v1.1-013500-013550-InactivityScores.idx' = '4d7cd04b2023e2bdc0fac60c7509c9658acb1dcc'
'caplin/v1.1-013500-013550-InactivityScores.seg' = '4b1026079e4411841661d2a1717baee6892541a1'
'caplin/v1.1-013500-013550-IntraRandaoMixes.idx' = '6951f855302893f09911a9b2a731bfaf3341bd38'
'caplin/v1.1-013500-013550-IntraRandaoMixes.seg' = '101842040b0b34bc1a08fc56015ade465eed80f9'
'caplin/v1.1-013500-013550-NextSyncCommittee.idx' = '906809c071868e253bcf4b8a2b658920420d69e1'
'caplin/v1.1-013500-013550-NextSyncCommittee.seg' = 'c28901e9a780b48787527dd1e6d264dcd410d59b'
'caplin/v1.1-013500-013550-PendingConsolidations.idx' = '5966a76b660d238fcf59e0a7a0105da900b4e999'
'caplin/v1.1-013500-013550-PendingConsolidations.seg' = '79f7aad0232943caa7293678cf4aea944604e95b'
'caplin/v1.1-013500-013550-PendingConsolidationsDump.idx' = '0627887b674dfb33e024de5b25ed71455ab7e140'
'caplin/v1.1-013500-013550-PendingConsolidationsDump.seg' = 'eed2cc9184320fdcd8149bea9b6496afb249cb6a'
'caplin/v1.1-013500-013550-PendingDeposits.idx' = '399a0a04dc15b1d3f2e66d9de65679312ff76964'
'caplin/v1.1-013500-013550-PendingDeposits.seg' = '013a714131adc56e1f2f2f90bf7455c68a91289b'
'caplin/v1.1-013500-013550-PendingDepositsDump.idx' = '5ef194165c4ead8cfecba4651c83f055adb31115'
'caplin/v1.1-013500-013550-PendingDepositsDump.seg' = '9a4a11d8135229d9647d8c33d2358fef79eca4a7'
'caplin/v1.1-013500-013550-PendingPartialWithdrawals.idx' = 'dee72f7a03bf1ff725be111fac13b91da3660751'
'caplin/v1.1-013500-013550-PendingPartialWithdrawals.seg' = 'a1b1d1efcfc2b1403baff1bf16e18303cebc1717'
'caplin/v1.1-013500-013550-PendingPartialWithdrawalsDump.idx' = '63dedddc060a18679c4308a4427f5dd4f71bd839'
'caplin/v1.1-013500-013550-PendingPartialWithdrawalsDump.seg' = '219015cad040190d522b32cffeb5aafd34048e0a'
'caplin/v1.1-013500-013550-RandaoMixes.idx' = 'b18a1ad3dfeed69c2d42c5f18a4457fec9c07d5c'
'caplin/v1.1-013500-013550-RandaoMixes.seg' = '8be9ccd66f93df527e0e41151118cd5a78f7b0b4'
'caplin/v1.1-013500-013550-SlotData.idx' = '513719aa5e2329fcbdbdf09fc71d24dd83d1edf4'
'caplin/v1.1-013500-013550-SlotData.seg' = 'c5c51f7d18066fea7b2da5aa84b20a834a5a4076'
'caplin/v1.1-013500-013550-StateEvents.idx' = '5a5c56021a12eeabb132d19f8bd9ea9be984f09b'
'caplin/v1.1-013500-013550-StateEvents.seg' = 'b6bedd06f722e8689a4ac7ef8101feb6162889c2'
'caplin/v1.1-013500-013550-StateRoot.idx' = 'b9b762c601e03e7559d6cc9c53051f70f150730e'
'caplin/v1.1-013500-013550-StateRoot.seg' = '222c49ef528f673840ceea0110220f42b9cb2a82'
'caplin/v1.1-013500-013550-ValidatorBalance.idx' = '8f5fc54bdc486a2db025e4428f5de75cc62f1efb'
'caplin/v1.1-013500-013550-ValidatorBalance.seg' = '49e74fb03f784195698e76cb7b861c59f62f641b'
'caplin/v1.1-013500-013550-ValidatorEffectiveBalance.idx' = '1be02fb7fe0b0b079ab46787d4566e8ef1be3076'
'caplin/v1.1-013500-013550-ValidatorEffectiveBalance.seg' = '1c82282a3e84197cdd4a11cadf91d795d038baab'
'caplin/v1.1-013500-013550-ValidatorSlashings.idx' = '8fbdf33a9956ef5b57510561f9bb1d5555886ccc'
'caplin/v1.1-013500-013550-ValidatorSlashings.seg' = 'e5d063560a18787b9773a065b594870a560ba8dd'
```
Contributor guide
Research direction
Start with mainnet.toml and gnosis.toml on the release/3.6 branch of the erigon-snapshot repository, then inspect the seg retire and seg integrity commands. Determine whether offline-merging Caplin files preserves the existing integrity checks, and investigate the earlier block-file merge attempt and its rollback bug before defining completion criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- blockchain, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100