erigontech / erigontech/erigon
db: visibleFiles evolution
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 463
Description
tasks:
**remove txNumMinimax logic, making entities work with unaligned orientation**
- [ ] BuildFilesInBackground - lastInDb use -- each domain/ii considered independently (`from=domain.visibleFilesTxNum; to=lastIdInDb/lastIdInDbNoHistory)`)
- [ ] `recalcVisibleFiles(dirtyFilesEndTxNumMinimax)` -> `dirtyFilesEndTxNum` considered separately for each domain/ii. (only constraint is foreign key)
- [ ] add tests for above
- [x] https://github.com/erigontech/erigon/issues/15081
- [x] https://github.com/erigontech/erigon/pull/15055
- [ ] https://github.com/erigontech/erigon/issues/13174
- [ ] https://github.com/erigontech/erigon/issues/13243
---
context:
1. about strong-aligment/weak-alignment: maybe after [https://github.com/erigontech/erigon/pull/15166](https://github.com/erigontech/erigon/pull/15166 "https://github.com/erigontech/erigon/pull/15166") we don't need alignment. Because this is reason why LogAddrs index was broken: it was not aligned to acc.kv - but used acc.kv progress as "prune to" target. Maybe we must add similar check to RoSnapshots (it's type visible files progress).,
2. And foreign keys in recalcVisible files likely will save us when: acc.kv merge done, but commitment.kv merge not done yet
```
1. 2./3. ah, okay to sum up:
- so we don't need weak alignment (miniMaxTxNum);,
- Major reason was prune, and that's fixed now. But another reason for weak alignment was was to "have less number of states" to test; should we keep it still?,
- but foreign keys in garbage and visiblesFiles - yes,
In forkables, I kept alignment (weak) as an optional property; So for normal erigon process, `alignment` is on. But for cmd regenerating some entity, it can be made `alignment` is off, and so can proceed independently. This is similar to what I did for foreign key in my PR too (rebuild_commitment turns foreign key checks off). (edited)
2. _[_10:41_]_
so assumption is that during normal erigon runs, backfill happens for all entity together. If you want to backfill or regen just one entity, write a cmd.
```
`"have less number of states" to test;` is nice thing - but as you can see seems we can't afford it. We already have non-aligned inverted_indices, they already suffer from over-prune because of non-alignment. So, yes alignment for state domains did reduce impact of over-prune (only InvertedIndices are affected) - so seems it anyway biting us. Also I see 1 cons of alignment: "devs starting use it" and write code like: `logAddressesIdx.Prune(to = commitmentDomain.Progress())` and when i writing `stage_custom_trace` it shoots. Another example: `BuildFilesInBackground`
`lastInDB := max( lastIdInDB(a.db, a.d[kv.AccountsDomain]), lastIdInDB(a.db, a.d[kv.CodeDomain]), lastIdInDB(a.db, a.d[kv.StorageDomain]), lastIdInDBNoHistory(a.db, a.d[kv.CommitmentDomain])) for ; step < lastInDB; step++ { if err := a.buildFiles(a.ctx, step); err != nil {`
So, i was needed to write `Build2` method to produce `ii` files in `stage_custom_trace` so, it's and biz-logic and assumptions: things will be aligned, commitment will not have history, etc... It makes `agg` not usable (for example in `Forkable` task)
Also: if we will have method to disable strong-align - then all (and strong and weak) will be our first-class sitizens - then we anyway need to support/test all their corner-case. Which - seems only increaing amount of possible states...
Contributor guide
Research direction
Start by tracing BuildFilesInBackground and recalcVisibleFiles, then inspect the related stage_custom_trace and RoSnapshots behavior described in the issue. Clarify how each domain/ii should progress independently, add tests for the changed behavior, and verify that foreign-key constraints remain correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, databases
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100