erigontech / erigontech/erigon
exec module: decouple startup, execution, prune, retire, and publish flows
- Dominant language
- Go
- Stars
- 3.6k
- Forks
- 1.5k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 455
Description
## Problem
The exec module's flows (ProcessFrozenBlocks, ValidateChain, UpdateForkChoice, prune, retire, publish) are heavily intertwined because they share a single RW transaction — an MDBX limitation where only one goroutine can hold an RwTx at a time.
This creates a serialization chain:
```
execute → flush → commit → prune → retire → publish
all serialized because they share one RwTx
```
Three resources compete:
1. **ExecModule semaphore(1)** — serializes Start/Validate/FCU; background prune holds it
2. **snBuildAllowed semaphore** — serializes block retirement vs state aggregation
3. **MDBX write lock** — only one RwTx at a time
## Current Flow Issues
### Startup (ProcessFrozenBlocks)
- `ExecModule.Start()` holds the semaphore for the entire duration of `ProcessFrozenBlocks` (potentially hours during initial sync)
- During this time, `Ready()` returns false, `ValidateChain()`/`UpdateForkChoice()` return `Busy`
- Snapshot download runs inside `ProcessFrozenBlocks` — downloading blocks the entire exec module
- No error recovery between startup and steady state: if `ProcessFrozenBlocks` fails partway, the exec module releases the semaphore in an inconsistent state
### Steady State (forkchoice/newpayload)
- `ValidateChain` does NOT persist state — validates in a throwaway tx, stores in `forkValidator.sharedDom`
- `UpdateForkChoice` merges extending fork state, then runs execution pipeline
- Two separate stage pipelines: `ProcessFrozenBlocks` uses full pipeline, `ValidateChain` uses in-memory pipeline with different `badBlockHalt` settings
- No distinction between "catching up" (big jump) and "synced" — CL-driven catch-up takes a different code path than startup catch-up
### Background Prune/Merge
- Triggered per-forkchoice via `runPostForkchoice()`
- Semaphore is transferred to background goroutine — while pruning, all CL requests return `Busy`
- Domain pruning bounded by slot time / 6
### Block Retirement
- Triggered per-cycle in `SnapshotsPrune` stage
- Non-blocking (background goroutine), but `snBuildAllowed` semaphore serializes it with state aggregation
- `FrozenBlocks` boundary advances when indices become visible (eventually consistent)
### Snapshot Publishing
- Only happens after merge (not initial dump)
- Publishing via seeder → BitTorrent after merge step completes
## Architectural Direction
The driver for the interconnection is the single-writer RwTx constraint. This is changing:
1. **Execution no longer needs RwTx** — it writes to memory (SharedDomains). **Pending:** a final refactor to move remaining `rawdb` RW interactions (`WriteCanonicalHash`, `WriteHeader`, `SaveStageProgress`, etc.) into SD writes + an additional memorydb overlay.
2. **Async transactions** allow cooperative sharing.
Once (1) is complete, flush/commit is the only point where execution needs a tx, and this can happen asynchronously:
```
BEFORE (current):
execute ──RwTx──► flush ──RwTx──► commit ──RwTx──► prune ──RwTx──► retire
[all serialized on one goroutine]
AFTER (target):
execute (memory only, no RwTx) ← can run continuously
│
▼ async handoff
flush/commit (brief RwTx window) ← only serialization point
│
▼ independent
prune (own RwTx, own schedule) ← no longer blocks execution
retire (own RwTx, own schedule) ← no longer blocks prune
publish (no RwTx needed) ← fully independent
```
## Separation Opportunities
- [ ] Snapshot download could run independently of execution (doesn't need the exec semaphore)
- [ ] Block retirement could be event-driven (trigger when execution advances N blocks) rather than polled per-cycle
- [ ] Domain pruning could have its own scheduling, decoupled from forkchoice timing pressure
- [ ] The `ProcessFrozenBlocks` "catch up" path and the `updateForkChoice` "big jump" path are essentially the same work with different error handling — unify them
- [ ] Move remaining rawdb RW writes out of execution into SD/memorydb to break the RwTx dependency
- [ ] Distinguish "busy doing initial sync" from "busy processing a forkchoice" in semaphore status
## Related PRs
- #19870: halt on invalid block during parallel initial sync (gated on Exec3Parallel)
- #19803: BAD_BLOCK_HALT + STOP_AFTER_BLOCK fix
## Entry Points Reference
| Chain type | Code path | File |
|---|---|---|
| PoS (mainnet) | `ExecModule.Start()` → `ProcessFrozenBlocks` | `backend.go:1496` |
| Bor (polygon) | `ExecModule.Start()` → `ProcessFrozenBlocks` | `backend.go:1502` |
| PoW (legacy) | `StageLoop()` → `ProcessFrozenBlocks` + iteration loop | `backend.go:1523` |
Contributor guide
Assessment
This issue has not been assessed yet.