erigontech / erigontech/erigon

exec module: decouple startup, execution, prune, retire, and publish flows

Open
#19,873 1 comment 0 reactions 1 assignee Claimed by @mh0lt View on GitHub
ErigonDB tech debt reduction
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

## Problem

The exec module's flows (ProcessFrozenBlocks, ValidateChain, UpdateForkChoice, prune, retire, publish) are heavily intertwined because they share a single RW transaction — an MDBX limitation where only one goroutine can hold an RwTx at a time.

This creates a serialization chain:
```
execute → flush → commit → prune → retire → publish
all serialized because they share one RwTx
```

Three resources compete:
1. **ExecModule semaphore(1)** — serializes Start/Validate/FCU; background prune holds it
2. **snBuildAllowed semaphore** — serializes block retirement vs state aggregation
3. **MDBX write lock** — only one RwTx at a time

## Current Flow Issues

### Startup (ProcessFrozenBlocks)
- `ExecModule.Start()` holds the semaphore for the entire duration of `ProcessFrozenBlocks` (potentially hours during initial sync)
- During this time, `Ready()` returns false, `ValidateChain()`/`UpdateForkChoice()` return `Busy`
- Snapshot download runs inside `ProcessFrozenBlocks` — downloading blocks the entire exec module
- No error recovery between startup and steady state: if `ProcessFrozenBlocks` fails partway, the exec module releases the semaphore in an inconsistent state

### Steady State (forkchoice/newpayload)
- `ValidateChain` does NOT persist state — validates in a throwaway tx, stores in `forkValidator.sharedDom`
- `UpdateForkChoice` merges extending fork state, then runs execution pipeline
- Two separate stage pipelines: `ProcessFrozenBlocks` uses full pipeline, `ValidateChain` uses in-memory pipeline with different `badBlockHalt` settings
- No distinction between "catching up" (big jump) and "synced" — CL-driven catch-up takes a different code path than startup catch-up

### Background Prune/Merge
- Triggered per-forkchoice via `runPostForkchoice()`
- Semaphore is transferred to background goroutine — while pruning, all CL requests return `Busy`
- Domain pruning bounded by slot time / 6

### Block Retirement
- Triggered per-cycle in `SnapshotsPrune` stage
- Non-blocking (background goroutine), but `snBuildAllowed` semaphore serializes it with state aggregation
- `FrozenBlocks` boundary advances when indices become visible (eventually consistent)

### Snapshot Publishing
- Only happens after merge (not initial dump)
- Publishing via seeder → BitTorrent after merge step completes

## Architectural Direction

The driver for the interconnection is the single-writer RwTx constraint. This is changing:

1. **Execution no longer needs RwTx** — it writes to memory (SharedDomains). **Pending:** a final refactor to move remaining `rawdb` RW interactions (`WriteCanonicalHash`, `WriteHeader`, `SaveStageProgress`, etc.) into SD writes + an additional memorydb overlay.

2. **Async transactions** allow cooperative sharing.

Once (1) is complete, flush/commit is the only point where execution needs a tx, and this can happen asynchronously:

```
BEFORE (current):
execute ──RwTx──► flush ──RwTx──► commit ──RwTx──► prune ──RwTx──► retire
[all serialized on one goroutine]

AFTER (target):
execute (memory only, no RwTx) ← can run continuously

▼ async handoff
flush/commit (brief RwTx window) ← only serialization point

▼ independent
prune (own RwTx, own schedule) ← no longer blocks execution
retire (own RwTx, own schedule) ← no longer blocks prune
publish (no RwTx needed) ← fully independent
```

## Separation Opportunities

- [ ] Snapshot download could run independently of execution (doesn't need the exec semaphore)
- [ ] Block retirement could be event-driven (trigger when execution advances N blocks) rather than polled per-cycle
- [ ] Domain pruning could have its own scheduling, decoupled from forkchoice timing pressure
- [ ] The `ProcessFrozenBlocks` "catch up" path and the `updateForkChoice` "big jump" path are essentially the same work with different error handling — unify them
- [ ] Move remaining rawdb RW writes out of execution into SD/memorydb to break the RwTx dependency
- [ ] Distinguish "busy doing initial sync" from "busy processing a forkchoice" in semaphore status

## Related PRs

- #19870: halt on invalid block during parallel initial sync (gated on Exec3Parallel)
- #19803: BAD_BLOCK_HALT + STOP_AFTER_BLOCK fix

## Entry Points Reference

| Chain type | Code path | File |
|---|---|---|
| PoS (mainnet) | `ExecModule.Start()` → `ProcessFrozenBlocks` | `backend.go:1496` |
| Bor (polygon) | `ExecModule.Start()` → `ProcessFrozenBlocks` | `backend.go:1502` |
| PoW (legacy) | `StageLoop()` → `ProcessFrozenBlocks` + iteration loop | `backend.go:1523` |

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.