erigontech / erigontech/erigon

perf: pipeline commitment with execution in parallel path (async ComputeCommitment + BAL-driven warmup)

Open
#19,791 3 comments 0 reactions 0 assignees View on GitHub
performance
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

## Background

Performance analysis of parallel and serial mainnet sync nodes (~24.4M–24.6M blocks) shows the per-10,000-block cost breakdown:

| Mode | Exec (EVM + apply) | Commitment | Effective |
|------|-------------------|------------|-----------|
| Parallel | ~845s | ~1477s ← bottleneck | ~1477s (commit hides exec via pipeline) |
| Serial | ~904s | ~1420s | ~2324s (sequential) |

Two observations drive this issue:

1. **Commitment is the bottleneck in the parallel path during sync** — exec is already hidden behind the previous batch's commitment across batch boundaries. But at tip (one block at a time, 12s slots) this pipeline collapses: each block commits before the next starts.

2. **At tip, commitment dominates per-block latency** — with small key sets per block (~1–5k vs ~300k during sync), commitment takes seconds while exec takes hundreds of ms. Pipelining them is the primary lever for reducing tip latency.

Serial path is out of scope — it remains unchanged and serves as the **correctness baseline**.

---

## Two execution modes

This work introduces two distinct paths depending on whether a Block Access List (EIP-7928, Amsterdam+) is present.

### Exec-driven path (no BAL — sync, pre-Amsterdam blocks)

Normal execution → state writes → async pipelined commitment.

```
Block N+1 arrives (no BAL)
├── [B] Sync: consume block N commitment result (check root hash)
│ FlushDeferredCommitmentUpdates(tx) → MDBX
├── [C] Execute block N+1 (parallel EVM workers, writes to VersionedWrites)
├── [D] applyVersionedWrites → SharedDomains.mem
│ Flush mem → MDBX
└── [E] Launch async commitment goroutine (paraTrieDB, deferred updates)
Block N+2 starts immediately at [B]
```

### BAL-driven path (BAL present — Amsterdam+ blocks at tip or in full block sync)

The BAL contains the **complete per-tx state diff with values** (balance, nonce, code, storage changes). This means we can apply state changes and compute commitment from the BAL directly, without waiting for execution results.

```
Block N+1 arrives (BAL present — all state diffs known)

├── [A] Validate BAL against header BAL hash (fast, before exec)

├── [B] Sync: consume block N commitment result (check root hash)
│ FlushDeferredCommitmentUpdates(tx) → MDBX

├── [C] Apply BAL final-state writes → SharedDomains.mem ← NO exec needed for state
│ Flush mem → MDBX
│ Launch async commitment goroutine (paraTrieDB, deferred updates)

├── [D] Execute block N+1 in **comparison mode** (concurrent with [E])
│ EVM runs normally, but writes go to a comparison buffer (not SharedDomains)
│ At end: compare exec outputs against BAL values
│ If mismatch → reject block (BAL is invalid)

└── [E] (commitment goroutine from [C] runs concurrently with [D])
reads BAL key set via paraTrieDB (branch nodes pre-loaded)
hashes Patricia trie
defers branch updates in memory

Block N+2 can start (BAL apply + exec) once both [D] and [E] complete.
```

**Key property**: The exec path does not write to SharedDomains. The BAL is the source of truth for state changes. Execution validates correctness but does not drive the write path. This means commitment starts as soon as the BAL is applied — fully concurrent with EVM execution.

**Must work without BAL**: The exec-driven path is always the fallback. The code must select the path at runtime based on whether a BAL is present.

---

## Why the scaffolding already exists

1. **`paraTrieDB` / `EnableParaTrieDB`** — read-only MDBX handle opened alongside the write tx. Commitment goroutine reads branch nodes here without blocking the write tx.

2. **`SetDeferCommitmentUpdates` / `FlushPendingUpdates`** — branch updates accumulate in memory, applied at a sync point. Decouples the commitment goroutine from the write tx.

3. **`VersionedIO` / BAL comparison** — `AsBlockAccessList()` already derives a BAL from exec outputs. The inverse (applying a BAL to state, then comparing exec outputs to it) uses the same data structures.

4. **`ProcessBAL`** — already validates exec-derived BAL against header BAL hash. The BAL-driven path inverts this: apply the header BAL, then validate exec produces the same diff.

---

## Required changes

### Phase 1 — async `ComputeCommitment` (exec-driven path, core change)

**`db/state/execctx/domain_shared.go`**
- Make `ComputeCommitment` operate entirely via `paraTrieDB` + deferred writes
- Add `ComputeCommitmentAsync(ctx) <-chan commitResult`
- `commitResult` carries `{rootHash []byte, err error}`

**`execution/stagedsync/exec3_parallel.go`**
- Replace blocking `ComputeCommitment` with `ComputeCommitmentAsync`
- Store `pendingCommitment <-chan commitResult` across loop iterations
- Sync at top of apply cycle, `FlushDeferredCommitmentUpdates` before next flush

### Phase 2 — BAL-driven path

**`execution/stagedsync/exec3_parallel.go`**
- On `newPayload` for Amsterdam+ blocks with BAL:
1. Validate BAL hash against block header
2. Apply BAL final-state writes to `SharedDomains.mem` (new `ApplyBALWrites` function)
3. Flush mem → MDBX, launch async commitment goroutine
4. Run EVM workers in **comparison mode**: writes to comparison buffer, not SharedDomains
5. After exec: compare comparison buffer against BAL values → accept/reject

**`execution/state/rw_v3.go`**
- Add `comparisonCollector` — a `StateWriter` that records writes into a buffer without touching SharedDomains
- Add comparison function: diff comparison buffer vs BAL's `AccountChanges` slice

**`db/state/execctx/domain_shared.go`**
- Add `ApplyBALWrites(bal types.BlockAccessList)` — applies BAL final-state values directly to shared domains mem buffer

**`execution/stagedsync/exec3_parallel.go`** (continued)
- Runtime selection: `if bal != nil { balDrivenPath(...) } else { execDrivenPath(...) }`
- Both paths converge at the sync point (step [B]) consuming the pending commitment result

---

## Correctness constraints

- **Single outstanding commitment**: only one goroutine at a time — enforced by sync point [B]
- **BAL validation first**: BAL hash validated against block header before any state writes
- **Comparison is mandatory**: EVM must still execute fully in BAL-driven mode; mismatch → bad block
- **Fallback path**: any block without a BAL (sync, pre-Amsterdam) uses exec-driven path
- **MDBX snapshot isolation**: `paraTrieDB` sees block N's committed state; unaffected by block N+1 mem writes
- **Error handling**: goroutine errors surface at sync point [B]; bad-block handling unchanged

---

## Performance impact

**Exec-driven (sync)**: Commitment already pipelined across batch boundaries. Phase 1 extends this to single-block boundaries at tip — removes commitment from the critical path.

**BAL-driven (tip)**: Commitment starts as soon as BAL is applied (before EVM starts). For a 12s slot with ~200ms exec + ~2s commitment, tip latency collapses from `~2.2s` to `max(0.2s exec, 2s commit started earlier)` ≈ 0, bounded by block propagation time.

---

## Dependencies

- **PR #19711** (`ibs-2cache-phase2-3`) must merge first — it cleans the `VersionedWrites → applyVersionedWrites → SharedDomains` write path that this work builds directly on.

## Out of scope

- Serial execution path (unchanged, used as correctness baseline)
- Parallel Patricia hash (`ParallelHashSort`) — can be layered on top as a follow-on
- Tx-level conflict reduction (separate workstream)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.