BAL Related Performance Evaluation

# Parallel vs Serial Execution Performance Analysis

## Goal

Establish baseline execution performance metrics for parallel and serial block execution at chain tip on mainnet. Use these numbers to direct optimization effort — specifically identifying whether EVM compute, state I/O, or commitment (state root) is the bottleneck, and how this varies with block size.

## Background

Both instances process the same blocks in real-time on the same machine, providing a direct apples-to-apples comparison. The serial path will eventually be decommissioned once parallel is stable, but we first need to check for serial-specific optimizations worth porting.

**Machine:** AMD EPYC 4244P (12 cores), 125GB RAM, NVMe storage
**Branch:** `fix/bal-selfdestruct-netzero` @ `741ad77cf6`
**Config:** Both use `--prune.mode=minimal`, `--experimental.bal=false`

## Initial Results (bal-devnet-2 branch, at chain tip)

### Headline Comparison

| Metric | Parallel (4197 blks) | Serial (100 blks) | Ratio |
|--------|---------------------|-------------------|-------|
| **Combined throughput** | **274 mgas/s** (p50) | **294 mgas/s** (avg) | 0.93x |
| EVM throughput | 496 mgas/s | 372 mgas/s | 1.37x |
| Commitment throughput | 510 mgas/s | 807 mgas/s | 0.63x |
| EVM % of wall time | 48.7% | 54.5% | — |
| Commitment % of wall time | 47.4% | 45.5% | — |
| Overhead | 3.9% | 0.0% | — |
| CPU utilization | 3.9 cores | 1 core | 3.9x |
| Speculative re-execution | ~23% repeat rate | N/A | — |

**Key takeaway:** Parallel execution uses 3.9x more CPU but delivers roughly the same combined throughput as serial. The EVM phase achieves only a 1.37x wall-clock speedup (496 vs 372 mgas/s EVM-only from earlier measurements), but this gain is offset by commitment being 37% slower in parallel mode (510 vs 807 mgas/s). This suggests parallel execution is missing some of the performance enhancements from serial execution.

### Throughput by Block Gas (serial, n=100)

| Block Gas | n | Avg EVM | EVM mgas/s | Avg Commit | Commit mgas/s |
|-----------|---|---------|------------|------------|---------------|
| 0-10M | 8 | 10ms | 537 | 7ms | 849 |
| 10-20M | 18 | 35ms | 491 | 15ms | 1,213 |
| 20-30M | 30 | 51ms | 519 | 35ms | 879 |
| 30M+ | 44 | 100ms | 484 | 96ms | 584 |

### Key Finding: Commitment scales poorly with block size

| Block Gas | EVM time | Commitment time | EVM % | Commit % |
|-----------|----------|-----------------|-------|----------|
| < 10M | faster | faster | ~66% | ~34% |
| 30-40M | ~equal | ~equal | ~50% | ~50% |
| 60M+ | slower | **2x slower** | ~33% | ~66% |

At large blocks (60M+ gas), commitment is the dominant bottleneck — **2x slower than EVM**. Pipelining EVM+commitment would give ~2x improvement since they currently run in series.

## Instrumentation Added

We added per-block **state gas accounting** to separate storage I/O gas from compute gas:

- **State gas**: gas charged for `SLOAD`, `SSTORE`, `BALANCE`, `EXTCODESIZE`, `EXTCODEHASH`, `EXTCODECOPY`, `CALL` cold access, `SELFDESTRUCT` cold access (EIP-2929 model)
- **Compute gas**: total gas − state gas

This is an internal profiling metric, not defined by any EIP.

### Files modified (9 files, +110 lines)

| File | Change |
|------|--------|
| `execution/vm/evm.go` | `StateGasUsed uint64` field on EVM struct, reset per block |
| `execution/vm/operations_acl.go` | `evm.StateGasUsed +=` in all EIP-2929 gas functions (19 sites) |
| `execution/vm/evmtypes/evmtypes.go` | `StateGasUsed` on `ExecutionResult` (per-transaction delta) |
| `execution/protocol/state_transition.go` | Snapshot before/after EVM call in both `ApplyMessageWithEVM` and `TransitionDb` |
| `execution/exec/state.go` | `StateGasUsed` atomic counter on `WorkerMetrics`, getter on `Worker` |
| `execution/stagedsync/exec3_serial.go` | Per-block state gas capture, debug log: `stateGas=X/Y(Z%)` |
| `execution/stagedsync/exec3_parallel.go` | Per-block state gas on `blockExecutor`, debug log: `stateGas=X/Y(Z%)` |
| `execution/stagedsync/exec3_metrics.go` | `sgas%` in periodic `LogExecution` summary (both parallel and serial) |

### Stats tooling (outside repo)

| Script | Purpose |
|--------|---------|
| `gather_stats.sh` | Parses debug logs → quartile-based throughput analysis + state gas breakdown → JSON |
| `compare_stats.sh` | Side-by-side comparison of two JSON result files |

## Current Status

Both instances are running on `fix/bal-selfdestruct-netzero`, currently in the Execution stage (~block 24,572,000), catching up to chain tip. The code changes have been built but not committed yet — the running instances use the previous binary. They need to be restarted with the new binary once they reach a convenient stopping point.

## Work Plan

### Phase 1: Baseline Collection (current)

- [x] Instrument state gas accounting (EVM, serial executor, parallel executor)
- [x] Add `sgas%` to periodic `LogExecution` summary
- [x] Update `gather_stats.sh` to parse `stateGas=X/Y(Z%)` from debug logs
- [x] Build passes
- [ ] Both instances reach chain tip on current branch
- [ ] Restart both with new binary (state gas instrumentation active)
- [ ] Collect 500+ blocks at tip for each mode
- [ ] Run `gather_stats.sh` for parallel and serial, compare with `compare_stats.sh`

**Deliverable:** Baseline numbers for parallel vs serial, with state gas breakdown by block size quartile.

### Phase 2: Analysis — Where Does Time Go?

Use the baseline to answer:

1. **What fraction of gas is state access?** — `sgas%` tells us. If state gas is 30% of total gas but storage reads take 60% of wall time, state I/O is disproportionately expensive.
2. **How does this vary with block size?** — Quartile breakdown shows if large blocks are more state-heavy.
3. **EVM vs commitment vs overhead** — Which phase dominates at each block size?
4. **Parallel speculative waste** — `repeat%` shows how much work is wasted on aborted re-executions.

### Phase 3: Cache Effectiveness (separate task)

A local caching layer was recently introduced for state reads. The state gas metrics provide the foundation for analyzing cache effectiveness:

- Correlate `sgas%` with `read` durations (`a=account, s=storage, c=code`) from `LogExecution`
- Compare cache hit rates (from domain metrics) against state gas intensity
- Identify block patterns where caching is most/least effective
- Measure how cache warming affects throughput over time

This is a **separate task** — the current instrumentation just provides the data foundation.

### Phase 4: Pipelining Feasibility

The initial results show EVM and commitment run in series and are roughly equal in duration. Pipelining them (EVM block N+1 while committing block N) could theoretically halve wall-clock time.

- Measure `max(EVM, commitment)` vs `EVM + commitment` by quartile to quantify potential savings
- Identify dependencies that prevent pipelining (state reads during commitment that conflict with next block's EVM)
- Design pipelining architecture

### Phase 5: Serial-Specific Optimizations

Before decommissioning the serial path:

- Compare serial vs parallel EVM throughput at equivalent block sizes (serial has no speculative overhead)
- Identify any serial-specific optimizations worth porting to parallel
- Measure single-core EVM efficiency (serial path is the cleanest measurement)

## Metrics Reference

### Per-block debug log (new)
```
[prefix] executed block 24572000 in 55ms stateGas=1234567/5678901(22%)
```

### Periodic LogExecution summary (enhanced)
```
[4/6 Execution] parallel executed ... gas/s=534M ... sgas%=22.5 ... tdur=12µs exec=8µs(67%) read=4µs(33%),a=1µs,s=2µs,c=1µs rd=3.98M ...
```

### Key Prometheus metrics
| Metric | Description |
|--------|-------------|
| `exec_mgas_sec` | Execution mgas/s |
| `exec_block_dur` | Block execution duration |
| `exec_txn_dur` / `exec_txn_read_dur` | Transaction timing breakdown |
| `exec_read_rate` / `exec_account_read_rate` / `exec_storage_read_rate` | Read rates |
| `commit_mgas_sec` | Commitment mgas/s |
| `commit_block_dur` | Commitment duration |