ethereum / ethereum/execution-specs
feat: consume direct support for all EL clients with engine fixture runners
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 505
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 116
Description
## Summary
Add direct fixture execution support (`consume direct`) for all EL clients across all three fixture formats: state tests, blockchain tests, and engine tests. This eliminates the need for Hive in execution correctness testing, reducing test times from hours to seconds.
## Background
We've built engine test runners for go-ethereum ([ethereum/go-ethereum#34650](https://github.com/ethereum/go-ethereum/pull/34650)), erigon ([erigontech/erigon#20315](https://github.com/erigontech/erigon/pull/20315)), nethermind ([NethermindEth/nethermind#11035](https://github.com/NethermindEth/nethermind/pull/11035)), and besu ([besu-eth/besu#10184](https://github.com/besu-eth/besu/pull/10184)) — direct runners for `blockchain_test_engine` fixtures that exercise the Engine API code path without Hive. Combined with the existing `blocktest` and `statetest` runners, this gives us three levels of direct testing.
## The Three Runners
### `statetest` — single transaction execution
- Executes state test fixtures (one transaction per test)
- Tests EVM opcode correctness, gas accounting, state transitions
- Fastest runner — no block-level overhead
### `blocktest` — block-level execution
- Executes blockchain test fixtures (RLP-encoded blocks)
- Tests block validation, multi-tx execution, uncle handling, state root verification
- **Equivalent to `consume rlp`** — exercises the exact same block import function across all clients
- **Does NOT exercise**: Engine API version validation, forkchoice state, two-phase insert
### `enginetest` — engine API execution via `newPayload` + `forkchoiceUpdated`
- Executes `blockchain_test_engine` fixtures (Engine API payloads)
- Tests version-specific parameter validation (V1-V5), payload-to-block conversion, forkchoice via `SetCanonical`, invalid block tracking, `PayloadStatusV1` responses, error codes
- **Equivalent to `consume engine`** — exercises the same engine code paths across all clients
- **Critical for BAL (EIP-7928)**: Only engine payloads carry pre-attached Block Access Lists, which trigger the parallel state processor. `blocktest` cannot test this.
## Code Path Equivalence
Each client's direct runner uses the **exact same block import function** as Hive's `consume rlp` / `consume engine`. The only thing Hive adds is the transport layer (devp2p / Engine API JSON-RPC), not different execution logic.
### `blocktest` = `consume rlp` minus devp2p transport
| Client | blocktest entry point | consume rlp entry point | Same function? |
|--------|----------------------|------------------------|----------------|
| **geth** | `blockchain.InsertChain()` → `ProcessBlock()` | devp2p → `InsertChain()` → `ProcessBlock()` | Yes |
| **erigon** | `InsertBlocksAndWait()` → `UpdateForkChoice()` | devp2p → staged sync → same execution module | Yes |
| **nethermind** | `blockTree.SuggestBlock()` | devp2p → `blockTree.SuggestBlock()` | Yes |
| **besu** | `blockImporter.importBlock()` | devp2p → `blockImporter.importBlock()` | Yes |
### `enginetest` = `consume engine` minus Engine API JSON-RPC transport
| Client | enginetest path | How close to production? |
|--------|----------------|------------------------|
| **geth** | Lightweight handler mirroring `ConsensusAPI`: `ExecutableDataToBlock` → `InsertBlockWithoutSetHead` → `SetCanonical` | Mirrors production logic |
| **erigon** | Real `EngineServer.HandleNewPayload()` (InsertBlocks + ValidateChain) → `HandleForkChoice()` | Real production code |
| **nethermind** | **Real** `IEngineRpcModule` via `IJsonRpcService.SendRequestAsync("engine_newPayloadVX")` | Exact production code |
| **besu** | Real `AbstractEngineNewPayload.syncResponse()` → `EngineForkchoiceUpdatedVX.syncResponse()` | Real production code |
### `statetest` — single transaction execution
| Client | statetest entry point |
|--------|----------------------|
| **geth** | `core.ApplyMessage()` |
| **erigon** | `TransactionProcessor.Execute()` |
| **nethermind** | `TransactionProcessor.Execute()` |
| **besu** | `TransactionProcessor.processTransaction()` |
## Why we need all three
| What it tests | statetest | blocktest | enginetest |
|--------------|-----------|-----------|------------|
| EVM opcodes | ✅ | ✅ | ✅ |
| Multi-tx blocks | ❌ | ✅ | ✅ |
| Block validation | ❌ | ✅ | ✅ |
| Engine API versions | ❌ | ❌ | ✅ |
| Forkchoice (FCU) | ❌ | ❌ | ✅ |
| Invalid payload handling | ❌ | ❌ | ✅ |
| Error codes (-32602 etc) | ❌ | ❌ | ✅ |
| BAL parallel execution | ❌ | ❌ | ✅ |
| Hive equivalent | — | consume rlp | consume engine |
## Benchmarks (v5.3.0 stable fixtures)
### go-ethereum
Hive: [consume engine 2h48m](https://hive.ethpandaops.io/#/test/generic/1775205963-db3a27b1aa7523fe68699dbc4b9a5445), [consume rlp 4h29m](https://hive.ethpandaops.io/#/test/generic/1774953007-2c7c20609223f35105c9a06a1533413a)
| Runner | Tests | w=1 | w=8 | vs Hive |
|--------|-------|-----|-----|---------|
| enginetest | 40,523 | 1m02s | **12.8s** | **~790x** faster |
| blocktest | 43,924 | 1m06s | **12.7s** | **~1,272x** faster |
| statetest | 40,553 | 21.8s | **4.4s** | — |
### erigon
| Runner | Tests | w=8 | w=12 |
|--------|-------|-----|------|
| enginetest | 2,351 (Prague) | — | **74s** |
| enginextest | 2,313 (Prague) | **27.5s** | — |
| enginextest | ~40,000 (full) | **~8 min** | — |
### nethermind
Hive: consume engine 16h39m, consume rlp >24h
| Runner | Tests | w=1 | w=8 | vs Hive |
|--------|-------|-----|-----|---------|
| enginetest | 40,519 | 3m10s | **2m48s** | **~356x** faster |
| blocktest | 43,912 | 1m47s | **57s** | **>1,500x** faster |
| statetest | 40,553 | 1m42s | **1m12s** | — |
### besu
Hive: consume engine 1d2h37m, consume rlp >36h
| Runner | Tests | w=1 | w=8 | vs Hive |
|--------|-------|-----|-----|---------|
| enginetest | 40,523 | 2m13s | **1m18s** | **~1,230x** faster |
| blocktest | 43,795 | 1m01s | **16.9s** | **>7,600x** faster |
| statetest | 40,551 | 42.6s | **25.2s** | — |
### Hive parity
- **geth**: Exact same 4 failures as Hive consume engine (system contract deployment tests)
- **nethermind**: All 7 Hive failures are exception mapper issues in EELS (correct INVALID status, unmapped error string). Direct runner correctly passes.
- **erigon**: 2,351/2,351 Prague pass (0 failures). Full suite pending.
- **besu**: 40,523/40,523 (0 failures). Full Hive parity.
## EELS consume direct integration
The EELS side needs `BlockchainEngineFixture` added to the `consume direct` plugin's supported formats, and each client's fixture consumer needs a `consume_engine_test()` method. Branch: [consume direct engine support](https://github.com/spencer-tb/execution-specs/tree/feat/consume-direct-engine).
## Client Checklist
### go-ethereum
- [x] `statetest` — exists upstream + `--workers`, `--run`
- [x] `blocktest` — exists upstream + `--workers`, `--run`
- [x] `enginetest` — [ethereum/go-ethereum#34650](https://github.com/ethereum/go-ethereum/pull/34650)
- [x] `--workers N` parallel flag — all three runners
- [x] Hive parity confirmed (v5.3.0)
- [ ] `--bal.exec` flag for BAL parallel execution mode switching (requires bal-devnet-3)
### erigon
- [x] `statetest` — improved with `--run`, `--workers`, `--jsonout`, statetest fixes
- [x] `blocktest` — improved with `--run`, `--workers`, `--jsonout`, stdin batch mode
- [x] `enginetest` — [erigontech/erigon#20315](https://github.com/erigontech/erigon/pull/20315)
- [x] `enginextest` — pre-alloc cached runner for `blockchain_test_engine_x` fixtures
- [x] `--workers N` parallel flag — all runners
- [x] Hive parity confirmed (v5.3.0, Prague)
### nethermind
- [x] `--stateTest` — exists upstream, improved with `--workers`, `--run` alias
- [x] `--blockTest` — exists upstream, improved with `--workers`, `--jsonout`
- [x] `--engineTest` — [NethermindEth/nethermind#11035](https://github.com/NethermindEth/nethermind/pull/11035)
- [x] `--workers N` parallel flag — all three runners
- [x] Hive parity confirmed (v5.3.0) — all 7 failures are exception mapper issues
### besu
- [x] `state-test` — exists upstream, improved with `--workers`, `--run`
- [x] `block-test` — exists upstream, improved with `--workers`, `--run`, directory support
- [x] `engine-test` — [besu-eth/besu#10184](https://github.com/besu-eth/besu/pull/10184)
- [x] `--workers N` parallel flag — all three runners
- [x] Hive parity confirmed (v5.3.0)
### reth
- [ ] Investigate existing test runner infrastructure
- [ ] `statetest` runner
- [ ] `blocktest` runner
- [ ] `enginetest` runner
### ethrex
- [ ] Investigate existing test runner infrastructure
- [ ] `statetest` runner
- [ ] `blocktest` runner
- [ ] `enginetest` runner
### nimbus-el
- [ ] Investigate existing test runner infrastructure
- [ ] `statetest` runner
- [ ] `blocktest` runner
- [ ] `enginetest` runner
### ethereumjs
- [ ] Investigate existing test runner infrastructure
- [ ] `statetest` runner
- [ ] `blocktest` runner
- [ ] `enginetest` runner
## Ideal consume direct flow
```
consume direct --bin --input
```
1. Discovers all fixture formats from the input directory
2. For each fixture file, dispatches to the appropriate runner:
- `state_test` → `statetest`
- `blockchain_test` → `blocktest`
- `blockchain_test_engine` → `enginetest`
3. Caches results per file (one subprocess per file, not per test case)
4. Reports via pytest with HTML output
This replaces Hive for execution correctness testing while keeping Hive for integration testing (devp2p, sync, client startup).
Contributor guide
Assessment
This issue has not been assessed yet.