ethereum / ethereum/execution-specs

Consider reducing fixture size for `account_query` (16 GB) and `unchunkified_bytecode` (13 GB)

Open
#2,223 3 comments 0 reactions 0 assignees View on GitHub
A-test-benchmark C-chore S-needs-discussion stale
Dominant language
Python
Stars
1.2k
Forks
505
Avg merge
2d 14h
Merged PRs (30d)
116

Description

## Summary

Grab the latest artifact like this:
```
gh run download -R ethereum/execution-specs -n fixtures_benchmark_fast -D /mnt/eest_artifacts
```

Two benchmark fixture files are extreme size outliers — 100-1000× larger than all others. This issue documents the specific redundancies driving the size and suggests possible approaches to reduce it.

## Top-10 Benchmark Fixture Sizes

| Rank | Fixture | Size |
|------|---------|------|
| 1 | `account_query.json` | **16 GB** |
| 2 | `unchunkified_bytecode.json` | **13 GB** |
| 3 | `block_full_of_ether_transfers.json` | 123 MB |
| 4 | `block_full_data.json` | 42 MB |
| 5 | `auth_transaction.json` | 19 MB |
| 6 | `block_full_of_precompile.json` | 15 MB |
| 7 | `block_full_of_sha256_hashes.json` | 14 MB |
| 8 | `secp256k1.json` | 13 MB |
| 9 | `kzg_point_evaluation.json` | 13 MB |
| 10 | `modexp.json` | 9.3 MB |

The gap between #2 (13 GB) and #3 (123 MB) is >100×. The remaining fixtures (#3–#10) are proportionate to their workload.

---

## Outlier 1: `account_query.json` (16 GB)

**Test:** [`test_account_query`](https://github.com/ethereum/execution-specs/blob/forks/amsterdam/tests/benchmark/compute/instruction/test_account_query.py#L428) at `tests/benchmark/compute/instruction/test_account_query.py:428`

**Parametrization:** 8 opcodes × 2 `access_warm` × 4 `mem_size` × 5 `code_size` × 2 `value_sent` = 640 theoretical combinations, reduced to ~94 by skip logic (lines 440–452).

**Root cause:** The 8 EXTCODECOPY + `max_code_size` cases (4 `mem_size` × 2 `access_warm`) each generate ~1.8 GB, totalling ~14.4 GB (roughly 96% of the file). Each case deploys ~44K contracts of 24,576 bytes via `CustomSizedContractFactory`.

### Redundancy flags

#### 1. `mem_size=0` with EXTCODECOPY is a no-op copy

At [line 482–489](https://github.com/ethereum/execution-specs/blob/forks/amsterdam/tests/benchmark/compute/instruction/test_account_query.py#L482), the EXTCODECOPY call uses `size=mem_size`. When `mem_size=0`, zero bytes are copied — this benchmarks account access cost only, which is functionally identical to what BALANCE/EXTCODESIZE already measure. It doesn't exercise EXTCODECOPY's actual code-copying behavior. This generates ~3.6 GB (2 warm variants) of fixtures that don't measure what EXTCODECOPY uniquely does.

#### 2. `mem_size` variations produce near-identical fixtures

The 4 `mem_size` values (0, 32, 256, 1024) change gas cost per iteration slightly, resulting in slightly different numbers of max-size contracts deployed. But the pre-state structure is essentially the same — thousands of 24 KB contracts. This creates 4× fixture duplication for the EXTCODECOPY + max_code_size parameter space.

#### 3. `access_warm=True` generates massive access lists

Warm variants include an `AccessList` entry for every target contract (~44K entries), roughly doubling each already-massive fixture. See [lines 549–565](https://github.com/ethereum/execution-specs/blob/forks/amsterdam/tests/benchmark/compute/instruction/test_account_query.py#L549) for the access list generation.

### Possible approaches

- Remove `mem_size=0` for EXTCODECOPY (it doesn't test code copying)
- Reduce `mem_size` to fewer values (e.g., just 0 and 1024 for boundary testing)
- Consider whether `access_warm=True` with `max_code_size` is worth the ~7 GB it adds

---

## Outlier 2: `unchunkified_bytecode.json` (13 GB)

**Test:** [`test_unchunkified_bytecode`](https://github.com/ethereum/execution-specs/blob/forks/amsterdam/tests/benchmark/compute/scenario/test_unchunkified_bytecode.py#L37) at `tests/benchmark/compute/scenario/test_unchunkified_bytecode.py:37`

**Parametrization:** 7 opcodes (CALL, CALLCODE, DELEGATECALL, STATICCALL, EXTCODESIZE, EXTCODEHASH, EXTCODECOPY), all cold access against max-code-size contracts.

**Root cause:** Each opcode deploys its own independent set of ~44K max-size (24,576-byte) contracts. 7 opcodes × ~1.7 GB each ≈ 12 GB.

### Redundancy flag: All 7 opcodes deploy identical contract sets independently

Every test case calls `CustomSizedContractFactory(pre=pre, fork=fork)` with default `contract_size=None` (max). The setup phase is identical across all 7 — the only difference is the opcode used in the execution phase. The contract deployment dominates the fixture size.

The 7 opcodes group into 3 distinct gas-cost categories:

| Category | Opcodes | Cold gas cost |
|----------|---------|---------------|
| Metadata | EXTCODESIZE, EXTCODEHASH | 2,600 |
| Copy | EXTCODECOPY | 2,696 (with 1000-byte copy) |
| Execution | CALL, CALLCODE, DELEGATECALL, STATICCALL | 2,700 |

Within each category, the gas cost is identical, so the benchmarks are effectively duplicates. EXTCODESIZE and EXTCODEHASH produce the same benchmark result. The 4 CALL variants produce the same benchmark result.

### Possible approaches

- Reduce to 3 representative opcodes (one per gas-cost category) instead of 7
- Alternatively, if all 7 are needed for client-specific behavior differences, document the rationale

---

## Context

The remaining large fixtures (#3–#10) were reviewed and their sizes are proportionate to their workloads — no action needed for those.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.