ethereum / ethereum/execution-specs

Opcode count verification & counting for `execute remote` under `fixed-opcode-count` mode

Open
#1,981 3 comments 0 reactions 0 assignees View on GitHub
A-test-execute C-feat stale
Dominant language
Python
Stars
1.2k
Forks
505
Avg merge
2d 8h
Merged PRs (30d)
106

Description

## Background

We currently support two benchmark modes:
- worst-case benchmark: Uses the `--gas-benchmark-values` flag to specify the target block gas limits.
- fixed-opcode-count benchmark: Uses the `--fixed-opcode-count` flag, where users provide explicit values or a configuration mapping. This mode is primarily used for gas repricing analysis and typically runs under execute-remote mode.

Now we already support opcode-count verification in local benchmark tests. Users can specify a `target_opcode` attribute in the benchmark test wrapper, `fill` the test, and then compare the opcode counts produced by the t8n tool against the expected values.

We now need to extend this capability to execute-remote mode and achieve the following goals (sorted in priority order):
1. Validate opcode counts against expectations under execute-remote mode and integrate this into CI for fast feedback.
2. Collect full opcode-count distributions for each run, so repricing analysis can quantify execution “noise.”
3. Expose richer execution metrics, such as warm vs. cold SSTORE counts and maximum memory expansion during execution.

## Description

### Validate opcode counts against expectations under execute-remote mode and integrate this into CI for fast feedback.

For the opcode count verification, we now have the `fill` mode implementation.
https://github.com/ethereum/execution-specs/blob/3ac8ce8865e2239a0d5d576091770573accc04c8/packages/testing/src/execution_testing/specs/benchmark.py#L487-L493

And we should support it in execute-remote mode, the entry is this function:
https://github.com/ethereum/execution-specs/blob/3ac8ce8865e2239a0d5d576091770573accc04c8/packages/testing/src/execution_testing/specs/benchmark.py#L497-L509

First, we need to implement a geth RPC method `debug_traceTransaction`, or `trace_transaction` method in other clients. Parse the _actual_ benchmark block, please do not include the setup blocks that create contracts or fund EOA, or any preparation block for the benchmark target block. We could add a flag `--cmp-opcode-count` to enable this feature

With this feature, integrate it into our current CI, using `execute hive` might be a solution so that we do not need extra kurtosis setup.

### Collect full opcode-count distributions for each run, so repricing analysis can quantify execution `noise`.
In the first step, we only calculate the opcode count for the targeted opcode, but this should be extended and provide the same feature implemented in evmone, that provide the opcode count for every opcode.

### Expose richer execution metrics, such as warm vs. cold SSTORE counts and maximum memory expansion

Instead of the opcode, we need to know the current status of the execution, like (1) warm / cold access, (2) maximum memory expansion size (3) account creation and more metric.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.