ethereum / ethereum/execution-specs
Opcode count verification & counting for `execute remote` under `fixed-opcode-count` mode
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 505
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 106
Description
## Background
We currently support two benchmark modes:
- worst-case benchmark: Uses the `--gas-benchmark-values` flag to specify the target block gas limits.
- fixed-opcode-count benchmark: Uses the `--fixed-opcode-count` flag, where users provide explicit values or a configuration mapping. This mode is primarily used for gas repricing analysis and typically runs under execute-remote mode.
Now we already support opcode-count verification in local benchmark tests. Users can specify a `target_opcode` attribute in the benchmark test wrapper, `fill` the test, and then compare the opcode counts produced by the t8n tool against the expected values.
We now need to extend this capability to execute-remote mode and achieve the following goals (sorted in priority order):
1. Validate opcode counts against expectations under execute-remote mode and integrate this into CI for fast feedback.
2. Collect full opcode-count distributions for each run, so repricing analysis can quantify execution “noise.”
3. Expose richer execution metrics, such as warm vs. cold SSTORE counts and maximum memory expansion during execution.
## Description
### Validate opcode counts against expectations under execute-remote mode and integrate this into CI for fast feedback.
For the opcode count verification, we now have the `fill` mode implementation.
https://github.com/ethereum/execution-specs/blob/3ac8ce8865e2239a0d5d576091770573accc04c8/packages/testing/src/execution_testing/specs/benchmark.py#L487-L493
And we should support it in execute-remote mode, the entry is this function:
https://github.com/ethereum/execution-specs/blob/3ac8ce8865e2239a0d5d576091770573accc04c8/packages/testing/src/execution_testing/specs/benchmark.py#L497-L509
First, we need to implement a geth RPC method `debug_traceTransaction`, or `trace_transaction` method in other clients. Parse the _actual_ benchmark block, please do not include the setup blocks that create contracts or fund EOA, or any preparation block for the benchmark target block. We could add a flag `--cmp-opcode-count` to enable this feature
With this feature, integrate it into our current CI, using `execute hive` might be a solution so that we do not need extra kurtosis setup.
### Collect full opcode-count distributions for each run, so repricing analysis can quantify execution `noise`.
In the first step, we only calculate the opcode count for the targeted opcode, but this should be extended and provide the same feature implemented in evmone, that provide the opcode count for every opcode.
### Expose richer execution metrics, such as warm vs. cold SSTORE counts and maximum memory expansion
Instead of the opcode, we need to know the current status of the execution, like (1) warm / cold access, (2) maximum memory expansion size (3) account creation and more metric.
Contributor guide
Assessment
This issue has not been assessed yet.