ethereum / ethereum/execution-specs
feat(benchmark): add opcode count field to fixture formats for easier relative opcode benchmarking
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 505
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 116
Description
Assuming that timing Engine API payload execution becomes the canonical method to benchmark opcodes, it'd be valuable to be able to report the absolute average time taken to execute a single opcode from a benchmark test. For example, how long does it take (on average) to execute an `ADD` versus an `ADDMOD`?
If the `--opcode.count` functionality, cc:
- https://github.com/ethereum/execution-spec-tests/issues/1852, and,
- https://github.com/ethereum/execution-spec-tests/pull/1956
becomes standard in `t8n` tools, then we could include this information in state (for the single state test tx) and blockchain (for every block in the fixtuer) in test fixtures in order to calculate and report the relative average time taken directly in `consume`.
The worst case block execution time for a crafted benchmark test targeting a particular opcode is obviously a valuable result and almost equivalent; but knowing the average time taken per opcode could be easier as a basis for discussion.
Considerations:
- The number of, resp. gas used by, other opcodes executed within the block. Can we normalize the result by taking this into consideration? Perhaps it's negligible?
- We could add this information to every test fixture, but perhaps it only make sense for benchmark test fixtures? (Only benchmark tests target a specific opcode adequately for the result to make sense). Perhaps there are other use cases though?
- ...
Perhaps there's a better way? We should check how single opcodes have been benchmarked on a relative basis in the past.
Contributor guide
Assessment
This issue has not been assessed yet.