Benchmarking the ethdebug implementation
- Dominant language
- C++
- Stars
- 25.7k
- Forks
- 6.2k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 29
Description
## Abstract
Stringing along debugging data
- externally, e.g., serializing it into a comment-style format or into a separate file to reparse it when the language is being lowered, or
- internally, in terms of data structures and on/off switches in the code that decide whether debug data should be recorded,
cannot have significant impact on the time it takes to invoke the compiler if debugging info is switched off. Any regression must be justified.
To ensure this, benchmarks should be run regularly.
## Specification
Benchmarking real-world contracts can be slow for big projects which are also most representative. For faster feedback loops, performance proxy contracts / source files should be devised by, e.g., programmatically generating code. These are expected to be smaller and faster to compile and yet give reasonable indication of the expected outcome on benchmarks run on real-world data. Said proxy contracts should explore the characteristics of real-world contracts in terms of structural patterns - e.g., deeply nested vs wide - as well as feature usage - e.g., heavy ABI, storage, control flow, etc.
For bigger PRs and changes, run the real-world ones, too. For these, a single standard-json file should be extracted out of the external benchmark projects so that costs and variances inherent to the frameworks are not considered.
All the benchmarks themselves run on a dedicated machine with minimal external load and as-stable-as-possible CPU frequency to get reproducible results, triggered manually and/or via labels. Each benchmark should run a minimum of three times (ideally more often, depending on invocation speed) to get more reliable data. Report at least median, mean, and variance.
The scripts to run the benchmarks should be structured so that they can also be run locally with ease to inform about performance implication while, e.g., developing code. To this end there should be a single entry point to run performance tests with the ability to select individual benchmarks (real-world as well as generated) as well as the full suite or a subset thereof.
Contributor guide
Research direction
No files, tests, or entry points are named. Start by locating the ethdebug implementation and any existing benchmark scripts, then compare them with the requested proxy contracts, real-world standard-json inputs, repeated runs, and local single-entry-point workflow; done means the benchmark suite measures debug-off overhead reproducibly and reports median, mean, and variance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, solidity
- Domain
- compilers, performance, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100