ethereum / ethereum/execution-specs
tracker(fill): improve `fill`'s test dev-ex
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 505
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 116
Description
EEST test developer experience should be improved, especially to help new testers. There's definitely a few low hanging fruit that could really help test writers.
### Plan
- [ ] Get feedback from newer test developers.
### Framework
- [ ] Ensure EEST types have good (concatenated) string representations to help debug test fails (via `__repr__` or even `__rich__`? `rich` offers more customized formatting and can even display table representations, etc.).
- [ ] Improve tracing output, cf https://github.com/ethereum/execution-spec-tests/pull/474
### Fill
- [ ] Introduce standard terminology for "fill test failures" - be consistent in docs/docstrings/error messages.
- [ ] Detect OOG without having to enable tracing and manually check the traces. At least point a test implementer to enable tracing if we don't enable it by default.
- [ ] Consider printing more mismatching storage keys if more than one differs (instead of only failing on the first detected mismatch), perhaps only with increased verbosity (`-v` or `-vv`).
### Tooling
- [ ] Add trace diff tooling: cli tools, Claude backslash command, ...
### Docs
- [ ] Improve documentation on how to set gas limits (to avoid failures).
- [ ] Add best practices and troubleshooting section.
- [ ] Add a section on how to debug a failing test using `fill --pdb` and inspecting objects, respectively using `rich.print()` in `pdb`.
Contributor guide
Assessment
This issue has not been assessed yet.