ethereum / ethereum/execution-specs

tracker(fill): improve `fill`'s test dev-ex

Open
#1,555 4 comments 0 reactions 0 assignees View on GitHub
A-doc A-test-fill C-feat C-tracker stale
Dominant language
Python
Stars
1.2k
Forks
505
Avg merge
2d 14h
Merged PRs (30d)
116

Description

EEST test developer experience should be improved, especially to help new testers. There's definitely a few low hanging fruit that could really help test writers.

### Plan
- [ ] Get feedback from newer test developers.

### Framework
- [ ] Ensure EEST types have good (concatenated) string representations to help debug test fails (via `__repr__` or even `__rich__`? `rich` offers more customized formatting and can even display table representations, etc.).
- [ ] Improve tracing output, cf https://github.com/ethereum/execution-spec-tests/pull/474

### Fill
- [ ] Introduce standard terminology for "fill test failures" - be consistent in docs/docstrings/error messages.
- [ ] Detect OOG without having to enable tracing and manually check the traces. At least point a test implementer to enable tracing if we don't enable it by default.
- [ ] Consider printing more mismatching storage keys if more than one differs (instead of only failing on the first detected mismatch), perhaps only with increased verbosity (`-v` or `-vv`).

### Tooling
- [ ] Add trace diff tooling: cli tools, Claude backslash command, ...

### Docs
- [ ] Improve documentation on how to set gas limits (to avoid failures).
- [ ] Add best practices and troubleshooting section.
- [ ] Add a section on how to debug a failing test using `fill --pdb` and inspecting objects, respectively using `rich.print()` in `pdb`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.