ethereum / ethereum/execution-specs
verify-traces: asymmetry between in-memory and disk-loaded traces for failed-during-execution txs (extend #2709)
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 505
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 116
Description
## Summary
[#2709](https://github.com/ethereum/execution-specs/pull/2709) made `collect_traces` tolerate a missing `trace-N-.jsonl` for transactions that produce a receipt but no `TransactionEnd` tracer event (e.g. EIP-3607 collisions, CREATE-into-non-empty failures during `process_transaction`). The fix appends an in-memory `TransactionTraces(traces=[])` placeholder per missing file.
The same placeholder is **not** written to disk, which causes `--verify-traces` to report spurious divergences for these tests. Reproducer: any test where a tx's `TransactionEnd` event never fires (the same set #2709 was written for).
## Reproducer
Trio of tests that exercise the path:
- `tests/ported_static/stCreateTest/test_transaction_collision_to_empty_but_code.py`
- `tests/ported_static/stCreateTest/test_transaction_collision_to_empty_but_nonce.py`
- `tests/ported_static/stEIP3607/test_init_colliding_with_non_empty_account.py`
Both runs (baseline dump + current live run) use post-#2709 code:
```bash
# 1. Dump baseline
TMPDIR=./.tmp uv run fill \
tests/ported_static/stCreateTest/test_transaction_collision_to_empty_but_code.py \
--evm-dump-dir .tmp/baseline --traces \
-n 4 --output .tmp/fix_baseline --clean
# 2. Verify against the freshly-dumped baseline
TMPDIR=./.tmp uv run fill \
tests/ported_static/stCreateTest/test_transaction_collision_to_empty_but_code.py \
--evm-dump-dir .tmp/current --traces \
--verify-traces .tmp/baseline \
--verify-traces-comparator exact-no-stack \
--verify-traces-json .tmp/report.json \
-n 4 --output .tmp/fix_current --clean
```
Result: `Total: 24, Equivalent: 0, Divergent: 24` even though the inputs and the runs are identical.
## Root cause
The two sides of the comparison go through different code paths:
| Side | Source | Output for these tests |
|------|--------|------------------------|
| `baseline_traces_list` | `_load_traces_from_dump_dir(dump_dir)` — globs `trace-*.jsonl` from disk | `[Traces(root=[])]` — **0** TransactionTraces (no trace files exist) |
| `current_traces_list` | `t8n.get_traces()` → `collect_traces(...)` — in-memory | `[Traces(root=[TransactionTraces(traces=[])])]` — **1** empty TransactionTraces (post-#2709 padding) |
The `compare_traces` outer length check (`len(baseline.root) != len(current.root)`) then fires.
#2709 only patched the in-memory `collect_traces`. Neither `output_traces` (`src/ethereum_spec_tools/evm_tools/t8n/evm_trace/eip3155.py`) nor the `--evm-dump-dir` plumbing (`shutil.copy` in `transition_tool.py::collect_traces`) emit anything for the missing-trace case, so the placeholder never reaches disk.
## Proposed fix
Make the on-disk representation symmetric with the in-memory one. Two equivalent approaches:
### Option A — write empty placeholder files
In `transition_tool.py::collect_traces`, when the missing-file branch (the #2709 branch) runs and `debug_output_path` is set, also write an empty `trace-{i}-{hash}.jsonl` to the dump dir. Then `_load_traces_from_dump_dir` reads it as a zero-line `TransactionTraces` and matches the in-memory shape.
```python
if not trace_file_path.exists():
traces.append(TransactionTraces(traces=[]))
if debug_output_path:
(Path(debug_output_path) / trace_file_name).write_text("")
continue
```
Trivial, ~3 lines. Doesn't change any other comparator behaviour.
### Option B — have `_load_traces_from_dump_dir` reconcile from `output/result.json`
When loading a dump dir, also read `output/result.json` to count receipts; if there are more receipts than `trace-*.jsonl` files, pad the result with `TransactionTraces(traces=[])` for the missing ones.
More invasive (touches the loader), but doesn't change the on-disk format and has no risk of empty files confusing other tooling.
Either approach makes the verify-traces comparison symmetric. Option A is the smallest diff and pairs naturally with the existing #2709 patch.
## Why it matters
Currently the only workaround is to put every test that fails-during-execution into an allowlist (e.g. EEST's static-port `FORCE_HARDCODED_TESTS`) so the test passes via baseline-equivalent state assertions, never actually exercising the verify-traces path. That defeats the point of trace verification for an entire class of tests.
## Context
Found while wiring up `--verify-traces` on PR [#2695](https://github.com/ethereum/execution-specs/pull/2695). After rebasing onto a `forks/amsterdam` containing #2709, the 3 STRUCTURAL tests above pass `fill` but report 78 spurious divergences in the trace report.
Contributor guide
Assessment
This issue has not been assessed yet.