ethereum / ethereum/execution-specs

suggestion: Increase invalid test coverage using the framework

Open
#2,705 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.2k
Forks
505
Avg merge
2d 14h
Merged PRs (30d)
116

Description

## suggestion: Increase invalid test coverage using the framework

BAL validation can be broken down into three dimensions:

1. Correctness: each entry has the right value
2. Exactness: exactly the right entries exist (no more, no less)
3. Sequence: entries are in canonical order

BAL equivalence = Correctness + Exactness + Sequence.

Our tests should not assume how clients verify equivalence of computed vs provided BAL (hash, item-by-item, or otherwise).

A client that does zero BAL validation (one that simply passes through the provided BAL) will pass every happy path test. For every happy path test, we should have a negative test and ensure client rejects them:

* happy path: Alice sends bob 1 ETH. BAL Balance change 1 ETH. accept block. test pass
* invalid path: Alice sends bob 1 ETH. BAL Balance change 2 ETH. reject block. test pass

However, instead of writing invalid tests by hand, we should automate them using the framework. We can derive these invalid tests from a single valid one:
* Correctnes
* corrupting an existing entry's value,
* Exactness
* adding a bogus entry,
* removing an entry,
* duplicating an entry,
* Sequence
* (if there are multiple items) swapping entries.

The required modifiers for these mutations already exists. So some kind of pytest hook over
existing tests that introspects a valid BAL expectation, enumerates its entries, produces N invalid variants,
one per applicable mutation.

this way our invalid coverage grows organically with valid tests.

_Originally posted by @raxhvl in https://github.com/ethereum/execution-specs/issues/2653#issuecomment-4235678713_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.