crytic / crytic/echidna

[Bug-Candidate]: Echidna `stopOnFail` does not behave as expected when `coverage` is turned on

Open
#1,126 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Haskell
Stars
3.2k
Forks
432
Avg merge
1d 20h
Merged PRs (30d)
5

Description

### Describe the issue:

While running Echidna on a complex codebase in the cloud, I am trying to optimize the time it takes to break assertions, which is why I've set `stopOnFail: true` in the config file, which also includes `coverage: true` and `corpusDir: echidna`.

With these settings, I expect Echidna to exit as soon as a property is falsified, which _may_ occur if the corpus contains a sequence that causes an assertion to fail.

However, the issue is that Echidna finishes replaying the entire corpus _before_ exiting.

This behavior is a bit counterintuitive to me and prevents the "short circuit" effect I desire. Instead of concluding in 1 minute, it takes approximately 4 hours due to the number of sequences in the corpus.

### Code example to reproduce the issue:

N/A

### Version:

Echidna 2.2.1
slither 0.9.3

### Relevant log output:

```shell
$ echidna contracts/TestContracts/invariants/echidna/EchidnaTester.sol --contract EchidnaTester --config config.yaml --corpus-dir echidna --test-mode assertion --test-limit 300000
[2023-09-28 14:28:05.87] Compiling contracts/TestContracts/invariants/echidna/EchidnaTester.sol... Done! (6.789608304s)
Analyzing contract: /home/ubuntu/contracts/TestContracts/invariants/echidna/EchidnaTester.sol:EchidnaTester
[2023-09-28 14:28:15.16] Running slither on contracts/TestContracts/invariants/echidna/EchidnaTester.sol... Done! (286.605246548s)
Loaded 8 transaction sequences from echidna/reproducers
Loaded 113 transaction sequences from echidna/coverage
[2023-09-28 14:33:06.57] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 22754, corpus: 0
[2023-09-28 14:33:09.91] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 26199, corpus: 0
[2023-09-28 14:33:11.62] [Worker 0] New coverage: 33685 instr, 11 contracts, 1 seqs in corpus
[2023-09-28 14:33:11.62] [Worker 0] Sequence replayed from corpus (1/121)
[2023-09-28 14:33:12.93] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 33685, corpus: 1
[2023-09-28 14:33:15.95] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 33685, corpus: 1
[2023-09-28 14:33:18.97] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 34263, corpus: 1
[2023-09-28 14:33:21.99] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 35559, corpus: 1
[2023-09-28 14:33:25.01] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 44497, corpus: 1
[2023-09-28 14:33:26.07] [Worker 0] New coverage: 49057 instr, 12 contracts, 2 seqs in corpus
[2023-09-28 14:33:26.07] [Worker 0] Sequence replayed from corpus (2/121)
[2023-09-28 14:33:28.03] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 49057, corpus: 2
[2023-09-28 14:33:31.03] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 50971, corpus: 2
[2023-09-28 14:33:31.75] [Worker 0] New coverage: 51769 instr, 12 contracts, 3 seqs in corpus
[2023-09-28 14:33:31.75] [Worker 0] Sequence replayed from corpus (3/121)
[2023-09-28 14:33:34.05] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 51769, corpus: 3
[2023-09-28 14:33:36.88] [Worker 0] New coverage: 52134 instr, 12 contracts, 4 seqs in corpus
[2023-09-28 14:33:36.88] [Worker 0] Sequence replayed from corpus (4/121)
[2023-09-28 14:33:37.07] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 52134, corpus: 4
[2023-09-28 14:33:38.55] [Worker 0] Sequence replayed from corpus (5/121)
[2023-09-28 14:33:40.08] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 52134, corpus: 4
[2023-09-28 14:33:40.20] [Worker 0] Sequence replayed from corpus (6/121)
[2023-09-28 14:33:43.11] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 52994, corpus: 4
[2023-09-28 14:33:46.12] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 56381, corpus: 4
[2023-09-28 14:33:49.13] [status] tests: 0/14, fuzzing: 0/300000, values: [], cov: 57134, corpus: 4
[2023-09-28 14:33:51.48] [Worker 0] Test redeemCollateral(uint256,uint256,uint256,uint256) falsified!
// ...
[2023-09-28 18:35:48.85] [status] tests: 1/14, fuzzing: 0/300000, values: [], cov: 71921, corpus: 26
[2023-09-28 18:35:51.69] [Worker 0] Sequence replayed from corpus (121/121)
[2023-09-28 18:35:51.69] [Worker 0] A test was falsified. Stopping.
[2023-09-28 18:35:51.70] [status] tests: 1/14, fuzzing: 21699/300000, values: [], cov: 71921, corpus: 26
// ...
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.