Improve echidna heuristics documentation
- Dominant language
- Haskell
- Stars
- 3.2k
- Forks
- 432
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 5
Description
I am developing [fuzzy.fyi](https://fuzzy.fyi/), a project that helps execute long runs of echidna on AWS, testing it on a [ERC4626 vault from Pods](https://github.com/pods-finance/yield-contracts). The idea is to test smart contracts on the cloud without compromising the developer's workflow with intensive resource-consuming fuzzy campaigns.
While working on this tool, we stumbled upon some choices of parameters that influence the fuzzer performance, which don't seem to be very well documented. It seems that some of these parameters are "rules of thumb"/"heuristics", so I would like to ask them here:
1. What are generally "good" `testLimit` and `seqLen` values? What values does Trail of Bits usually use during its audits? How long should a "good" run last (hours, days, weeks)?
2. Assuming Trail of Bits performs long runs on the cloud, what is generally the best AWS instance for echidna? Another way to ask: is echidna constrained by CPU or RAM? Should I go after a CPU-optimized instance (such as `c5`) or a memory-optimized instance (such as `r5`)?
3. Are there any benchmarks on echidna's performance vs hardware specifications? For example, if we pick a `2xlarge` instance, should we expect half of the time to run a campaign from a `xlarge` instance?
4. On what depends the choice of `testLimit` and `seqLen`? Meaning: when should you increase one or the other? How can we calculate the fuzzer "performance" (meaning, probability to find bugs), assuming the choice of these variables has an impact on the performance?
5. When should we reutilize the corpus? Does it make sense to reuse the corpus if the contract interface changes? Does it make sense to reuse the corpus for different pull requests for the same project? Does it make sense to reuse the corpus from a different project?
6. Is it better to test 10 runs with `testLimit` 100k and corpus enabled or test 1 run with `testLimit` 1M?
7. Sometimes, long runs (`testLimit` 1M) are terminated by the OOM killer after many hours. What is the recommendation when that happens? I think getting a bigger instance would just hide the problem.
Contributor guide
Assessment
This issue has not been assessed yet.