Regression testing using `fuzzy` and the archival data
- Dominant language
- Rust
- Stars
- 697
- Forks
- 200
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 65
Description
**Issue summary**
The entire Filecoin blockchain is stored in CloudFlare. With this data, we can do regression testing over arbitrary ranges of epochs.
Regression testing is likely to be too slow for the per-PR CI. Manually triggering the regression tests might be the best option.
The blockchain data covers all of the network versions (currently from NV1 to NV20). Forest has support for NV16 and newer. Epochs lower than `1_960_320` should not be tested.
Tasks:
- [ ] Create a script that automatically downloads snapshots and validates Forest over a range of epochs. The script should accept filters such as network version and epoch range. How many epochs to validate should also be configurable (default to 30k).
- [ ] The script should pick a random block of epochs in the given range and validate using `forest-tool snapshot validate --check-links 0 --check-stateroots EPOCHS`.
- [ ] Make sure files are properly cleaned-up. Otherwise, the server will run out of disk space.
- [ ] Trigger the script to run on `fuzzy` when `@regression-check` is commented in a PR.
- [ ] Support arguments such as `@regression-check nv18` that limit the testing to a specific network version.
- [ ] Write results in a comment response.
**Other information and links**
See Hubert's `grpc_bench` for inspiration w.r.t. workflows triggered by comments: https://github.com/LesnyRumcajs/grpc_bench/blob/master/.github/workflows/issue_bench.yml
Contributor guide
Assessment
This issue has not been assessed yet.