cockroachdb / cockroachdb/cockroach
asim: enhance asim testing framework with randomized testing integration
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Currently, the allocator simulator assesses the allocator’s behavior under
explicit and user-defined conditions. This could restrict the exposure of
complex scenarios.
**Issues:** https://github.com/cockroachdb/cockroach/issues/106311
This issue tracks work that needs to be done to integrate randomness into the
simulator framework, eliminating manual test setup and explicit assertions.
- [x] Extend the `gen_cluster` command to support randomized parameters for zone
configurations (including voter and non-voter constraints, localities) and a random number of nodes and stores per node.
- [x] Add randomness to initial node placement across clusters
- [x] Extend `gen_ranges`, `gen_load` to support randomized parameters
- [ ] Scope out what assertions would be interesting based on the random cluster
setup
- [x] Ensure consistency on assertion evaluation
- [ ] Extend `set_liveness` to support randomized node status change
- [x] Currently, it is hard to verify stats without using a plot or assertion.
Simplifying this by generating text-based statistics would make it easier.
**Future work:**
- Add new types of assertions for zone configuration, range distribution,
node liveness, load balance, and when multiple samples are run
- Simulate network partition, high latency, load-related latency, and
randomized inter-region latency based on node proximity
- Different gossip delay for different regions
**Anticipated difficulties:**
- Test setup and flakiness: randomized testing could lead to complex and
unexpected test setups. It could be hard to validate the generated input.
**Some ideas to help alleviate these issues:**
- Use a seed number for every randomization
- Record test inputs for every failed test
- Write unit tests on things to make them more deterministic
- Add new parameters to allow multiple test runs
**Potentially useful libraries:**
- https://go.dev/doc/tutorial/fuzz
- https://pkg.go.dev/pgregory.net/rapid
Note that this issue just outlines potential project directions. Some ideas
might be out of scope of this project.
Jira issue: CRDB-29441
Contributor guide
Assessment
This issue has not been assessed yet.