cockroachdb / cockroachdb/cockroach

asim: enhance asim testing framework with randomized testing integration

Open
#106,192 1 comment 0 reactions 0 assignees View on GitHub
A-kv A-kv-simulation C-enhancement T-kv
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Currently, the allocator simulator assesses the allocator’s behavior under
explicit and user-defined conditions. This could restrict the exposure of
complex scenarios.

**Issues:** https://github.com/cockroachdb/cockroach/issues/106311

This issue tracks work that needs to be done to integrate randomness into the
simulator framework, eliminating manual test setup and explicit assertions.

- [x] Extend the `gen_cluster` command to support randomized parameters for zone
configurations (including voter and non-voter constraints, localities) and a random number of nodes and stores per node.
- [x] Add randomness to initial node placement across clusters
- [x] Extend `gen_ranges`, `gen_load` to support randomized parameters
- [ ] Scope out what assertions would be interesting based on the random cluster
setup
- [x] Ensure consistency on assertion evaluation
- [ ] Extend `set_liveness` to support randomized node status change
- [x] Currently, it is hard to verify stats without using a plot or assertion.
Simplifying this by generating text-based statistics would make it easier.

**Future work:**
- Add new types of assertions for zone configuration, range distribution,
node liveness, load balance, and when multiple samples are run
- Simulate network partition, high latency, load-related latency, and
randomized inter-region latency based on node proximity
- Different gossip delay for different regions

**Anticipated difficulties:**
- Test setup and flakiness: randomized testing could lead to complex and
unexpected test setups. It could be hard to validate the generated input.

**Some ideas to help alleviate these issues:**
- Use a seed number for every randomization
- Record test inputs for every failed test
- Write unit tests on things to make them more deterministic
- Add new parameters to allow multiple test runs

**Potentially useful libraries:**
- https://go.dev/doc/tutorial/fuzz
- https://pkg.go.dev/pgregory.net/rapid

Note that this issue just outlines potential project directions. Some ideas
might be out of scope of this project.

Jira issue: CRDB-29441

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.