cockroachdb / cockroachdb/cockroach
roachtest: make the failure injection framework a first class citizen of roachtest
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
Now that we have a base set of failure modes in the failure injection library, we should switch focus to integrating it in roachtests, i.e. use the failure injection library in roachtests instead of adhoc implementations.
While there are no blockers for using the library as is, further refinement goals involve:
- [ ] Validate the cluster/test spec before injecting any failures. Most if not all failures can cause non trivial side effects if recovery goes awry. To shield the blast radius of failure injection, we should ensure that all clusters are marked as no reuse. Other examples may include failure modes that are not compatible on all clouds, or interact differently if there are multiple stores per node.
- [ ] Integrate with the monitor. When a failure mode is injected/recovered, it should update the state of the cluster/node. For example, if a process kill is injected, we update the node health status in the monitor to be marked as unavailable. The purpose of this is two fold. The monitor itself will know not to fail the test if a node death is expected. This state can also be accessible by the roachtest itself, to determine which actions are valid. This is especially pertinent for https://github.com/cockroachdb/cockroach/issues/148084.
- [ ] Clear observability into failure injection induced failures. This is a broad goal but ranges from ensuring we have the proper tooling (e.g. a cgroup metrics exporter) to having adequate logging so the test debugger can construct a clear timeline of when and what failures were injected/recovered.
Jira issue: CRDB-51409
Contributor guide
Assessment
This issue has not been assessed yet.