cockroachdb / cockroachdb/cockroach

roachtest: make the failure injection framework a first class citizen of roachtest

Open
#148,109 0 comments 0 reactions 0 assignees View on GitHub
C-enhancement
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

Now that we have a base set of failure modes in the failure injection library, we should switch focus to integrating it in roachtests, i.e. use the failure injection library in roachtests instead of adhoc implementations.

While there are no blockers for using the library as is, further refinement goals involve:

- [ ] Validate the cluster/test spec before injecting any failures. Most if not all failures can cause non trivial side effects if recovery goes awry. To shield the blast radius of failure injection, we should ensure that all clusters are marked as no reuse. Other examples may include failure modes that are not compatible on all clouds, or interact differently if there are multiple stores per node.
- [ ] Integrate with the monitor. When a failure mode is injected/recovered, it should update the state of the cluster/node. For example, if a process kill is injected, we update the node health status in the monitor to be marked as unavailable. The purpose of this is two fold. The monitor itself will know not to fail the test if a node death is expected. This state can also be accessible by the roachtest itself, to determine which actions are valid. This is especially pertinent for https://github.com/cockroachdb/cockroach/issues/148084.
- [ ] Clear observability into failure injection induced failures. This is a broad goal but ranges from ensuring we have the proper tooling (e.g. a cgroup metrics exporter) to having adequate logging so the test debugger can construct a clear timeline of when and what failures were injected/recovered.

Jira issue: CRDB-51409

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.