cockroachdb / cockroachdb/cockroach
roachtest: remove remnants of PostValidationNoDeadNodes
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
`PostValidationNoDeadNodes` functionality currently does not function in master and is a no-op. The mechanism relies on finding dead nodes via `roachprod.Monitor`. It passes `IgnoreEmptyNodes` as an option to the `Monitor` call. This is supposed to detect old dead processes by inspecting the contents of the data dir for certain markers. This is not a reliable way of determining dead nodes.
This functionality stopped working when `Monitor` was refactored to start with a list of active processes. After this change [1] it completely stopped functioning, as the directory to look for markers only works for tenants, but tenants do not have the same marker files. And if the original list only contains active processes to start monitoring, it's not possible to find dead ones.
Ultimately, a refactor of `roachprod.Monitor` to also take into account new processes and become a default for the duration of a roachtest will replace this check.
The original reasoning behind this check was to avoid a stall that occurred if some of the post validations ran on a failed cluster. If this issue persists we can devise another plan for determining if validation should run.
Ideally as part of this issue each test that specifies `PostValidationNoDeadNodes` should be inspected; and we should be
certain it has some other way of validating process death. And ensure it was only added to avoid the stall, but not as a way to confirm nodes have not died.
[1] https://github.com/cockroachdb/cockroach/pull/111064
Jira issue: CRDB-45525
Contributor guide
Assessment
This issue has not been assessed yet.