cockroachdb / cockroachdb/cockroach

roachtest: remove remnants of PostValidationNoDeadNodes

Open
#137,329 2 comments 0 reactions 0 assignees View on GitHub
A-testeng-foundations autosolve C-enhancement T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

`PostValidationNoDeadNodes` functionality currently does not function in master and is a no-op. The mechanism relies on finding dead nodes via `roachprod.Monitor`. It passes `IgnoreEmptyNodes` as an option to the `Monitor` call. This is supposed to detect old dead processes by inspecting the contents of the data dir for certain markers. This is not a reliable way of determining dead nodes.

This functionality stopped working when `Monitor` was refactored to start with a list of active processes. After this change [1] it completely stopped functioning, as the directory to look for markers only works for tenants, but tenants do not have the same marker files. And if the original list only contains active processes to start monitoring, it's not possible to find dead ones.

Ultimately, a refactor of `roachprod.Monitor` to also take into account new processes and become a default for the duration of a roachtest will replace this check.

The original reasoning behind this check was to avoid a stall that occurred if some of the post validations ran on a failed cluster. If this issue persists we can devise another plan for determining if validation should run.

Ideally as part of this issue each test that specifies `PostValidationNoDeadNodes` should be inspected; and we should be
certain it has some other way of validating process death. And ensure it was only added to avoid the stall, but not as a way to confirm nodes have not died.

[1] https://github.com/cockroachdb/cockroach/pull/111064

Jira issue: CRDB-45525

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.