cockroachdb / cockroachdb/cockroach

roachtest: no `RefetchCertsFromNode` after `DistributeCerts`

Open
#138,799 3 comments 0 reactions 0 assignees View on GitHub
A-testeng-foundations C-bug T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

`DistributeCerts` is executed by _every_ invocation of `c.Start`, whenever `c.Secure`. Before actually doing expensive work, it invokes `c.checkForCertificates`, which in turn calls `c.fileExistsOnFirstNode`, looking for `certs.tar` on `n1`. If it's found, we bail out from `DistributeCerts`, thus skipping any other work. This appears to be a reasonable optimization strategy–we don't need to regenerate the certs and redistribute them upon every restart.

The problem is that `c.fileExistsOnFirstNode` _can_ fail since it is a remote command. The failure amounts to returning `false`, meaning that `DistributeCerts` will continue by regenerating new certs and _replacing_ the old ones, on every cluster node. So far, the side-effect of `c.fileExistsOnFirstNode` failing doesn't appear to be a big deal; i.e., it causes unnecessary work. However, the `test_runner` also needs the certs for its pgurl. (It uses `AuthUserCert` mode [1].) This is done in `RefetchCertsFromNode`, immediately after `c.Start`; i.e.,`certs.tar` is fetched from `n1` and unpacked into `c.localCertsDir` on the `test_runner` node. This ensures the `test_runner` pg client can authenticate to any cluster node using the same certs.

Except `RefetchCertsFromNode` is invoked exactly _once_ [2]; i.e., on restart, we're stuck with the same certs which were created initially. This is indeed the root cause of the recent failures [3], [4].

[1] https://github.com/cockroachdb/cockroach/blob/3ce8f44d1e033036783687e3c7ccb125d8de100b/pkg/roachprod/install/cockroach.go#L559
[2] https://github.com/cockroachdb/cockroach/blob/3ce8f44d1e033036783687e3c7ccb125d8de100b/pkg/cmd/roachtest/cluster.go#L2195-L2201
[3] https://github.com/cockroachdb/cockroach/issues/138722
[4] https://github.com/cockroachdb/cockroach/issues/136878

Jira issue: CRDB-46348

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.