cockroachdb / cockroachdb/cockroach
roachtest: no `RefetchCertsFromNode` after `DistributeCerts`
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
`DistributeCerts` is executed by _every_ invocation of `c.Start`, whenever `c.Secure`. Before actually doing expensive work, it invokes `c.checkForCertificates`, which in turn calls `c.fileExistsOnFirstNode`, looking for `certs.tar` on `n1`. If it's found, we bail out from `DistributeCerts`, thus skipping any other work. This appears to be a reasonable optimization strategy–we don't need to regenerate the certs and redistribute them upon every restart.
The problem is that `c.fileExistsOnFirstNode` _can_ fail since it is a remote command. The failure amounts to returning `false`, meaning that `DistributeCerts` will continue by regenerating new certs and _replacing_ the old ones, on every cluster node. So far, the side-effect of `c.fileExistsOnFirstNode` failing doesn't appear to be a big deal; i.e., it causes unnecessary work. However, the `test_runner` also needs the certs for its pgurl. (It uses `AuthUserCert` mode [1].) This is done in `RefetchCertsFromNode`, immediately after `c.Start`; i.e.,`certs.tar` is fetched from `n1` and unpacked into `c.localCertsDir` on the `test_runner` node. This ensures the `test_runner` pg client can authenticate to any cluster node using the same certs.
Except `RefetchCertsFromNode` is invoked exactly _once_ [2]; i.e., on restart, we're stuck with the same certs which were created initially. This is indeed the root cause of the recent failures [3], [4].
[1] https://github.com/cockroachdb/cockroach/blob/3ce8f44d1e033036783687e3c7ccb125d8de100b/pkg/roachprod/install/cockroach.go#L559
[2] https://github.com/cockroachdb/cockroach/blob/3ce8f44d1e033036783687e3c7ccb125d8de100b/pkg/cmd/roachtest/cluster.go#L2195-L2201
[3] https://github.com/cockroachdb/cockroach/issues/138722
[4] https://github.com/cockroachdb/cockroach/issues/136878
Jira issue: CRDB-46348
Contributor guide
Assessment
This issue has not been assessed yet.