cockroachdb / cockroachdb/cockroach
roachprod: clock sync issues on azure-provisioned machines
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
See for example https://github.com/cockroachdb/cockroach/issues/135154#issuecomment-2579596780. Anecdotally, we frequently see wonky azure clusters with extremely significant clock sync as well as other diffuse stability issues, which causes significant toil.
We should investigate whether clock synchronization is set up properly on these machines, and we should consider disabling live migrations or other maintenance that may be going on in the background while we run these tests.
If we cannot improve on this, perhaps most tests should be opted out of azure by default to reduce the surface area on which we encounter these infrastructure flakes.
Jira issue: CRDB-46321
Contributor guide
Assessment
This issue has not been assessed yet.