cockroachdb / cockroachdb/cockroach

roachprod: clock sync issues on azure-provisioned machines

Open
#138,726 8 comments 0 reactions 1 assignee Claimed by @herkolategan View on GitHub
A-testeng-infra A-testing C-enhancement C-investigation T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

See for example https://github.com/cockroachdb/cockroach/issues/135154#issuecomment-2579596780. Anecdotally, we frequently see wonky azure clusters with extremely significant clock sync as well as other diffuse stability issues, which causes significant toil.

We should investigate whether clock synchronization is set up properly on these machines, and we should consider disabling live migrations or other maintenance that may be going on in the background while we run these tests.

If we cannot improve on this, perhaps most tests should be opted out of azure by default to reduce the surface area on which we encounter these infrastructure flakes.

Jira issue: CRDB-46321

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.