cockroachdb / cockroachdb/cockroach

roachtest: better reporting of infrastructure flakes

Open
#121,696 2 comments 1 reaction 0 assignees View on GitHub
A-testeng-infra A-testing C-enhancement E-starter T-testeng
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

We have made several improvements to infrastructure flake detection: issues like ssh-flakes, timeouts, DNS issues, VM preemptions, public cloud errors or general blips rarely bubble up to teams. Instead, an issue is still created (or a comment added to an existing issue) and the issue is owned by us (examples: [cluster creation](https://github.com/cockroachdb/cockroach/issues/78601), [ssh_problem](https://github.com/cockroachdb/cockroach/issues/78601), [dns_problem](https://github.com/cockroachdb/cockroach/issues/120959), etc).

The reporting still relies on creating an issue on GitHub just like regular failures. This approach has a few downsides:

* failures due to infrastructure flakes are marked as test failures on TeamCity; our existing data pipeline ingestion (data consumed by other teams and execs) cannot see what failures are legitimate vs infrastructure flakes, leading to a distorted view of the daily failure ratio in these nightly runs.
* It's very noisy for Test Eng: every night, test eng gets dozens of emails due to these flakes (primarily VM preemptions).
* It's hard to analyze the frequency of these errors or monitor patterns: we would need to analyze GitHub data "manually", which is not ideal.

A better approach to report these errors would be to expose them in a way that can be consumed Snowflake. This would eliminate the daily noise, allow us to analyze behaviour over time, and perhaps even set up alerts if something is outside "expected" ratios.

Jira issue: CRDB-37456

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.