erigontech / erigontech/erigon

ci: move flaky tests off main/release branch triggers

Open
#20,301 1 comment 0 reactions 0 assignees View on GitHub
github_actions QA
Dominant language
Go
Stars
3.6k
Forks
1.5k
Avg merge
1d 16h
Merged PRs (30d)
455

Description

Flaky tests currently run on `main` and `release/**` branches. Failures there can't be corrected — there's no PR to fix, no author to notify. The branch looks broken to outside observers.

Flaky tests should not run on main or release if they're known to be flaky. The preferred destination is `pull_request` triggers where failures are visible to the author and can be prioritised. However, some flaky tests are also too long-running for PR checks — this is an unsolved problem. Even so, running them on main/release is not the right answer since failures there are permanent and public.

For long-running flaky tests that can't move to PR checks, the failure attribution mechanism in #20300 would at least make scheduled failures actionable by linking them to the commit that introduced the regression.

Known flaky tests to evaluate:
- #19446 — Flaky Hive test "Blob Transaction Ordering"
- #15939 — Flaky TestState/stEIP1559 timeout
- #8789 — Unstable bellatrix ssz_static test
- #14975 — TestEvmRun failing on Windows CI
- #14468 — global variable race in tests

Related: #19136, #20300

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.