ampproject / ampproject/amphtml

Non-deterministic flakiness in E2E tests

Open
#32,442 5 comments 0 reactions 1 assignee Claimed by @rileyajones View on GitHub
P2: Soon Related to: Flaky Tests Stale Type: Bug WG: infra
Dominant language
JavaScript
Stars
14.9k
Forks
4.1k
PR merge metrics
No merged PRs in 30d

Description

Of late, we're seeing an uptick in flakiness on our E2E tests. Many of the recent failures appear to be in the experiment tests run on the `master` branch. In addition, PR jobs are showing failures too.

[Here is a link](https://app.circleci.com/pipelines/github/ampproject/amphtml?branch=master) to the `master` builds on CircleCI. All failures in the past 24h seem to come from E2E tests. (**Pro-tip:** Enable "auto expand" if not already enabled, and refresh the page. You'll see every failing job on one screen without having to click through.)

This is a tracking bug to investigate and fix the root cause of the problem. (New Chrome version? A new dependency version that's causing more flakiness? Bit-rotting of E2E infrastructure code?)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.