Smoke tests are flaky because of CSS transition
- Dominant language
- JavaScript
- Stars
- 7.2k
- Forks
- 621
- PR merge metrics
- No merged PRs in 30d
Description
I'm separating this issue out of #747 for clarity. This CSS line:
```css
.lemurInYourFace {
transition: transform 750ms ease, opacity 750ms ease, visibility 800ms ease;
```
Means that any tests which grabs its screenshot in the first 750ms of the page will get a different result. If I repeatedly run the smoke test on my Macbook, I can get anywhere from 0 to 8 failures.
For websites with a transition like this, the right way to handle it would be to detect the transitionend event, and not grab any screenshots until that event is triggered. That could be a new backstop feature, or it could be something users do manually and then use the readyEvent.
You probably don't want to comment out the tests, as suggested in the PR, because you'd have to comment out lots of tests - any tests that doesn't wait for the transition in theory could be affected, in practice only fast tests have the problem.
If you remove this transition, then I get 100% consistent smoke test results. I'll submit a 1 line PR for this.
Contributor guide
Assessment
This issue has not been assessed yet.