box / box/flaky

Rerun flaky tests after the others

Open
#40 5 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
398
Forks
57
PR merge metrics
No merged PRs in 30d

Description

If a flaky test fails, it looks like it's rerun immediately.

For flaky integration tests which run against a third party, the external service may only be temporarily down. Running the same test immediately won't help much.

But deferring it, and sticking it at the end of the queue so it's run after all the remaining tests, might give the external service enough time to sort itself out and let the flaky test pass.

I'm seeing this now (using pytest). A flaky test passes in one Travis job, but fails in the next.

Is there an option for deferring flaky tests until after the rest of the test suite?

If not, would it be possible to add one?

Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.