OpenFreeEnergy / OpenFreeEnergy/openfe
Evaluate how we want to run flaky tests
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 331
- Forks
- 56
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 13
Description
See https://github.com/OpenFreeEnergy/openfe/pull/1152
We need a more comprehensive plan on how we want to handle flaky tests.
There are two different pytest plugins that do it.
Another consideration is how they seem to not play well with xdist (sometimes I've noticed if the test crashes hard enough, it can't retry it)
codecov also now offers tracking of flaky tests, so maybe we remove all the retry marks and just kinda see what happens.
I don't look at the logs close enough to see how often things need to be retried, and if we set codcov to track this (which is easy to do) then we can get some stats on what tests are actually flaky.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading pull request 1152 and compare the two pytest plugin approaches mentioned in the issue. Investigate their interaction with xdist and whether Codecov flaky-test tracking can provide useful statistics. Done means documenting a decided strategy for retries, tracking, and test execution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100