Occasional test failures due to Hypothesis' health check
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 11
- Forks
- 77
- PR merge metrics
- No merged PRs in 30d
Description
From time to time we get failures in tests running locally or in Travis' builds because the generation of data with Hypothesis is too slow. The canonical failure message is: Data generation is extremely slow: Only produced *N* valid examples in *M* seconds (0 invalid ones and 0 exceeded maximum size). Whether we get this failure or not depends on multiple factors: the machine the tests are running on, the load the processor has and maybe D. Trump's mood or how the dinosaurs actually became extinct.
The type of tests where this happens is very clear: those using compound data. For example, in test_PMap_s1s we give the test one pmap that is produced with a hypothesis composite. The composite needs to produce a (small) number of peaks, which come themselves from other composites and so on. Every composite has been carefully thought so the number of draws is kept small, and still, here we are.
I know we can turn off the health check for any individual test we want, but I want to ask whether we can soften a bit the Health Check globally. I have another case where this happens in #426.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the test_PMap_s1s case and the other compound-data tests described in the issue, then reproduce the intermittent Hypothesis health-check failure locally or in Travis. Review the related discussion in issue #426 and determine what global mitigation is acceptable; done means the affected tests no longer fail intermittently without unnecessarily disabling health checks.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100