next-exp / next-exp/IC

Occasional test failures due to Hypothesis' health check

Open
#441 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question testing
Dominant language
Python
Stars
11
Forks
77
PR merge metrics
No merged PRs in 30d

Description

From time to time we get failures in tests running locally or in Travis' builds because the generation of data with Hypothesis is too slow. The canonical failure message is: Data generation is extremely slow: Only produced *N* valid examples in *M* seconds (0 invalid ones and 0 exceeded maximum size). Whether we get this failure or not depends on multiple factors: the machine the tests are running on, the load the processor has and maybe D. Trump's mood or how the dinosaurs actually became extinct.

The type of tests where this happens is very clear: those using compound data. For example, in test_PMap_s1s we give the test one pmap that is produced with a hypothesis composite. The composite needs to produce a (small) number of peaks, which come themselves from other composites and so on. Every composite has been carefully thought so the number of draws is kept small, and still, here we are.

I know we can turn off the health check for any individual test we want, but I want to ask whether we can soften a bit the Health Check globally. I have another case where this happens in #426.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the test_PMap_s1s case and the other compound-data tests described in the issue, then reproduce the intermittent Hypothesis health-check failure locally or in Travis. Review the related discussion in issue #426 and determine what global mitigation is acceptable; done means the affected tests no longer fail intermittently without unnecessarily disabling health checks.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.