apache / apache/arrow

[CI][Python] Have a way to reproduce hypothesis failures from CI

Open
#33,324 0 comments 0 reactions 0 assignees View on GitHub
Component: Continuous Integration Component: Python Type: test
Dominant language
C++
Stars
17.1k
Forks
4.3k
Avg merge
3d 13h
Merged PRs (30d)
88

Description

We have a nightly test build with hypothesis enabled, and those tests fail / crash from time to time, eg https://github.com/ursacomputing/crossbow/actions/runs/3286024804/jobs/5413689973

Ideally, if there is such a failure, we should actually fix that test case. But that requires us to be able to reproduce the failure locally.
If it's an actual test failure, hypothesis should print some information to re-run it locally with the same input (https://hypothesis.readthedocs.io/en/latest/reproducing.html#reproducing-an-example-with-reproduce-failure).
But if it is segfaulting, this information is not printed by default.

Another idea might to save the ./hypothesis/examples directory as artifact on the CI build, to use it locally, but probably that might have the same issue of not having the information we need in case of a crash.

**Reporter**: [Joris Van den Bossche](https://issues.apache.org/jira/browse/ARROW-18127) / @jorisvandenbossche

**Note**: *This issue was originally created as [ARROW-18127](https://issues.apache.org/jira/browse/ARROW-18127). Please see the [migration documentation](https://github.com/apache/arrow/issues/14542) for further details.*

Contributor guide

Open the contributing guide

Research direction

Start with the nightly CI test build and the linked failed GitHub Actions job, then read Hypothesis' reproducing-an-example documentation. Investigate how the ./hypothesis/examples directory is handled during failures; done means a locally rerunnable example is preserved or printed even when a test segfaults.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, python
Domain
ci-cd, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.