dask / dask/dask-image

Property-based testing with hypothesis

Open
#208 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
226
Forks
56
PR merge metrics
No merged PRs in 30d

Description

Hi @Zac-HD and @FudgeMunkey,

We talked a while ago about whether it would be good to add property-based testing with hypothesis to dask-image.

I'd like to suggest a good place to start, which is writing an equivalence test for the `affine_transform` function (found in the `ndinterp` subpackage). You can see some (non-property-based) equivalence testing happening here in [`validate_affine_transform`](https://github.com/dask/dask-image/blob/main/tests/test_dask_image/test_ndinterp/test_affine_transformation.py#L13).

Unlike most of the rest of dask-image, the affine transform function doesn't simply call the equivalent scipy function on chunks of the dask array. It's a bit more complicated, which it a really good one to try. Here's a recent bug report where the user found a mismatch between the default kwarg values expected by `dask-image` and `scipy`, which led to this error: https://github.com/dask/dask-image/issues/204

Earlier discussion: https://github.com/napari/napari/issues/2444

> As background, [Hypothesis](https://hypothesis.readthedocs.io/en/latest/) is basically super-powered random testing, and [I wrote this short paper](http://conference.scipy.org/proceedings/scipy2020/zac_hatfield-dodds.html) on applying it to scientific code. ... For Napari's first property-based tests, I'd use the following as a checklist (and [try the ghostwriter](https://hypothesis.readthedocs.io/en/latest/ghostwriter.html)):
>
> - "fuzz tests": feed in weird but technically-possible inputs, and see if anything crashes. In Numpy (for example) this has found issues with unicode strings, zero-dimensional arrays, and comparing arrays-of-structs containing `nan`. The key is to think carefully about how to generate *all possible* values, as bugs often rely on the interaction of two or more edge cases. Once you have some library-specific generators, it's easy to reuse them for later steps.
> - Round-trip tests: save your data, load it, assert no changes! Simple, important, and highly effective. Test format conversions in-memory, as well as persistent storage formats.
> - Equivalence: fantastic when you have a function that should behave identically to a reference implementation, even if only over a subset of inputs. Traditionally great for refactoring, but perhaps you also have Dask- or Napari-specific implementations of functions in `scipy` or `scikit-*`, and could run both on small arrays? Presenting identical data as Numpy vs Dask arrays could also be interesting.
> - Look for parametrized tests, and consider changing them to use Hypothesis. This is best when the full product is too slow to run regularly, e.g. array dimensionality * size * dtype * bitwidth * endianness * random contents (for each operand...) and using Hypothesis would allow a less-restricted test. Note that you can also supply some arguments from `@pytest.mark.parametrize` and others from `@hypothesis.given` in the same test!
> - "Classic" properties like asserting bounds on outputs, idempotence, commutative operations, etc. These mostly apply to "algorithmic" code and are far from the only use of a PBT library, but still very useful when applicable.

(No worries if you're swamped, or have stuff you're working on for other projects. It's good to document good places to start, so people can pick them up in the future.)

Contributor guide

Open the contributing guide

Research direction

Start with tests/test_dask_image/test_ndinterp/test_affine_transformation.py, especially validate_affine_transform, and inspect the affine_transform implementation in the ndinterp subpackage. Review issue #204 for the reported dask-image and SciPy keyword mismatch, then use Hypothesis to exercise small valid inputs against the SciPy reference. Done means the property-based equivalence test passes and can expose mismatches between the implementations.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.