JuliaDiff / JuliaDiff/ChainRulesTestUtils.jl

Test Suite: AbstractArrays and Numbers

Open
#98 12 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Julia
Stars
50
Forks
18
PR merge metrics
No merged PRs in 30d

Description

This PR is the latest to propose a rule for AbstractArrays that, in my opinion, is too broadly-typed. While I obviously sympathise with rule authors wanting to do this (it feels like the natural thing to do), we know that it causes real problems.

I wonder whether we should implement a test util comprising rrule_tests and frule_tests on a suite of different types of arrays that we insist any new rule that purports to support AbstractArrays, or some subtype thereof, must pass. The idea being would would include

  1. examples of every concrete type of array that we can find in Base / the standard libraries
  2. examples from popular external packages, such as StaticArrays, FillArrays, and BlockArrays.

Crucially, I propose that the tests should pass if, for all arrays in the suite, either

  1. the standard rule tests pass, or
  2. no rule is found for that array.

This means that the rule author can pick and choose the kinds of arrays that they support, and obvious "gaming" of the tests would be addressed at review time (i.e. the idea is not to implement a rule of AbstractArrays and then implement rules that return nothing for each non-Array array in the test suite).

This approach is obviously not a panacea since you clearly can't test against all concrete subtypes of AbstractArray that will ever be written. Rather, the goal is to have enough coverage of types of arrays that break assumptions that hold for Array-like AbstractArrays to ensure that rule authors and reviewers don't accidentally implement something too loosely-typed. My hope is that this will encourage rule authors (and reviewers) to be cognisant of the issues surrounding implementing rules for abstract types.

While Numbers have generally been less troublesome in practice, they obviously suffer from the same issues and have caused problems from time to time. It might make sense to consider a similar thing for Numbers.

Maybe there are some other common types where this stuff goes wrong on a regular basis?

I'm not entirely sure how this should look in practice (do we want a formal testing function that actually runs a load of tests, or do we just want to provide a global const Vector of subtypes of AbstractArray that we insist gets used when writing tests?), but if people like the general idea then we can think a little more carefully about how to make it work.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the issue and the linked PR 335, then examine the existing rrule_test and frule_test utilities. Define what a representative AbstractArray and Number suite should cover, how missing rules are handled, and what acceptance criteria the project agrees on.

Written by the indexing model from the issue text.

Assessment

Tech stack
julia
Domain
testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.