Testing weird types

Couldn't think of a better place to raise this, so 🤷‍♂ .

Proposal

A long time ago Jarrett made DiffTests.jl. That package is useful for testing whether your AD tool works well on problems that are tricky to differentiate. What we don't currently have is a package to test whether your package is compatible with the API that ChainRules specifies because, when DiffTests was written, the Julia AD world only really believed in numbers and arrays of numbers.

In short, I would like a package that tells me: "does my AD tool work with this weird type input and it's differential.", for increasingly complicated set of types. Bonus points for producing a nice human-readable report at the end.

IIRC e.g. ForwardDiff, ReverseDiff, Tracker etc only believe in numbers and dense arrays of numbers. ForwardDiff2 is a bit more flexible, and Zygote is more flexible still. FiniteDifferencing should really be almost as flexible as Zygote on this front, but it's not clear that it currently is, and it's hard to know what it should look like for this to be the case. FiniteDiff also only believes in numbers and arrays of numbers. This range of capabilities provides strong motivation for this package.

To summarise, this package would help

• everyone to better understand the landscape,
• package authors can get a feel for where they sit,
• any time we find a type that feels hard to work with, we have a package that we can encode this in and test everything else against.

Start by reviewing the existing DiffTests.jl test-function suite and the ChainRules API referenced in the proposal. Compare how ForwardDiff, ReverseDiff, Tracker, ForwardDiff2, Zygote, FiniteDifferencing, and FiniteDiff handle non-numeric inputs. Done should be a package that exercises increasingly complex types and produces a useful human-readable compatibility report.