Feature request: Coverage in DifferentiationInterfaceTest

To preface, this may be impossible and I really like DIT as it stands.

Ask

I would like to be able to use DifferentiaionInterfaceTest as my unit testing interface, benchmarking tool, and code coverage generator when creating packages meant to work well with AD (both a specific AD package and for multiple as far as I am aware, none of them has good tools for this). Related package level issue: https://github.com/EpiAware/CensoredDistributions.jl/issues/272

The missing part of this is having an understanding of what is being covered by my DIT scenarios which it seems code coverage is designed for.

The reason I want this is we are moving towards making a larger Julia ecosystem of packages and we really want them on a firm base/with some good practices established before hand.

What we are doing at the moment:

We currently run DIT in our CI per benchmark, in our docs (https://censoreddistributions.epiaware.org/stable/getting-started/tutorials/ad-backends) and in our benchmarking suite from a shared set of scenarios. Early days but very happy with DIT as it stands. The only additional nice to have would be a shared way to test rules but that seems like the real wild west so I imagine is out of scope.

Solution

I don't know. I don't know anything about how the AD version of the code interacts with the original version or if there is enough commonality between backends to make this a thing that is plausible to do across backends.

Start by reading DifferentiationInterfaceTest (DIT) scenarios and the existing CI, documentation, and benchmarking entry points described in the issue. Determine whether coverage can be defined consistently across the supported AD backends and how it would connect to the shared scenarios. Done means a documented, working coverage workflow or a clearly justified explanation of why it is not feasible.