JuliaDiff / JuliaDiff/ChainRulesTestUtils.jl
Test `frule`s with `NotImplemented` properly
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 50
- Forks
- 18
- PR merge metrics
- No merged PRs in 30d
Description
Currently, test_frule is pretty useless if one of the partial derivatives is a NotImplemented since the derivative will always be a NotImplemented and hence tests always pass (even though marked as broken), regardless of the other derivatives. With https://github.com/JuliaDiff/ChainRulesCore.jl/pull/477 it is possible to test the other partial derivatives by setting the tangents of the arguments for which the partial derivative is a NotImplemented to NoTangent() to test these partial derivatives properly. However, currently one has to do this manually, as in https://github.com/JuliaMath/SpecialFunctions.jl/blob/0af956882245e3b07340002c7c95c319e51af52a/test/chainrules.jl#L56-L57. It would be nice if these proper tests would be performed automatically.
Since it can't be inferred automatically which argument caused the NotImplemented derivative one approach might be to force users to always specify a NoTangent() tangent in the frule tests if the partial derivative is NotImplemented (if this is not already the default rand_tangent, in which case the partial derivative probably should just be NoTangent()), and basically not allow NotImplemented derivatives in the tests. A problem might be that this hides the fact that the implementation is broken and should be fixed in the frule tests. Maybe it would be better to let users specify a tangent of type NotImplemented (e.g. with @not_implemented()) and then use a NoTangent() internally but mark the tests as broken?
Originally posted by @devmotion in https://github.com/JuliaDiff/ChainRulesCore.jl/issues/477#issuecomment-932890666
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the test_frule implementation and the SpecialFunctions example at test/chainrules.jl#L56-L57, then review ChainRulesCore.jl pull request #477 for the available tangent behavior. Define and test how NotImplemented partial derivatives should be represented while still exercising the other partial derivatives. Done means frule tests no longer pass solely because one derivative is NotImplemented, with regression coverage for that case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100