JuliaDiff / JuliaDiff/ChainRulesCore.jl
Make inplace accumulation support Hermitian and Symmetric matrices
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 267
- Forks
- 66
- PR merge metrics
- No merged PRs in 30d
Description
Follow up to https://github.com/JuliaDiff/ChainRulesCore.jl/pull/234
In that PR we are just saying that Hermitian and Symmetric matrices can not be used for in-place accumulation of gradients.
Conceptually though they really can; because pullbacks and pushforwards are linear operators, they preserve this structure: the differentials are also going to be Hermitian/Symmetric.
Its just that right now julia is really fussy about what operations you are allowed to do to them in such a way that its not easy to just treat them like any other matrix type.
https://github.com/JuliaLang/LinearAlgebra.jl/issues/773
In contrast we get away just fine treating (e.g.) Diagonal matrix's like any other matrix type (which is likewise safe because linear operator avoids doing anything illegal)
Anyway, we could do something to work around julia being fussy.
For example we could use parent to unwrap it to get the matrix behind it,
perform the inplace accumulation,
and then wrap it back up again.
This might mean we end up doing 2x as much work as we conceptually need to, since we only need to actually update one half.
and e.g. operations like parent(H1) .= H2 (where H1 and H2 are both Hermitian/Symmetric) will update both halves.
Similar for in-place multiplications.
It also feels pretty nasty; and means internally it will look like some functions violate the principle that the primal and the differential have the same structure. Which we might like to be asserting more and won't really be able to do as well with it internally manipulating the parent type.
But maybe that OK.
One might think that we could unwrap then rewrap as UpperTriangular(parent(H1)),
but that won't workout.
Even thought its much less fussy, and we don't run into these kinds of problems, it has has different invariants.
In this case that -- that you can't see lower triangular part to nonzero.
And we will end up doing operations like adding inplace a Symmetric/Hermitian matrix to it.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing PR #234 and JuliaLang/LinearAlgebra.jl issue #773, since no source files or tests are identified. Determine how in-place gradient accumulation handles Hermitian and Symmetric matrices and choose a way to support them without violating their invariants. Done means the supported accumulation paths work for both matrix types with appropriate coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100